Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
29b6f30f2b | ||
|
|
72e979f1d9 | ||
|
|
6c6207a132 | ||
|
|
0effe485f4 | ||
|
|
32407bc781 | ||
|
|
df870e9ed3 | ||
|
|
323892b36c | ||
|
|
36ee06c7cc | ||
|
|
9bfd8f9a13 | ||
|
|
cb6c72b409 | ||
|
|
a721c1fdcf | ||
|
|
80f10289fe | ||
|
|
a0ad314736 | ||
|
|
1808a373a1 | ||
|
|
67789fb31c | ||
|
|
da4bef8e42 | ||
|
|
0076d0067e | ||
|
|
faf34348c8 | ||
|
|
69ce697ee5 | ||
|
|
93a3cb260a | ||
|
|
33379890ad | ||
|
|
7f8a42ce51 | ||
|
|
4de3c824d0 | ||
|
|
11fbbd3f6d | ||
|
|
f546f04c8e | ||
|
|
270b3b1c4c | ||
|
|
5a7d9cc05c | ||
|
|
5871ec9d28 | ||
|
|
58b023ff16 | ||
|
|
b61d17dd61 | ||
|
|
93aab6dadc | ||
|
|
7098be1c8b | ||
|
|
8520b84b68 | ||
|
|
549fa4009f | ||
|
|
dfe2bd1bc3 | ||
|
|
324d417d27 | ||
|
|
637020b82b | ||
|
|
dab6456581 | ||
|
|
015d29cafb | ||
|
|
1a03c59f82 | ||
|
|
8281b7c798 | ||
|
|
efe57ee49d | ||
|
|
7f77f4d7bf | ||
|
|
087309259b | ||
|
|
453db55f12 | ||
|
|
620321a9cc | ||
|
|
738d45f1c7 | ||
|
|
d959aae41b | ||
|
|
de99bc3bd5 | ||
|
|
c47633a409 | ||
|
|
4aa9abe22a | ||
|
|
d90cfde1bb | ||
|
|
bd92729623 | ||
|
|
afe819f498 | ||
|
|
6b91205036 | ||
|
|
1dfa032be4 | ||
|
|
fd7d20fe1e | ||
|
|
27ff5846c2 | ||
|
|
d14b37fab2 | ||
|
|
edf86c1800 | ||
|
|
437a995a0f | ||
|
|
03eb8f7d04 | ||
|
|
beca8cfea8 | ||
|
|
32103ad7b7 | ||
|
|
46c4445258 | ||
|
|
8bbab3fcc3 | ||
|
|
9ff5f38cb3 | ||
|
|
359c70bc21 | ||
|
|
0579ec91ad | ||
|
|
b6ec4e23f3 | ||
|
|
eb188931b5 | ||
|
|
86f23326ff | ||
|
|
ba4f64240a | ||
|
|
8ca7de89eb | ||
|
|
ed0b0b63cf | ||
|
|
0d80ab2cb0 | ||
|
|
6e3e014826 | ||
|
|
5bd8968aea | ||
|
|
48a7154810 | ||
|
|
5e135b9655 | ||
|
|
eeb3bf4452 | ||
|
|
126f23fd3e | ||
|
|
85b0db65bc | ||
|
|
45a5a5ce21 | ||
|
|
d57ecea804 | ||
|
|
6feafbd3be | ||
|
|
8f71c90f20 | ||
|
|
a481d459bd | ||
|
|
af6690840e | ||
|
|
75864a597f | ||
|
|
16294fed44 | ||
|
|
e2c4ea93b0 | ||
|
|
b3c18db33f | ||
|
|
e7fc37d438 | ||
|
|
b825d99337 | ||
|
|
b6a3eba1ac | ||
|
|
bef688e9dd | ||
|
|
958a79ef7f | ||
|
|
72d137e1f3 | ||
|
|
e8fdf0e244 | ||
|
|
83e71c5689 | ||
|
|
3bb401f4e5 | ||
|
|
86f5213055 | ||
|
|
6e600c48cb | ||
|
|
14f1257d1f | ||
|
|
be1ec3ade0 | ||
|
|
d58010fe1b | ||
|
|
da9315815d | ||
|
|
bb492086c9 | ||
|
|
641e660677 | ||
|
|
62035435e8 | ||
|
|
4eed552153 | ||
|
|
ae516cae63 | ||
|
|
85fd9ca799 | ||
|
|
86c701bff9 | ||
|
|
f4e318f9f2 | ||
|
|
287a6cb3a2 | ||
|
|
c29c276dc1 | ||
|
|
3d0705fdb7 | ||
|
|
8f7b242610 | ||
|
|
d0aa2fbd52 | ||
|
|
df845f45a0 | ||
|
|
d55976eff0 | ||
|
|
b2e578b21d | ||
|
|
f7b7e2c13a | ||
|
|
1550ce2976 | ||
|
|
7489bef085 | ||
|
|
7c0b0c2572 | ||
|
|
5e2d314efe | ||
|
|
29269b9cf0 | ||
|
|
c5c57508b3 | ||
|
|
5e0d6826da | ||
|
|
e347f99542 | ||
|
|
38b8c55e52 | ||
|
|
255ac1bad0 | ||
|
|
522bbddccf | ||
|
|
66ad03948b | ||
|
|
18de6d26d3 | ||
|
|
424ab000dd | ||
|
|
28aeacbc39 | ||
|
|
48ccb1da30 | ||
|
|
2345888c4e | ||
|
|
df194e2a93 | ||
|
|
c7b5d99133 | ||
|
|
e97c297113 | ||
|
|
c9843479db | ||
|
|
7264c321f8 | ||
|
|
8541eb75c5 | ||
|
|
ff7d8d08dc | ||
|
|
ce74ac8777 | ||
|
|
94799b4a63 | ||
|
|
b3690e8d31 | ||
|
|
7ccb2da19f | ||
|
|
0f014129ad | ||
|
|
60ac321a1b | ||
|
|
c931d8f4f8 | ||
|
|
b940eb460f | ||
|
|
8a9c2f1ce4 | ||
|
|
b033b8e9da | ||
|
|
bafe9677d9 | ||
|
|
c44bae0d0d | ||
|
|
093e85e47a | ||
|
|
630912df55 | ||
|
|
3d17104228 | ||
|
|
992eb67143 | ||
|
|
b14ad0cc1a | ||
|
|
c7220a45ab | ||
|
|
4e03e363ed | ||
|
|
40a97daa9a | ||
|
|
12f125c77e | ||
|
|
e6067dfaec | ||
|
|
86b30ece98 | ||
|
|
836d69178c | ||
|
|
cb70c2b1af | ||
|
|
200a559183 | ||
|
|
30b3886f9f | ||
|
|
9d79bb8e36 | ||
|
|
ab87ef734d | ||
|
|
3d88779eff | ||
|
|
d88e57675d | ||
|
|
3c17d19b7e | ||
|
|
0b16481633 | ||
|
|
ecd2545fcf | ||
|
|
930b403799 | ||
|
|
4a6da3bf84 | ||
|
|
16a8995a9a | ||
|
|
5673e2e772 | ||
|
|
349a40a261 | ||
|
|
5b6f49ee0c | ||
|
|
740a690644 | ||
|
|
9bb3cc2664 | ||
|
|
a0a4bcc903 | ||
|
|
2b76518c22 | ||
|
|
7cad4633dc | ||
|
|
5c9b7ca313 | ||
|
|
f293f10e7e | ||
|
|
721cb34a9d | ||
|
|
7b4033bc35 | ||
|
|
68d456bb58 | ||
|
|
87f884d7a1 | ||
|
|
294a5db4c1 | ||
|
|
8765f766e1 | ||
|
|
b9707e0d8f | ||
|
|
a7f813b7a9 | ||
|
|
f33bdc731d | ||
|
|
e2f02327c3 | ||
|
|
39385feb78 | ||
|
|
fa66c6a025 | ||
|
|
4a01ecdfae | ||
|
|
cc95f526e0 | ||
|
|
9fc18dbd89 | ||
|
|
de3d83f14b | ||
|
|
0fc9f2afec | ||
|
|
a9948de99e | ||
|
|
7f9a97b0fa | ||
|
|
935b38a962 | ||
|
|
ea4d5e7839 | ||
|
|
d566e5bc8e | ||
|
|
2c2e493df8 | ||
|
|
2fc90c44e6 | ||
|
|
aa17e3319f | ||
|
|
883a06e9c0 | ||
|
|
85fd8b6494 | ||
|
|
598b1bd911 | ||
|
|
ea3b565f9c | ||
|
|
97a020ecf0 | ||
|
|
5e6b23aeae | ||
|
|
5a650eeee7 | ||
|
|
53d00ab76a | ||
|
|
823e222bd0 | ||
|
|
70857bd1ad | ||
|
|
6a64db0b8c | ||
|
|
ee638bda6c | ||
|
|
25105605f1 | ||
|
|
022a3bd6b9 | ||
|
|
e7d358b541 | ||
|
|
2339cc8e85 | ||
|
|
0af744bc3a | ||
|
|
87cce2e2b8 | ||
|
|
8c45d4a9f2 | ||
|
|
dba8934aa2 | ||
|
|
b7cecde57e | ||
|
|
46abe2e29a | ||
|
|
81a007552b | ||
|
|
db016d4675 | ||
|
|
9d2e395437 | ||
|
|
10b9d6950d | ||
|
|
e87c13e919 | ||
|
|
252f0d4fac | ||
|
|
d3cebe58a0 | ||
|
|
1f29e374be | ||
|
|
4eac1b50d5 | ||
|
|
1dc8eb1adb | ||
|
|
41e866b46c | ||
|
|
8e4044bb37 | ||
|
|
d8b059813a | ||
|
|
6819feb8b2 | ||
|
|
79f3e35682 | ||
|
|
b15acbaae9 | ||
|
|
0cf6bb3087 | ||
|
|
a28a19dc63 | ||
|
|
a63c8e851b | ||
|
|
1fe68ba11e | ||
|
|
d21fb765e2 | ||
|
|
de80856cd9 | ||
|
|
1575baca36 | ||
|
|
31ba6d3d27 | ||
|
|
2ad83f37fd | ||
|
|
2767e2995d | ||
|
|
7b70d82322 | ||
|
|
2549d9539e | ||
|
|
6e3e4bcfdd | ||
|
|
c965a7fbdd | ||
|
|
8f2c4bbc5c | ||
|
|
b8e219e7d1 | ||
|
|
7393ae80f5 | ||
|
|
21b0b1f0b2 | ||
|
|
0e17caa52a | ||
|
|
ff168048be | ||
|
|
e14644f77a | ||
|
|
b0c93a598e | ||
|
|
a22f029c2c | ||
|
|
8e6344ee4e | ||
|
|
8ed76b40a6 | ||
|
|
ad78a418bc | ||
|
|
b8cc0de44a | ||
|
|
a7ab148483 | ||
|
|
51ad469ef2 | ||
|
|
7678d4cca3 | ||
|
|
6dccf0390d | ||
|
|
3f23a77eab | ||
|
|
cb8ff16d86 | ||
|
|
915256cb8c | ||
|
|
1ee4b3c2a0 | ||
|
|
9a1d549686 | ||
|
|
3777d3a62c | ||
|
|
80891d3239 | ||
|
|
c0727c261d | ||
|
|
2eac93c645 | ||
|
|
4ec0ece457 | ||
|
|
50fa5c206f | ||
|
|
ef93835e22 | ||
|
|
25a471fe59 | ||
|
|
e02b7941a8 | ||
|
|
ad443e853a | ||
|
|
1fa2dda7c8 | ||
|
|
abb5f00d81 | ||
|
|
69b8c7c0bc | ||
|
|
f4e7a76886 | ||
|
|
74d63f099c | ||
|
|
f448c1c73c | ||
|
|
35e7dbc11c | ||
|
|
05369e748a | ||
|
|
ade951688e | ||
|
|
5a28c04c44 | ||
|
|
8db5feccc7 | ||
|
|
0337e15a2e | ||
|
|
e58106d552 | ||
|
|
364611f7d7 | ||
|
|
4471bc0770 |
+6
-3
@@ -1,6 +1,9 @@
|
||||
# These are supported funding model platforms
|
||||
# GitHub Sponsors isn't set up for this account — fund via Ko-fi or PayPal.
|
||||
|
||||
github: [debpalash]
|
||||
# ko_fi: omnivoice
|
||||
ko_fi: debpalash
|
||||
custom:
|
||||
- "https://paypal.me/palashCoder"
|
||||
- "https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md"
|
||||
# github: [debpalash] # not available
|
||||
# open_collective: omnivoice-studio
|
||||
# custom: ["https://omnivoice.palash.dev/sponsor"]
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
name: 🤝 Sponsorship inquiry
|
||||
description: Support OmniVoice and (optionally) claim a logo slot. Not for bugs or feature requests.
|
||||
title: "Sponsorship inquiry: "
|
||||
labels: ["sponsor"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for considering sponsoring **OmniVoice Studio** 💛
|
||||
|
||||
OmniVoice is free, local-first, and AGPL-3.0 — sponsorship keeps development going.
|
||||
See **[SPONSORS.md](https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md)** for tiers, placements, and logo guidelines.
|
||||
Prefer to just donate? [Ko-fi](https://ko-fi.com/debpalash) (recurring) or [PayPal](https://paypal.me/palashCoder) (one-time) — you don't need this form for that.
|
||||
- type: input
|
||||
id: name
|
||||
attributes:
|
||||
label: Name or organization
|
||||
description: How you'd like to be credited (person or company).
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: website
|
||||
attributes:
|
||||
label: Website / link
|
||||
description: The URL your name or logo should link to (homepage, product page, profile…).
|
||||
placeholder: https://example.com
|
||||
- type: input
|
||||
id: logo
|
||||
attributes:
|
||||
label: Logo URL (optional)
|
||||
description: Link to your logo (SVG preferred, else 2× PNG, transparent background). You can also attach it in the description below.
|
||||
placeholder: https://example.com/logo.svg
|
||||
- type: dropdown
|
||||
id: tier
|
||||
attributes:
|
||||
label: Tier you're interested in
|
||||
description: See SPONSORS.md for what each tier includes. Not sure? Pick "Not sure yet".
|
||||
options:
|
||||
- Backer
|
||||
- Bronze
|
||||
- Silver
|
||||
- Gold
|
||||
- Not sure yet — let's talk
|
||||
- Custom / annual arrangement
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: method
|
||||
attributes:
|
||||
label: How you'd like to support
|
||||
options:
|
||||
- Ko-fi (recurring)
|
||||
- Ko-fi (one-time)
|
||||
- PayPal (one-time)
|
||||
- Not sure yet — let's discuss
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: contact
|
||||
attributes:
|
||||
label: How should we reach you?
|
||||
description: Email or another contact. (GitHub will also notify you on this issue.)
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: notes
|
||||
attributes:
|
||||
label: Anything else?
|
||||
description: Questions, constraints, timeline, or context. Attach your logo here if you didn't link it above.
|
||||
- type: checkboxes
|
||||
id: ack
|
||||
attributes:
|
||||
label: Acknowledgements
|
||||
options:
|
||||
- label: I understand sponsorship is a thank-you, not a paywall — OmniVoice stays fully free and AGPL-3.0, and sponsors don't get gated features.
|
||||
required: true
|
||||
- label: If I provide a logo, I have the right to use it and grant OmniVoice permission to display it in the README, the app, and the project website.
|
||||
required: false
|
||||
@@ -105,9 +105,25 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: bun run typecheck:ci
|
||||
|
||||
# oxlint gate — fast Rust linter, blocks on errors so lint debt can't
|
||||
# re-accumulate (warnings, incl. the react-compiler advisories in
|
||||
# `lint:hooks`, are non-blocking). See frontend/.oxlintrc.json.
|
||||
- name: Frontend lint (oxlint)
|
||||
working-directory: frontend
|
||||
run: bun run lint
|
||||
|
||||
# oxfmt format gate — JS/TS/JSX only (CSS/JSON/Tauri excluded; see
|
||||
# frontend/.oxfmtrc.json). `bun run format` fixes locally.
|
||||
- name: Frontend format check (oxfmt)
|
||||
working-directory: frontend
|
||||
run: bun run format:check
|
||||
|
||||
# `bun run test` (frontend/package.json), not `bunx vitest` — bunx
|
||||
# resolves by npm package name and can miss workspace-hoisted bins,
|
||||
# then falls back to fetching from npm (#962 class).
|
||||
- name: Run Vitest (frontend)
|
||||
working-directory: frontend
|
||||
run: bunx vitest run
|
||||
run: bun run test
|
||||
|
||||
# Legacy node:test runner for tests/frontend/*.test.mjs
|
||||
- name: Run frontend node:test (legacy)
|
||||
@@ -133,7 +149,7 @@ jobs:
|
||||
- os: windows-2022
|
||||
label: Windows
|
||||
rust_target: x86_64-pc-windows-msvc
|
||||
- os: ubuntu-22.04
|
||||
- os: ubuntu-24.04
|
||||
label: Linux
|
||||
rust_target: x86_64-unknown-linux-gnu
|
||||
runs-on: ${{ matrix.os }}
|
||||
|
||||
+135
-32
@@ -4,12 +4,16 @@
|
||||
# - push of a tag matching `v*` (e.g. `v0.2.0`) → full STABLE release,
|
||||
# publishes artifacts + signed updater manifest (`latest.json`) to the
|
||||
# tag's GH Release. This is the default Stable updater channel.
|
||||
# - workflow_dispatch (publish_preview=true) → builds the selected branch and
|
||||
# publishes a rolling `preview` PRERELEASE with its own signed
|
||||
# `latest.json` at releases/download/preview/. This feeds the opt-in
|
||||
# Preview updater channel (Settings → About → Update channel). The stable
|
||||
# `latest` release is untouched. Run this manually whenever you want to cut
|
||||
# a preview from `main`.
|
||||
# - schedule (nightly, 07:00 UTC) → rolling `preview` PRERELEASE built from
|
||||
# `main` with its own signed `latest.json` at releases/download/preview/.
|
||||
# Feeds the opt-in Preview updater channel (Settings → About → Update
|
||||
# channel). The `preview-gate` job skips the matrix on nights when `main`
|
||||
# didn't move, so an idle day costs only a ~30s gate job — keeping Preview
|
||||
# ≤24h behind `main` at a predictable ~1-matrix/day cost. The stable
|
||||
# `latest` release is untouched.
|
||||
# - workflow_dispatch (publish_preview=true) → the same preview build on
|
||||
# demand from the selected branch (e.g. to preview a feature branch, or to
|
||||
# refresh immediately without waiting for the nightly).
|
||||
# - workflow_dispatch (publish_preview=false) → on-demand build (prior
|
||||
# behavior; draft release named after the branch).
|
||||
#
|
||||
@@ -28,6 +32,10 @@ name: Desktop Release
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
schedule:
|
||||
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
|
||||
# preview-gate job no-ops the matrix when main hasn't moved in a day.
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
draft:
|
||||
@@ -121,8 +129,41 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
|
||||
|
||||
# Decide preview-vs-stable, and for nightly runs whether `main` actually
|
||||
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
|
||||
# `preview-notes` job, so a no-commit night costs only this ~30s job.
|
||||
preview-gate:
|
||||
name: Preview gate
|
||||
runs-on: ubuntu-22.04
|
||||
outputs:
|
||||
is_preview: ${{ steps.decide.outputs.is_preview }}
|
||||
proceed: ${{ steps.decide.outputs.proceed }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 50
|
||||
- id: decide
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
event="${{ github.event_name }}"
|
||||
if [ "$event" = "schedule" ] || { [ "$event" = "workflow_dispatch" ] && [ "${{ inputs.publish_preview }}" = "true" ]; }; then
|
||||
echo "is_preview=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "is_preview=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
# Nightly: skip the matrix when main hasn't moved in the last day.
|
||||
if [ "$event" = "schedule" ] && [ -z "$(git log --since='25 hours ago' --oneline)" ]; then
|
||||
echo "No new commits on main in the last day — skipping nightly preview."
|
||||
echo "proceed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "proceed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
build:
|
||||
needs: test
|
||||
needs: [test, preview-gate]
|
||||
# Nightly runs with no new commits on main skip the 4-platform matrix.
|
||||
if: needs.preview-gate.outputs.proceed == 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -149,6 +190,14 @@ jobs:
|
||||
# backlog that motivated the original drop is contained by
|
||||
# fail-fast:false — a slow Intel leg can delay the release run but
|
||||
# can't fail the other targets.
|
||||
#
|
||||
# #889 (2026-07): Intel macOS is now UNSUPPORTED for the local
|
||||
# backend — torch ≥2.3 ships no macOS x86_64 wheels, so the venv
|
||||
# bootstrap can never succeed on Intel. The shipped x64 artifact is
|
||||
# effectively UI-only (usable with a remote backend); the app now
|
||||
# pre-fails first-run bootstrap with an honest message on Intel.
|
||||
# Whether to keep shipping this x64 leg (UI-only) or drop it is an
|
||||
# OWNER CALL — deliberately not changed in the #889 PR.
|
||||
- os: macos-15-intel
|
||||
arch: x86_64-apple-darwin
|
||||
label: "macOS Intel"
|
||||
@@ -164,13 +213,24 @@ jobs:
|
||||
bundles: "msi,updater"
|
||||
|
||||
# Linux: ship .AppImage only. AppImage is universal (no distro
|
||||
# package-manager dep), runs on any glibc-2.31+ host, and is the
|
||||
# package-manager dep), runs on any glibc-2.39+ host, and is the
|
||||
# Linux auto-update target. The .deb target was dropped: tauri-bundler
|
||||
# fails it with "Failed to create control scripts: No such file or
|
||||
# directory" (no custom deb config of ours is at fault) — revisit on a
|
||||
# tauri-cli bump. FUSE unavailability on GH runners is handled via
|
||||
# APPIMAGE_EXTRACT_AND_RUN=1.
|
||||
- os: ubuntu-22.04
|
||||
#
|
||||
# Bumped from ubuntu-22.04 → ubuntu-24.04 (#961): the AppImage
|
||||
# bundles whatever `libwebkit2gtk-4.1-dev` the build runner's apt
|
||||
# repos resolve (see the "Linux system deps" step below) — 22.04's
|
||||
# was meaningfully stale relative to what current Ubuntu/Fedora
|
||||
# ship, and AppRun's LD_LIBRARY_PATH makes that bundled, stale copy
|
||||
# take priority over a healthy system WebKitGTK at runtime. Raises
|
||||
# the AppImage's glibc floor from 2.35 to 2.39 — pre-2022 distros
|
||||
# (Ubuntu <22.04, Debian <12) lose support; no report of anyone on
|
||||
# something that old has come in, and the project's own install
|
||||
# docs already assume Debian 12 / Ubuntu 22.04+.
|
||||
- os: ubuntu-24.04
|
||||
arch: x86_64-unknown-linux-gnu
|
||||
label: "Linux x64"
|
||||
rust_target: x86_64-unknown-linux-gnu
|
||||
@@ -427,11 +487,14 @@ jobs:
|
||||
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
|
||||
# is also correctly above the last stable.
|
||||
- name: Stamp preview version
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
CONF=frontend/src-tauri/tauri.conf.json
|
||||
# package.json is the single source of truth; tauri.conf.json reads its
|
||||
# version from it ("version": "../package.json"), so stamping
|
||||
# package.json restamps the whole bundle.
|
||||
CONF=frontend/package.json
|
||||
BASE=$(jq -r .version "$CONF")
|
||||
# MSI/WiX requires the semver pre-release identifier to be numeric-only
|
||||
# (and <= 65535). "preview.N" hard-fails the Windows bundler, so the
|
||||
@@ -471,11 +534,13 @@ jobs:
|
||||
# rolling `preview` prerelease for the updater's Preview channel.
|
||||
# Every other invocation — crucially the `v*` tag-push stable release
|
||||
# — evaluates these expressions to exactly their prior values.
|
||||
tagName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'preview' || github.ref_name }}
|
||||
releaseName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
|
||||
tagName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
|
||||
# Version-first so the tag is readable in GitHub's truncated
|
||||
# release-list sidebar (which clips the title mid-string).
|
||||
releaseName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'Preview — OmniVoice Studio' || format('{0} — OmniVoice Studio', github.ref_name) }}
|
||||
releaseBody: ${{ steps.changelog.outputs.body }}
|
||||
releaseDraft: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) || false }}
|
||||
releaseDraft: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ needs.preview-gate.outputs.is_preview == 'true' }}
|
||||
updaterJsonPreferNsis: false
|
||||
includeUpdaterJson: true
|
||||
|
||||
@@ -661,8 +726,8 @@ jobs:
|
||||
# preview-only — stable `v*` releases keep their CHANGELOG section + the
|
||||
# appended checksums.
|
||||
preview-notes:
|
||||
needs: build
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
needs: [build, preview-gate]
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
@@ -693,16 +758,51 @@ jobs:
|
||||
echo ""
|
||||
echo "$CONTRIB"
|
||||
} > /tmp/preview-notes.md
|
||||
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
|
||||
# --prerelease re-asserts the flag every run: a non-prerelease
|
||||
# `preview` release is eligible to become GitHub's "Latest", which is
|
||||
# the exact URL the Stable updater channel reads — so it must never
|
||||
# flip off.
|
||||
gh release edit preview --repo "$REPO" --prerelease --notes-file /tmp/preview-notes.md
|
||||
echo "Applied auto-generated release notes + contributors to the preview release."
|
||||
|
||||
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
|
||||
# main is always last-release + 1 patch. The moment a stable v* tag is
|
||||
# released, bump the three version sources on main to the next patch so every
|
||||
# PR and preview build identifies as the next version. Pushes directly to
|
||||
# main with the workflow token (a metadata-only commit; CI runs on PRs).
|
||||
- name: Verify preview updater manifest (prerelease + platform parity)
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The preview release must stay a prerelease (or it can hijack the
|
||||
# Stable channel's releases/latest endpoint), and its updater manifest
|
||||
# must cover every platform stable does (else those users — e.g. Intel
|
||||
# Mac — silently get no preview updates).
|
||||
is_pre=$(gh release view preview --repo "$REPO" --json isPrerelease -q .isPrerelease)
|
||||
test "$is_pre" = "true" || { echo "::error::preview release is not a prerelease"; exit 1; }
|
||||
curl -fsSL "https://github.com/$REPO/releases/download/preview/latest.json" -o /tmp/preview-latest.json
|
||||
curl -fsSL "https://github.com/$REPO/releases/latest/download/latest.json" -o /tmp/stable-latest.json
|
||||
python3 - <<'PY'
|
||||
import json, re
|
||||
prev = json.load(open("/tmp/preview-latest.json"))
|
||||
stab = json.load(open("/tmp/stable-latest.json"))
|
||||
v = prev.get("version", "")
|
||||
assert re.fullmatch(r"\d+\.\d+\.\d+-\d+", v), f"preview version not X.Y.Z-N: {v!r}"
|
||||
pk, sk = set(prev.get("platforms", {})), set(stab.get("platforms", {}))
|
||||
missing = sk - pk
|
||||
assert not missing, f"preview manifest missing platforms vs stable: {sorted(missing)}"
|
||||
print(f"preview manifest OK: {v} platforms={sorted(pk)}")
|
||||
PY
|
||||
|
||||
# ── Post-release version bump (OWNER-GATED as of 2026-07-01) ──────────────
|
||||
# Previously auto-ran after every stable v* tag to keep main = release + 1.
|
||||
# The owner now controls bumps manually ("keep 0.3.8; I say when to bump"), so
|
||||
# this job is OPT-IN: it runs ONLY when the repo variable AUTO_VERSION_BUMP is
|
||||
# set to 'true' (Settings → Secrets and variables → Actions → Variables).
|
||||
# Unset/anything-else → main stays at whatever it is after release. Re-enable
|
||||
# by setting the variable; disable again by unsetting it.
|
||||
version-bump:
|
||||
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
|
||||
if: >-
|
||||
github.event_name == 'push' && github.ref_type == 'tag'
|
||||
&& !contains(github.ref, '-')
|
||||
&& vars.AUTO_VERSION_BUMP == 'true'
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
@@ -718,23 +818,26 @@ jobs:
|
||||
RELEASED="${GITHUB_REF_NAME#v}"
|
||||
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
|
||||
NEXT="$MAJ.$MIN.$((PAT + 1))"
|
||||
CURRENT=$(jq -r .version frontend/src-tauri/tauri.conf.json)
|
||||
# frontend/package.json is the SINGLE SOURCE OF TRUTH: vite injects
|
||||
# __APP_VERSION__ from it, and tauri.conf.json reads its bundle version
|
||||
# from it ("version": "../package.json"). Read CURRENT from it.
|
||||
CURRENT=$(jq -r .version frontend/package.json)
|
||||
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
|
||||
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
|
||||
fi
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/src-tauri/tauri.conf.json > "$tmp"
|
||||
mv "$tmp" frontend/src-tauri/tauri.conf.json
|
||||
# frontend/package.json drives __APP_VERSION__ (vite.config.js) — the
|
||||
# first-run footer + every auto bug report. Keep it in lockstep too,
|
||||
# set absolutely (jq) so any prior drift self-heals. (#248-sweep finding)
|
||||
# Bump the canonical (package.json), set absolutely so any prior drift
|
||||
# self-heals. tauri.conf.json needs no edit — it derives from this.
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
|
||||
mv "$tmp" frontend/package.json
|
||||
# The remaining files are CI-guarded mirrors (cargo/uv require a
|
||||
# literal; the version.py literal is the frozen-backend last resort) —
|
||||
# bump them in lockstep with the canonical.
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
|
||||
sed -i "0,/_FALLBACK_VERSION = \"$CURRENT\"/s//_FALLBACK_VERSION = \"$NEXT\"/" backend/core/version.py
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add frontend/src-tauri/tauri.conf.json frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml
|
||||
git add frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml backend/core/version.py
|
||||
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
|
||||
git push origin main
|
||||
|
||||
+1047
File diff suppressed because it is too large
Load Diff
@@ -3,7 +3,7 @@
|
||||
|
||||
**OmniVoice Studio**
|
||||
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The current version lives in `frontend/package.json` (the single source of truth — see Versioning); the latest stable tag is on the [Releases page](https://github.com/debpalash/OmniVoice-Studio/releases/latest). With `AUTO_VERSION_BUMP` off (the current owner setting), `main` holds at the released version between releases.
|
||||
|
||||
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
|
||||
|
||||
@@ -16,188 +16,36 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
|
||||
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
|
||||
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
<!-- GSD:project-end -->
|
||||
|
||||
<!-- GSD:stack-start source:research/STACK.md -->
|
||||
## Technology Stack
|
||||
|
||||
## Recommended Stack — Per Capability
|
||||
### Capability 1 — HuggingFace Token Persistence (issue #35)
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| `huggingface_hub` (already pinned transitively by `transformers>=5.3.0`) | `≥1.12.x` (latest 2026) | Auth + cache + token storage | Canonical, used by every HF library already in the stack. `HfFolder` is **superseded** in v1.x by the higher-level `login()` / `auth_list()` / `auth_switch()` API. |
|
||||
| `keyring` (Python) | `≥25.x` | Optional OS-keychain backing | Only adopt if a future hardening pass wants Keychain/Credential-Manager/SecretService. **Not recommended for this milestone** — adds a native dep (`dbus`, `pywin32`) per platform with no real security win over `0600` file storage in `HF_HOME`. |
|
||||
| Shell | One-liner to persist `HF_TOKEN` |
|
||||
|-------|---------------------------------|
|
||||
| macOS zsh (default since 10.15) | `echo 'export HF_TOKEN=hf_xxx' >> ~/.zshrc && source ~/.zshrc` |
|
||||
| Linux bash | `echo 'export HF_TOKEN=hf_xxx' >> ~/.bashrc && source ~/.bashrc` |
|
||||
| Windows PowerShell (user scope) | `[Environment]::SetEnvironmentVariable("HF_TOKEN","hf_xxx","User")` (new shells only) |
|
||||
| Windows cmd | `setx HF_TOKEN "hf_xxx"` (user scope, new shells only) |
|
||||
- [HF environment variables docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH confidence (official, current)
|
||||
- [HF authentication API docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH confidence
|
||||
- [Microsoft `setx` docs](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH confidence
|
||||
### Capability 2 — In-App Structured Bug Reporting (opt-in, GitHub Issues)
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| GitHub REST API `POST /repos/{owner}/{repo}/issues` | `2026-03-10` API version | Server-side issue creation | Official, stable. Requires auth. |
|
||||
| **Prefilled-URL pattern** (`github.com/{owner}/{repo}/issues/new?title=…&body=…&labels=…`) | n/a | Zero-auth fallback | **This is the recommended primary path for v0.3.x.** No token needed, no GitHub App registration needed, user's browser opens with a prefilled form, they review and click Submit. They own the issue, the OSS project gets the report, and OmniVoice never holds a credential. |
|
||||
| `gh-app-jwt` + GitHub App (Rust crate `octocrab` or Python `pygithub`) | only if we later want fully-automated submission | Programmatic posting under an app identity | **Defer to a later milestone.** Requires registering a public GitHub App, hosting a token-exchange endpoint, and managing rate-limit quotas — disproportionate for stabilization scope. |
|
||||
| `platform`, `psutil`, `torch.cuda` (already in deps) | already pinned | Capture OS, CPU/GPU/VRAM info | No new deps. |
|
||||
| `httpx` (already in `dev-dependencies`, promote to runtime if needed) | `≥0.28.1` | HTTP for the API call path (if/when we add auth) | Modern async-first, already used in test suite. |
|
||||
- ✓ No token storage in OmniVoice → no security surface
|
||||
- ✓ Opt-in by definition (user has to click Submit on github.com)
|
||||
- ✓ User owns the issue → can be replied to, edited, closed by them
|
||||
- ✓ Zero infra cost — no proxy, no app, no rate-limit management
|
||||
- ✓ Works identically on macOS / Windows / Linux via Tauri's `shell.open`
|
||||
- ✓ Survives our project being forked (just change the URL)
|
||||
- OS name + version (`platform.platform()`)
|
||||
- Python version (`sys.version`)
|
||||
- OmniVoice version (`pyproject.toml`)
|
||||
- Backend git SHA (if installed from source) or installer build ID
|
||||
- CPU model, RAM (`psutil.cpu_count()`, `psutil.virtual_memory()`)
|
||||
- GPU vendor/model/VRAM (`torch.cuda.get_device_name()`, `torch.cuda.mem_get_info()`, MPS detect)
|
||||
- Active TTS engine + list of installed engines
|
||||
- Frontend: bun version, OS shell
|
||||
- Last error message + stack trace if launched from an error toast
|
||||
- Audio file contents (privacy — reference samples may contain user's voice)
|
||||
- File paths containing `/Users/<name>/` (strip home dir → `~/`)
|
||||
- HF token, OpenAI keys, any env var matching `*TOKEN*|*KEY*|*SECRET*`
|
||||
- [GitHub URL query parameters for issues](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/creating-an-issue#creating-an-issue-from-a-url-query) — HIGH confidence
|
||||
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (widely used reference impl)
|
||||
- [GitHub REST API: Create an issue](https://docs.github.com/en/rest/issues/issues#create-an-issue) — HIGH confidence (for the future auto-submit path)
|
||||
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — reviewed, **rejected for milestone** due to local-first constraint
|
||||
### Capability 3 — `uv venv` Mirror Fallback for Restricted Networks (issues #57, #60)
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| `uv` (already used) | `≥0.5.x` | Python+venv bootstrap | Existing dep. |
|
||||
| `UV_PYTHON_INSTALL_MIRROR` env var | uv `0.4.x`+ | Override python-build-standalone download URL | **Official, current.** Replaces `https://github.com/astral-sh/python-build-standalone/releases/download/...` in download URL construction. No built-in fallback if mirror fails. |
|
||||
| `UV_PYTHON_PREFERENCE=only-system` (or CLI flag `--python-preference only-system`) | uv `0.4.x`+ | Skip the python-build-standalone download entirely; use the user's system Python | **The reliable escape hatch** when no mirror works. Requires a compatible Python `>=3.11` to already be on PATH. |
|
||||
| `UV_HTTP_TIMEOUT`, `UV_HTTP_CONNECT_TIMEOUT`, `UV_HTTP_RETRIES` | uv `0.4.x`+ | Tune retry behavior for flaky links | Defaults are 30s / 10s / 3 — bump to 120s / 30s / 5 for restricted networks. |
|
||||
# Pseudocode for the bootstrap
|
||||
# Final fallback: don't download Python at all
|
||||
- `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (Tsinghua — fastest in China)
|
||||
- `UV_DEFAULT_INDEX=https://mirrors.aliyun.com/pypi/simple` (Aliyun fallback)
|
||||
- Russia: no major government-blessed PyPI mirror; users typically tunnel via VPN. Document this honestly rather than ship a broken default.
|
||||
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (official)
|
||||
- [uv issue #5224 — python-build-standalone mirror support](https://github.com/astral-sh/uv/issues/5224) — HIGH (the feature was added)
|
||||
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms real user pain, no built-in fallback)
|
||||
- [uv python-versions concepts](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH (documents `python-preference` semantics)
|
||||
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained mirror list; verify each URL still works before shipping)
|
||||
### Capability 4 — Supertonic-3 TTS Engine
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| `supertonic` (PyPI) | `1.3.1` (latest, May 18 2026 — Phase 3 Wave 1 to verify constructor signature before bump) | Official Supertonic-3 inference SDK | Authoritative wrapper from Supertone Inc. Wraps the ONNX session orchestration so we don't have to. |
|
||||
| `onnxruntime` | `≥1.17.x` (any recent) | ONNX inference runtime | Already a transitive dep of WhisperX (via CTranslate2 path is separate, but `onnxruntime` itself ships for kittentts and audioseal). Verify with `uv tree` after adding — should resolve cleanly. |
|
||||
| `huggingface_hub` (already pinned) | `≥1.12.x` | Model weight download (~400 MB on first use) | Reuses existing HF token + cache infrastructure. The user's existing `HF_TOKEN` (Capability 1) works for the Supertonic model download too. |
|
||||
| `numpy`, `soundfile` (already pinned) | already pinned | Audio I/O + array math | No new deps. |
|
||||
- `text_encoder.onnx`
|
||||
- `latent_denoiser.onnx`
|
||||
- `voice_decoder.onnx`
|
||||
- 44.1 kHz sample rate, 24-dim latent, 128-dim style
|
||||
- ~99M parameters total
|
||||
- Tokenizer: `AutoTokenizer.from_pretrained(model_path)` — loads from `tokenizer.json` shipped with model
|
||||
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official)
|
||||
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official)
|
||||
- [supertonic PyPI page](https://pypi.org/project/supertonic/) — HIGH (`1.3.1` confirmed 2026-05-18; same publisher, MIT, same 4 deps)
|
||||
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure details)
|
||||
### Capability 5 — Cross-Platform Documentation Tooling
|
||||
| Technology | Version | Purpose | Why Recommended |
|
||||
|------------|---------|---------|-----------------|
|
||||
| Plain Markdown in `docs/` + GitHub-rendered (current state) | n/a | Install tutorial, troubleshooting | Zero new infra. Renders inline on GitHub for issue-replies. No build step to break. |
|
||||
| Existing `scripts/smoke-test.sh` + Playwright `tests/` (already in `package.json`) | already pinned | Verify install paths actually work | **This is the real solution to "docs drift."** If smoke-test exercises the install path described in docs, docs that drift will break CI. |
|
||||
| **Future** (defer): Astro Starlight | `≥0.30` | Standalone docs site at `docs.omnivoice.studio` | Adopt only when docs exceed ~20 markdown files and need search/versioning. Tauri, the framework OmniVoice already depends on, uses Starlight — well-traveled choice. Material for MkDocs entered maintenance mode in November 2025 per Docsio's 2026 review — **avoid** for new docs. |
|
||||
| Project | What they do |
|
||||
|---------|--------------|
|
||||
| **OBS Studio** | Docs at `obsproject.com/docs` (Sphinx, separate repo). Install paths in README, wiki for community-contributed. CI doesn't gate on docs drift. |
|
||||
| **Audacity** | Manual at `manual.audacityteam.org` (MediaWiki). README is minimal. Install path = "use the installer." No automated sync. |
|
||||
| **Tauri** | Docs at `v2.tauri.app` (Astro Starlight, separate repo `tauri-apps/tauri-docs`). README is minimal. Heavy reliance on community contributions and PR review. |
|
||||
| **VS Code** | Docs at `code.visualstudio.com/docs` (separate repo, Markdown). README is minimal. Manual sync; docs team is staffed. |
|
||||
- [Tauri docs (Astro Starlight)](https://github.com/tauri-apps/tauri-docs) — HIGH (reference for "if we ever move off README")
|
||||
- [OBS Studio docs](https://docs.obsproject.com/) — HIGH (Sphinx, separate site reference)
|
||||
- [Audacity Manual](https://manual.audacityteam.org/) — HIGH (MediaWiki reference)
|
||||
- [Docsio: Material for MkDocs 2026 review (maintenance mode)](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with project's own GitHub activity)
|
||||
- [Docsio: Starlight 2026 review](https://docsio.co/blog/starlight-docs) — MEDIUM
|
||||
## Installation
|
||||
# No new Python dependencies needed for Capabilities 1, 2, 3, 5.
|
||||
# Only Capability 4 adds a runtime dep:
|
||||
# Verify no regressions:
|
||||
# Should show single versions of each; no duplicates.
|
||||
## Alternatives Considered
|
||||
| Recommended | Alternative | When to Use Alternative |
|
||||
|-------------|-------------|-------------------------|
|
||||
| HF token via in-app Settings → `huggingface_hub.login()` | OS keyring via `keyring` package | Only if a security hardening milestone later demands OS-native credential storage. Not worth the cross-platform native-dep cost for v0.3.x. |
|
||||
| Prefilled-URL GitHub Issues | GitHub App + device flow + authenticated POST | When milestone budget can afford registering a public GitHub App and hosting a token-exchange function. Defer. |
|
||||
| Prefilled-URL GitHub Issues | Sentry / `sentry-tauri` | Never — violates the "no third-party telemetry endpoint" constraint in PROJECT.md. |
|
||||
| `UV_PYTHON_INSTALL_MIRROR` chain + `only-system` fallback | Bundle Python in the Tauri installer | Adds ~30 MB to every installer for ~5% of users. Revisit if the bootstrap is still a top complaint in v0.4. |
|
||||
| In-repo Markdown docs | Astro Starlight standalone site | When docs grow past ~20 pages and need full-text search. Tauri provides a precedent if/when we get there. |
|
||||
| In-repo Markdown docs | MkDocs / Material for MkDocs | **Avoid** for new sites — Material for MkDocs is in maintenance mode as of Nov 2025. |
|
||||
## What NOT to Use
|
||||
| Avoid | Why | Use Instead |
|
||||
|-------|-----|-------------|
|
||||
| `HfFolder.save_token()` directly | Older API; v1.x `login()` does the same plus git-credential integration and is the documented path | `huggingface_hub.login(token=val, add_to_git_credential=False)` |
|
||||
| Setting `HF_TOKEN` via shell rc files as the *only* persistence mechanism | Different per OS, fragile, opaque to the user, breaks in installer-launched processes that don't source shell rc | Write to `$HF_HOME/token` via `login()`. Document env var as override only. |
|
||||
| `setx` for HF token persistence | Doesn't propagate to current shell; common source of "I set it but it's empty" bug reports | `[Environment]::SetEnvironmentVariable(...,"User")` in PowerShell, or the in-app Settings field |
|
||||
| PAT-based GitHub Issues posting from OmniVoice | Would require shipping or asking for a token; breaks local-first promise | Prefilled-URL pattern (user submits from their browser) |
|
||||
| `sentry-tauri` for OmniVoice | Third-party telemetry endpoint — violates PROJECT.md constraint | Local-only `backend.log` rotation + opt-in prefilled-URL reporter |
|
||||
| `hf_transfer` for downloads | Deprecated in favor of `hf-xet` per HF docs | Default `huggingface_hub` (uses `hf-xet` automatically when available) |
|
||||
| `--python-preference managed` (default) without mirror config in restricted-network installers | Hits GitHub CDN, times out, user sees raw `uv` error | Configure `UV_PYTHON_INSTALL_MIRROR` + retry chain + `only-system` final fallback |
|
||||
| Material for MkDocs as a *new* docs choice | Entered maintenance mode November 2025 | If docs site is eventually needed, use Astro Starlight (Tauri precedent) |
|
||||
## Stack Patterns by Variant
|
||||
- Set `UV_PYTHON_INSTALL_MIRROR` to one of the gh-proxy URLs at install time
|
||||
- Set `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (China) or document VPN requirement (Russia)
|
||||
- Fall back to `UV_PYTHON_PREFERENCE=only-system` if all mirrors fail
|
||||
- Increase `UV_HTTP_TIMEOUT=120`, `UV_HTTP_RETRIES=5`
|
||||
- Default path: in-app Settings field → `login()` → file at `$HF_HOME/token`
|
||||
- Power-user path: `export HF_TOKEN=...` in shell rc (documented but not promoted)
|
||||
- Both paths are read at HF library import time; env var wins on conflict
|
||||
- Default path: in-app "Report a bug" → prefilled GitHub Issues URL → user reviews + submits in browser
|
||||
- All optional capture toggles default ON except "include reproduction file" (privacy)
|
||||
- No path posts to any URL except `github.com/{owner}/{repo}/issues/new` (rendered locally as a URL, opened via `shell.open`)
|
||||
- `uv add supertonic` → new TTSBackend subclass in `backend/services/tts_backend.py`
|
||||
- Auto-detected and added to the engine picker in Settings
|
||||
- ~400 MB model download on first synthesize call, cached in `$HF_HUB_CACHE`
|
||||
- Existing IndexTTS/CosyVoice/etc. installs are untouched (no shared model weights)
|
||||
## Version Compatibility
|
||||
| Package A | Compatible With | Notes |
|
||||
|-----------|-----------------|-------|
|
||||
| `supertonic@1.3.1` | `onnxruntime>=1.17`, `numpy>=1.24`, `huggingface_hub>=0.20` | All deps already satisfied transitively by current `pyproject.toml`. |
|
||||
| `huggingface_hub>=1.12` | `transformers>=5.3.0` (current pin) | `HfFolder` retained as deprecated alias; `login()`/`get_token()` are the canonical APIs. |
|
||||
| `uv>=0.5` | `UV_PYTHON_INSTALL_MIRROR`, `UV_PYTHON_PREFERENCE` | Both env vars stable since uv 0.4.x. |
|
||||
| Tauri v2 + `@tauri-apps/api/shell` | `shell.open()` for the prefilled-URL pattern | Already in the desktop app; no new permission needed beyond what the existing "open external link" plugin grants. |
|
||||
## Sources
|
||||
- [Hugging Face Hub environment variables](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH (verified against v1.12.1 docs, current 2026)
|
||||
- [Hugging Face Hub authentication API](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH (verified `login()` is the canonical 1.x API)
|
||||
- [Microsoft `setx` reference](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH (confirms "current shell" gotcha)
|
||||
- [PowerShell `about_Environment_Variables`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables) — HIGH
|
||||
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (verified all mirror + retry env vars)
|
||||
- [uv issue #5224 — python-build-standalone mirror](https://github.com/astral-sh/uv/issues/5224) — HIGH (feature shipped)
|
||||
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms user pain, justifies fallback chain)
|
||||
- [uv `python-preference` semantics](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH
|
||||
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official, 99M params, 31 languages, OpenRAIL-M)
|
||||
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official inference API)
|
||||
- [supertonic 1.3.1 on PyPI](https://pypi.org/project/supertonic/) — HIGH (released 2026-05-18, MIT code license; bumped from 1.2.3 after Phase 3 research)
|
||||
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure)
|
||||
- [GitHub Docs: Authenticating to the REST API](https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api) — HIGH
|
||||
- [GitHub Docs: Generating a user access token for a GitHub App](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app) — HIGH (device flow reference)
|
||||
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (canonical prefilled-URL reference impl)
|
||||
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — MEDIUM (reviewed, rejected on PROJECT.md constraint, not on quality)
|
||||
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained, verify URLs are still live before pinning in production)
|
||||
- [Tauri 2 docs (Astro Starlight reference)](https://v2.tauri.app/) — HIGH (precedent for docs framework if we ever migrate)
|
||||
- [Docsio: Material for MkDocs entered maintenance mode Nov 2025](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with the project's own GitHub commit activity)
|
||||
The May-2026 stack research that used to live here served five capabilities that have all since shipped (HF-token Settings panel, prefilled-URL bug reporting, uv mirror fallback for restricted networks, the Supertonic-3 engine, in-repo Markdown docs). Follow the patterns in the code itself; the durable *don'ts* that research established:
|
||||
|
||||
- **No third-party telemetry endpoints, ever** (`sentry-tauri` was evaluated and rejected) — bug reporting stays opt-in via prefilled GitHub-issue URLs.
|
||||
- **No PAT/token-based GitHub posting from the app** — the user submits from their own browser.
|
||||
- **Don't recommend `setx` for env vars on Windows** (silent truncation, no current-shell propagation) — use the in-app Settings panel or PowerShell `[Environment]::SetEnvironmentVariable`.
|
||||
- **Don't adopt Material for MkDocs** for any future docs site (maintenance mode since Nov 2025) — Astro Starlight is the precedent if docs ever outgrow the repo.
|
||||
- **`hf_transfer` is deprecated** — default `huggingface_hub` (hf-xet) handles downloads.
|
||||
|
||||
For anything new: prefer what's already pinned in `pyproject.toml` / `frontend/package.json`, and check `uv tree` for conflicts before adding a dependency.
|
||||
<!-- GSD:stack-end -->
|
||||
|
||||
<!-- GSD:conventions-start source:CONVENTIONS.md -->
|
||||
## Conventions
|
||||
|
||||
**Versioning (hard rule, owner-set 2026-06-11):** main is always **latest release + 1 patch**. The moment `vX.Y.Z` is released, main's version files (`frontend/src-tauri/tauri.conf.json`, `frontend/src-tauri/Cargo.toml`, `pyproject.toml`, **and `frontend/package.json`** — keep all **four** in lockstep; `package.json` drives the runtime `__APP_VERSION__` via vite, shown in the first-run footer + every auto bug report, so a drift ships a build that misreports its own version — guarded by `tests/test_app_version.py::test_all_version_files_in_lockstep`) bump to `X.Y.(Z+1)`. Consequences:
|
||||
**Versioning (hard rule, owner-set 2026-06-11; single-source 2026-06-16):** main is always **latest release + 1 patch**. **`frontend/package.json` is the SINGLE SOURCE OF TRUTH for the app version** — vite injects `__APP_VERSION__` from it (first-run footer + every auto bug report), and `frontend/src-tauri/tauri.conf.json` reads its bundle version from it (`"version": "../package.json"`, so the MSI/dmg/updater version can't drift from the UI). Three toolchain-required **mirrors** are kept equal to it and bumped in lockstep — `frontend/src-tauri/Cargo.toml` + `pyproject.toml` (cargo/uv need a literal) and `backend/core/version.py`'s `_FALLBACK_VERSION` (the frozen-backend last resort; at runtime the backend reads its version from package metadata via `importlib.metadata`, which `backend.spec`'s `copy_metadata('omnivoice')` makes work in the frozen build too). Never hand-edit any mirror or re-hardcode a literal in `tauri.conf.json`. Guarded by `tests/test_app_version.py` (`test_all_version_files_in_lockstep` + `test_tauri_version_derives_from_package_json`). The moment `vX.Y.Z` is released, bump `package.json` (+ the mirrors) to `X.Y.(Z+1)`. Consequences:
|
||||
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
|
||||
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
|
||||
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. **Owner override (2026-07-01): the post-release bump is now MANUAL — the `version-bump` job in release.yml is opt-in behind the `AUTO_VERSION_BUMP` repo variable (default off), so `main` stays at the released version until the owner explicitly asks to bump.** (Historically the bump auto-ran; re-enable that by setting `AUTO_VERSION_BUMP=true`.) When pinned, `main` == the released tag; preview-build ordering and "release + 1" only resume once a bump is requested.
|
||||
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
|
||||
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
|
||||
|
||||
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
|
||||
|
||||
**Release notes / changelog (hard rule, owner-set 2026-06-16):** every tagged release gets a **high-quality, user-facing `## [X.Y.Z] — DATE` section in `CHANGELOG.md`** before (or in the same hour as) the tag — never the "Auto-generated release for vX.Y.Z…" fallback. `release.yml` extracts that section verbatim as the GitHub Release body (the `Extract CHANGELOG section for tag` step), so a missing/empty section ships a bare release. Quality bar = the existing house style: a one-paragraph headline, then `### Added` / `### Fixed` / `### Changed` / `### License` / `### CI` subsections; each entry is a **bold one-line lead** (what the user gets), 1–3 lines of plain-English why, and the `(#NNN)` issue/PR ref — grouped by theme, written for users, **not** raw commit dumps. This applies to **preview builds too**: preview release notes summarize what's new on `main` since the last stable, in the same style. Workflow: as features merge, keep `## [Unreleased]` current; at release time rename it to the version + date. If a release was already cut with the fallback body, the next action is to backfill `CHANGELOG.md` **and** `gh release edit <tag>` the live body — not backlog.
|
||||
|
||||
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
|
||||
|
||||
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
|
||||
@@ -220,16 +68,9 @@ No project skills found. Add skills to any of: `.claude/skills/`, `.agents/skill
|
||||
<!-- GSD:skills-end -->
|
||||
|
||||
<!-- GSD:workflow-start source:GSD defaults -->
|
||||
## GSD Workflow Enforcement
|
||||
## Workflow
|
||||
|
||||
Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.
|
||||
|
||||
Use these entry points:
|
||||
- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks
|
||||
- `/gsd-debug` for investigation and bug fixing
|
||||
- `/gsd-execute-phase` for planned phase work
|
||||
|
||||
Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.
|
||||
Direct repo edits are authorized (owner decision, 2026-07-08). The GSD command gate that used to live here referenced `/gsd-quick` / `/gsd-debug` / `/gsd-execute-phase` skills that are not installed in this environment; the owner chose to keep working directly rather than restore them. The working conventions that matter are in **Conventions** above — versioning, docs-sync, changelog, localization, fix quality, keep-main-green — plus: gate every merge on the "Tests (backend + frontend)" check passing and the PR being MERGEABLE, and check the open-PR queue before implementing any community-reported fix (contributors may have already submitted one).
|
||||
<!-- GSD:workflow-end -->
|
||||
|
||||
|
||||
|
||||
+26
-1
@@ -18,6 +18,7 @@ Thanks for your interest in improving OmniVoice Studio! This guide covers everyt
|
||||
### Prerequisites
|
||||
|
||||
- [Git](https://git-scm.com/)
|
||||
- `curl` (used by the Bun / uv / rustup install one-liners on macOS and Linux)
|
||||
- [Bun](https://bun.sh/) (frontend package manager)
|
||||
- [uv](https://docs.astral.sh/uv/) (Python environment manager)
|
||||
- [ffmpeg](https://ffmpeg.org/) (audio/video processing)
|
||||
@@ -159,7 +160,7 @@ class MyEngineBackend(TTSBackend):
|
||||
|
||||
- **Components**: Functional components with hooks
|
||||
- **State**: Zustand stores in `src/stores/`, organized by slice
|
||||
- **CSS**: Vanilla CSS in component-level files — no Tailwind
|
||||
- **CSS**: **Utilities-first + shadcn/ui, one stylesheet.** UI is built on the shadcn/ui primitives in `src/components/ui/` (wrapped by the `src/ui/` barrel, themed to the OmniVoice palette), composed with Tailwind v4 utility classes. **All styling now lives in a single file — `src/index.css`**: the `@theme` / `[data-theme]` token foundation plus the irreducible set utilities can't express (`@keyframes`, glassmorphism/`backdrop-filter`, pseudo-elements, `:has()`, unlayered cascade overrides, and styling hooks on library-generated DOM like virtualized rows / WaveSurfer). The per-component `.css` files were eliminated in the CSS→Tailwind/shadcn migration — **do not create new ones.** Reach for shadcn primitives + utilities; if a rule is genuinely irreducible, add it to `src/index.css` with a provenance comment. (The only other `.css` is the test-only visual harness. See `docs/shadcn-migration.md`.)
|
||||
- **Naming**: `PascalCase` for components, `camelCase` for hooks and utils
|
||||
|
||||
### Rust (Tauri)
|
||||
@@ -169,6 +170,30 @@ class MyEngineBackend(TTSBackend):
|
||||
|
||||
---
|
||||
|
||||
## Frontend file structure & size limits
|
||||
|
||||
Frontend code stays modular so an edit loads one small file, not a 1900-line
|
||||
one. The rules:
|
||||
|
||||
- **Size caps:** **soft 300 lines**, **hard 500 lines** per `.jsx` file.
|
||||
Anything over 500 lines must be split. (The cap does **not** apply to
|
||||
`src/index.css` — it is the single, intentional styling foundation and the
|
||||
only app stylesheet; see the CSS rule above.)
|
||||
- **Pages are thin orchestrators.** A file in `frontend/src/pages/` is just
|
||||
layout + routing + state wiring that composes feature components — no inline
|
||||
sub-component over ~50 lines.
|
||||
- **One component per file.** Co-locate `Foo.jsx` + `Foo.test.jsx` together in a
|
||||
per-page feature folder under `frontend/src/components/` (e.g.
|
||||
`components/settings/`, `components/dub/`). Styling is **not** co-located —
|
||||
it's utilities + shadcn, with any irreducible rules in `src/index.css`.
|
||||
- **Shared bits go in a `primitives/` folder** inside the feature folder
|
||||
(`components/settings/primitives/` is the existing example).
|
||||
- **Enforced by ESLint `max-lines`** (`max: 500`) — **warn-only for now** so it
|
||||
never breaks CI, with the goal of upgrading to `error` once the backlog of
|
||||
oversized files clears.
|
||||
|
||||
---
|
||||
|
||||
## Commit Messages
|
||||
|
||||
Write clear, concise messages. The PR title becomes the squash-merge commit.
|
||||
|
||||
@@ -4,45 +4,49 @@
|
||||
<h3>The open-source ElevenLabs alternative.</h3>
|
||||
<p>Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.<br/>Open-source, no API keys, fully local. <b>646 languages.</b></p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="#quickstart">Quickstart</a> ·
|
||||
<a href="#features">Features</a> ·
|
||||
<a href="#why-omnivoice-studio">Why OmniVoice Studio?</a> ·
|
||||
<a href="#why-ovs">Why OVS</a> ·
|
||||
<a href="#tts-engines">TTS Engines</a> ·
|
||||
<a href="#asr-engines">ASR Engines</a> ·
|
||||
<a href="#openai-api">API</a> ·
|
||||
<a href="#sponsors">Sponsors</a> ·
|
||||
<a href="#sponsor--donate">Donate</a> ·
|
||||
<a href="#contributing">Contributing</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README_CN.md"><strong>简体中文</strong></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<!-- Pre-built macOS bundle is Apple Silicon. Intel Macs: build from source (docs/install/macos.md); a pre-built Intel target is tracked in #279. -->
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
</p>
|
||||
<p>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
<div align="center">
|
||||
<img src=".github/assets/social-preview.png" alt="OmniVoice Studio — The open-source ElevenLabs alternative" width="100%"/>
|
||||
<img src="docs/screenshot-launchpad.png" alt="OmniVoice Studio — Launchpad" width="100%"/>
|
||||
</div>
|
||||
|
||||
> **Your voice is the most personal data you have. So why rent it back from a cloud?** Every mainstream voice tool ships your audio to someone else's server and bills you monthly for the privilege. OmniVoice Studio flips that: clone, design, dub, and dictate on your own hardware — 646 languages, no meter running, nothing leaving your machine.
|
||||
|
||||
<div align="center">
|
||||
|
||||
| 🔑 No API keys | 🙅 No accounts | ☁️ No cloud | 💳 No subscription |
|
||||
|:---:|:---:|:---:|:---:|
|
||||
| nothing to paste in | nothing to sign up for | your audio stays home | it's your computer |
|
||||
|
||||
</div>
|
||||
|
||||
> [!WARNING]
|
||||
> **OmniVoice Studio is in active beta.** Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome — [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
|
||||
> **OmniVoice Studio is in active beta.** Things may break between releases — for the latest features and fixes, clone the repo and run from source rather than the pre-built installers. Bug reports and PRs are very welcome: [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
@@ -54,79 +58,157 @@
|
||||
|
||||
<br/>
|
||||
|
||||
## Features
|
||||
<a id="screenshots"></a>
|
||||
|
||||
## 📸 See it in action
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-studio.png" alt="Studio" width="100%"/>
|
||||
<br/><b>Studio</b><br/>
|
||||
<sub>Generate & clone in one workspace — a 3-second clip mirrors any voice, 646 languages, zero-shot.</sub>
|
||||
</td>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
|
||||
<br/><b>Voice Design</b><br/>
|
||||
<sub>Build new voices from scratch — gender, age, accent, pitch, emotion, dialect.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
|
||||
<br/><b>Voice Gallery</b><br/>
|
||||
<sub>Browse ready-made archetype voices with language filters — or build your own library.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
|
||||
<br/><b>Video Dubbing</b><br/>
|
||||
<sub>A real dub, end to end: 37 segments transcribed, translated to Bengali, re-voiced, and timed — ready to export as MP4.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-engines.png" alt="Settings — Engines" width="100%"/>
|
||||
<br/><b>Settings → Engines</b><br/>
|
||||
<sub>The engine compatibility matrix — 14 TTS engines with per-engine GPU preflight, no silent CPU fallback.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
|
||||
<br/><b>Settings → Models</b><br/>
|
||||
<sub>One-click model store — auto-detects your platform (CUDA / MPS / CPU) and recommends the right models.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-openapi.png" alt="Settings — API Reference" width="100%"/>
|
||||
<br/><b>API Reference</b><br/>
|
||||
<sub>The full local REST API, embedded — every endpoint documented with copy-paste client snippets.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-updates.png" alt="Settings — What's New" width="100%"/>
|
||||
<br/><b>What's New</b><br/>
|
||||
<sub>In-app changelog reader — see exactly what shipped in each release without leaving the app.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## ✨ Features
|
||||
|
||||
The eight headliners — and twelve more waiting under the fold.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="33%">
|
||||
<td align="center" width="25%">
|
||||
<h3>🎙️ Voice Cloning</h3>
|
||||
<p>3-second clip → mirror any voice.<br/><b>646 languages</b>, zero-shot.</p>
|
||||
</td>
|
||||
<td align="center" width="33%">
|
||||
<td align="center" width="25%">
|
||||
<h3>🎨 Voice Design</h3>
|
||||
<p>Gender, age, accent, pitch, speed,<br/>emotion, dialect — <b>dial it in</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="33%">
|
||||
<td align="center" width="25%">
|
||||
<h3>🎬 Video Dubbing</h3>
|
||||
<p>YouTube URL or file → transcribe →<br/>translate → re-voice → <b>MP4</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>📖 Audiobook Editor</h3>
|
||||
<p>Import text, EPUB, or PDF. Auto-chapter,<br/>loudnorm, metadata. Export <b>.m4b</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎭 Stories</h3>
|
||||
<p>Multi-voice editor. Assign voices<br/>per-line, preview, <b>export full cast</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ Dictation Widget</h3>
|
||||
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
<p><kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔊 Vocal Isolation</h3>
|
||||
<p>Demucs-powered. Splits speech<br/>from music, <b>keeps the background</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>👥 Speaker Diarization</h3>
|
||||
<p>Pyannote + WhisperX.<br/><b>Auto-identifies</b> who said what.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>📦 Batch Queue</h3>
|
||||
<p>Drop <b>50 videos</b>, walk away.<br/>Progress bars per job.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🤖 MCP Server</h3>
|
||||
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🛡️ AI Watermark</h3>
|
||||
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔐 100% Local</h3>
|
||||
<p>No keys, no cloud, no accounts.<br/><b>Your machine only</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⚡ GPU Auto-Detect</h3>
|
||||
<p>CUDA · MPS · ROCm · CPU.<br/>≤8 GB? <b>Auto-offloads</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧩 Extensible</h3>
|
||||
<p>Subclass <code>TTSBackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
<h3>🤖 MCP Server</h3>
|
||||
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<details>
|
||||
<summary><b>…and 12 more</b> — isolation, diarization, batch, watermarking, diagnostics, and friends</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
- 🔊 **Vocal Isolation** — Demucs-powered: splits speech from music and keeps the background bed.
|
||||
- 👥 **Speaker Diarization** — Pyannote + WhisperX auto-identify who said what.
|
||||
- 📦 **Batch Queue** — drop 50 videos, walk away; per-job progress bars.
|
||||
- 🛡️ **AI Watermark** — AudioSeal (Meta): invisible, survives compression.
|
||||
- 🔬 **Diagnostics** — self-check suite, error journal, scrubbed diagnostic bundles.
|
||||
- ⚡ **GPU Auto-Detect** — CUDA · MPS · ROCm (Linux, opt-in) · CPU; ≤8 GB VRAM auto-offloads.
|
||||
- 🧭 **Engine routing** — preflight GPU check per engine; no silent CPU fallback.
|
||||
- 🧩 **Extensible** — subclass `TTSBackend`, add any engine in ~50 lines.
|
||||
- 🎒 **Portable personas** — export voices as `.ovsvoice` bundles: identity + watermark.
|
||||
- ♾️ **Unlimited TTS** — sentence-chunked generation, no length cap, streaming via WebSocket.
|
||||
- 🌐 **Remote backend** — point the UI at a remote server; Tailscale-friendly, bearer auth.
|
||||
- 🧠 **Dictation + LLM** — local-LLM cleanup of transcripts, optional echo cancellation.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
<a id="quickstart"></a>
|
||||
|
||||
Per-OS install guides — pick yours and follow it end-to-end:
|
||||
## ⚡ Quickstart
|
||||
|
||||
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<br/>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
<br/>
|
||||
<sub><b>Intel Macs are not supported for the local backend:</b> the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac (x86_64) wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — see <a href="docs/install/macos.md">docs/install/macos.md</a>.</sub>
|
||||
</div>
|
||||
|
||||
Stuck? Run the built-in self-check first — **Settings → About → "Run
|
||||
Pick your OS and follow the guide end-to-end:
|
||||
|
||||
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
|
||||
<details>
|
||||
<summary><b>🧰 Stuck? Self-checks, tokens & restricted networks</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
Run the built-in self-check first — **Settings → About → "Run
|
||||
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
|
||||
a checkout (`--deep` also test-loads the active engine). Then see
|
||||
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) for the
|
||||
@@ -141,57 +223,13 @@ diarization-specific gating, see
|
||||
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
|
||||
options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-clone.png" alt="Voice Clone" width="100%"/>
|
||||
<br/><b>Voice Clone</b><br/>
|
||||
<sub>Drop a 3-second clip → mirror any voice. 646 languages, zero-shot.</sub>
|
||||
</td>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
|
||||
<br/><b>Voice Design</b><br/>
|
||||
<sub>Build new voices from scratch — gender, age, accent, pitch, style.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
|
||||
<br/><b>Video Dubbing</b><br/>
|
||||
<sub>Upload or paste a YouTube URL. Transcribe, translate, re-voice, export.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
|
||||
<br/><b>Voice Gallery</b><br/>
|
||||
<sub>Search YouTube, browse categories, download clips, build your library.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
|
||||
<br/><b>Settings → Models</b><br/>
|
||||
<sub>15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU).</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-libraryprojects.png" alt="Projects" width="100%"/>
|
||||
<br/><b>Projects</b><br/>
|
||||
<sub>Dub projects, voice profiles, generation history, exports — all searchable.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" colspan="2">
|
||||
<img src="docs/screenshot-logs.png" alt="Settings — Logs" width="100%"/>
|
||||
<br/><b>Settings → Logs</b><br/>
|
||||
<sub>Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Why OmniVoice Studio?
|
||||
<a id="why-ovs"></a>
|
||||
|
||||
## 💡 Why OmniVoice?
|
||||
|
||||
ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
|
||||
|
||||
@@ -200,15 +238,20 @@ ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. O
|
||||
| **Pricing** | $5–$330/mo, per-character billing | Free & open-source (AGPL-3.0) · [Commercial license](#license) for proprietary use |
|
||||
| **Voice Cloning** | ✅ 3s clip | ✅ 3s clip, zero-shot |
|
||||
| **Voice Design** | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
|
||||
| **Audiobook / Stories** | ❌ | ✅ Full audiobook editor + multi-voice stories (EPUB/PDF import, .m4b export) |
|
||||
| **Languages** | 32 | **646** |
|
||||
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
|
||||
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
|
||||
| **API Keys** | Required | Not needed |
|
||||
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
|
||||
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm (Linux) · CPU |
|
||||
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS Engines** | 1 | **14** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS) |
|
||||
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
|
||||
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
|
||||
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
|
||||
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
|
||||
|
||||
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
|
||||
Professional-grade voice AI, minus the subscription and the cloud.
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
@@ -219,125 +262,271 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
|
||||
|
||||
---
|
||||
|
||||
## System Requirements
|
||||
## 🖥️ System Requirements
|
||||
|
||||
| | **Minimum** | **Recommended** |
|
||||
|---|---|---|
|
||||
| **OS** | Windows 10, macOS 12+, Ubuntu 20.04+ | Any modern 64-bit OS |
|
||||
| **OS** | Windows 10, macOS 12+ (Apple Silicon), Ubuntu 24.04+ (glibc 2.39+) | Any modern 64-bit OS |
|
||||
| **RAM** | 8 GB | 16 GB+ |
|
||||
| **VRAM (GPU)** | 4 GB (auto-offloads TTS to CPU) | 8 GB+ (NVIDIA RTX 3060+) |
|
||||
| **Disk** | 10 GB free (models + cache) | 20 GB+ SSD |
|
||||
| **Python** | 3.10+ (managed by `uv`) | 3.11–3.12 |
|
||||
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm |
|
||||
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm (Linux only) |
|
||||
|
||||
> [!TIP]
|
||||
> On GPUs with **≤8 GB VRAM**, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).
|
||||
|
||||
### TTS Engines
|
||||
> [!NOTE]
|
||||
> **AMD GPUs:** ROCm acceleration is **Linux-only and opt-in** — pick **"AMD GPU (ROCm)"** on the first-run setup screen or set `OMNIVOICE_TORCH_VARIANT=rocm` ([docs/install/linux.md](docs/install/linux.md#amd-gpu-rocm)). **On Windows, AMD GPUs (incl. Ryzen AI iGPUs) run CPU-only**: PyTorch has no Windows ROCm wheels, so Windows GPU acceleration is NVIDIA/CUDA-only ([docs/install/windows.md](docs/install/windows.md#gpu-support)).
|
||||
|
||||
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
|
||||
> [!IMPORTANT]
|
||||
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
|
||||
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
### 🗣️ TTS Engines
|
||||
|
||||
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var — the selection applies everywhere synthesis happens: single-clip generation, Voice Cloning, Video Dubbing, and Batch TTS.
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full matrix</b> — 14 engines × platform × clone/instruct × license</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **OmniVoice** (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
|
||||
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2** ⚡ | Multi | ✅ | — | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
|
||||
>
|
||||
> **Clone** matters beyond single-clip generation: Video Dubbing (and any Batch job with a pinned voice) needs reference-audio cloning to preserve speaker identity, so picking a Clone-less engine (KittenTTS, Sherpa-ONNX, Supertonic 3) as the active engine fails those jobs up front with an actionable message instead of silently falling back to OmniVoice.
|
||||
>
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
|
||||
|
||||
### ASR Engines
|
||||
</details>
|
||||
|
||||
OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictation, video dubbing, and subtitle generation — all fully local. **WhisperX** is the cross-platform default; the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### 🎧 ASR Engines
|
||||
|
||||
**10 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → Engines** (the ASR Engines table — same picker TTS has), or pin one with the `OMNIVOICE_ASR_BACKEND` env var (the env var wins over the Settings pick). Nine run fully on-device; one (OpenAI-compatible) is an optional remote client for pointing at Qwen3-ASR or another compatible server — see below.
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full lineup</b> — 10 engines, what each is best at, and compute-type notes</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Engine | `OMNIVOICE_ASR_BACKEND` | Languages | Best for |
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
| **WhisperX** (default) | `whisperx` | ~100 | Dubbing & subtitles — word-level timing via wav2vec2 forced alignment |
|
||||
| **Faster-Whisper** | `faster-whisper` | ~100 | Fast transcription on Linux / macOS / Windows (CTranslate2) |
|
||||
| **Faster-Whisper (isolated)** | `faster-whisper-isolated` | ~100 | Same as Faster-Whisper but crash-isolated in a subprocess — an ASR crash won't take down the app |
|
||||
| **MLX Whisper** | `mlx-whisper` | ~100 | Native Apple Silicon speed (Apple MLX / Metal) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers (no cuDNN 8 needed) |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA accuracy at ~10× realtime even on CPU, auto language detection (NVIDIA NeMo, CUDA/CPU) |
|
||||
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
|
||||
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
|
||||
| **sherpa-onnx** (live dictation) | `sherpa-onnx-asr` | 25 EU + 90+ | Live, faster-than-real-time dictation — small streaming/offline ONNX models (Parakeet TDT v3/v2, streaming Zipformer & Paraformer, Whisper Tiny), CPU, identical on macOS / Windows / Linux. Picked per-model in **Settings → Voice**. |
|
||||
| **OpenAI-compatible** ⚠️ remote | `openai-compat-asr` | Server-dependent | A path to **Qwen3-ASR** today (self-hosted server, no transformers wait), any OpenAI-compatible transcription endpoint, or OpenAI's own API — no install, configure in **Settings → Models**. Audio leaves your machine to whatever server you point it at; see [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md). |
|
||||
|
||||
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.
|
||||
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. **sherpa-onnx** powers the live dictation model picker — you talk and text appears as you speak. Every engine runs on-device — no API keys, no cloud.
|
||||
|
||||
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoicePreview · BatchQueue · Gallery │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 97 API endpoints · SSE streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
|
||||
│ ASR │ Source │ TTS │ Diarization │
|
||||
│ │ Sep. │ │ │
|
||||
└──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected)
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery │
|
||||
│ Dictation · BatchQueue · Diagnostics · MCP Client │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │ Engine Routing │
|
||||
│ (+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight │
|
||||
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
```
|
||||
|
||||
<a id="openai-api"></a>
|
||||
|
||||
## 🔌 OpenAI-compatible API
|
||||
|
||||
Already have a script, agent, or tool that speaks OpenAI's audio API? Point it at `http://localhost:3900/v1` — no key needed, no code changes. The backend ships a drop-in surface for the audio endpoints, wired to whichever TTS/ASR engine you have active (and yes, `voice` accepts your cloned voice-profile IDs).
|
||||
|
||||
| Endpoint | What it does |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS — text in; `mp3` / `wav` / `flac` / `opus` / `pcm` out. `tts-1` / `tts-1-hd` map to your active engine; OpenAI voice names (`alloy`, …) are accepted. |
|
||||
| `POST /v1/audio/transcriptions` | STT — audio file in; `json`, `text`, `verbose_json`, `srt`, or `vtt` out. `whisper-1` maps to your active ASR engine. |
|
||||
| `GET /v1/audio/voices` | OmniVoice extension — lists every voice profile and engine, so clients can discover your clones. |
|
||||
|
||||
```sh
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
|
||||
|
||||
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
|
||||
print(result.text)
|
||||
```
|
||||
|
||||
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
|
||||
|
||||
### 📓 Run on Google Colab (community)
|
||||
|
||||
No local GPU? A community member ([@shakib30](https://github.com/shakib30)) maintains a working Colab notebook: [shakib30/OmniVoice-Studio-google-colab](https://github.com/shakib30/OmniVoice-Studio-google-colab). Community-maintained — issues with the notebook go there; issues with OmniVoice itself come here.
|
||||
|
||||
### 🤝 Agent Skills
|
||||
|
||||
Teach your AI agent (Claude Code, Cursor, Codex, …) to use OmniVoice with one command:
|
||||
|
||||
```sh
|
||||
npx skills add debpalash/omnivoice-studio
|
||||
```
|
||||
|
||||
Ships two [skills](https://skills.sh): **`omnivoice`** — speak and transcribe through your local install (including your cloned voices) from any agent, free and offline; and **`oss-maintainer`** — the maintainer methodology this project is run with, for anyone running their own OSS project with an agent.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
### ✅ Shipped
|
||||
|
||||
| Category | Features |
|
||||
|----------|----------|
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
|
||||
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
|
||||
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
|
||||
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
|
||||
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
|
||||
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
|
||||
| **Windows Hardening** | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste |
|
||||
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
|
||||
## 🗺️ Roadmap
|
||||
|
||||
### 🔜 Up Next
|
||||
|
||||
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
|
||||
- 📖 **Audiobook Editor** — chapter-aware long-form narration
|
||||
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
|
||||
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
|
||||
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
|
||||
|
||||
<details>
|
||||
<summary><b>✅ Everything shipped so far</b> — the receipts, by category</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Category | Features |
|
||||
|----------|----------|
|
||||
| **Longform** | Audiobook editor (text/EPUB/PDF → chaptered .m4b), Stories multi-voice editor, two-pass loudnorm mastering, crash-resume for interrupted renders, pronunciation control + SSML-lite prosody |
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS, per-speaker voice assignment, Smart Fit timing + second-pass QC, dedicated Dub home |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags, portable persona bundles (`.ovsvoice`), voice console workspace |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export, unlimited-length TTS via sentence-chunked generation |
|
||||
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
|
||||
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
|
||||
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
|
||||
| **TTS** | 14 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS), engine routing with GPU preflight |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
|
||||
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
|
||||
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
|
||||
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG — Apple Silicon; Intel unsupported for the local backend, #889 — Windows MSI, Linux deb/AppImage), auto-update infrastructure, single-instance enforcement, close-to-tray, macOS Gatekeeper fix |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste, customizable hotkey, local-LLM transcript refinement |
|
||||
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
|
||||
| **MCP Server** | OmniVoice as a local TTS/STT provider for Claude, Cursor, and any MCP client |
|
||||
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
|
||||
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Community
|
||||
<a id="sponsor--donate"></a>
|
||||
|
||||
## 💜 Sponsor / Donate
|
||||
|
||||
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
|
||||
|
||||
<div align="center">
|
||||
|
||||
**This month's agent bill fund**
|
||||
|
||||
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="$10 / $200 raised" />
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
|
||||
<br/>
|
||||
<sub>Every dollar goes directly to agent bills — keeping OmniVoice development continuous.</sub>
|
||||
|
||||
</div>
|
||||
|
||||
<a id="sponsors"></a>
|
||||
|
||||
### 🌟 Sponsors
|
||||
|
||||
OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponsors keep development going, and in return get a logo slot here, in the app, and (for top tiers) on the project website. It's a thank-you, never a paywall. **[See tiers & become a sponsor →](SPONSORS.md)**
|
||||
|
||||
<div align="center">
|
||||
|
||||
<!-- SPONSORS:START — logo slots are filled here as sponsors come aboard; see SPONSORS.md -->
|
||||
|
||||
**Your logo here** — [become a sponsor](SPONSORS.md)
|
||||
|
||||
<!-- SPONSORS:END -->
|
||||
|
||||
</div>
|
||||
|
||||
<sub>💡 GitHub also shows a **Sponsor** button at the top of this repo, wired to the same links via <a href=".github/FUNDING.yml"><code>.github/FUNDING.yml</code></a>.</sub>
|
||||
|
||||
---
|
||||
|
||||
## 💬 Community
|
||||
|
||||
<div align="center">
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join Discord" /></a>
|
||||
<br/>
|
||||
<sub>We respond to setup questions within hours, not days.</sub>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary><b>What happens in there</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Channel | What happens there |
|
||||
|---------|--------------------|
|
||||
| `#showcase` | Members share their dubs, clones, and voice designs |
|
||||
| `#help` | Setup issues, GPU troubleshooting, model questions |
|
||||
| `#feature-requests` | Vote on what gets built next |
|
||||
| `#dev` | Architecture discussions, PR reviews, engine integrations |
|
||||
| `#announcements` | Release notes, breaking changes, early access |
|
||||
| `#announcements` | Release news and the big moments — new versions land here first |
|
||||
| `#releases` + `#changelog` | Every build and exactly what's inside it |
|
||||
| `#issues` | Bug reports as forum posts — triaged straight into GitHub issues |
|
||||
| `#ideas` | Feature requests, discussed and voted on |
|
||||
| `#discuss-ideas` | Design talk before things get built |
|
||||
| `#general` | Setup help, GPU troubleshooting, and showing off your dubs |
|
||||
|
||||
**[→ Join the Discord](https://discord.gg/bzQavDfVV9)** — we respond to setup questions within hours, not days.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
<a id="contributing"></a>
|
||||
|
||||
We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.
|
||||
## 🤝 Contributing
|
||||
|
||||
Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, translations. All of it.
|
||||
|
||||
- 📖 Read the **[Contributing Guide](CONTRIBUTING.md)** for setup, code style, and PR workflow
|
||||
- 🐛 Browse [good first issues](https://github.com/debpalash/OmniVoice-Studio/labels/good%20first%20issue)
|
||||
@@ -345,18 +534,26 @@ We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
## ❓ FAQ
|
||||
|
||||
<details>
|
||||
<summary><b>Is this really as good as ElevenLabs?</b></summary>
|
||||
<br/>
|
||||
For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.
|
||||
Honest answer: <b>it depends on what you're doing.</b>
|
||||
|
||||
<b>Where OmniVoice is genuinely competitive:</b> voice cloning from a clean reference clip (state-of-the-art open diffusion TTS), language coverage (646 languages vs. their 32), and everything structural — no per-character billing, no usage caps, no audio leaving your machine, full pipeline customizability (10 TTS engines, 10 ASR engines, your choice of translation).
|
||||
|
||||
<b>Where ElevenLabs still wins:</b> out-of-the-box consistency and polish, especially for English TTS. Their one model is heavily tuned; our quality depends on which engine you pick, your hardware, and — for cloning — the reference audio (a dry, close-mic clip clones dramatically better than a noisy or echoey one).
|
||||
|
||||
<b>For dubbing specifically:</b> a dub is a chain — transcription → translation → cloning → synthesis — and the output is only as good as its weakest link on <i>your</i> source material. Noisy or accented source audio degrades transcription, which degrades everything downstream; some language pairs translate better than others. If parts of a dub come out incoherent, check the segment table's <i>original</i> text first: if the transcription is already wrong there, switch the ASR engine (Settings → Engines) or use cleaner source audio — that's usually the fix, not the voice.
|
||||
|
||||
Try it on your real material — it's free and takes one download. Many users find it replaces ElevenLabs outright; some keep both for different jobs. Both outcomes are fine with us.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Does it work on Apple Silicon (M1/M2/M3/M4)?</b></summary>
|
||||
<br/>
|
||||
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware.
|
||||
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware. <b>Intel Macs are not supported</b>: the app UI installs, but the local Python backend cannot run because PyTorch no longer ships Intel-Mac wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — an Intel Mac can only be used with a remote backend.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
@@ -380,12 +577,14 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
|
||||
<details>
|
||||
<summary><b>Can I add my own TTS engine?</b></summary>
|
||||
<br/>
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Fourteen engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, and Confucius4-TTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
<a id="license"></a>
|
||||
|
||||
## 📜 License
|
||||
|
||||
OmniVoice Studio is free and open-source software under the [**GNU Affero General Public License v3.0 (AGPL-3.0)**](https://www.gnu.org/licenses/agpl-3.0.html).
|
||||
|
||||
@@ -397,7 +596,7 @@ The bundled `omnivoice/` TTS model by Han Zhu remains Apache-2.0 upstream. See [
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
@@ -410,6 +609,20 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | Optimized Transformer inference on CPU and GPU |
|
||||
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | Invisible neural audio watermarking for AI provenance |
|
||||
| [**Tauri**](https://tauri.app) | Native desktop app framework |
|
||||
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS engine — 31 languages, CPU-efficient |
|
||||
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | WASM-ready universal TTS/ASR runtime |
|
||||
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | Zero-shot TTS engine — 5 languages, RTF 0.014 |
|
||||
|
||||
---
|
||||
|
||||
## 🧰 More local open-source from the maker
|
||||
|
||||
Like the local-first philosophy? It runs in the family:
|
||||
|
||||
| Project | What it is |
|
||||
|---------|------------|
|
||||
| [**Opal**](https://github.com/debpalash/Opal) 💠 | **Play everything.** The evolved media player for the next decades of entertainment — video, anime, comics, torrents, Jellyfin/Plex, with local AI built in. |
|
||||
| [**memxt**](https://github.com/debpalash/memxt) 🧠 | **The fastest benchmarked open-source AI memory system.** 100% local memory for AI agents, with MCP support. |
|
||||
|
||||
---
|
||||
|
||||
@@ -419,7 +632,8 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
If you read this far, you're our kind of person.<br/>
|
||||
**[⭐ Star this repo](https://github.com/debpalash/OmniVoice-Studio)** so others can find it too.<br/>
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.<br/>
|
||||
**[❤️ Support development](https://ko-fi.com/debpalash)** — fund the AI agent bills that keep OmniVoice shipping.
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
@@ -372,9 +372,13 @@ OmniVoice 配备多引擎 TTS 后端。默认引擎(OmniVoice)始终可用
|
||||
| **MLX-Audio**(Kokoro, Qwen3-TTS, CSM, Dia 等) | 多语言 | 因引擎而异 | 因引擎而异 | ❌ | ✅ 原生 | ❌ | 因引擎而异 |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-v1.5**(8B,可选装) | 31 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts**(2B,可选装) | 24 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **KittenTTS** | 英语 | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
|
||||
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行,大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon。
|
||||
>
|
||||
> **MOSS-TTS-v1.5**(8B,约 16 GB 权重)和 **dots.tts**(2B,约 9 GB 权重)是重量级可选引擎,从本地克隆在独立 venv 中运行——参见 [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) 和 [dots.tts](docs/engines/dots-tts.md)。两者均不支持 Apple Silicon **MPS**(上游仅支持 CUDA/CPU;在 Mac 上以 CPU 运行)。dots.tts 上游仅支持 Linux/macOS——无 Windows 路径。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
<div align="center">
|
||||
<img src="docs/logo.png" alt="OmniVoice Logo" width="96" />
|
||||
<h1>Sponsor OmniVoice Studio</h1>
|
||||
<p><b>Keep the open-source ElevenLabs alternative free, local, and shipping.</b></p>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Why sponsor?
|
||||
|
||||
OmniVoice Studio is built by one developer, in the open, using Claude Code and AI agents — and the agent bills are real. Over the last few months I've spent thousands of dollars on Claude subscriptions to keep features shipping, bugs fixed, and your issues answered.
|
||||
|
||||
OmniVoice is **free**, **fully local**, and **AGPL-3.0**. There's no paid tier, no accounts, no cloud, and no SaaS revenue — nothing runs on a server we bill you for, because nothing runs on a server at all. That's the whole point, and it's also why there's no recurring revenue to fund development. Sponsorship is what makes continued full-time work possible.
|
||||
|
||||
If OmniVoice has created value for you or your company, sponsoring means the next release keeps coming — and you get a thank-you (and, at most tiers, a logo slot) in return.
|
||||
|
||||
### Where your money goes
|
||||
|
||||
Every dollar goes to the cost of building OmniVoice — chiefly the **AI agent bills that keep it shipping** (Claude subscriptions and API usage), plus the occasional signing certificate, test hardware, and model-hosting costs. It is not a salary top-up; it's what keeps the lights on for continuous development.
|
||||
|
||||
---
|
||||
|
||||
## Sponsorship tiers
|
||||
|
||||
Tiers are about **visibility and gratitude** — what you get is placement, not gated features (see [Not a paywall](#not-a-paywall)). Higher tiers include everything in the tiers below them.
|
||||
|
||||
| Tier | Suggested monthly | What you get |
|
||||
|------|-------------------|--------------|
|
||||
| **🥉 Backer** | _set by owner_ <!-- OWNER: set amounts --> | Your name or handle listed in the **Backers** section of this file, with a link of your choice. |
|
||||
| **🟫 Bronze** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a small logo in `SPONSORS.md` **and** in the README [Sponsors section](README.md#sponsors). |
|
||||
| **🥈 Silver** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** your logo in the **README** and in the app's **in-app Sponsors page footer** (as that page ships). |
|
||||
| **🥇 Gold** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a **prominent logo slot** and link on the project **website / landing page**. |
|
||||
|
||||
> **Amounts are set by the maintainer** — look for the `<!-- OWNER: set amounts -->` markers in this file's source. If you don't see a price that fits, say so in your inquiry; custom and annual arrangements are welcome.
|
||||
|
||||
Placements marked "as that page ships" (the in-app Sponsors page and the project website) are on the near-term roadmap. Until they exist, Silver/Gold logos live in `SPONSORS.md` and the README, and are added to the app and site the moment those land — no re-application needed.
|
||||
|
||||
---
|
||||
|
||||
## How to become a sponsor
|
||||
|
||||
**1. Open a sponsorship inquiry (recommended).** This opens a short GitHub form (name/org, logo, tier, contact) so we can get you set up:
|
||||
|
||||
> **[→ Open a sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml)**
|
||||
|
||||
**2. Or start recurring support directly:**
|
||||
|
||||
- **Ko-fi (recurring or one-time):** [ko-fi.com/debpalash](https://ko-fi.com/debpalash)
|
||||
- **PayPal (one-time):** [paypal.me/palashCoder](https://paypal.me/palashCoder)
|
||||
|
||||
If you sponsor via Ko-fi/PayPal and want a logo slot, still open an inquiry (or drop a note there) so we know who to credit and where to link.
|
||||
|
||||
**3. Prefer to talk first?** Reach out directly:
|
||||
|
||||
- Email: <!-- OWNER: add your sponsor contact email here if you want one public -->
|
||||
- Or ask in the `#dev` / `#announcements` channels on [Discord](https://discord.gg/bzQavDfVV9).
|
||||
|
||||
---
|
||||
|
||||
## Logo & asset guidelines
|
||||
|
||||
To make your logo look sharp everywhere (README on GitHub, the in-app page, the website), please send:
|
||||
|
||||
- **Format:** **SVG preferred** (scales cleanly); otherwise **PNG at 2× resolution**.
|
||||
- **Background:** **transparent** — no baked-in white/black box.
|
||||
- **Contrast:** send a variant that stays legible on **both light and dark** backgrounds, or one light-mode and one dark-mode file (GitHub and the app both render in either theme).
|
||||
- **Dimensions:** legible at **~40px tall**; keep the wordmark within roughly **480px wide**. Landscape/wordmark shapes work best in the README row.
|
||||
- **File size:** keep SVGs under ~50 KB and PNGs under ~100 KB.
|
||||
- **Link target:** the destination URL you want the logo to point to (usually your homepage).
|
||||
|
||||
**How your logo gets added:**
|
||||
|
||||
- **Easiest:** attach the asset and link in your [sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml) — the maintainer places it.
|
||||
- **Or open a PR:** add your asset under `docs/sponsors/` and an entry to the tables in this file. Silver/Gold logos are also wired into the app's in-app Sponsors page (via the `sponsors.js` manifest) and the project website as those surfaces ship.
|
||||
|
||||
By sponsoring you confirm you have the right to use the submitted logo and grant OmniVoice permission to display it in the contexts above. We won't alter your logo beyond scaling, and we'll remove it promptly on request.
|
||||
|
||||
---
|
||||
|
||||
## Current sponsors
|
||||
|
||||
OmniVoice doesn't have any sponsors yet — **you could be the first.** These slots fill in as sponsors come aboard.
|
||||
|
||||
### 🥇 Gold
|
||||
|
||||
_Be the first Gold sponsor — [claim this slot](#how-to-become-a-sponsor)._
|
||||
|
||||
### 🥈 Silver
|
||||
|
||||
_Open — [become a Silver sponsor](#how-to-become-a-sponsor)._
|
||||
|
||||
### 🟫 Bronze
|
||||
|
||||
_Open — [become a Bronze sponsor](#how-to-become-a-sponsor)._
|
||||
|
||||
### 🥉 Backers
|
||||
|
||||
_Open — [become a Backer](#how-to-become-a-sponsor)._
|
||||
|
||||
<!-- When a sponsor joins, add them to the matching section above:
|
||||
- Logo tiers (Bronze+): <a href="https://sponsor.example"><img src="docs/sponsors/name.svg" alt="Name" height="48" /></a>
|
||||
- Backers: - [Name / handle](https://link) -->
|
||||
|
||||
---
|
||||
|
||||
## Not a paywall
|
||||
|
||||
Sponsorship is a **thank-you, never a paywall.**
|
||||
|
||||
Every feature of OmniVoice Studio is and will remain **free** and **open-source under [AGPL-3.0](LICENSE)**. Sponsors do **not** get private builds, gated features, license exceptions, or anything that degrades the experience for people who don't (or can't) pay. What sponsors get is **visibility and our gratitude** — and the knowledge that they're directly funding the next release.
|
||||
|
||||
OmniVoice stays local-first and fully functional with zero dollars spent. Sponsoring just helps it keep getting better, faster.
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
<sub>Thank you for keeping local-first voice AI alive and free. ❤️</sub><br/>
|
||||
<sub>Questions? <a href="https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml">Open an inquiry</a> · <a href="https://discord.gg/bzQavDfVV9">Discord</a></sub>
|
||||
</div>
|
||||
+15
-1
@@ -13,12 +13,18 @@
|
||||
# Run: uv run pyinstaller backend.spec --noconfirm --clean
|
||||
import platform
|
||||
import sys
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules, copy_metadata
|
||||
|
||||
IS_MAC_ARM = sys.platform == "darwin" and platform.machine() == "arm64"
|
||||
|
||||
datas = []
|
||||
binaries = []
|
||||
|
||||
# Bundle the omnivoice package's .dist-info so importlib.metadata.version()
|
||||
# resolves inside the frozen build. Without it the backend can't read its own
|
||||
# version and falls back to the literal in backend/core/version.py — which is
|
||||
# how a 0.3.6 desktop build shipped reporting "0.3.5" in About + bug reports.
|
||||
datas += copy_metadata('omnivoice')
|
||||
hiddenimports = [
|
||||
# Web stack
|
||||
'uvicorn', 'uvicorn.logging', 'uvicorn.loops', 'uvicorn.loops.auto',
|
||||
@@ -27,6 +33,14 @@ hiddenimports = [
|
||||
'uvicorn.lifespan', 'uvicorn.lifespan.on',
|
||||
'fastapi', 'fastapi.responses', 'starlette',
|
||||
'multipart',
|
||||
# SOCKS proxy support (#959). httpx imports socksio lazily inside a
|
||||
# try/except (only when a socks5:// proxy env var is set), so
|
||||
# PyInstaller's static tracer never sees it — without this entry the
|
||||
# frozen installers keep raising "Using SOCKS proxy, but the 'socksio'
|
||||
# package is not installed" on every model load under a SOCKS proxy,
|
||||
# even though pyproject.toml ships the package. Guarded by
|
||||
# tests/test_socks_proxy.py.
|
||||
'socksio',
|
||||
|
||||
# Core
|
||||
'uuid', 'asyncio',
|
||||
|
||||
@@ -18,7 +18,6 @@ Design notes
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
@@ -137,7 +136,7 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
from api.routers.generation import ( # noqa: WPS433 — intentional lazy import
|
||||
get_model,
|
||||
_run_inference,
|
||||
_gpu_pool,
|
||||
run_on_gpu_pool_guarded,
|
||||
_safe_torchaudio_save,
|
||||
)
|
||||
|
||||
@@ -147,8 +146,6 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
language = None
|
||||
text = (a.get("sample_script") or "").strip() or _FALLBACK_SCRIPT
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _infer(seed: int):
|
||||
return _run_inference(
|
||||
model, # _model
|
||||
@@ -171,14 +168,18 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
"broadcast", # effect_preset
|
||||
)
|
||||
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED)
|
||||
# Bounded + pool-reset on hang so a wedged preview render can't starve the
|
||||
# GPU pool and brick the backend (#730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _infer(_PREVIEW_SEED), what="Archetype preview generate")
|
||||
if _is_unusable_audio(audio_tensor):
|
||||
# Blank OR a degenerate tonal buzz — retry once on a different seed to
|
||||
# step off the bad diffusion trajectory. Static message only: the
|
||||
# archetype id is request-derived (CodeQL log-injection); the seed is a
|
||||
# module constant, safe to log.
|
||||
logger.warning("Archetype rendered unusable at seed %d — retrying once", _PREVIEW_SEED)
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED + 1)
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _infer(_PREVIEW_SEED + 1), what="Archetype preview generate")
|
||||
if _is_unusable_audio(audio_tensor):
|
||||
raise RuntimeError("the voice engine returned no audible audio for this archetype")
|
||||
|
||||
|
||||
@@ -164,6 +164,7 @@ async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
|
||||
class AudiobookRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None # voice profile id; None = engine default
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b" # "m4b" | "mp3"
|
||||
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
|
||||
@@ -215,7 +216,35 @@ def _resolve_voice(profile_id: str | None) -> dict:
|
||||
return out
|
||||
|
||||
|
||||
def _build_synth(default_voice: str | None) -> dict:
|
||||
def _resolve_default_language(language: str | None, default_voice: str | None) -> str | None:
|
||||
"""Pick the language to thread into the longform synth callable.
|
||||
|
||||
Priority (mirrors the single-shot /generate path, #533): an explicit
|
||||
non-Auto request ``language`` wins; otherwise the selected profile's stored
|
||||
language drives it; otherwise ``None`` (genuine Auto — the engine
|
||||
autodetects, exactly as before). Hardcoding ``None`` here (#505 B2) let the
|
||||
engine re-autodetect per chunk, so a non-English clone flipped to the wrong
|
||||
language on short/ambiguous chapters.
|
||||
"""
|
||||
if language and language != "Auto":
|
||||
return language
|
||||
if default_voice:
|
||||
from core.db import db_conn
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT language FROM voice_profiles WHERE id=?", (default_voice,)
|
||||
).fetchone()
|
||||
if row:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
return prof_lang
|
||||
return None
|
||||
|
||||
|
||||
def _build_synth(default_voice: str | None, language: str | None = None) -> dict:
|
||||
"""Describe how to synthesize for the active TTS engine.
|
||||
|
||||
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
|
||||
@@ -223,6 +252,11 @@ def _build_synth(default_voice: str | None) -> dict:
|
||||
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
|
||||
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
|
||||
engine_id)`` once the (async) model is in hand.
|
||||
|
||||
``language`` (already resolved by :func:`_resolve_default_language`) is
|
||||
threaded into every chunk's ``generate`` so a non-English clone stays in its
|
||||
language instead of re-autodetecting per chunk (#505 B2). ``None`` keeps the
|
||||
engine's autodetect behavior unchanged.
|
||||
"""
|
||||
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
|
||||
|
||||
@@ -239,14 +273,14 @@ def _build_synth(default_voice: str | None) -> dict:
|
||||
if cls is OmniVoiceBackend:
|
||||
from services.model_manager import get_model
|
||||
return {"mode": "omnivoice", "resolve": resolve,
|
||||
"engine_id": engine_id, "get_model": get_model}
|
||||
"engine_id": engine_id, "get_model": get_model, "language": language}
|
||||
|
||||
backend = cls()
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return backend.generate(
|
||||
text, language=None, ref_audio=v["ref_audio"],
|
||||
text, language=language, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)
|
||||
@@ -254,20 +288,22 @@ def _build_synth(default_voice: str | None) -> dict:
|
||||
"synth": synth, "sample_rate": backend.sample_rate}
|
||||
|
||||
|
||||
async def _prepare_synth(default_voice: str | None):
|
||||
async def _prepare_synth(default_voice: str | None, language: str | None = None):
|
||||
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
|
||||
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
|
||||
full job and the per-chapter preview."""
|
||||
info = _build_synth(default_voice)
|
||||
full job and the per-chapter preview. ``language`` is threaded into every
|
||||
chunk so a non-English clone holds its language (#505 B2)."""
|
||||
info = _build_synth(default_voice, language=language)
|
||||
resolve, engine_id = info["resolve"], info["engine_id"]
|
||||
if info["mode"] == "omnivoice":
|
||||
lang = info["language"]
|
||||
model = await info["get_model"]()
|
||||
sr = getattr(model, "sampling_rate", 24000)
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return model.generate(
|
||||
text=text, language=None, ref_audio=v["ref_audio"],
|
||||
text=text, language=lang, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)[0]
|
||||
@@ -323,6 +359,7 @@ class AudiobookPreviewRequest(BaseModel):
|
||||
text: str
|
||||
chapter_index: int = 0
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@@ -346,7 +383,10 @@ async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
|
||||
chapter = plan.chapters[req.chapter_index]
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(req.default_voice)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
req.default_voice,
|
||||
language=_resolve_default_language(req.language, req.default_voice),
|
||||
)
|
||||
loop = asyncio.get_running_loop()
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
|
||||
@@ -364,6 +404,7 @@ async def _render_longform_sse(
|
||||
plan,
|
||||
*,
|
||||
default_voice: str | None,
|
||||
language: str | None = None,
|
||||
fmt: str = "m4b",
|
||||
bitrate: str = "128k",
|
||||
loudness: str | None = None,
|
||||
@@ -412,7 +453,8 @@ async def _render_longform_sse(
|
||||
for c in plan.chapters
|
||||
],
|
||||
params={
|
||||
"default_voice": default_voice, "fmt": fmt, "bitrate": bitrate,
|
||||
"default_voice": default_voice, "language": language,
|
||||
"fmt": fmt, "bitrate": bitrate,
|
||||
"loudness": loudness, "cover_path": cover_path,
|
||||
"metadata": metadata, "lexicon": lexicon,
|
||||
},
|
||||
@@ -453,7 +495,9 @@ async def _render_longform_sse(
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
try:
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(default_voice)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
default_voice, language=_resolve_default_language(language, default_voice)
|
||||
)
|
||||
|
||||
total = len(plan.chapters)
|
||||
chapter_files: list[str] = []
|
||||
@@ -557,7 +601,8 @@ async def audiobook_synthesize(req: AudiobookRequest):
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="audiobook",
|
||||
),
|
||||
@@ -582,6 +627,7 @@ class LongformChapter(BaseModel):
|
||||
class LongformRenderRequest(BaseModel):
|
||||
chapters: list[LongformChapter] = []
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b"
|
||||
loudness: str | None = None
|
||||
@@ -612,7 +658,8 @@ async def longform_render(req: LongformRenderRequest):
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="story",
|
||||
),
|
||||
@@ -702,7 +749,7 @@ async def resume_longform(job_id: str):
|
||||
# never names a work dir / output file (defence-in-depth path-injection).
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=p.get("default_voice"),
|
||||
plan, default_voice=p.get("default_voice"), language=p.get("language"),
|
||||
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
|
||||
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
|
||||
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
|
||||
|
||||
@@ -142,7 +142,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
_set_progress(job, "transcribe", 0)
|
||||
|
||||
from services.asr_backend import get_active_asr_backend
|
||||
from services.model_manager import _gpu_pool, _cpu_pool
|
||||
from services.model_manager import _gpu_pool, _cpu_pool, run_on_gpu_pool_guarded
|
||||
from services.segmentation import (
|
||||
segment_transcript, assign_speakers_heuristic,
|
||||
)
|
||||
@@ -162,7 +162,12 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
pass
|
||||
return segments, detected_lang
|
||||
|
||||
segments, source_lang = await loop.run_in_executor(_gpu_pool, _transcribe)
|
||||
# Bound the batch transcribe (#730) so a wedged whisperx/CTranslate2 call
|
||||
# can't hold its GPU-pool worker forever and starve the rest of the backend
|
||||
# ("can't reach backend"); run_transcribe_guarded also resets the pool on
|
||||
# timeout to restore capacity.
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
segments, source_lang = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Batch")
|
||||
source_lang = (source_lang or "en").split("_")[0][:2].lower()
|
||||
job["segments"] = segments
|
||||
job["source_lang"] = source_lang
|
||||
@@ -174,6 +179,21 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
job["status"] = "failed"
|
||||
return
|
||||
|
||||
# ── Engine resolution (issue #312 class) ────────────────────────────
|
||||
# Batch used to hardcode OmniVoice via get_model() regardless of the
|
||||
# engine selected in Settings → Engines. require_cloning only when a
|
||||
# specific voice is pinned (job["voice_id"]) — an unpinned job is fine on
|
||||
# any active engine. Resolved ONCE for the whole job (every language
|
||||
# below shares the same active engine); an uncaught ValueError here
|
||||
# propagates to _worker()'s existing except-Exception handling, which
|
||||
# already records a structured job failure via core.failure.build_failure.
|
||||
from services.tts_backend import resolve_generation_backend
|
||||
backend = await resolve_generation_backend(
|
||||
require_cloning=bool(job.get("voice_id")),
|
||||
cloning_purpose="this batch job's pinned voice",
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
# ── 3. Translate + Generate per language ───────────────────────────
|
||||
total_langs = len(langs)
|
||||
outputs = {}
|
||||
@@ -238,13 +258,10 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
total_segments=len(translated_segments),
|
||||
)
|
||||
|
||||
from services.model_manager import get_model
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
from services.audio_io import atomic_save_wav
|
||||
import torch
|
||||
|
||||
_model = await get_model()
|
||||
sr = _model.sampling_rate
|
||||
total_samples = int(duration * sr)
|
||||
full_audio = torch.zeros(1, total_samples)
|
||||
total_segs = len(translated_segments)
|
||||
@@ -290,28 +307,24 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
ref_text = row.get("ref_text")
|
||||
|
||||
try:
|
||||
audios = _model.generate(
|
||||
audio_out = backend.generate(
|
||||
text=text, language=lang,
|
||||
ref_audio=ref_audio, ref_text=ref_text,
|
||||
duration=dur, num_step=16,
|
||||
guidance_scale=2.0, speed=1.0,
|
||||
denoise=True, postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=sr,
|
||||
)
|
||||
return normalize_audio(mastered, target_dBFS=-2.0)
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
audio_out = apply_mastering(audio_out, sample_rate=sr)
|
||||
return normalize_audio(audio_out, target_dBFS=-2.0)
|
||||
except Exception as e:
|
||||
logger.warning("TTS failed for seg %d (lang=%s): %s", i, lang, e)
|
||||
return torch.zeros(1, int(dur * sr))
|
||||
|
||||
try:
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
|
||||
# Bounded + pool-reset on hang so a wedged batch segment can't
|
||||
# starve the GPU pool and brick the backend (#730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Batch generate")
|
||||
|
||||
# Fit to slot
|
||||
target_samples_seg = int(seg_duration * sr)
|
||||
|
||||
@@ -18,7 +18,7 @@ import os
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from fastapi import APIRouter, File, Form, HTTPException, UploadFile
|
||||
from typing import Optional
|
||||
|
||||
router = APIRouter()
|
||||
@@ -96,9 +96,17 @@ async def transcribe_audio(
|
||||
return result, backend.id
|
||||
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
|
||||
t0 = time.perf_counter()
|
||||
result, engine_id = await loop.run_in_executor(_gpu_pool, _run)
|
||||
try:
|
||||
result, engine_id = await run_transcribe_guarded(
|
||||
_gpu_pool, _run, what="Dictation",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
# Backend is alive — ASR couldn't finish. 504 with guidance, not a
|
||||
# silent hang the UI reads as "can't reach the local backend".
|
||||
logger.warning("Capture transcription timed out: %s", e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
elapsed = round(time.perf_counter() - t0, 2)
|
||||
|
||||
# Normalize result shape
|
||||
@@ -112,6 +120,15 @@ async def transcribe_audio(
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
# Cross-transport parity: deterministically polish the final text
|
||||
# (leading capital + terminal punctuation) exactly like the live
|
||||
# dictation socket (capture_ws) does, so the widget's POST fallback and
|
||||
# MCP/CLI callers get the same typed-looking result the WS returns —
|
||||
# not the raw "...test" the REST path used to leak. Segments stay raw
|
||||
# (their timings/verbatim recognition are the contract).
|
||||
from services.text_polish import polish_text
|
||||
full_text = polish_text(full_text)
|
||||
|
||||
# Calculate audio duration from segments if available
|
||||
duration = 0.0
|
||||
if segments:
|
||||
@@ -127,8 +144,12 @@ async def transcribe_audio(
|
||||
if _truthy(refine) and full_text:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full_text)
|
||||
if refined and refined != full_text:
|
||||
refined_text = refined
|
||||
if refined:
|
||||
# Polish the refined text too, so both surfaced strings read as
|
||||
# typed text (mirrors the raw-vs-refined contract of the WS).
|
||||
refined = polish_text(refined)
|
||||
if refined != full_text:
|
||||
refined_text = refined
|
||||
|
||||
logger.info(
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
|
||||
|
||||
@@ -19,7 +19,15 @@ Protocol:
|
||||
"segments": [...], "language": "en",
|
||||
"duration_s": 4.2, "transcription_time_s": 0.8,
|
||||
"engine": "mlx-whisper"}
|
||||
{"type": "error", "detail": "..."} — error
|
||||
{"type": "status", "stage": "downloading"|"loading"|"ready"}
|
||||
— model cold-start
|
||||
{"type": "error", "message": "...", "kind": "...",
|
||||
"detail": "..."} — error ("detail"
|
||||
kept for legacy)
|
||||
|
||||
Every ``final`` text is normalised by services.text_polish (leading
|
||||
capital for Latin scripts, terminal punctuation, single-spaced) so the
|
||||
pasted result reads like typed text. Partials are raw.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -32,6 +40,7 @@ import time
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
|
||||
|
||||
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
|
||||
from services.text_polish import polish_text
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture_ws")
|
||||
@@ -101,6 +110,30 @@ def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _select_sherpa_spec(websocket: WebSocket):
|
||||
"""Resolve the sherpa dictation model for this WS session, or None.
|
||||
|
||||
A ``?model=<id>`` query param wins (the frontend can pin a model per
|
||||
session); otherwise the persisted ``dictation.model_id`` pref is used (only
|
||||
when dictation is enabled). Returns the :class:`SherpaModelSpec` or None
|
||||
(None → the legacy Whisper/WebM path runs unchanged).
|
||||
"""
|
||||
try:
|
||||
from services import sherpa_dictation as sd
|
||||
except Exception:
|
||||
return None
|
||||
requested = websocket.query_params.get("model")
|
||||
if requested:
|
||||
return sd.get_spec(requested) # explicit selection (may be None if bad)
|
||||
# Fall back to the persisted dictation pref.
|
||||
try:
|
||||
from services.asr_backend import dictation_model_id
|
||||
mid = dictation_model_id()
|
||||
except Exception:
|
||||
mid = None
|
||||
return sd.get_spec(mid) if mid else None
|
||||
|
||||
|
||||
@router.websocket("/ws/transcribe")
|
||||
async def ws_transcribe(websocket: WebSocket):
|
||||
"""Stream audio in, get partial + final transcription out."""
|
||||
@@ -119,6 +152,24 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
|
||||
await websocket.accept()
|
||||
|
||||
# Live-dictation engine selection. When a sherpa-onnx model is selected
|
||||
# (via ?model= or the dictation.model_id pref) AND sherpa is installed,
|
||||
# run the dedicated low-latency handler. Otherwise fall through to the
|
||||
# legacy Whisper/WebM path, byte-for-byte unchanged.
|
||||
spec = _select_sherpa_spec(websocket)
|
||||
if spec is not None:
|
||||
from services.asr_backend import SherpaDictationBackend
|
||||
ok, _reason = SherpaDictationBackend.is_available()
|
||||
if ok:
|
||||
if spec.streaming:
|
||||
await _run_sherpa_streaming(websocket, spec)
|
||||
else:
|
||||
await _run_sherpa_offline(websocket, spec)
|
||||
return
|
||||
# sherpa not installed → fall through to the legacy path so the user
|
||||
# still gets dictation (just not live partials).
|
||||
logger.info("sherpa dictation selected but unavailable — legacy path")
|
||||
|
||||
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
|
||||
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
|
||||
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
|
||||
@@ -260,20 +311,29 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
if total_bytes > MIN_FINAL_BUFFER_BYTES:
|
||||
try:
|
||||
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
|
||||
# Dictation v2: deterministic polish so the pasted final reads
|
||||
# like typed text (leading capital, terminal punctuation).
|
||||
result["text"] = polish_text(result.get("text", ""))
|
||||
# Wave 2.1: optional local-LLM refinement of the final text.
|
||||
# Off-thread (network call, not GPU); pass-through on any
|
||||
# failure or when no LLM backend is configured. The raw text
|
||||
# always ships too — clients paste refined_text ?? text.
|
||||
# HARD-BOUNDED (maybe_refine_async, ~4s OMNIVOICE_REFINE_TIMEOUT_S):
|
||||
# a slow/dead LLM can never delay this `final` beyond the budget —
|
||||
# it falls back to the unrefined (but polished) text. Best-effort:
|
||||
# never let refinement turn a good final into an error. The raw
|
||||
# text always ships too — clients paste refined_text ?? text.
|
||||
if result.get("text"):
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, result["text"])
|
||||
if refined and refined != result["text"]:
|
||||
result["refined_text"] = refined
|
||||
try:
|
||||
from services.refinement import maybe_refine_async
|
||||
refined = await maybe_refine_async(result["text"])
|
||||
if refined and refined != result["text"]:
|
||||
result["refined_text"] = refined
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.debug("Dictation refinement skipped: %s", e)
|
||||
if not await _safe_send({"type": "final", **result}):
|
||||
logger.debug("Skipped final send — client already disconnected")
|
||||
except Exception as e:
|
||||
logger.error("Final transcription failed: %s", e)
|
||||
await _safe_send({"type": "error", "detail": str(e)})
|
||||
await _safe_send({"type": "error", "message": str(e),
|
||||
"kind": "transcribe", "detail": str(e)})
|
||||
else:
|
||||
await _safe_send({
|
||||
"type": "final",
|
||||
@@ -292,6 +352,413 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
pass
|
||||
|
||||
|
||||
# ── sherpa-onnx live dictation handlers ─────────────────────────────────────
|
||||
#
|
||||
# Both handlers read raw int16 mono PCM frames (reusing the AEC framing: an
|
||||
# opt-in 1-byte type prefix when ?aec=1, else bare PCM) at ?sr= (default 16000).
|
||||
# This is the low-latency transport — no WebM/ffmpeg in the hot path.
|
||||
|
||||
# How often the offline-kind handler re-decodes the live window for a partial
|
||||
# (streaming-kind decodes every frame, no cadence needed).
|
||||
SHERPA_OFFLINE_PARTIAL_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_PARTIAL", "0.8"))
|
||||
|
||||
# Utterance gate for the offline-kind handler: once the trailing this-many
|
||||
# seconds of the live buffer fall below the RMS floor, the utterance is
|
||||
# COMMITTED — decoded, flushed as a `final`, and dropped from the buffer. Each
|
||||
# decode is thereby bounded by one utterance instead of the whole session
|
||||
# (the old full-buffer re-decode was O(n²)), and a sentence commits ~0.6s
|
||||
# after the user stops speaking instead of only at EOF.
|
||||
SHERPA_OFFLINE_SILENCE_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_SILENCE", "0.6"))
|
||||
SHERPA_OFFLINE_RMS_FLOOR = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_RMS", "0.01"))
|
||||
|
||||
|
||||
def _pcm16_to_f32(pcm: bytes):
|
||||
"""int16 little-endian mono PCM bytes → float32 numpy in [-1, 1]."""
|
||||
import numpy as np
|
||||
if not pcm:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
# Guard against an odd trailing byte from a split frame.
|
||||
if len(pcm) % 2:
|
||||
pcm = pcm[:-1]
|
||||
return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
async def _sherpa_session(websocket: WebSocket):
|
||||
"""Shared WS receive setup for the sherpa handlers.
|
||||
|
||||
Returns ``(get_frame, state)`` where ``get_frame`` is an async callable
|
||||
that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref
|
||||
frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC,
|
||||
and the disconnect flag for the caller's finaliser.
|
||||
"""
|
||||
pcm_sr = 16000
|
||||
try:
|
||||
pcm_sr = int(websocket.query_params.get("sr", "16000"))
|
||||
except (TypeError, ValueError):
|
||||
pcm_sr = 16000
|
||||
aec = None
|
||||
if websocket.query_params.get("aec") in ("1", "true", "on"):
|
||||
try:
|
||||
from services.aec import NlmsEchoCanceller
|
||||
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
|
||||
except Exception as e:
|
||||
logger.warning("AEC requested but disabled (sherpa): %s", e)
|
||||
aec = None
|
||||
return pcm_sr, aec
|
||||
|
||||
|
||||
async def _recv_pcm_frame(websocket: WebSocket, aec):
|
||||
"""Receive one frame; return (kind, pcm_bytes).
|
||||
|
||||
kind ∈ {"near","eof","skip"}. Demuxes AEC-tagged frames when ``aec`` is on
|
||||
and feeds the playback reference into the canceller. A text "EOF" or an
|
||||
empty/closed socket yields kind "eof".
|
||||
"""
|
||||
msg = await websocket.receive()
|
||||
mtype = msg.get("type")
|
||||
if mtype == "websocket.disconnect":
|
||||
return "eof", b""
|
||||
if mtype != "websocket.receive":
|
||||
return "skip", b""
|
||||
data = msg.get("bytes")
|
||||
if data is not None:
|
||||
if len(data) == 0:
|
||||
return "eof", b""
|
||||
if aec is not None:
|
||||
kind, payload = _demux_aec_frame(data)
|
||||
if kind == "far":
|
||||
aec.push_far_end(payload)
|
||||
return "skip", b""
|
||||
if not payload:
|
||||
return "skip", b""
|
||||
return "near", aec.process_near_end(payload)
|
||||
return "near", data
|
||||
if msg.get("text") == "EOF":
|
||||
return "eof", b""
|
||||
return "skip", b""
|
||||
|
||||
|
||||
async def _sherpa_load_with_status(websocket: WebSocket, backend, spec) -> bool:
|
||||
"""Build the recognizer off the event loop, narrating cold-start progress.
|
||||
|
||||
Sends ``{"type":"status","stage":"downloading"|"loading"}`` before the
|
||||
load ("downloading" when the pinned assets aren't in the HF cache yet;
|
||||
stage-only — HF's per-file progress isn't worth a callback plumb-through)
|
||||
and ``{"type":"status","stage":"ready"}`` after, so the widget can show
|
||||
*why* the first dictation takes a moment. Returns False when the load
|
||||
failed (the error frame is sent and the socket closed here).
|
||||
"""
|
||||
try:
|
||||
from services import sherpa_dictation as _sd
|
||||
stage = "loading" if _sd.is_installed(spec) else "downloading"
|
||||
except Exception:
|
||||
stage = "loading"
|
||||
try:
|
||||
await websocket.send_json({"type": "status", "stage": stage})
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await asyncio.to_thread(backend.ensure_loaded)
|
||||
except Exception as e:
|
||||
logger.error("sherpa dictation load failed (%s): %s", spec.id, e)
|
||||
try:
|
||||
await websocket.send_json({"type": "error", "message": str(e),
|
||||
"kind": "load", "detail": str(e)})
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
try:
|
||||
await websocket.send_json({"type": "status", "stage": "ready"})
|
||||
except Exception:
|
||||
pass
|
||||
return True
|
||||
|
||||
|
||||
async def _run_sherpa_streaming(websocket: WebSocket, spec):
|
||||
"""True streaming: feed the OnlineRecognizer frame-by-frame, emit `partial`
|
||||
every time the decoded text grows, and `final` on sherpa's endpoint (silence)
|
||||
detection and on EOF. <300ms perceived latency on CPU for the tiny models.
|
||||
"""
|
||||
import numpy as np
|
||||
from services.asr_backend import get_sherpa_dictation_backend
|
||||
|
||||
pcm_sr, aec = await _sherpa_session(websocket)
|
||||
logger.info("sherpa streaming dictation: model=%s sr=%d aec=%s",
|
||||
spec.id, pcm_sr, bool(aec))
|
||||
|
||||
# Reuse the shared, per-model warm backend (#888): the recognizer is built
|
||||
# once and shared across sessions instead of rebuilt (1.3–2.5s) per connect,
|
||||
# so the first dictation is instant when the preload warmed it. Each session
|
||||
# still gets its own decode stream below.
|
||||
backend = get_sherpa_dictation_backend(spec.id)
|
||||
# Build the recognizer off the event loop if it isn't warm yet
|
||||
# (download-on-first-use + ONNX session init can take a moment); status
|
||||
# frames keep the widget honest.
|
||||
if not await _sherpa_load_with_status(websocket, backend, spec):
|
||||
return
|
||||
rec = backend._rec
|
||||
stream = rec.create_stream()
|
||||
|
||||
last_partial = ""
|
||||
committed: list[str] = [] # finalized utterances this session
|
||||
client_disconnected = False
|
||||
|
||||
async def _send(payload) -> bool:
|
||||
nonlocal client_disconnected
|
||||
if client_disconnected:
|
||||
return False
|
||||
try:
|
||||
await websocket.send_json(payload)
|
||||
return True
|
||||
except Exception:
|
||||
client_disconnected = True
|
||||
return False
|
||||
|
||||
def _decode_after_feed(pcm: bytes):
|
||||
"""Blocking: feed one PCM frame, decode, return (text, is_endpoint).
|
||||
Runs in a thread so the ONNX work never blocks the event loop."""
|
||||
samples = _pcm16_to_f32(pcm)
|
||||
if len(samples):
|
||||
stream.accept_waveform(pcm_sr, samples)
|
||||
while rec.is_ready(stream):
|
||||
rec.decode_stream(stream)
|
||||
endpoint = rec.is_endpoint(stream)
|
||||
text = (rec.get_result(stream) or "").strip()
|
||||
return text, endpoint
|
||||
|
||||
def _flush_final():
|
||||
"""Blocking: pad + drain the stream for the trailing utterance."""
|
||||
tail = np.zeros(int(0.5 * pcm_sr), dtype=np.float32)
|
||||
stream.accept_waveform(pcm_sr, tail)
|
||||
stream.input_finished()
|
||||
while rec.is_ready(stream):
|
||||
rec.decode_stream(stream)
|
||||
return (rec.get_result(stream) or "").strip()
|
||||
|
||||
try:
|
||||
while True:
|
||||
kind, pcm = await _recv_pcm_frame(websocket, aec)
|
||||
if kind == "eof":
|
||||
break
|
||||
if kind == "skip":
|
||||
continue
|
||||
text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm)
|
||||
if endpoint:
|
||||
# Commit this utterance (polished — it gets pasted); reset
|
||||
# for the next one.
|
||||
text = polish_text(text)
|
||||
if text:
|
||||
committed.append(text)
|
||||
await _send({"type": "final", "text": text,
|
||||
"segments": [{"start": 0.0, "end": None, "text": text}],
|
||||
"language": "auto", "engine": backend.id})
|
||||
rec.reset(stream)
|
||||
last_partial = ""
|
||||
elif text and text != last_partial:
|
||||
last_partial = text
|
||||
await _send({"type": "partial", "text": text})
|
||||
except WebSocketDisconnect:
|
||||
client_disconnected = True
|
||||
except Exception as e:
|
||||
logger.warning("sherpa streaming loop ended: %s", e)
|
||||
client_disconnected = True
|
||||
|
||||
# Drain the trailing (un-endpointed) utterance on EOF.
|
||||
try:
|
||||
tail_text = await asyncio.to_thread(_flush_final)
|
||||
except Exception as e:
|
||||
logger.debug("sherpa streaming flush failed: %s", e)
|
||||
tail_text = ""
|
||||
tail_text = polish_text(tail_text)
|
||||
if tail_text and tail_text != (committed[-1] if committed else None):
|
||||
committed.append(tail_text)
|
||||
|
||||
# Pieces are already polished; the join is too (polish is idempotent).
|
||||
full = " ".join(t for t in committed if t).strip()
|
||||
segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t]
|
||||
if not client_disconnected:
|
||||
if full:
|
||||
# Hard-bounded refinement (~4s): never delays this summary `final`
|
||||
# beyond OMNIVOICE_REFINE_TIMEOUT_S even with a dead LLM endpoint.
|
||||
try:
|
||||
from services.refinement import maybe_refine_async
|
||||
refined = await maybe_refine_async(full)
|
||||
except Exception:
|
||||
refined = None
|
||||
payload = {"type": "final", "text": full, "segments": segments,
|
||||
"language": "auto", "engine": backend.id}
|
||||
if refined and refined != full:
|
||||
payload["refined_text"] = refined
|
||||
await _send(payload)
|
||||
else:
|
||||
await _send({"type": "final", "text": "", "segments": [],
|
||||
"language": "auto", "engine": backend.id})
|
||||
try:
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _run_sherpa_offline(websocket: WebSocket, spec):
|
||||
"""Offline-kind sherpa model with live partials, utterance-windowed.
|
||||
|
||||
Raw PCM accumulates in a *live* buffer holding only the current
|
||||
(uncommitted) utterance. Every ~800ms the live window is re-decoded for a
|
||||
``partial``; when the trailing ~0.6s of it fall below the RMS floor the
|
||||
utterance is committed — decoded once more, flushed as a ``final``, and
|
||||
its samples dropped — so per-partial cost is bounded by one utterance
|
||||
(not the whole session) and sentences commit as the user pauses instead
|
||||
of only at EOF."""
|
||||
from services.asr_backend import get_sherpa_dictation_backend
|
||||
|
||||
pcm_sr, aec = await _sherpa_session(websocket)
|
||||
logger.info("sherpa offline dictation: model=%s sr=%d aec=%s",
|
||||
spec.id, pcm_sr, bool(aec))
|
||||
|
||||
# Shared, per-model warm backend (#888) — built once, reused per session.
|
||||
backend = get_sherpa_dictation_backend(spec.id)
|
||||
if not await _sherpa_load_with_status(websocket, backend, spec):
|
||||
return
|
||||
|
||||
buf = bytearray() # live (uncommitted) PCM only
|
||||
committed: list[str] = [] # polished utterances already flushed
|
||||
last_partial = ""
|
||||
running = True
|
||||
client_disconnected = False
|
||||
last_audio = time.monotonic()
|
||||
# Trailing-silence gate window, in bytes of int16 mono PCM.
|
||||
sil_bytes = max(2, int(SHERPA_OFFLINE_SILENCE_S * pcm_sr) * 2)
|
||||
|
||||
async def _send(payload) -> bool:
|
||||
nonlocal client_disconnected
|
||||
if client_disconnected:
|
||||
return False
|
||||
try:
|
||||
await websocket.send_json(payload)
|
||||
return True
|
||||
except Exception:
|
||||
client_disconnected = True
|
||||
return False
|
||||
|
||||
def _rms(pcm: bytes) -> float:
|
||||
samples = _pcm16_to_f32(pcm)
|
||||
if not len(samples):
|
||||
return 0.0
|
||||
return float((samples * samples).mean() ** 0.5)
|
||||
|
||||
def _decode_window(pcm: bytes) -> str:
|
||||
samples = _pcm16_to_f32(pcm)
|
||||
if not len(samples):
|
||||
return ""
|
||||
return backend._decode_offline(samples, pcm_sr)
|
||||
|
||||
async def receive():
|
||||
nonlocal running, client_disconnected, last_audio
|
||||
try:
|
||||
while running:
|
||||
kind, pcm = await _recv_pcm_frame(websocket, aec)
|
||||
if kind == "eof":
|
||||
running = False
|
||||
break
|
||||
if kind == "skip":
|
||||
continue
|
||||
buf.extend(pcm)
|
||||
last_audio = time.monotonic()
|
||||
except WebSocketDisconnect:
|
||||
client_disconnected = True
|
||||
running = False
|
||||
except Exception as e:
|
||||
logger.debug("sherpa offline receive ended: %s", e)
|
||||
running = False
|
||||
|
||||
async def _commit(snapshot: bytes):
|
||||
"""Finalize one utterance: decode it off-thread, flush a polished
|
||||
`final`, drop its samples from the live buffer. `receive()` may
|
||||
append while we decode — only the snapshot's prefix is dropped."""
|
||||
nonlocal last_partial
|
||||
try:
|
||||
text = await asyncio.to_thread(_decode_window, snapshot)
|
||||
except Exception as e:
|
||||
logger.debug("sherpa offline commit decode failed: %s", e)
|
||||
return
|
||||
del buf[:len(snapshot)]
|
||||
last_partial = ""
|
||||
text = polish_text(text)
|
||||
if text:
|
||||
committed.append(text)
|
||||
await _send({"type": "final", "text": text,
|
||||
"segments": [{"start": 0.0, "end": None, "text": text}],
|
||||
"language": "auto", "engine": backend.id})
|
||||
|
||||
async def partials():
|
||||
nonlocal last_partial, running
|
||||
while running:
|
||||
await asyncio.sleep(SHERPA_OFFLINE_PARTIAL_S)
|
||||
if not running or len(buf) < 2000:
|
||||
continue
|
||||
snapshot = bytes(buf)
|
||||
if len(snapshot) > sil_bytes and \
|
||||
_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR:
|
||||
if _rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR:
|
||||
await _commit(snapshot)
|
||||
else:
|
||||
# Pure silence — drop it (keep the gate window for
|
||||
# continuity) so a long pause can't grow the buffer.
|
||||
del buf[:len(snapshot) - sil_bytes]
|
||||
continue
|
||||
try:
|
||||
text = await asyncio.to_thread(_decode_window, snapshot)
|
||||
except Exception as e:
|
||||
logger.debug("sherpa offline partial failed: %s", e)
|
||||
continue
|
||||
if text and text != last_partial:
|
||||
last_partial = text
|
||||
await _send({"type": "partial", "text": text})
|
||||
|
||||
recv_task = asyncio.create_task(receive())
|
||||
part_task = asyncio.create_task(partials())
|
||||
await asyncio.wait([recv_task, part_task], return_when=asyncio.FIRST_COMPLETED)
|
||||
running = False
|
||||
for t in (recv_task, part_task):
|
||||
if not t.done():
|
||||
t.cancel()
|
||||
try:
|
||||
await t
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
|
||||
# Drain the trailing (un-committed) utterance on EOF.
|
||||
try:
|
||||
tail = await asyncio.to_thread(_decode_window, bytes(buf))
|
||||
except Exception as e:
|
||||
logger.error("sherpa offline final failed: %s", e)
|
||||
tail = ""
|
||||
tail = polish_text(tail)
|
||||
if tail:
|
||||
committed.append(tail)
|
||||
# Pieces are already polished; the join is too (polish is idempotent).
|
||||
full = " ".join(committed).strip()
|
||||
segments = [{"start": 0.0, "end": None, "text": t} for t in committed]
|
||||
if not client_disconnected:
|
||||
payload = {"type": "final", "text": full, "segments": segments,
|
||||
"language": "auto", "engine": backend.id}
|
||||
if full:
|
||||
# Hard-bounded refinement (~4s) — never delays the `final`.
|
||||
try:
|
||||
from services.refinement import maybe_refine_async
|
||||
refined = await maybe_refine_async(full)
|
||||
if refined and refined != full:
|
||||
payload["refined_text"] = refined
|
||||
except Exception:
|
||||
pass
|
||||
await _send(payload)
|
||||
try:
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
|
||||
"""Quick partial transcription of the current audio buffer."""
|
||||
|
||||
@@ -301,15 +768,17 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
|
||||
|
||||
try:
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
|
||||
|
||||
def _run():
|
||||
backend = get_capture_asr_backend()
|
||||
result = backend.transcribe(tmp, word_timestamps=False)
|
||||
return result.get("text", "")
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
text = await loop.run_in_executor(_gpu_pool, _run)
|
||||
# Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call
|
||||
# must not hold its GPU-pool worker forever and starve TTS / other ASR
|
||||
# into a "can't reach backend"; on timeout the pool is reset to recover.
|
||||
text = await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
|
||||
return text.strip()
|
||||
finally:
|
||||
try:
|
||||
@@ -327,7 +796,7 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
|
||||
try:
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
|
||||
|
||||
def _run():
|
||||
backend = get_capture_asr_backend()
|
||||
@@ -362,8 +831,9 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
"engine": backend.id,
|
||||
}
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(_gpu_pool, _run)
|
||||
# Bounded + pool-resetting on timeout (#730), same rationale as the
|
||||
# partial path above.
|
||||
return await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
"""
|
||||
Dictation router — sherpa-onnx live-dictation engine.
|
||||
|
||||
Exposes the seven sherpa-onnx dictation models and the dictation prefs the
|
||||
frontend dictation UI binds to.
|
||||
|
||||
GET /dictation/models → the 7 models + install state (frontend model list)
|
||||
GET /dictation/prefs → { enabled, mode, model_id }
|
||||
POST /dictation/prefs → persist any subset of those prefs
|
||||
|
||||
Install state reuses the same HF-cache check the model store uses, so a model
|
||||
shown "installed" here is the same snapshot the backend will load.
|
||||
|
||||
Prefs are stored in the shared ``prefs.json`` store under the ``dictation.*``
|
||||
namespace (``dictation.enabled``, ``dictation.mode``, ``dictation.model_id``),
|
||||
mirroring how the ASR/TTS engine picks persist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from core import prefs
|
||||
from services import sherpa_dictation as sd
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.dictation")
|
||||
|
||||
# Pref keys (the binding contract — the frontend writes exactly these).
|
||||
PREF_ENABLED = "dictation.enabled"
|
||||
PREF_MODE = "dictation.mode"
|
||||
PREF_MODEL_ID = "dictation.model_id"
|
||||
|
||||
_DEFAULT_ENABLED = True
|
||||
_DEFAULT_MODE = "toggle"
|
||||
_VALID_MODES = ("toggle", "hold")
|
||||
|
||||
|
||||
def _read_prefs() -> dict:
|
||||
mid = prefs.get(PREF_MODEL_ID, sd.DEFAULT_MODEL_ID)
|
||||
if not sd.is_sherpa_model(mid):
|
||||
mid = sd.DEFAULT_MODEL_ID
|
||||
mode = prefs.get(PREF_MODE, _DEFAULT_MODE)
|
||||
if mode not in _VALID_MODES:
|
||||
mode = _DEFAULT_MODE
|
||||
return {
|
||||
"enabled": bool(prefs.get(PREF_ENABLED, _DEFAULT_ENABLED)),
|
||||
"mode": mode,
|
||||
"model_id": mid,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dictation/models", dependencies=[Depends(require_loopback)])
|
||||
def list_dictation_models():
|
||||
"""The seven sherpa-onnx dictation models + install state.
|
||||
|
||||
Each entry: id, repo_id, label, tag ("offline"|"streaming"), recommended,
|
||||
size_gb, languages, kind, and install state (installed/installing). The
|
||||
``installed`` flag is computed from the same HF cache the model store reads,
|
||||
so it matches the model-store row state.
|
||||
"""
|
||||
available, reason = sd.sherpa_available()
|
||||
out = []
|
||||
for spec in sd.list_specs():
|
||||
out.append({
|
||||
"id": spec.id,
|
||||
"repo_id": spec.repo_id,
|
||||
"label": spec.label,
|
||||
"tag": spec.tag,
|
||||
"recommended": spec.recommended,
|
||||
"size_gb": spec.size_gb,
|
||||
"languages": spec.languages,
|
||||
"kind": spec.kind,
|
||||
"installed": sd.is_installed(spec),
|
||||
})
|
||||
return {
|
||||
"models": out,
|
||||
"engine_available": available,
|
||||
"engine_reason": None if available else reason,
|
||||
"default_model_id": sd.DEFAULT_MODEL_ID,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dictation/prefs", dependencies=[Depends(require_loopback)])
|
||||
def get_dictation_prefs():
|
||||
return _read_prefs()
|
||||
|
||||
|
||||
class DictationPrefsUpdate(BaseModel):
|
||||
enabled: Optional[bool] = None
|
||||
mode: Optional[str] = None
|
||||
model_id: Optional[str] = None
|
||||
|
||||
|
||||
@router.post("/dictation/prefs", dependencies=[Depends(require_loopback)])
|
||||
def set_dictation_prefs(req: DictationPrefsUpdate):
|
||||
"""Persist any subset of the dictation prefs. Validates ``mode`` and
|
||||
``model_id`` so a bad value can't wedge the capture engine."""
|
||||
if req.mode is not None:
|
||||
if req.mode not in _VALID_MODES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"mode must be one of {_VALID_MODES}",
|
||||
)
|
||||
prefs.set_(PREF_MODE, req.mode)
|
||||
if req.model_id is not None:
|
||||
if not sd.is_sherpa_model(req.model_id):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"unknown dictation model_id {req.model_id!r}",
|
||||
)
|
||||
# Normalise to the canonical dictation id (accept repo_id too).
|
||||
prefs.set_(PREF_MODEL_ID, sd.get_spec(req.model_id).id)
|
||||
if req.enabled is not None:
|
||||
prefs.set_(PREF_ENABLED, bool(req.enabled))
|
||||
# Rebuild the cached capture singleton so the change takes effect at once.
|
||||
try:
|
||||
from services import asr_backend
|
||||
asr_backend._capture_backend = None
|
||||
asr_backend._capture_backend_key = None
|
||||
except Exception:
|
||||
pass
|
||||
return _read_prefs()
|
||||
+395
-88
@@ -16,6 +16,7 @@ from core.tasks import task_manager
|
||||
from core import event_bus
|
||||
from schemas.requests import DubIngestUrlRequest
|
||||
from services.model_manager import get_model, _gpu_pool, _cpu_pool, get_diarization_pipeline, offload_tts_for_asr, restore_tts_after_asr
|
||||
from services.asr_backend import ASRTimeoutError, reset_pool_after_wedge, run_transcribe_guarded
|
||||
from services.audio_io import _safe_soundfile_write
|
||||
from services.ffmpeg_utils import find_ffmpeg
|
||||
from services.segmentation import (
|
||||
@@ -23,6 +24,9 @@ from services.segmentation import (
|
||||
assign_speakers_from_diarization,
|
||||
assign_speakers_from_turns,
|
||||
assign_speakers_heuristic,
|
||||
resplit_segments_by_diarization,
|
||||
resplit_segments_by_turns,
|
||||
_words_from_whisper,
|
||||
clean_up_segments,
|
||||
)
|
||||
from services.onset_align import snap_segment_starts
|
||||
@@ -31,6 +35,7 @@ from services import dub_pipeline
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
|
||||
# ── Legacy-name aliases to services/dub_pipeline.py ────────────────────────
|
||||
# Phase 2.4 moved the business logic into a service. Other routers
|
||||
# (dub_generate, dub_translate, dub_export) + internal call sites below still
|
||||
@@ -360,11 +365,41 @@ async def dub_ingest_url(req: DubIngestUrlRequest):
|
||||
|
||||
TRANSCRIBE_CHUNK_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_S", "30.0"))
|
||||
TRANSCRIBE_CHUNK_TIMEOUT_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S", "120.0"))
|
||||
#: How many times to attempt each transcribe chunk before giving up on it. A
|
||||
#: transient wedge (esp. the first chunk, where whisperx cold-loads its model)
|
||||
#: shouldn't silently drop that whole window — retry once on a fresh pool so the
|
||||
#: transcript doesn't come back "missing the beginning".
|
||||
_CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_ATTEMPTS", "2")))
|
||||
|
||||
|
||||
_sse_event = dub_pipeline.sse_event
|
||||
_prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local _prep_event below for the inline one-liner shape
|
||||
|
||||
#: User-facing warning emitted when auto voice cloning is skipped because the
|
||||
#: speaker labels came from the silence-gap heuristic (see _diarize /
|
||||
#: extract_speaker_clones — gap-based labels routinely mix two people's audio
|
||||
#: into one reference, which is how "made up" clone voices happen).
|
||||
CLONE_SKIP_HEURISTIC_MSG = (
|
||||
"auto voice cloning skipped: speaker labels are gap-based estimates — "
|
||||
"set up diarization (Settings → Models → pyannote) for per-speaker clones"
|
||||
)
|
||||
|
||||
|
||||
def _clamp_num_speakers(value) -> Optional[int]:
|
||||
"""Clamp the user's speaker-count hint to a sane 1–20 range.
|
||||
|
||||
Shared by the SSE and legacy transcribe endpoints so the two can't drift.
|
||||
None / non-int / out-of-range → None (auto-detect), so a bad query string
|
||||
can never break a diarization call.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
value = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value if 1 <= value <= 20 else None
|
||||
|
||||
|
||||
@router.get("/dub/transcribe-stream/{job_id}")
|
||||
async def dub_transcribe_stream(
|
||||
@@ -383,15 +418,14 @@ async def dub_transcribe_stream(
|
||||
pyannote auto-detects the count — but its auto-detect can collapse a
|
||||
multi-speaker clip to a single speaker (issue #274). When the user knows
|
||||
the exact count, supplying it forces pyannote to return that many speakers.
|
||||
On paths that can't honor the hint exactly (inline ASR turns, the
|
||||
silence-gap heuristic) it is never silently dropped: the heuristic cycles
|
||||
the requested count and a `warning` SSE event tells the user how far the
|
||||
labels can be trusted.
|
||||
"""
|
||||
# Clamp to a sane range; ignore anything non-positive / absurd so a bad
|
||||
# query string can never break the diarization call. None → auto-detect.
|
||||
if num_speakers is not None:
|
||||
try:
|
||||
num_speakers = int(num_speakers)
|
||||
num_speakers = num_speakers if 1 <= num_speakers <= 20 else None
|
||||
except (TypeError, ValueError):
|
||||
num_speakers = None
|
||||
num_speakers = _clamp_num_speakers(num_speakers)
|
||||
|
||||
job = _get_job(job_id)
|
||||
|
||||
@@ -418,6 +452,11 @@ async def dub_transcribe_stream(
|
||||
asr_audio_target = job.get("vocals_path")
|
||||
if not asr_audio_target or not os.path.exists(asr_audio_target):
|
||||
asr_audio_target = job.get("audio_path")
|
||||
# #963: onset snapping is only trustworthy on the Demucs vocals
|
||||
# track. When separation failed/was skipped, dub_pipeline sets
|
||||
# vocals_path to the mixed audio_path — so compare paths instead
|
||||
# of trusting the key's presence.
|
||||
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
|
||||
if not asr_audio_target or not os.path.exists(asr_audio_target):
|
||||
preflight_error = "No audio available for transcription."
|
||||
else:
|
||||
@@ -425,10 +464,22 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
# The PyTorch-Whisper backend lazily builds its own pipeline
|
||||
# when no preloaded `_asr_pipe` is present (issue #255), so it
|
||||
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1 — don't reject it
|
||||
# here; any load failure surfaces per-chunk with a real cause.
|
||||
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1.
|
||||
_asr_backend = get_active_asr_backend(asr_pipe=getattr(_model, "_asr_pipe", None))
|
||||
# Eagerly load the model HERE so a real load failure (e.g.
|
||||
# WhisperX: missing weights, CTranslate2/cuDNN mismatch, the
|
||||
# torch-2.6 weights-only VAD regression) surfaces once, with
|
||||
# its actual cause, as a clean preflight `error` event —
|
||||
# instead of being buried in N cryptic per-chunk failures
|
||||
# and retried on every chunk (#578). Run in a thread so the
|
||||
# (blocking) load doesn't stall the event loop.
|
||||
_ensure_loaded = getattr(_asr_backend, "ensure_loaded", None)
|
||||
if callable(_ensure_loaded):
|
||||
await asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, _ensure_loaded
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception("transcribe preflight: ASR load failed (job=%s)", job_id)
|
||||
from core.failure import build_failure
|
||||
f = build_failure(e, stage="transcribe-preflight", include_diagnostic=False)
|
||||
preflight_error = "ASR backend initialization failed: " + f["reason"] + (
|
||||
@@ -436,9 +487,16 @@ async def dub_transcribe_stream(
|
||||
)
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
|
||||
async def gen():
|
||||
async def _gen_body():
|
||||
if preflight_error:
|
||||
yield _sse_event("error", {"detail": preflight_error})
|
||||
# Always follow a terminal `error` with `done` so the stream closes
|
||||
# via a named event, not a raw connection drop. A bare error+close
|
||||
# races the browser's native EventSource error (which carries no
|
||||
# `data`); if that native error wins, the client falls back to the
|
||||
# misleading generic "stream dropped … ASR backend failed" message
|
||||
# and the real cause (in `detail`) is lost (#578).
|
||||
yield _sse_event("error", {"detail": preflight_error, "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
return
|
||||
import math
|
||||
import tempfile
|
||||
@@ -453,7 +511,9 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
audio_np, sr = await loop.run_in_executor(_cpu_pool, _load)
|
||||
except Exception as e:
|
||||
yield _sse_event("error", {"detail": f"audio load failed: {e}"})
|
||||
# Terminal error → always emit `done` (see preflight note, #578).
|
||||
yield _sse_event("error", {"detail": f"audio load failed: {e}", "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
return
|
||||
|
||||
total = float(len(audio_np)) / float(sr) if sr else 0.0
|
||||
@@ -470,6 +530,9 @@ async def dub_transcribe_stream(
|
||||
logger.warning("offload_tts_for_asr failed (continuing): %s", e)
|
||||
|
||||
all_segments: list[dict] = []
|
||||
# Words (global-timeline) retained so diarization can re-split a segment
|
||||
# that spans two speakers' turns at the word boundary (#486).
|
||||
all_words: list = []
|
||||
detected_lang = None
|
||||
next_seg_id = 0
|
||||
chunk_errors: list[str] = []
|
||||
@@ -521,31 +584,59 @@ async def dub_transcribe_stream(
|
||||
logger.exception("chunk transcribe failed (backend=%s)", _asr_backend.id)
|
||||
return {"chunks": [], "language": None, "error": str(e)}
|
||||
|
||||
try:
|
||||
# wait_for in a loop to yield pings so the EventSource connection doesn't drop
|
||||
fut = loop.run_in_executor(_gpu_pool, _transcribe_chunk)
|
||||
waited = 0.0
|
||||
part = None
|
||||
# Retry a failed/timed-out chunk once on a fresh pool before giving
|
||||
# up. Otherwise a transient wedge on the FIRST chunk (whisperx often
|
||||
# cold-loads its model there, the #730 hang) drops that whole window
|
||||
# and the transcript is "missing the beginning, only middle+end".
|
||||
# The retry reuses the same audio window, so a recovered chunk fills
|
||||
# the hole instead of leaving silent gaps.
|
||||
part = None
|
||||
for _attempt in range(1, _CHUNK_TRANSCRIBE_ATTEMPTS + 1):
|
||||
# A wedged chunk gets the SAME guarded-timeout + pool-reset
|
||||
# semantics as the whole-file paths (#730/#851):
|
||||
# run_transcribe_guarded bounds the call, abandons the poisoned
|
||||
# pool so the retry (and any concurrent TTS work) gets a fresh
|
||||
# worker, and raises the actionable ASRTimeoutError. Run it as
|
||||
# a task and poll so we can keep yielding pings — the
|
||||
# EventSource connection drops without them.
|
||||
pool_reset_by_guard = False
|
||||
task = asyncio.ensure_future(run_transcribe_guarded(
|
||||
_gpu_pool, _transcribe_chunk,
|
||||
what=f"Dub chunk {i + 1}/{chunks_n}",
|
||||
timeout=TRANSCRIBE_CHUNK_TIMEOUT_S,
|
||||
timeout_env="OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S",
|
||||
))
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut], timeout=5.0)
|
||||
done, _pending = await asyncio.wait({task}, timeout=5.0)
|
||||
if done:
|
||||
part = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
waited += 5.0
|
||||
if waited >= TRANSCRIBE_CHUNK_TIMEOUT_S:
|
||||
# Re-raise TimeoutError if we exceed the overall limit
|
||||
raise asyncio.TimeoutError()
|
||||
except asyncio.TimeoutError:
|
||||
logger.error(
|
||||
"Transcribe chunk %d/%d timed out after %.0fs (job=%s)",
|
||||
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, job_id,
|
||||
)
|
||||
part = {
|
||||
"chunks": [], "language": None,
|
||||
"error": f"Chunk {i+1} timed out after {TRANSCRIBE_CHUNK_TIMEOUT_S:.0f}s — "
|
||||
f"ASR backend may be stuck. Try restarting the server.",
|
||||
}
|
||||
try:
|
||||
part = task.result()
|
||||
except ASRTimeoutError as e:
|
||||
# The guard already reset the pool; keep the actionable
|
||||
# message (it names the durable fixes, and — after repeated
|
||||
# timeouts — the crash-isolated engine escape hatch).
|
||||
pool_reset_by_guard = True
|
||||
logger.error(
|
||||
"Transcribe chunk %d/%d timed out after %.0fs (attempt %d/%d, job=%s)",
|
||||
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, _attempt,
|
||||
_CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
|
||||
)
|
||||
part = {"chunks": [], "language": None, "error": str(e)}
|
||||
# Success → keep it. Failure/timeout → retry once on a fresh
|
||||
# worker (the internal _transcribe_chunk except returns an
|
||||
# error-part; the timeout path already reset the pool).
|
||||
if part is not None and not part.get("error"):
|
||||
break
|
||||
if _attempt < _CHUNK_TRANSCRIBE_ATTEMPTS:
|
||||
logger.warning(
|
||||
"Retrying transcribe chunk %d/%d after failure/timeout (next attempt %d/%d, job=%s)",
|
||||
i + 1, chunks_n, _attempt + 1, _CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
|
||||
)
|
||||
if not pool_reset_by_guard:
|
||||
reset_pool_after_wedge(
|
||||
_gpu_pool, what=f"Dub chunk {i + 1}/{chunks_n}")
|
||||
if part.get("error"):
|
||||
chunk_errors.append(part["error"])
|
||||
logger.warning("Chunk %d/%d error: %s", i + 1, chunks_n, part["error"])
|
||||
@@ -553,16 +644,29 @@ async def dub_transcribe_stream(
|
||||
detected_lang = part["language"]
|
||||
asr_speaker_turns.extend(part.get("speaker_turns") or [])
|
||||
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
|
||||
# Same word source segment_transcript used (already global-timeline),
|
||||
# kept for the post-diarization speaker re-split (#486).
|
||||
try:
|
||||
all_words.extend(_words_from_whisper(part))
|
||||
except Exception:
|
||||
pass
|
||||
# #280: Whisper often stretches a segment's start back over
|
||||
# leading music/silence (classic case: speech begins at 0:03,
|
||||
# transcript says 0.0 → the dub plays 3 s early). Snap starts
|
||||
# forward to the actual speech onset. `audio_np` is the same
|
||||
# track ASR ran on — vocals.wav when Demucs succeeded.
|
||||
# track ASR ran on — vocals.wav when Demucs succeeded. #963:
|
||||
# when it didn't (mixed audio), snapping is disabled — every
|
||||
# footstep/sigh/score cue is a false onset candidate there.
|
||||
try:
|
||||
snap_segment_starts(chunk_segs, audio_np, sr)
|
||||
snap_segment_starts(chunk_segs, audio_np, sr,
|
||||
separated_vocals=asr_on_vocals)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs)
|
||||
# Provisional per-chunk labels for the streaming UI only — the
|
||||
# final diarization pass below overwrites them. Honor the user's
|
||||
# speaker-count hint here too so the interim view doesn't flip
|
||||
# between 2 and N speakers.
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs, num_speakers)
|
||||
for s in chunk_segs:
|
||||
s["id"] = f"s{next_seg_id:05x}"
|
||||
s["text_original"] = s.get("text", "")
|
||||
@@ -616,30 +720,110 @@ async def dub_transcribe_stream(
|
||||
return
|
||||
|
||||
def _diarize():
|
||||
"""Returns (segments, warning_payload_or_None).
|
||||
"""Returns (segments, warning_payload_or_None, labels_source).
|
||||
|
||||
`labels_source` records where the speaker labels came from —
|
||||
`"pyannote"` | `"turns"` | `"heuristic"` — so downstream
|
||||
auto-clone extraction can refuse to cut reference audio from
|
||||
gap-based estimates (a mixed-speaker reference is how "made up"
|
||||
clone voices happen).
|
||||
|
||||
`warning_payload` is a structured dict
|
||||
`{detail, error_class, docs_url}` whenever we silently fell back
|
||||
to the silence-gap heuristic (no HF_TOKEN, model unavailable,
|
||||
license not accepted, or pyannote raised). The heuristic only
|
||||
detects speaker turns from >1.2s silences, so a rapid-fire
|
||||
man↔woman exchange will read as one speaker. Issue #78 — we
|
||||
attach an `error_class` so the front-end's errorDocsMap can
|
||||
render a "See docs" deeplink instead of a dead-end toast.
|
||||
license not accepted, or pyannote raised) — or whenever the
|
||||
user's `num_speakers` hint could not be honored exactly. The
|
||||
heuristic only detects speaker turns from >1.2s silences, so a
|
||||
rapid-fire man↔woman exchange will read as one speaker. Issue
|
||||
#78 — we attach an `error_class` so the front-end's errorDocsMap
|
||||
can render a "See docs" deeplink instead of a dead-end toast.
|
||||
"""
|
||||
# The active ASR backend already diarized inline (FunASR cam++):
|
||||
# use its speaker turns directly and skip pyannote entirely (#182).
|
||||
if asr_speaker_turns:
|
||||
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
|
||||
return assign_speakers_from_turns(all_segments, asr_speaker_turns), None
|
||||
|
||||
from services.model_manager import (
|
||||
DIARIZATION_ERR_LICENSE,
|
||||
DIARIZATION_ERR_NO_TOKEN,
|
||||
)
|
||||
from core import error_docs_map
|
||||
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
def _hint_suffix() -> str:
|
||||
"""Honest caveat appended to heuristic-fallback warnings when a
|
||||
multi-speaker hint is set: the count is now honored, but the
|
||||
heuristic can't attribute voices. (A hint of 1 IS fully
|
||||
honored — one label — so it needs no caveat.)"""
|
||||
if not num_speakers or num_speakers < 2:
|
||||
return ""
|
||||
return (
|
||||
f" Your speaker-count setting ({num_speakers}) is only "
|
||||
f"approximately honored: the heuristic cycles "
|
||||
f"{num_speakers} speaker labels on silence gaps instead "
|
||||
f"of recognizing voices, so lines may be attributed to "
|
||||
f"the wrong speaker."
|
||||
)
|
||||
|
||||
def _use_turns(crash: Exception | None = None, err_sentinel=None):
|
||||
"""Label from the ASR backend's inline speaker turns; warn when
|
||||
that means the user's explicit count can't be enforced."""
|
||||
logger.info(
|
||||
"Using inline ASR diarization (%d turns)%s.",
|
||||
len(asr_speaker_turns),
|
||||
"" if crash else "; skipping pyannote",
|
||||
)
|
||||
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
resplit = resplit_segments_by_turns(assigned, all_words, asr_speaker_turns)
|
||||
if not num_speakers:
|
||||
return resplit, None, "turns"
|
||||
error_class = (
|
||||
"HF_AUTH_FAILED"
|
||||
if err_sentinel == DIARIZATION_ERR_NO_TOKEN
|
||||
else "PYANNOTE_LICENSE_REQUIRED"
|
||||
)
|
||||
if crash:
|
||||
detail = (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(crash).__name__}); falling back to the ASR "
|
||||
f"engine's built-in speaker turns. Speaker-count hint "
|
||||
f"ignored: the detected count may differ from the "
|
||||
f"{num_speakers} you set."
|
||||
)
|
||||
else:
|
||||
detail = (
|
||||
f"Speaker-count hint ignored: pyannote diarization is "
|
||||
f"unavailable, so the ASR engine's built-in speaker "
|
||||
f"turns were used and the detected count may differ "
|
||||
f"from the {num_speakers} you set. Set up diarization "
|
||||
f"(Settings → Models → pyannote) to enforce an exact "
|
||||
f"speaker count."
|
||||
)
|
||||
return resplit, {
|
||||
"detail": detail,
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
"speaker_hint": {"requested": num_speakers, "status": "ignored"},
|
||||
}, "turns"
|
||||
|
||||
# The active ASR backend already diarized inline (FunASR cam++):
|
||||
# its turns are the fast path and skip pyannote entirely (#182) —
|
||||
# but ONLY when the user didn't set an explicit speaker count.
|
||||
# Inline turns are labeled per-30s-chunk and can't be forced to N
|
||||
# speakers, so a set num_speakers prefers pyannote — the one
|
||||
# engine that honors an exact count. When pyannote can't load,
|
||||
# the turns are still the best labels available; use them and say
|
||||
# so instead of silently eating the hint.
|
||||
diar_pipe = None
|
||||
err_sentinel = None
|
||||
if asr_speaker_turns:
|
||||
if num_speakers:
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
if not diar_pipe:
|
||||
return _use_turns(err_sentinel=err_sentinel)
|
||||
logger.info(
|
||||
"num_speakers=%d set: preferring pyannote over %d inline "
|
||||
"ASR turns (only pyannote honors an exact count).",
|
||||
num_speakers, len(asr_speaker_turns),
|
||||
)
|
||||
else:
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
if not diar_pipe:
|
||||
# Phase 1 AUTH-01: ask the resolver (App → Env → HF-CLI),
|
||||
# not just the env var. This is the #35 fix — users who
|
||||
@@ -690,13 +874,20 @@ async def dub_transcribe_stream(
|
||||
f"heuristic; rapid speaker turns may be merged."
|
||||
)
|
||||
error_class = "PYANNOTE_LICENSE_REQUIRED"
|
||||
warning = {
|
||||
"detail": detail + _hint_suffix(),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
}
|
||||
if num_speakers:
|
||||
warning["speaker_hint"] = {
|
||||
"requested": num_speakers,
|
||||
"status": "approximate" if num_speakers > 1 else "honored",
|
||||
}
|
||||
return (
|
||||
assign_speakers_heuristic(all_segments),
|
||||
{
|
||||
"detail": detail,
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
},
|
||||
assign_speakers_heuristic(all_segments, num_speakers),
|
||||
warning,
|
||||
"heuristic",
|
||||
)
|
||||
try:
|
||||
# Pass the user's speaker-count hint through to pyannote when
|
||||
@@ -708,9 +899,17 @@ async def dub_transcribe_stream(
|
||||
diar = diar_pipe(asr_audio_target, num_speakers=num_speakers)
|
||||
else:
|
||||
diar = diar_pipe(asr_audio_target)
|
||||
return assign_speakers_from_diarization(all_segments, diar), None
|
||||
assigned = assign_speakers_from_diarization(all_segments, diar)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
return resplit_segments_by_diarization(assigned, all_words, diar), None, "pyannote"
|
||||
except Exception as e:
|
||||
logger.error(f"Diarization failed: {e}")
|
||||
# Inline ASR turns beat the silence-gap heuristic as a crash
|
||||
# fallback (this path is reachable with turns present since a
|
||||
# set num_speakers routes turns-jobs through pyannote).
|
||||
if asr_speaker_turns:
|
||||
return _use_turns(crash=e)
|
||||
# Mid-run failure — classify against the same sentinels so a
|
||||
# post-load 401 (rare but possible after a token rotation)
|
||||
# still gets the right docs deeplink.
|
||||
@@ -721,36 +920,50 @@ async def dub_transcribe_stream(
|
||||
if err_class_post == DIARIZATION_ERR_LICENSE
|
||||
else "PYANNOTE_LICENSE_REQUIRED" # LOAD failures land here too
|
||||
)
|
||||
warning = {
|
||||
"detail": (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(e).__name__}); falling back to a silence-gap "
|
||||
f"heuristic. Rapid speaker turns may be merged."
|
||||
+ _hint_suffix()
|
||||
),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
}
|
||||
if num_speakers:
|
||||
warning["speaker_hint"] = {
|
||||
"requested": num_speakers,
|
||||
"status": "approximate" if num_speakers > 1 else "honored",
|
||||
}
|
||||
return (
|
||||
assign_speakers_heuristic(all_segments),
|
||||
{
|
||||
"detail": (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(e).__name__}); falling back to a silence-gap "
|
||||
f"heuristic. Rapid speaker turns may be merged."
|
||||
),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
},
|
||||
assign_speakers_heuristic(all_segments, num_speakers),
|
||||
warning,
|
||||
"heuristic",
|
||||
)
|
||||
|
||||
fut_diar = loop.run_in_executor(_gpu_pool, _diarize)
|
||||
final_segs = None
|
||||
diar_warning = None
|
||||
labels_source = "heuristic"
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_diar], timeout=5.0)
|
||||
if done:
|
||||
final_segs, diar_warning = done.pop().result()
|
||||
final_segs, diar_warning, labels_source = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
if diar_warning:
|
||||
logger.warning("diarization fallback: %s", diar_warning.get("detail"))
|
||||
yield _sse_event("warning", {
|
||||
payload = {
|
||||
"detail": diar_warning.get("detail"),
|
||||
"source": "diarization",
|
||||
"error_class": diar_warning.get("error_class"),
|
||||
"docs_url": diar_warning.get("docs_url"),
|
||||
})
|
||||
}
|
||||
# Machine-readable trail of what happened to the user's
|
||||
# speaker-count hint (the `detail` text carries the human story).
|
||||
if diar_warning.get("speaker_hint"):
|
||||
payload["speaker_hint"] = diar_warning["speaker_hint"]
|
||||
yield _sse_event("warning", payload)
|
||||
|
||||
job["segments"] = final_segs
|
||||
|
||||
@@ -762,17 +975,55 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
from services.speaker_clone import extract_speaker_clones, auto_profile_id
|
||||
vocals_for_clone = job.get("vocals_path") or asr_audio_target
|
||||
fut_clones = loop.run_in_executor(
|
||||
_cpu_pool, extract_speaker_clones,
|
||||
vocals_for_clone, final_segs, os.path.dirname(vocals_for_clone),
|
||||
)
|
||||
clones = None
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
|
||||
if done:
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
clones = {}
|
||||
if labels_source == "heuristic":
|
||||
# Clone-purity guard: heuristic labels are silence-gap
|
||||
# estimates, not voice identity — a per-speaker reference cut
|
||||
# from them routinely concatenates two people's audio and the
|
||||
# clone sounds "made up". Skip auto-clones and say so instead
|
||||
# of shipping bad ones. (extract_speaker_clones enforces the
|
||||
# same guard internally; this branch exists to surface the
|
||||
# warning to the user.)
|
||||
logger.info(
|
||||
"auto speaker clones skipped (labels_source=heuristic, job=%s)",
|
||||
job_id,
|
||||
)
|
||||
yield _sse_event("warning", {
|
||||
"detail": CLONE_SKIP_HEURISTIC_MSG,
|
||||
"source": "speaker_clone",
|
||||
})
|
||||
else:
|
||||
fut_clones = loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_speaker_clones(
|
||||
vocals_for_clone, final_segs,
|
||||
os.path.dirname(vocals_for_clone),
|
||||
labels_source=labels_source,
|
||||
),
|
||||
)
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
|
||||
if done:
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
if clones:
|
||||
from services.speaker_clone import refine_ref_texts
|
||||
# Bound the re-transcribe like every other ASR dispatch in
|
||||
# this file (#730): a wedged transcribe would otherwise hold
|
||||
# the GPU-pool worker forever and starve later work into a
|
||||
# "can't reach backend". On timeout the guard resets the pool
|
||||
# and raises — keep the original (unrefined) clones, matching
|
||||
# refine_ref_text's own "failure is a strict no-op" fallback.
|
||||
try:
|
||||
clones = await run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
lambda: refine_ref_texts(clones, _asr_backend),
|
||||
what="Dub clone ref-text refine",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
logger.warning(
|
||||
"clone ref-text refine timed out; keeping original ref_text: %s", e
|
||||
)
|
||||
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
|
||||
# own reference from the vocals so the dub of each line matches the
|
||||
# prosody of its source line. Short lines fall back to the
|
||||
@@ -792,6 +1043,19 @@ async def dub_transcribe_stream(
|
||||
),
|
||||
)
|
||||
if seg_clones:
|
||||
from services.speaker_clone import refine_ref_texts
|
||||
# Same guard as the per-speaker refine above (#730):
|
||||
# keep the original seg_clones on a wedge/timeout.
|
||||
try:
|
||||
seg_clones = await run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
lambda: refine_ref_texts(seg_clones, _asr_backend),
|
||||
what="Dub segment ref-text refine",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
logger.warning(
|
||||
"segment ref-text refine timed out; keeping original ref_text: %s", e
|
||||
)
|
||||
job["segment_clones"] = seg_clones
|
||||
except Exception as e:
|
||||
logger.warning("per-segment clone refs skipped: %s", e)
|
||||
@@ -856,6 +1120,25 @@ async def dub_transcribe_stream(
|
||||
})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
async def gen():
|
||||
# Terminal-event guard (#516): the SSE stream must NEVER close without a
|
||||
# terminal event. Any unanticipated exception in the body (e.g. an ASR
|
||||
# load that escapes the per-chunk handler) previously dropped the
|
||||
# connection, which the frontend can only report as "stream dropped,
|
||||
# likely ASR failed" — hiding the real cause. Emit a structured `error`
|
||||
# (with the actionable hint from build_failure) then `done`, so the user
|
||||
# sees the real failure + a Retry instead of a silent disconnect.
|
||||
try:
|
||||
async for ev in _gen_body():
|
||||
yield ev
|
||||
except Exception as e: # noqa: BLE001 — last-resort stream finalizer
|
||||
logger.exception("transcribe stream crashed (job=%s)", job_id)
|
||||
from core.failure import build_failure
|
||||
f = build_failure(e, stage="transcribe", include_diagnostic=False)
|
||||
detail = f["reason"] + (f" — {f['hint']}" if f.get("hint") else "")
|
||||
yield _sse_event("error", {"detail": detail, "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
return StreamingResponse(
|
||||
gen(),
|
||||
media_type="text/event-stream",
|
||||
@@ -867,18 +1150,28 @@ async def dub_transcribe_stream(
|
||||
|
||||
|
||||
@router.post("/dub/transcribe/{job_id}")
|
||||
async def dub_transcribe(job_id: str):
|
||||
async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
|
||||
"""Legacy synchronous transcribe (kept for the headless CLI).
|
||||
|
||||
`num_speakers` mirrors the SSE endpoint's query param (same 1–20 clamp):
|
||||
an exact speaker count forwarded to pyannote, or cycled by the silence-gap
|
||||
heuristic when pyannote is unavailable. None → auto-detect.
|
||||
"""
|
||||
num_speakers = _clamp_num_speakers(num_speakers)
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
_model = await get_model()
|
||||
|
||||
def _transcribe():
|
||||
|
||||
|
||||
asr_audio_target = job.get("vocals_path")
|
||||
if not asr_audio_target or not os.path.exists(asr_audio_target):
|
||||
asr_audio_target = job.get("audio_path")
|
||||
|
||||
# #963: same source-awareness as the SSE endpoint — vocals_path
|
||||
# falls back to the mixed audio_path when Demucs failed/skipped.
|
||||
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
|
||||
|
||||
import torch
|
||||
|
||||
detected_lang = None
|
||||
@@ -922,10 +1215,13 @@ async def dub_transcribe(job_id: str):
|
||||
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
|
||||
|
||||
# #280: snap segment starts forward to the actual speech onset so the
|
||||
# dub doesn't begin seconds before the original speaker does.
|
||||
# dub doesn't begin seconds before the original speaker does. #963:
|
||||
# only on the separated vocals track — on mixed audio every ambient
|
||||
# sound is a false onset candidate, so snapping is disabled.
|
||||
try:
|
||||
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
|
||||
snap_segment_starts(segments, audio_for_onset, onset_sr)
|
||||
snap_segment_starts(segments, audio_for_onset, onset_sr,
|
||||
separated_vocals=asr_on_vocals)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped: %s", e)
|
||||
|
||||
@@ -933,13 +1229,20 @@ async def dub_transcribe(job_id: str):
|
||||
if diar_pipe:
|
||||
try:
|
||||
diar_target = job.get("vocals_path") or job.get("audio_path")
|
||||
diarization = diar_pipe(diar_target)
|
||||
# Same hint pass-through as the SSE endpoint (#274): omit the
|
||||
# kwarg entirely when unset so we don't depend on it existing
|
||||
# in every pyannote build.
|
||||
if num_speakers:
|
||||
logger.info("Diarizing with num_speakers=%d (user hint)", num_speakers)
|
||||
diarization = diar_pipe(diar_target, num_speakers=num_speakers)
|
||||
else:
|
||||
diarization = diar_pipe(diar_target)
|
||||
segments = assign_speakers_from_diarization(segments, diarization)
|
||||
except Exception as e:
|
||||
logger.error(f"Pyannote diarization failed during inference: {e}. Falling back to heuristic.")
|
||||
segments = assign_speakers_heuristic(segments)
|
||||
segments = assign_speakers_heuristic(segments, num_speakers)
|
||||
else:
|
||||
segments = assign_speakers_heuristic(segments)
|
||||
segments = assign_speakers_heuristic(segments, num_speakers)
|
||||
|
||||
# Previously ran `segment_for_subtitles(segments)` here. Removed 2026-04-21 —
|
||||
# that splitter enforces Netflix's 17 CPS reading-speed ceiling which
|
||||
@@ -959,7 +1262,11 @@ async def dub_transcribe(job_id: str):
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
segments_result = await loop.run_in_executor(_gpu_pool, _transcribe)
|
||||
# Bound the whole-file transcribe (#730): a wedged whisperx/CTranslate2
|
||||
# call would otherwise hold its GPU-pool worker forever and starve
|
||||
# every other request into a "can't reach backend". run_transcribe_guarded
|
||||
# also resets the pool on timeout so capacity is restored.
|
||||
segments_result = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Dub")
|
||||
except asyncio.CancelledError:
|
||||
job["aborted"] = True
|
||||
raise
|
||||
|
||||
@@ -170,8 +170,42 @@ async def dub_list_tracks(job_id: str):
|
||||
return {"tracks": job.get("dubbed_tracks", {})}
|
||||
|
||||
|
||||
def _segments_for_lang(job: dict, lang: "str | None") -> list:
|
||||
"""Job segments with `text` overlaid from ``job["segments_i18n"][lang]``.
|
||||
|
||||
P1.2 — ``job["segments"]`` is single-slot: it holds whichever language was
|
||||
generated LAST, so exporting subtitles for track A after generating track B
|
||||
emitted B's text under A's language label (the "N identical subtitle
|
||||
files" class). ``segments_i18n`` ({lang: {segKey: text}}, written by
|
||||
``dub_generate._sync_job_segments``) preserves each generated track's text;
|
||||
this overlays it non-destructively when present.
|
||||
|
||||
Back-compat: no lang requested, no ``segments_i18n`` on the job (predates
|
||||
the field), no entry for this lang, or no text for a given segment — each
|
||||
falls back to the segment as-is, i.e. exactly today's behaviour.
|
||||
Segment keys are the stable id (str) with the list index (str) as the
|
||||
legacy fallback, mirroring how the map is written.
|
||||
"""
|
||||
segments = job.get("segments", [])
|
||||
if not lang:
|
||||
return segments
|
||||
i18n = job.get("segments_i18n")
|
||||
lang_texts = i18n.get(lang) if isinstance(i18n, dict) else None
|
||||
if not isinstance(lang_texts, dict) or not lang_texts:
|
||||
return segments
|
||||
out = []
|
||||
for i, seg in enumerate(segments):
|
||||
key = str(seg.get("id")) if seg.get("id") is not None else str(i)
|
||||
txt = lang_texts.get(key)
|
||||
if txt is None:
|
||||
txt = lang_texts.get(str(i))
|
||||
out.append(dict(seg, text=txt) if isinstance(txt, str) and txt.strip() else seg)
|
||||
return out
|
||||
|
||||
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
|
||||
fitted_segments: "list[dict] | None" = None) -> str | None:
|
||||
fitted_segments: "list[dict] | None" = None,
|
||||
lang: "str | None" = None) -> str | None:
|
||||
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
|
||||
|
||||
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
|
||||
@@ -181,8 +215,11 @@ def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
|
||||
fitted timeline — when provided, cue times come from there instead of
|
||||
the original ``job["segments"]`` timings, so burned subs track the
|
||||
retimed video / fitted audio rather than the source timeline.
|
||||
|
||||
``lang`` (P1.2): burn the named track's text (see ``_segments_for_lang``)
|
||||
instead of whatever language generated last.
|
||||
"""
|
||||
segments = job.get("segments", [])
|
||||
segments = _segments_for_lang(job, lang)
|
||||
if not segments:
|
||||
return None
|
||||
if fitted_segments:
|
||||
@@ -486,7 +523,9 @@ async def dub_download(
|
||||
# Smart Fit: cue times come from the fitted timeline — that's where the
|
||||
# dubbed audio actually sits, whether or not the video retime succeeds.
|
||||
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
|
||||
# Burn the DEFAULT track's text (P1.2) — it's the audio the viewer hears.
|
||||
_burn_lang = default_track if default_track and default_track != "original" else None
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments, lang=_burn_lang) if burn_subs else None
|
||||
|
||||
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
|
||||
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
|
||||
@@ -1125,20 +1164,40 @@ async def dub_get_audio(job_id: str):
|
||||
raise HTTPException(status_code=404, detail="Audio file not found")
|
||||
return FileResponse(audio, media_type="audio/wav")
|
||||
|
||||
def _seg_wav_candidates(job: dict, lang: "str | None", seg_keys: tuple) -> list:
|
||||
"""Per-segment WAV name candidates, language-keyed first (P1.3).
|
||||
|
||||
Generation writes ``seg_{lang}_{id}.wav`` now; ``lang`` defaults to the
|
||||
job's last-generated track. Legacy un-keyed names (``seg_{id}.wav`` /
|
||||
``seg_{index}.wav``) stay as fallbacks so jobs rendered by previous
|
||||
builds keep serving their audio — these read-only endpoints keep the
|
||||
permissive fallback that matches their historic behaviour (the strict
|
||||
single-track gate lives on the generate splice path, where a wrong-
|
||||
language read would be baked into a track).
|
||||
"""
|
||||
lang = lang or job.get("language_code")
|
||||
keys = []
|
||||
if lang:
|
||||
keys.extend(f"{lang}_{k}" for k in seg_keys)
|
||||
keys.extend(seg_keys)
|
||||
return keys
|
||||
|
||||
|
||||
@router.get("/dub/preview/{job_id}/{segment_index}")
|
||||
async def dub_preview_segment(job_id: str, segment_index: int):
|
||||
async def dub_preview_segment(job_id: str, segment_index: int, lang: str = Query(None)):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
# Resolve the stable-id-named WAV via the render manifest; fall back to the
|
||||
# legacy index name for jobs rendered before id-based naming (#185). Each
|
||||
# candidate is realpath-normalised and containment-checked BEFORE any
|
||||
# filesystem access, so the guard dominates every path sink.
|
||||
# Resolve the stable-id-named WAV via the render manifest — language-keyed
|
||||
# name first (P1.3), then the legacy id/index names for jobs rendered
|
||||
# before per-language (and before id-based, #185) naming. Each candidate
|
||||
# is realpath-normalised and containment-checked BEFORE any filesystem
|
||||
# access, so the guard dominates every path sink.
|
||||
order = job.get("seg_order") or []
|
||||
seg_id = order[segment_index] if 0 <= segment_index < len(order) else segment_index
|
||||
base = os.path.realpath(DUB_DIR)
|
||||
seg_path = None
|
||||
for _sid in (seg_id, segment_index):
|
||||
for _sid in _seg_wav_candidates(job, lang, (seg_id, segment_index)):
|
||||
cand = os.path.realpath(dub_seg_path(job_id, _sid))
|
||||
if cand.startswith(base + os.sep) and os.path.exists(cand):
|
||||
seg_path = cand
|
||||
@@ -1187,8 +1246,14 @@ async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: flo
|
||||
|
||||
try:
|
||||
from services.model_manager import _get_gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
recognized, engine_id = await loop.run_in_executor(_get_gpu_pool(), _recognize)
|
||||
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
|
||||
recognized, engine_id = await run_transcribe_guarded(
|
||||
_get_gpu_pool(), _recognize, what="QC",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
# Backend is alive; ASR just couldn't finish in time. 504, not 500/connection.
|
||||
logger.warning("dub QC ASR pass timed out for %s: %s", job_id, e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.exception("dub QC ASR pass failed for %s", job_id)
|
||||
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
|
||||
@@ -1336,13 +1401,16 @@ def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
|
||||
async def dub_export_srt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
lang: str = Query(None, description="Track language code. Emits that track's text (segments_i18n) when the job carries it; when that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
segments = job.get("segments", [])
|
||||
# P1.2 — text follows the REQUESTED track, not whichever language was
|
||||
# generated last (job["segments"] is single-slot). Legacy jobs without
|
||||
# segments_i18n fall back to today's behaviour.
|
||||
segments = _segments_for_lang(job, lang)
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
@@ -1385,13 +1453,14 @@ def _format_vtt_time(seconds):
|
||||
async def dub_export_vtt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
lang: str = Query(None, description="Track language code. Emits that track's text (segments_i18n) when the job carries it; when that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
segments = job.get("segments", [])
|
||||
# Same per-track text resolution as /dub/srt (see comment there, P1.2).
|
||||
segments = _segments_for_lang(job, lang)
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
@@ -1421,7 +1490,7 @@ async def dub_export_vtt(
|
||||
|
||||
|
||||
@router.get("/dub/export-segments/{job_id}")
|
||||
async def dub_export_segments_zip(job_id: str):
|
||||
async def dub_export_segments_zip(job_id: str, lang: str = Query(None)):
|
||||
import zipfile
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
@@ -1439,7 +1508,7 @@ async def dub_export_segments_zip(job_id: str):
|
||||
seg_id = order[i] if i < len(order) else i
|
||||
# realpath + containment guard before any filesystem access.
|
||||
seg_path = None
|
||||
for _sid in (seg_id, i):
|
||||
for _sid in _seg_wav_candidates(job, lang, (seg_id, i)):
|
||||
cand = os.path.realpath(dub_seg_path(job_id, _sid))
|
||||
if cand.startswith(base + os.sep) and os.path.exists(cand):
|
||||
seg_path = cand
|
||||
|
||||
+573
-237
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,7 @@ from fastapi.responses import JSONResponse
|
||||
|
||||
from schemas.requests import TranslateRequest
|
||||
from services.model_manager import _cpu_pool, _gpu_pool
|
||||
from services.translator import cinematic_available, cinematic_refine_many
|
||||
from services.translator import cinematic_available, cinematic_refine_many, _cinematic_budget
|
||||
from api.routers.dub_core import _get_job
|
||||
|
||||
router = APIRouter()
|
||||
@@ -302,15 +302,69 @@ async def dub_translate(req: TranslateRequest):
|
||||
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
|
||||
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
|
||||
_unload_nllb()
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
# Cinematic/Autofit refine + rate-ratio badges must run for NLLB too
|
||||
# (previously this returned before _maybe_cinematic, so a Cinematic
|
||||
# pick on NLLB silently produced plain Fast output). Unloading NLLB
|
||||
# first is fine — the refine LLM is a separate network provider.
|
||||
return await _maybe_cinematic(translated, req, src_lang, loop)
|
||||
|
||||
# OpenAI / Ollama Local LLM Translation
|
||||
# LLM translation — resolves through the LLM Skills registry: per-skill
|
||||
# "Dub translation" override → global active provider (Settings → LLM
|
||||
# Providers). The keys users configure + test in the app now actually
|
||||
# power this engine; the raw TRANSLATE_* env vars stay working as a
|
||||
# power-user override so pre-skills setups see zero behavior change.
|
||||
if provider == "openai":
|
||||
base_url = os.environ.get("TRANSLATE_BASE_URL")
|
||||
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-3.5-turbo")
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url=base_url, api_key=api_key or "local")
|
||||
from services import llm_skills
|
||||
|
||||
llm_timeout = llm_skills._default_timeout()
|
||||
handle = None
|
||||
try:
|
||||
handle = llm_skills.resolve_skill_client("dub_translation")
|
||||
except Exception: # noqa: BLE001 — resolution must never 500 a translate
|
||||
logger.exception("dub_translation skill resolution failed; trying env fallback")
|
||||
if handle is not None:
|
||||
client = handle.client
|
||||
model_name = handle.model
|
||||
llm_timeout = handle.timeout
|
||||
# The provider-store key never touches env; resolve it so the
|
||||
# error scrubber below can redact it if a provider echoes it.
|
||||
try:
|
||||
from services import llm_providers
|
||||
api_key = llm_providers.resolve_api_key(
|
||||
llm_skills.effective_provider("dub_translation")) or api_key
|
||||
except Exception: # noqa: BLE001 — scrub-key resolution is best-effort
|
||||
pass
|
||||
elif os.environ.get("TRANSLATE_BASE_URL") or api_key:
|
||||
# Legacy env-only setup (no provider configured in-app).
|
||||
from openai import OpenAI
|
||||
# max_retries=0: a 429 + long Retry-After must not let one segment's
|
||||
# SDK call sleep+retry and blow the overall translate wall time.
|
||||
client = OpenAI(base_url=os.environ.get("TRANSLATE_BASE_URL"),
|
||||
api_key=api_key or "local", max_retries=0)
|
||||
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
|
||||
else:
|
||||
# Nothing configured anywhere — name the exact next step instead
|
||||
# of letting an empty key surface as a raw 401 per segment.
|
||||
try:
|
||||
reason = llm_skills.resolve_skill("dub_translation").reason
|
||||
except Exception: # noqa: BLE001
|
||||
reason = None
|
||||
if reason == "disabled":
|
||||
friendly = (
|
||||
"The LLM translation engine is turned off — enable the "
|
||||
"'Dub translation' skill in Settings → LLM Skills, or "
|
||||
"pick another engine in the Engine dropdown."
|
||||
)
|
||||
else:
|
||||
friendly = (
|
||||
"The LLM translation engine has no provider configured. "
|
||||
"Add and test one in Settings → LLM Providers (it powers "
|
||||
"this engine; route it per-skill in Settings → LLM "
|
||||
"Skills), or set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
|
||||
"+ TRANSLATE_MODEL. Or pick another engine in the "
|
||||
"Engine dropdown."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
|
||||
def _build_prompt(src_code: str, tgt_code: str) -> str:
|
||||
"""Build a system prompt that resists hallucinations on small
|
||||
@@ -372,6 +426,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
res = client.chat.completions.create(
|
||||
model=model_name,
|
||||
temperature=0.2, # less drift than default 1.0
|
||||
timeout=llm_timeout, # bound per call (OMNIVOICE_LLM_TIMEOUT, 45s default)
|
||||
messages=[
|
||||
{"role": "system", "content": sys_for_attempt},
|
||||
{"role": "user", "content": seg.text},
|
||||
@@ -399,25 +454,38 @@ async def dub_translate(req: TranslateRequest):
|
||||
seg.id, attempt + 1, e,
|
||||
)
|
||||
# Both attempts failed — keep source text + flag error so the
|
||||
# frontend can surface "fallback to literal" warning.
|
||||
return {"id": seg.id, "text": seg.text, "error": last_err or "llm-failed"}
|
||||
# frontend can surface "fallback to literal" warning. Scrub the
|
||||
# provider error: some OpenAI-compatible providers echo the key
|
||||
# or a user_id in the body, which must not reach the UI verbatim.
|
||||
from core.scrub import scrub_provider_error
|
||||
return {"id": seg.id, "text": seg.text,
|
||||
"error": scrub_provider_error(last_err, api_key) or "llm-failed"}
|
||||
|
||||
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
|
||||
translated = await asyncio.gather(*tasks)
|
||||
translated.sort(key=lambda x: str(x["id"]))
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=True)}
|
||||
# provider="openai" is already an LLM translation — _maybe_cinematic
|
||||
# skips the reflect/adapt re-refine (already_llm) but still stamps
|
||||
# rate-ratio badges and runs the bounded Autofit fit pass. Before
|
||||
# this it returned here, so Cinematic/Autofit on the LLM engine did
|
||||
# nothing.
|
||||
return await _maybe_cinematic(translated, req, src_lang, loop, already_llm=True)
|
||||
|
||||
# Offline Argos Translate
|
||||
if provider == "argos" or provider == "libretranslate":
|
||||
try:
|
||||
import argostranslate # noqa: F401
|
||||
except ImportError:
|
||||
# Single-source the install command from the engine registry so
|
||||
# this 400 and the proactive Install button in the Engine
|
||||
# selector can never drift (see translation_engines.install_command).
|
||||
from services.translation_engines import install_command
|
||||
cmd = install_command("argos") or "uv pip install argostranslate"
|
||||
friendly = (
|
||||
f"The '{provider}' translation engine needs the optional "
|
||||
f"`argostranslate` Python package, which isn't installed in "
|
||||
f"this backend. Install it with `uv pip install argostranslate` "
|
||||
f"(or `pip install argostranslate`) and restart the server, or "
|
||||
f"this backend. Install it with `{cmd}` "
|
||||
f"and restart the server, or "
|
||||
f"switch the Engine dropdown to another provider."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
@@ -460,8 +528,11 @@ async def dub_translate(req: TranslateRequest):
|
||||
return results
|
||||
|
||||
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
# Argos is the DEFAULT engine — routing it through _maybe_cinematic is
|
||||
# the headline fix: a user who picks Cinematic/Autofit on Argos now
|
||||
# gets the LLM refine + fit pass (and rate-ratio badges in Fast mode)
|
||||
# instead of silent plain-Fast output.
|
||||
return await _maybe_cinematic(translated, req, src_lang, loop)
|
||||
|
||||
# Legacy / API Deep_Translator logic.
|
||||
# Preflight the optional `deep_translator` dep once so we fail with a
|
||||
@@ -470,11 +541,16 @@ async def dub_translate(req: TranslateRequest):
|
||||
try:
|
||||
import deep_translator # noqa: F401
|
||||
except ImportError:
|
||||
# Same single-source install command as the Engine selector's Install
|
||||
# button (translation_engines.install_command) — google/deepl/
|
||||
# microsoft/mymemory all share the deep_translator package.
|
||||
from services.translation_engines import install_command
|
||||
cmd = install_command(provider) or "uv pip install deep_translator"
|
||||
friendly = (
|
||||
f"The '{provider}' translation engine needs the optional "
|
||||
f"`deep_translator` Python package, which isn't installed in "
|
||||
f"this backend. Install it with `uv pip install deep_translator` "
|
||||
f"(or `pip install deep_translator`) and restart the server, or "
|
||||
f"this backend. Install it with `{cmd}` "
|
||||
f"and restart the server, or "
|
||||
f"switch the Engine dropdown to Argos (local, bundled), NLLB "
|
||||
f"(local, heavier), or OpenAI (LLM)."
|
||||
)
|
||||
@@ -530,7 +606,11 @@ async def dub_translate(req: TranslateRequest):
|
||||
)
|
||||
time.sleep(0.25 * (attempt + 1))
|
||||
logger.error("translate %s -> %s gave up (provider=%s): %s", src_arg, seg_lc, provider, last_err)
|
||||
return {"id": seg.id, "text": seg.text, "error": last_err or "unknown"}
|
||||
# Scrub before it reaches the UI — DeepL/Microsoft errors can echo
|
||||
# the API key (same class as the OpenAI user_id leak).
|
||||
from core.scrub import scrub_provider_error
|
||||
return {"id": seg.id, "text": seg.text,
|
||||
"error": scrub_provider_error(last_err, _deepl_key or _msft_key or api_key) or "unknown"}
|
||||
|
||||
tasks = [loop.run_in_executor(_cpu_pool, _translate_single, seg) for seg in req.segments]
|
||||
translated = await asyncio.gather(*tasks)
|
||||
@@ -544,24 +624,19 @@ async def dub_translate(req: TranslateRequest):
|
||||
return JSONResponse(status_code=500, content={"error": str(e)})
|
||||
|
||||
|
||||
async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
"""If quality=cinematic and a usable LLM is configured, run REFLECT+ADAPT.
|
||||
Otherwise return Fast-mode shape unchanged.
|
||||
def _stamp_predicted_rate_ratio(translated, req) -> None:
|
||||
"""Stamp a predicted ``rate_ratio`` on every row that has a known slot.
|
||||
|
||||
No LLM needed — just the per-language CPS table from ``services/speech_rate``.
|
||||
The UI's ``seg-rate-badge`` reads it (Fast mode included) to show which
|
||||
segments will compress hard at generation time, so users can edit text or
|
||||
pick a heavier quality. Mutates ``translated`` in place; never raises.
|
||||
"""
|
||||
quality = (getattr(req, "quality", None) or "fast").lower()
|
||||
# Stamp the predicted rate_ratio on every translated row that has a
|
||||
# known slot. Works for Fast mode too — no LLM needed; just the CPS
|
||||
# table from services/speech_rate. The UI's `seg-rate-badge` reads
|
||||
# this value and shows users which segments will compress hard at
|
||||
# generation time, so they can edit text or pick Cinematic quality.
|
||||
try:
|
||||
from services.speech_rate import rate_ratio as _predict_rate_ratio
|
||||
slots = {str(s.id): getattr(s, "slot_seconds", None) for s in req.segments}
|
||||
for row in translated:
|
||||
seg_ref = next(
|
||||
(s for s in req.segments if str(s.id) == str(row["id"])),
|
||||
None,
|
||||
)
|
||||
slot = getattr(seg_ref, "slot_seconds", None) if seg_ref else None
|
||||
slot = slots.get(str(row["id"]))
|
||||
text = (row.get("text") or "").strip()
|
||||
if slot and text and not row.get("error"):
|
||||
row["rate_ratio"] = round(
|
||||
@@ -570,19 +645,119 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
except Exception as e:
|
||||
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
|
||||
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
"quality_used": "fast", **_dialect_flags(req, applied=False)}
|
||||
|
||||
if quality != "cinematic":
|
||||
async def _apply_fit_pass(rows, req, slots_by_id, source_by_id, quality, loop, deadline) -> None:
|
||||
"""Run the Autofit slot-fit pass over ``rows`` concurrently, in place.
|
||||
|
||||
Bounded by ``deadline`` (shared with the cinematic refine) so a slow /
|
||||
rate-limited LLM can't spin the fit pass per-segment unbounded — the old
|
||||
behavior, which ran one blocking ``adjust_for_slot`` per segment in the
|
||||
merge loop, outside any budget. Segments still running at the deadline keep
|
||||
their current text and get ``rate_error='fit-budget'``. Only rows with a
|
||||
slot + text + no prior error participate.
|
||||
"""
|
||||
strict = (quality == "autofit")
|
||||
items = []
|
||||
for row in rows:
|
||||
seg_id = str(row["id"])
|
||||
slot = slots_by_id.get(seg_id)
|
||||
text = row.get("text") or ""
|
||||
if slot and text and not row.get("error"):
|
||||
items.append((seg_id, text, float(slot), req.target_lang,
|
||||
source_by_id.get(seg_id), strict))
|
||||
if not items:
|
||||
return
|
||||
try:
|
||||
from services.speech_rate import adjust_for_slot_many
|
||||
fits = await adjust_for_slot_many(
|
||||
items, executor=_cpu_pool, deadline=deadline, loop=loop,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("rate-fit pass skipped: %s", e)
|
||||
return
|
||||
for row in rows:
|
||||
f = fits.get(str(row["id"]))
|
||||
if not f:
|
||||
continue
|
||||
if f.get("text"):
|
||||
row["text"] = f["text"]
|
||||
if f.get("rate_ratio") is not None:
|
||||
row["rate_ratio"] = f["rate_ratio"]
|
||||
if f.get("error"):
|
||||
row["rate_error"] = f["error"]
|
||||
|
||||
|
||||
async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False):
|
||||
"""Post-process a literal translation into Cinematic/Autofit output.
|
||||
|
||||
Runs for EVERY provider now (Argos/NLLB/Google/…/OpenAI). The three
|
||||
LLM-independent branches (nllb/argos) and the openai branch used to return
|
||||
*before* reaching this, so a Cinematic/Autofit pick on them — including the
|
||||
DEFAULT Argos engine — silently produced plain Fast output with a success
|
||||
toast. Fast mode still returns the plain translation (plus rate-ratio badges).
|
||||
|
||||
``already_llm`` (provider="openai"): the translation was itself produced by
|
||||
an LLM, so the REFLECT+ADAPT *re*-refine is skipped, but the bounded Autofit
|
||||
fit pass + rate-ratio stamping still run, and the dialect the translate
|
||||
prompt already baked in is reported as applied.
|
||||
"""
|
||||
quality = (getattr(req, "quality", None) or "fast").lower()
|
||||
|
||||
_stamp_predicted_rate_ratio(translated, req)
|
||||
|
||||
# #280 item 2 — regional dialect hint, guarded against a stale dialect from
|
||||
# another language. For already_llm the initial translate prompt already
|
||||
# applied it, so it's reported applied in the Fast-shape base too.
|
||||
dialect_hint = ""
|
||||
_dialect = getattr(req, "dialect", None)
|
||||
if _dialect and str(_dialect).lower().startswith(str(req.target_lang).lower()[:2]):
|
||||
dialect_hint = dialect_clause(_dialect)
|
||||
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
"quality_used": "fast",
|
||||
**_dialect_flags(req, applied=(already_llm and bool(dialect_hint)))}
|
||||
|
||||
# Fast (and anything unrecognised) returns the plain translation unchanged.
|
||||
if quality not in ("cinematic", "autofit"):
|
||||
return base
|
||||
|
||||
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
|
||||
slots_by_id = {
|
||||
str(s.id): getattr(s, "slot_seconds", None)
|
||||
for s in req.segments
|
||||
if getattr(s, "slot_seconds", None)
|
||||
}
|
||||
|
||||
# One wall-clock deadline shared by the whole LLM phase (refine + fit), so a
|
||||
# slow/rate-limited provider can't run either pass unbounded. <=0 disables.
|
||||
budget = _cinematic_budget()
|
||||
deadline = (loop.time() + budget) if budget and budget > 0 else None
|
||||
|
||||
# provider="openai": already an LLM translation → skip REFLECT+ADAPT, keep
|
||||
# the rate-ratio badges, still run the bounded fit pass.
|
||||
if already_llm:
|
||||
merged = []
|
||||
for row in translated:
|
||||
out = {"id": row["id"],
|
||||
"text": row.get("text", "") or "",
|
||||
"literal": row.get("text", "") or ""}
|
||||
if row.get("error"):
|
||||
out["error"] = row["error"]
|
||||
if "rate_ratio" in row:
|
||||
out["rate_ratio"] = row["rate_ratio"]
|
||||
merged.append(out)
|
||||
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
|
||||
return {"translated": merged, "target_lang": req.target_lang,
|
||||
"source_lang": src_lang, "quality_used": quality,
|
||||
**_dialect_flags(req, applied=bool(dialect_hint))}
|
||||
|
||||
# Non-LLM provider → the reflect/adapt refine needs a separately-configured
|
||||
# LLM (Settings → LLM Providers). Without one, degrade to Fast with a flag.
|
||||
if not cinematic_available():
|
||||
logger.warning("cinematic requested but no LLM configured — returning Fast result.")
|
||||
logger.warning("%s requested but no LLM configured — returning Fast result.", quality)
|
||||
base["cinematic_skipped"] = "no-llm-configured"
|
||||
return base
|
||||
|
||||
# Build a map from id → original segment (to fetch source text + direction).
|
||||
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
|
||||
directions: dict[str, str] = {
|
||||
str(s.id): s.direction
|
||||
for s in req.segments
|
||||
@@ -590,7 +765,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
}
|
||||
pairs = []
|
||||
passthrough_index = {}
|
||||
for i, row in enumerate(translated):
|
||||
for row in translated:
|
||||
seg_id = str(row["id"])
|
||||
literal = row.get("text", "") or ""
|
||||
if row.get("error") or not literal.strip():
|
||||
@@ -601,12 +776,6 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
if not pairs:
|
||||
return base
|
||||
|
||||
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
|
||||
# prompts. Guard against a stale dialect from another language.
|
||||
dialect_hint = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
|
||||
dialect_hint = dialect_clause(req.dialect)
|
||||
|
||||
refined = await cinematic_refine_many(
|
||||
pairs,
|
||||
source_lang=src_lang,
|
||||
@@ -618,16 +787,6 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
|
||||
# Phase 4.4 — speech-rate fit pass. Segment boundaries aren't in the
|
||||
# translate request (by design — translator is boundary-agnostic), so we
|
||||
# only run it when the caller supplied `slot_seconds` on each segment.
|
||||
# The frontend populates this for Cinematic calls from the edit view.
|
||||
slots_by_id = {
|
||||
str(s.id): getattr(s, "slot_seconds", None)
|
||||
for s in req.segments
|
||||
if getattr(s, "slot_seconds", None)
|
||||
}
|
||||
|
||||
merged = []
|
||||
for row in translated:
|
||||
seg_id = str(row["id"])
|
||||
@@ -646,35 +805,15 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
}
|
||||
if r.get("error"):
|
||||
out["error"] = r["error"]
|
||||
|
||||
# Optional slot-fit pass — only when the caller asked for cinematic
|
||||
# *and* provided a slot. Runs best-effort; no-LLM or mid-loop failure
|
||||
# just leaves the cinematic text untouched.
|
||||
slot = slots_by_id.get(seg_id)
|
||||
if slot and out["text"]:
|
||||
try:
|
||||
from services.speech_rate import adjust_for_slot
|
||||
fit = await asyncio.to_thread(
|
||||
adjust_for_slot,
|
||||
out["text"],
|
||||
slot_seconds=float(slot),
|
||||
target_lang=req.target_lang,
|
||||
source_text=source_by_id.get(seg_id),
|
||||
)
|
||||
if fit.get("text"):
|
||||
out["text"] = fit["text"]
|
||||
out["rate_ratio"] = fit.get("rate_ratio")
|
||||
if fit.get("error"):
|
||||
out["rate_error"] = fit["error"]
|
||||
except Exception as e:
|
||||
logger.warning("rate-fit skipped for %s: %s", seg_id, e)
|
||||
|
||||
merged.append(out)
|
||||
|
||||
# Phase 4.4 speech-rate fit pass — now concurrent + bounded (see helper).
|
||||
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
|
||||
|
||||
return {
|
||||
"translated": merged,
|
||||
"target_lang": req.target_lang,
|
||||
"source_lang": src_lang,
|
||||
"quality_used": "cinematic",
|
||||
"quality_used": quality,
|
||||
**_dialect_flags(req, applied=bool(dialect_hint)),
|
||||
}
|
||||
|
||||
@@ -15,6 +15,9 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
|
||||
`OMNIVOICE_LLM_BACKEND`) still win over the UI choice so power-users can pin
|
||||
a backend without Settings silently undoing it.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
from time import perf_counter
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
@@ -261,9 +264,177 @@ def engine_health(engine_id: str):
|
||||
}
|
||||
|
||||
|
||||
# ── Real-synthesis self-test (in-process TTS engines) ──────────────────────
|
||||
#
|
||||
# ``/health`` above is a liveness/import probe — for an in-process backend it
|
||||
# only calls ``is_available()`` and the UI labels the result "deps OK". This
|
||||
# route goes one step further: for an AVAILABLE, IN-PROCESS TTS engine it runs
|
||||
# a *tiny real synthesis* from a fixed short phrase and reports duration +
|
||||
# sample-rate + sample count, proving the engine actually emits audio rather
|
||||
# than merely importing. The Compat Matrix's "Self-test" button calls it.
|
||||
#
|
||||
# Guardrails (kept identical across macOS/Windows/Linux per the default-feature
|
||||
# rule — the phrase, timeout and gating don't branch on OS):
|
||||
# * TTS family + available + in-process only. Subprocess engines keep their
|
||||
# spawn-and-ping ``health_check`` (a real synth there is a sidecar
|
||||
# cold-start — out of scope for a click-to-test affordance).
|
||||
# * Bounded wall-clock timeout (``OMNIVOICE_SELFTEST_TIMEOUT_S``, default 90s):
|
||||
# a runaway synth returns ``ok=False`` / ``timed_out=True`` instead of
|
||||
# hanging the Settings panel. The orphaned worker is best-effort daemon.
|
||||
# * A process-wide lock serialises self-tests so a click-storm can't stack
|
||||
# concurrent model loads.
|
||||
# * Only ever on user click (POST) — never on Settings load. Loopback-gated.
|
||||
|
||||
# Deliberately short + ASCII so the synth stays CPU-cheap and the phrase never
|
||||
# trips the no-hardcoded-CJK guard.
|
||||
_SELFTEST_PHRASE = "OmniVoice engine self test."
|
||||
_SELFTEST_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _selftest_timeout_s() -> float:
|
||||
try:
|
||||
return max(1.0, float(os.environ.get("OMNIVOICE_SELFTEST_TIMEOUT_S", "90")))
|
||||
except (TypeError, ValueError):
|
||||
return 90.0
|
||||
|
||||
|
||||
def _sample_count(audio) -> int:
|
||||
"""Total sample count of an engine's ``generate()`` return, tolerant of
|
||||
torch.Tensor / numpy.ndarray / list shapes. 0 when it can't be measured."""
|
||||
try:
|
||||
shape = getattr(audio, "shape", None)
|
||||
if shape is not None and len(shape) > 0:
|
||||
return int(shape[-1])
|
||||
return int(len(audio))
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def _run_synth_bounded(backend, timeout_s: float) -> dict | None:
|
||||
"""Run one tiny synthesis in a daemon thread, bounded by ``timeout_s``.
|
||||
|
||||
Returns ``{"audio": .., "duration_ms": ..}`` on success, ``{"error": exc}``
|
||||
on a synth exception, or ``None`` when the timeout elapsed (worker left
|
||||
running best-effort — Python threads can't be force-killed)."""
|
||||
box: dict = {}
|
||||
|
||||
def _worker():
|
||||
t0 = perf_counter()
|
||||
try:
|
||||
audio = backend.generate(_SELFTEST_PHRASE, language="en", num_step=8)
|
||||
box["audio"] = audio
|
||||
except Exception as exc: # noqa: BLE001 — surfaced to the caller as ok=False
|
||||
box["error"] = exc
|
||||
finally:
|
||||
box["duration_ms"] = (perf_counter() - t0) * 1000.0
|
||||
|
||||
th = threading.Thread(target=_worker, name="engine-selftest", daemon=True)
|
||||
th.start()
|
||||
th.join(timeout_s)
|
||||
if th.is_alive():
|
||||
return None
|
||||
return box
|
||||
|
||||
|
||||
class SelfTestResponse(BaseModel):
|
||||
id: str
|
||||
ok: bool
|
||||
message: str
|
||||
duration_ms: float
|
||||
sample_rate: int | None = None
|
||||
num_samples: int | None = None
|
||||
audio_seconds: float | None = None
|
||||
timed_out: bool = False
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/{engine_id}/selftest",
|
||||
response_model=SelfTestResponse,
|
||||
dependencies=[Depends(require_loopback)],
|
||||
)
|
||||
def engine_selftest(engine_id: str):
|
||||
"""Run a bounded, real synthesis on an available in-process TTS engine.
|
||||
|
||||
404 for an unknown TTS id; 400 when the engine is subprocess-isolated or
|
||||
not currently available (a real synth on either is meaningless). Never
|
||||
raises through to a 500 on a synth failure — the exception is captured into
|
||||
``ok=False`` / ``message`` so the panel renders a per-row failure."""
|
||||
if engine_id not in tts_backend._REGISTRY:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"unknown TTS engine id: {engine_id!r}",
|
||||
)
|
||||
cls = tts_backend._REGISTRY[engine_id]
|
||||
if getattr(cls, "_is_subprocess_isolated", False):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"{engine_id} is subprocess-isolated — self-test runs real "
|
||||
"synthesis for in-process engines only. Use Test engine "
|
||||
"(spawn-and-ping) for subprocess engines."
|
||||
),
|
||||
)
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"{engine_id} is not available: {tts_backend._mask_hf_tokens(msg)}. "
|
||||
"Install/enable the engine, then self-test."
|
||||
),
|
||||
)
|
||||
|
||||
timeout_s = _selftest_timeout_s()
|
||||
# Serialise so a click-storm can't stack concurrent model loads.
|
||||
with _SELFTEST_LOCK:
|
||||
backend = _get_engine_instance(cls)
|
||||
res = _run_synth_bounded(backend, timeout_s)
|
||||
|
||||
if res is None:
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=False,
|
||||
message=f"timed out after {timeout_s:.0f}s (model still loading?)",
|
||||
duration_ms=timeout_s * 1000.0,
|
||||
timed_out=True,
|
||||
)
|
||||
if "error" in res:
|
||||
exc = res["error"]
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=False,
|
||||
message=tts_backend._mask_hf_tokens(f"{type(exc).__name__}: {exc}"),
|
||||
duration_ms=res.get("duration_ms", 0.0),
|
||||
)
|
||||
|
||||
n = _sample_count(res.get("audio"))
|
||||
try:
|
||||
sr = int(getattr(backend, "sample_rate", 0) or 0) or None
|
||||
except Exception:
|
||||
sr = None
|
||||
secs = round(n / sr, 3) if (sr and n) else None
|
||||
return SelfTestResponse(
|
||||
id=engine_id,
|
||||
ok=n > 0,
|
||||
message="synthesized" if n > 0 else "engine returned no audio",
|
||||
duration_ms=res["duration_ms"],
|
||||
sample_rate=sr,
|
||||
num_samples=n or None,
|
||||
audio_seconds=secs,
|
||||
)
|
||||
|
||||
|
||||
class SelectEngineRequest(BaseModel):
|
||||
family: str # "tts" | "asr" | "llm"
|
||||
backend_id: str
|
||||
# Only meaningful for family="tts", backend_id="mlx-audio" (#981) — picks
|
||||
# which of mlx-audio's curated models is actually loaded. A curated key
|
||||
# ("kokoro") or a raw HF repo id ("mlx-community/Kokoro-82M-bf16") — the
|
||||
# same tolerance MLXAudioBackend.__init__ already has. Ignored otherwise.
|
||||
model_id: str | None = None
|
||||
|
||||
|
||||
class SelectEngineResponse(BaseModel):
|
||||
@@ -306,6 +477,24 @@ def select_engine(req: SelectEngineRequest):
|
||||
f"Backend {req.backend_id} can't run on this machine: {why}. "
|
||||
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
|
||||
)
|
||||
# #981: mlx-audio multiplexes 7+ curated models behind one backend id —
|
||||
# persist the model pick alongside the backend id so the UI can actually
|
||||
# select which curated model gets loaded (previously it always defaulted
|
||||
# to Kokoro no matter what the user downloaded in Settings → Models).
|
||||
if req.family == "tts" and req.backend_id == "mlx-audio" and req.model_id is not None:
|
||||
known_keys = tts_backend.MLXAudioBackend.CURATED_MODELS
|
||||
# Accept a curated key OR a raw HF repo id ("owner/name") — the same
|
||||
# tolerance MLXAudioBackend.__init__ already has for power users.
|
||||
# Anything else (typo'd key, malformed id) is rejected outright
|
||||
# rather than silently persisted as a "custom repo" that then fails
|
||||
# to resolve at load time.
|
||||
if req.model_id not in known_keys and not re.fullmatch(r"[\w.-]+/[\w.-]+", req.model_id):
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Unknown mlx-audio model: {req.model_id!r}. Expected one of "
|
||||
f"{sorted(known_keys)} or a HF repo id like 'owner/name'.",
|
||||
)
|
||||
prefs.set_("mlx_audio_model_id", req.model_id)
|
||||
prefs.set_(pref_key, req.backend_id)
|
||||
return {
|
||||
"family": req.family,
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import os
|
||||
import io
|
||||
import re
|
||||
import uuid
|
||||
import time
|
||||
import random
|
||||
import asyncio
|
||||
import tempfile
|
||||
import contextlib
|
||||
@@ -11,16 +13,36 @@ from typing import Optional
|
||||
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from core.db import db_conn
|
||||
import sqlite3
|
||||
from core.db import db_conn, ensure_schema
|
||||
from core.config import OUTPUTS_DIR, VOICES_DIR
|
||||
from services.model_manager import get_model, _gpu_pool
|
||||
import functools
|
||||
from services.model_manager import (
|
||||
get_model, _gpu_pool, run_on_gpu_pool_guarded, GpuJobTimeoutError,
|
||||
)
|
||||
from services.audio_io import _safe_torchaudio_save
|
||||
from core import event_bus
|
||||
from omnivoice.utils.voice_design import heal_design_instruct
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.generate")
|
||||
|
||||
|
||||
def _profile_instruct(row):
|
||||
"""Validator-safe instruct for a stored profile row.
|
||||
|
||||
Sanitizes the persisted instruct (dropping the ``"[object Object]"``
|
||||
sentinel / freeform prose that older builds saved) and, for a design row,
|
||||
rebuilds the tags from ``vd_states`` when the stored value is unusable — so
|
||||
a poisoned/legacy profile never 400-s generation (#550 #571 #594 #596).
|
||||
"""
|
||||
try:
|
||||
vd = row["vd_states"]
|
||||
except (KeyError, IndexError):
|
||||
vd = None
|
||||
return heal_design_instruct(row["instruct"], vd)
|
||||
|
||||
|
||||
def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
"""Synthesize ``[(text, pause_ms), ...]`` spans and stitch silence between
|
||||
them (issue #276).
|
||||
@@ -60,14 +82,31 @@ def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
return torch.cat(parts, dim=-1)
|
||||
|
||||
|
||||
def _sanitize_audio(audio_out):
|
||||
"""Replace non-finite samples (NaN / ±inf) with silence so a model glitch
|
||||
can't produce an unreadable WAV (#629). Returns the input unchanged when it's
|
||||
already finite or isn't a tensor. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.is_tensor(audio_out) and not bool(torch.isfinite(audio_out).all()):
|
||||
logger.warning(
|
||||
"Generated audio contained non-finite samples (NaN/inf) — "
|
||||
"sanitizing to silence to keep the WAV decodable (#629)."
|
||||
)
|
||||
return torch.nan_to_num(audio_out, nan=0.0, posinf=0.0, neginf=0.0)
|
||||
except Exception:
|
||||
pass
|
||||
return audio_out
|
||||
|
||||
|
||||
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
|
||||
"""Shared post-DSP for /generate: preset validation → mastering →
|
||||
effect chain → loudness normalization.
|
||||
|
||||
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
|
||||
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
|
||||
opt out of the broadcast Compressor + Reverb chain that's tuned for
|
||||
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
|
||||
opt out of the broadcast highpass + Compressor pre-stage that's tuned
|
||||
for OmniVoice's 24 kHz clone output. Loudness normalization still runs —
|
||||
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
|
||||
"""
|
||||
from services.audio_dsp import (
|
||||
@@ -75,6 +114,14 @@ def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering
|
||||
apply_effects_chain, get_effect_chain,
|
||||
)
|
||||
|
||||
# #629: a numerical glitch in the model (observed on MPS) can leave NaN/±inf
|
||||
# samples, which write an unreadable WAV that then fails decoding with an
|
||||
# opaque "ffmpeg returned error code: 183 / Invalid data" — surfaced to the
|
||||
# user as a misleading "ran out of memory". Replace non-finite samples with
|
||||
# silence here, before any DSP/encode touches the audio, so the output is
|
||||
# always a valid WAV. Covers the raw path too (it returns just below).
|
||||
audio_out = _sanitize_audio(audio_out)
|
||||
|
||||
preset = effect_preset or "broadcast"
|
||||
if preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
@@ -96,6 +143,158 @@ def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering
|
||||
return normalize_audio(audio_out, target_dBFS=-2.0)
|
||||
|
||||
|
||||
def _safe_exc_text(e: BaseException) -> str:
|
||||
"""``f"{type(e).__name__}: {e}"`` — the house style used for
|
||||
unrecognized-error formatting throughout the backend (grep
|
||||
``type(e).__name__`` in settings.py / asr_backend.py / model_manager.py
|
||||
/ engines.py) — with a guard against leaking a raw container repr.
|
||||
|
||||
#977: an AssertionError raised deep inside a vendored dependency
|
||||
(mlx-audio's Kokoro pipeline) had ``.args`` shaped like
|
||||
``('du', {'a': 'American English', ...})`` — a tuple containing a dict.
|
||||
``str(e)`` on that renders the WHOLE table straight into the user-facing
|
||||
message. Any engine's ``generate()`` can raise something shaped like
|
||||
this (not just Kokoro), so guard generically: if any element of
|
||||
``e.args`` is a container rather than a plain string, don't interpolate
|
||||
``str(e)`` at all — name the exception type and point at the log
|
||||
instead.
|
||||
"""
|
||||
args = getattr(e, "args", ())
|
||||
if any(isinstance(a, (dict, list, tuple, set, frozenset)) for a in args):
|
||||
return f"{type(e).__name__} — see Settings → Logs → Backend for details"
|
||||
return f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def _exception_chain(e):
|
||||
"""Yield ``e`` plus every ``__cause__``/``__context__`` beneath it
|
||||
(cycle-safe). Engines and hub libraries routinely wrap the original
|
||||
transport/allocator error, so classification must look at the whole
|
||||
chain, not just the outermost message."""
|
||||
seen = set()
|
||||
stack = [e]
|
||||
while stack:
|
||||
exc = stack.pop()
|
||||
if exc is None or id(exc) in seen:
|
||||
continue
|
||||
seen.add(id(exc))
|
||||
yield exc
|
||||
stack.append(exc.__cause__)
|
||||
stack.append(exc.__context__)
|
||||
|
||||
|
||||
# #880: transport-level exception type names from httpx (huggingface_hub ≥1.x
|
||||
# downloads over it) and requests/urllib3 (older engine deps). Any of these
|
||||
# anywhere in the exception chain means the network — not memory — killed the
|
||||
# generation.
|
||||
_NETWORK_EXC_NAMES = frozenset({
|
||||
# httpx
|
||||
"ConnectError", "ConnectTimeout", "ReadTimeout", "ReadError",
|
||||
"WriteError", "WriteTimeout", "PoolTimeout", "NetworkError",
|
||||
"TransportError", "RemoteProtocolError", "ProxyError", "CloseError",
|
||||
# requests / urllib3
|
||||
"ConnectionError", "ChunkedEncodingError", "MaxRetryError",
|
||||
"NewConnectionError", "ProtocolError",
|
||||
# stdlib socket-level drops mid-download
|
||||
"ConnectionResetError", "ConnectionAbortedError", "ConnectionRefusedError",
|
||||
# huggingface_hub: failed first-use download with nothing in the disk cache
|
||||
"LocalEntryNotFoundError",
|
||||
})
|
||||
|
||||
# Same class, but the transport error was stringified into a wrapper message
|
||||
# (so the type name is gone). All lowercase; matched against .lower().
|
||||
_NETWORK_MSG_SIGNATURES = (
|
||||
"client has been closed", # httpx closed-client lifecycle error (#880)
|
||||
"cannot send a request", # httpx: same error, message head
|
||||
"connection error", # requests / huggingface_hub wording
|
||||
"connection reset", # ECONNRESET mid-download
|
||||
"read timed out", # requests/urllib3 timeout wording
|
||||
"max retries exceeded", # urllib3 retry exhaustion
|
||||
"temporary failure in name resolution", # DNS down (glibc)
|
||||
"name or service not known", # DNS down (glibc)
|
||||
"getaddrinfo failed", # DNS down (Windows)
|
||||
)
|
||||
|
||||
|
||||
def _is_network_failure(e) -> bool:
|
||||
"""True iff the failure (anywhere in its chain) is an HTTP-client
|
||||
lifecycle / network-transport error — e.g. a first-use model download
|
||||
from the HF Hub dying mid-generation (#880)."""
|
||||
for exc in _exception_chain(e):
|
||||
if type(exc).__name__ in _NETWORK_EXC_NAMES:
|
||||
return True
|
||||
low = str(exc).lower()
|
||||
if any(sig in low for sig in _NETWORK_MSG_SIGNATURES):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# Signatures of an *actual* out-of-memory condition. All lowercase.
|
||||
_OOM_MSG_SIGNATURES = (
|
||||
"out of memory", # CUDA / MPS / generic torch wording
|
||||
"not enough memory", # torch CPU DefaultCPUAllocator
|
||||
"cannot allocate memory", # OS-level ENOMEM
|
||||
"std::bad_alloc", # C++ allocator failure
|
||||
"cublas_status_alloc_failed", # cuBLAS workspace allocation
|
||||
"cuda_error_out_of_memory", # raw CUDA driver error name
|
||||
"paging file is too small", # Windows [WinError 1455] mapping DLLs
|
||||
)
|
||||
|
||||
|
||||
def _is_oom_failure(e) -> bool:
|
||||
"""True iff the failure (anywhere in its chain) actually looks like an
|
||||
out-of-memory condition — the only case where the Flush hint is honest."""
|
||||
for exc in _exception_chain(e):
|
||||
if isinstance(exc, MemoryError):
|
||||
return True
|
||||
# torch.cuda.OutOfMemoryError subclasses RuntimeError; match by name
|
||||
# so this needs no torch import (and covers other frameworks' twins).
|
||||
if type(exc).__name__ == "OutOfMemoryError":
|
||||
return True
|
||||
low = str(exc).lower()
|
||||
if any(sig in low for sig in _OOM_MSG_SIGNATURES):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# #919: an engine that requires a model path / env var which isn't set (or is
|
||||
# set to a directory missing its model files) fails with a *configuration*
|
||||
# error, not a runtime one. The reporting user selected sherpa-onnx and hit
|
||||
# "OMNIVOICE_SHERPA_MODEL not set. Point it to a sherpa-onnx TTS model
|
||||
# directory …" — a pure setup problem — yet the OOM catch-all told them (on a
|
||||
# 63 GB-RAM box) to press Flush for memory they never ran out of. Classify the
|
||||
# whole CLASS of "engine not configured / required env var not set" errors so
|
||||
# any current or future opt-in engine (sherpa/Confucius4/dots/MOSS …) surfaces
|
||||
# actionable setup guidance instead of the memory hint. All lowercase; matched
|
||||
# over the whole exception chain (engines wrap the original error).
|
||||
_CONFIG_MSG_SIGNATURES = (
|
||||
"not set. point it to", # sherpa: OMNIVOICE_SHERPA_MODEL not set
|
||||
"no model.onnx found in", # sherpa: dir set but the model file is missing
|
||||
"not configured", # generic "engine not configured" wording
|
||||
"venv not found. set", # confucius4/dots/MOSS dedicated-venv opt-ins
|
||||
"unavailable: omnivoice_", # is_available() reason wrapped by _ensure_loaded
|
||||
)
|
||||
|
||||
# An OMNIVOICE_* engine env var named alongside "not set" / "point it to" /
|
||||
# "set omnivoice_…" is the strongest config-missing signal and generalizes to
|
||||
# any engine gated on such a var (issue #919 class).
|
||||
_CONFIG_ENV_RE = re.compile(r"omnivoice_[a-z0-9_]+")
|
||||
|
||||
|
||||
def _is_config_failure(e) -> bool:
|
||||
"""True iff the failure is a *configuration* problem — a required engine
|
||||
model path / env var that isn't set (or points nowhere) — rather than a
|
||||
runtime fault. The remedy is to set the value, never to Flush VRAM."""
|
||||
for exc in _exception_chain(e):
|
||||
low = str(exc).lower()
|
||||
if any(sig in low for sig in _CONFIG_MSG_SIGNATURES):
|
||||
return True
|
||||
if _CONFIG_ENV_RE.search(low) and (
|
||||
"not set" in low or "point it to" in low or "set omnivoice_" in low
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _oom_friendly_reraise(e):
|
||||
"""Best-effort cache flush + the user-facing OOM hint shared by both
|
||||
inference paths."""
|
||||
@@ -127,10 +326,116 @@ def _oom_friendly_reraise(e):
|
||||
f"or run `chmod +x` on the engine binary named in the error. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
# #629: a decode/ffmpeg failure on the rendered audio is NOT out of memory —
|
||||
# it's unreadable audio (usually a transient numerical glitch). Say so rather
|
||||
# than sending the user down the OOM path.
|
||||
if "ffmpeg returned error" in es or "Decoding failed" in es or "Invalid data found" in es:
|
||||
raise RuntimeError(
|
||||
f"The engine produced unreadable audio (a decode step failed) — this is "
|
||||
f"usually a transient glitch. Use the Flush button to reload the model, "
|
||||
f"then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
# #664: a bad voice-design instruct (free-form prose, mixed EN/ZH, or
|
||||
# conflicting tags) raises "Unsupported instruct items …" / "Cannot mix …
|
||||
# in a single instruct" / "Conflicting instruct items …" from omnivoice's
|
||||
# _resolve_instruct. That's a USER-INPUT validation error, not an OOM. Match
|
||||
# on the message signature (NOT the type — a lower layer can wrap the original
|
||||
# ValueError, which is why the route's `except ValueError` guard misses it)
|
||||
# and re-raise as a clean ValueError so the route returns a 400 with the
|
||||
# instruct guidance, instead of a 500 telling the user to Flush for memory
|
||||
# they never ran out of. (Complements the client-side guard in #658/#612.)
|
||||
_low = es.lower()
|
||||
if ("unsupported instruct items" in _low
|
||||
or "conflicting instruct items" in _low
|
||||
or "in a single instruct" in _low):
|
||||
raise ValueError(es) from e
|
||||
# #705: a corrupt or wrong-architecture native component (a .dll / .pyd / .exe
|
||||
# — torch, ffmpeg, or a bundled engine binary) fails to load/spawn on Windows
|
||||
# with "[WinError 193] %1 is not a valid Win32 application". That is NOT OOM,
|
||||
# and Flush won't help — reinstalling/repairing the component is the real fix.
|
||||
if "[winerror 193]" in _low or "is not a valid win32 application" in _low:
|
||||
raise RuntimeError(
|
||||
f"A native component (a DLL / .pyd / .exe — e.g. torch, ffmpeg, or an "
|
||||
f"engine binary) is corrupt or built for the wrong architecture "
|
||||
f"([WinError 193]). Reinstall or repair that component — the Flush "
|
||||
f"button won't help here. Underlying error: {e}"
|
||||
) from e
|
||||
# #715: a "[Errno 32] Broken pipe" (BrokenPipeError) surfacing from
|
||||
# generation is NOT out of memory — it means the backend's stdout/stderr
|
||||
# pipe to the desktop shell that launched it closed mid-render (an orphaned
|
||||
# backend whose parent shell exited or relaunched). main.py wraps
|
||||
# sys.stdout/stderr to swallow EPIPE, but a C-level write inside the native
|
||||
# engine/torch can still raise one past that guard. Flush won't help —
|
||||
# relaunching the app re-parents the backend to a live shell.
|
||||
# #756: the GPU's compute capability isn't in this PyTorch build's arch list,
|
||||
# so CUDA can't launch kernels ("no kernel image is available for execution").
|
||||
# NOT OOM. get_best_device() now falls back to CPU up front, but classify the
|
||||
# raw error too in case CUDA was forced (OMNIVOICE_FORCE_CUDA) or a sub-path
|
||||
# still ran on the GPU — point at the real fix, not the Flush button.
|
||||
if "no kernel image is available" in _low:
|
||||
raise RuntimeError(
|
||||
f"Your GPU isn't supported by the installed PyTorch build (CUDA can't "
|
||||
f"launch kernels for its compute capability). Switch the compute device "
|
||||
f"to CPU in Settings, or install a matching PyTorch (e.g. a cu128 build "
|
||||
f"for newer GPUs). The Flush button won't help. Underlying error: {e}"
|
||||
) from e
|
||||
if isinstance(e, BrokenPipeError) or "broken pipe" in _low or "errno 32" in _low:
|
||||
raise RuntimeError(
|
||||
f"The backend lost its output pipe mid-generation — the desktop app "
|
||||
f"that launched it closed or relaunched ([Errno 32] Broken pipe). "
|
||||
f"Restart the app and try again; the Flush button won't help here. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
# #880: an httpx/requests transport failure surfacing from generation —
|
||||
# most commonly a first-use model download from the HF Hub dying with
|
||||
# httpx's "Cannot send a request, as the client has been closed" (the
|
||||
# shared client got closed mid-lifecycle), a connect/read timeout, or a
|
||||
# dropped connection — is NOT out of memory. The model never finished
|
||||
# loading, so Flush is the wrong remedy; retrying is. Matched over the
|
||||
# whole exception chain (type names + stringified signatures) because
|
||||
# engines wrap the original transport error.
|
||||
if _is_network_failure(e):
|
||||
raise RuntimeError(
|
||||
f"A model download or network call failed mid-generation (usually "
|
||||
f"the engine fetching its model files on first use). This is a "
|
||||
f"network problem, not a memory problem — flushing VRAM won't "
|
||||
f"help. Retry the generation; if it keeps failing, check your "
|
||||
f"internet connection and any HF_ENDPOINT/mirror setting. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
# #919: a required engine model path / env var that isn't set is a pure
|
||||
# CONFIGURATION problem, not a runtime one. sherpa-onnx's
|
||||
# "OMNIVOICE_SHERPA_MODEL not set. Point it to …" used to fall through to
|
||||
# the OOM catch-all, telling a user with 63 GB of RAM to press Flush. Point
|
||||
# at the real fix — set the variable — and never mention memory or Flush.
|
||||
# The underlying error already names the exact variable + what to point it
|
||||
# at (and Settings → Engines shows a copy-paste setup line), so keep it
|
||||
# front-and-center. Checked before the OOM branch so a config error can
|
||||
# never be mislabeled as memory.
|
||||
if _is_config_failure(e):
|
||||
raise RuntimeError(
|
||||
f"This TTS engine isn't set up yet — it needs a model path or "
|
||||
f"environment variable that isn't configured, so nothing was "
|
||||
f"generated. Set it as the underlying error describes (it names the "
|
||||
f"exact variable and what to point it at), then restart OmniVoice — "
|
||||
f"or pick a ready engine in Settings → Engines. This is a setup "
|
||||
f"problem, not a memory one. Underlying error: {e}"
|
||||
) from e
|
||||
# #880 (the class bug): the OOM hint used to be the catch-all fallback,
|
||||
# so ANY unrecognized error told the user to press Flush for memory they
|
||||
# never ran out of. Only claim OOM when something in the chain actually
|
||||
# looks like one; everything else surfaces as what it is — unrecognized —
|
||||
# with the real error front and center.
|
||||
if _is_oom_failure(e):
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
f"TTS engine stopped mid-generation with an error OmniVoice doesn't "
|
||||
f"recognize. Retry once; if it keeps failing, please report it with "
|
||||
f"the full trace. Underlying error: {_safe_exc_text(e)}"
|
||||
) from e
|
||||
|
||||
|
||||
def _run_inference(
|
||||
@@ -293,6 +598,32 @@ def _run_backend_inference(
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
def _persist_profile_ref_text(profile_id: str, ref_text: str) -> None:
|
||||
"""Cache an auto-transcribed reference transcript onto its profile row.
|
||||
|
||||
#1032 perf regression: profiles saved without a transcript re-ran a FULL
|
||||
ASR model load + transcribe on every /generate (the #308 auto-transcribe
|
||||
path). Persisting the first transcript makes subsequent generates read it
|
||||
from the row like a user-entered one. The guarded UPDATE only ever fills
|
||||
an empty column — it can never overwrite a transcript the user typed or a
|
||||
lock wrote — and a failure is logged, never raised (best-effort, same
|
||||
contract as the transcribe itself)."""
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
updated = conn.execute(
|
||||
"UPDATE voice_profiles SET ref_text=? "
|
||||
"WHERE id=? AND (ref_text IS NULL OR ref_text='')",
|
||||
(ref_text, profile_id),
|
||||
).rowcount
|
||||
if updated:
|
||||
event_bus.emit("profiles", {"action": "updated", "id": profile_id})
|
||||
except Exception as e: # noqa: BLE001 — cache write must not break generate
|
||||
logger.warning(
|
||||
"could not persist auto-transcribed ref_text onto profile %s: %s",
|
||||
profile_id, e,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/generate")
|
||||
async def generate_speech(
|
||||
text: str = Form(...),
|
||||
@@ -318,7 +649,21 @@ async def generate_speech(
|
||||
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
|
||||
max_chunk_chars: int = Form(800, ge=0),
|
||||
crossfade_ms: int = Form(50, ge=0, le=1000),
|
||||
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
|
||||
# [[…]] overrides to the text before synthesis. Default ON; the global
|
||||
# OMNIVOICE_PRONUNCIATION pref can disable it for power users. Omitting it
|
||||
# with an empty dictionary is byte-identical to legacy behavior.
|
||||
pronounce: bool = Form(True),
|
||||
):
|
||||
# #502: NFC-normalize the input text so decomposed (NFD) diacritics — common
|
||||
# in pasted Vietnamese and other Latin-with-marks text — are composed to the
|
||||
# single codepoints the tokenizer/model expect, instead of base-letter +
|
||||
# combining-mark sequences that render as distorted/garbled speech. NFC is a
|
||||
# no-op for already-composed text; mirrors the duration estimator
|
||||
# (utils/duration.py) so the estimate and the synthesis see the same text.
|
||||
import unicodedata
|
||||
text = unicodedata.normalize("NFC", text)
|
||||
|
||||
# ── Engine resolution (issue #312) ──────────────────────────────────────
|
||||
# The request runs on the engine selected in Settings (POST /engines/select,
|
||||
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
|
||||
@@ -374,11 +719,51 @@ async def generate_speech(
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
|
||||
# ── #1033/#1037: warm the engine under the LOAD budget, not the generate
|
||||
# budget. A cold adapter lazily loads (and possibly downloads multi-GB
|
||||
# weights) inside generate(), so a fresh install's first request burned
|
||||
# its whole OMNIVOICE_GENERATE_TIMEOUT_S window on the download and died
|
||||
# with a misleading "too heavy for the available compute" 503 (#1014
|
||||
# measured it: 0% GPU util for the full 300s). Model loading gets its own,
|
||||
# larger budget (OMNIVOICE_MODEL_LOAD_TIMEOUT, default 1200s) — the same
|
||||
# split get_model() already has for the native engine. Once warm, this is
|
||||
# a no-op per request.
|
||||
if _backend is not None:
|
||||
from services.model_manager import _model_load_timeout
|
||||
try:
|
||||
await run_on_gpu_pool_guarded(
|
||||
_backend.ensure_ready,
|
||||
what=f"TTS engine '{engine_id}' model load",
|
||||
timeout=_model_load_timeout(),
|
||||
)
|
||||
# Builtin TimeoutError base, not GpuJobTimeoutError — reload-proof
|
||||
# class identity (see the twin catch in openai_compat.py).
|
||||
except TimeoutError as exc:
|
||||
logger.warning("engine load exceeded the model-load budget: %s", exc)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
f"TTS engine '{engine_id}' did not finish loading within its "
|
||||
f"model-load budget — on a first run this usually means the "
|
||||
f"weight download is slow or stalled (check Settings → Models "
|
||||
f"for progress), not that generation failed. Retry once the "
|
||||
f"model shows as installed."
|
||||
),
|
||||
) from exc
|
||||
|
||||
ref_audio_path = None
|
||||
cleanup_ref = False
|
||||
used_seed = seed
|
||||
resolved_profile_id = None
|
||||
history_mode = None # profile.kind when a profile drives; else inferred at insert
|
||||
# #1032: profile id to persist an auto-transcribed reference transcript to.
|
||||
# Set only for a plain (unlocked) clone profile whose stored ref_text is
|
||||
# empty — the case where every /generate re-ran a full ASR model load +
|
||||
# transcribe of the same clip. Locked profiles are excluded (their ref
|
||||
# audio is the locked take, and unlocking would leave a mismatched
|
||||
# transcript paired with the original reference); design profiles are
|
||||
# excluded (a re-render replaces the sample, stranding a stale transcript).
|
||||
persist_ref_text_profile_id = None
|
||||
|
||||
if profile_id:
|
||||
with db_conn() as conn:
|
||||
@@ -400,7 +785,7 @@ async def generate_speech(
|
||||
if not ref_text:
|
||||
ref_text = row["ref_text"]
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
instruct = _profile_instruct(row)
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif profile_kind == "design":
|
||||
@@ -410,26 +795,45 @@ async def generate_speech(
|
||||
if ref_audio_path and not ref_text and row["ref_text"]:
|
||||
ref_text = row["ref_text"]
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
instruct = _profile_instruct(row)
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
|
||||
# Legacy design-shaped row (pre-0004 archetype materialization
|
||||
# failure path): instruct-only conditioning.
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
instruct = _profile_instruct(row)
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
else:
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
if not ref_text and row["ref_text"]:
|
||||
ref_text = row["ref_text"]
|
||||
elif ref_audio_path and not ref_text:
|
||||
# Empty stored transcript → the auto-transcribe below will
|
||||
# run; cache its result onto the profile so it runs ONCE,
|
||||
# not on every generate (#1032 perf regression).
|
||||
persist_ref_text_profile_id = profile_id
|
||||
if not instruct and row["instruct"]:
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
if language == "Auto":
|
||||
language = None
|
||||
# #533: a profile's stored language must drive generation when the
|
||||
# request didn't pin one. Without this the German (etc.) archetype
|
||||
# generates with language=None and the model drifts to English —
|
||||
# even though the archetype PREVIEW renders correctly (archetypes.py
|
||||
# passes the language). An EXPLICIT non-Auto request language still
|
||||
# wins; we only fill the gap. `row` is a sqlite3.Row, so guard the
|
||||
# column lookup for pre-language DBs mid-upgrade.
|
||||
if language is None:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
language = prof_lang
|
||||
elif ref_audio is not None:
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
|
||||
@@ -446,32 +850,93 @@ async def generate_speech(
|
||||
# fallback behaves exactly as before.
|
||||
if ref_audio_path and not ref_text:
|
||||
from services.asr_backend import transcribe_reference
|
||||
ref_text = await asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, transcribe_reference, ref_audio_path
|
||||
)
|
||||
# Same #730 hang risk as any whisperx transcribe — bound + reset the pool
|
||||
# so a wedged reference transcribe can't brick the backend. This path is
|
||||
# best-effort (transcribe_reference returns None on failure → the model's
|
||||
# built-in ASR fallback), so a timeout degrades to None rather than
|
||||
# failing the whole generate.
|
||||
try:
|
||||
ref_text = await run_on_gpu_pool_guarded(
|
||||
functools.partial(transcribe_reference, ref_audio_path),
|
||||
what="Reference transcribe",
|
||||
)
|
||||
except GpuJobTimeoutError as e:
|
||||
logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
|
||||
ref_text = None
|
||||
# #1032: cache the transcript onto its clone profile so the ASR model
|
||||
# load + transcribe above happens once per profile, not per generate.
|
||||
# Only fills an empty column — a user-entered transcript always wins.
|
||||
if ref_text and persist_ref_text_profile_id:
|
||||
_persist_profile_ref_text(persist_ref_text_profile_id, ref_text)
|
||||
|
||||
# #526: materialize a concrete seed when none was supplied (and no profile
|
||||
# pinned one) so the take is reproducible and we can hand it back via the
|
||||
# X-Seed header for the "keep this seed" control. An explicit request seed
|
||||
# or a profile's stored seed still wins — used_seed is only filled when it
|
||||
# is still None here, never overwritten.
|
||||
if used_seed is None:
|
||||
used_seed = random.randint(0, 2**31 - 1)
|
||||
|
||||
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
|
||||
# [[…]] one-off overrides to the text, here — AFTER `language` is fully
|
||||
# resolved (a profile may fill it above) so per-language entries match the
|
||||
# real render language, and BEFORE the text reaches either inference path
|
||||
# (native OmniVoice or a pluggable backend) and the chunk splitter. This is
|
||||
# the single point user text → normalized text → model, so the transform
|
||||
# covers generate for every engine. Pure text substitution → identical on
|
||||
# mac/Win/Linux. A disabled pref or empty dictionary is a pass-through, so
|
||||
# plain text stays byte-identical (#G5 backward-compat).
|
||||
from core import prefs as _prefs
|
||||
_pron_env = os.environ.get("OMNIVOICE_PRONUNCIATION")
|
||||
if _pron_env is not None:
|
||||
# Env wins (power-user override); "0"/"false"/"no"/"off" disable it.
|
||||
_pron_enabled = _pron_env.strip().lower() not in ("0", "false", "no", "off", "")
|
||||
else:
|
||||
_pron_enabled = bool(_prefs.get("pronunciation_enabled", True))
|
||||
if pronounce and _pron_enabled:
|
||||
from services.pronunciation import apply_pronunciation, load_entries_from_db
|
||||
try:
|
||||
_pron_rows = load_entries_from_db()
|
||||
except Exception: # noqa: BLE001 — table missing / DB locked → no-op
|
||||
_pron_rows = []
|
||||
text = apply_pronunciation(text, _pron_rows, language)
|
||||
else:
|
||||
# Even with the dictionary off, inline [[…]] overrides are an explicit,
|
||||
# in-text authoring choice → always honored (and never left as literal
|
||||
# double-bracket text the model would mispronounce).
|
||||
from services.pronunciation import apply_inline_overrides
|
||||
text = apply_inline_overrides(text)
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
if _backend is not None:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_backend_inference,
|
||||
_backend, text, language, ref_audio_path, ref_text, instruct,
|
||||
duration, num_step, guidance_scale, speed, denoise,
|
||||
postprocess_output, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
# Bounded + pool-reset on hang so a wedged generate can't starve the
|
||||
# GPU pool and brick the backend ("can't reach backend", #730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
functools.partial(
|
||||
_run_backend_inference,
|
||||
_backend, text, language, ref_audio_path, ref_text, instruct,
|
||||
duration, num_step, guidance_scale, speed, denoise,
|
||||
postprocess_output, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
),
|
||||
what="TTS generate",
|
||||
)
|
||||
# Read after generation: engines with lazy model loading report
|
||||
# their real rate only once weights are up.
|
||||
sample_rate = _backend.sample_rate
|
||||
else:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
functools.partial(
|
||||
_run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
),
|
||||
what="TTS generate",
|
||||
)
|
||||
sample_rate = _model.sampling_rate
|
||||
# Invisible AudioSeal provenance watermark on the final audio. Embedding
|
||||
@@ -493,13 +958,29 @@ async def generate_speech(
|
||||
|
||||
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
# #710: the clip is already generated and saved above. A history-write
|
||||
# failure — e.g. "no such table: generation_history" on a DB that missed
|
||||
# schema init — must NOT 500 the user's generation. Self-heal the schema
|
||||
# once and retry; if it still fails, log and return the audio anyway.
|
||||
def _write_history():
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
try:
|
||||
_write_history()
|
||||
except sqlite3.OperationalError as e:
|
||||
logger.warning("generation history write failed (%s); healing schema + retrying", e)
|
||||
try:
|
||||
ensure_schema()
|
||||
_write_history()
|
||||
except Exception as e2:
|
||||
logger.warning("history write still failed after schema heal; returning audio anyway: %s", e2)
|
||||
except Exception as e:
|
||||
logger.warning("generation history write failed; returning audio anyway: %s", e)
|
||||
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
|
||||
|
||||
buffer = io.BytesIO()
|
||||
@@ -535,6 +1016,12 @@ async def generate_speech(
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except GpuJobTimeoutError as e:
|
||||
# A wedged GPU generate — the pool was already reset to restore capacity
|
||||
# (#730 class). Report the actionable timeout instead of the misleading
|
||||
# "can't reach backend" the frontend shows when the pool starves.
|
||||
logger.error("Generate timed out: %s", e)
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except ValueError as e:
|
||||
logger.error("Validation failed: %s", e)
|
||||
raise HTTPException(status_code=400, detail=str(e)) from e
|
||||
@@ -545,7 +1032,7 @@ async def generate_speech(
|
||||
status_code=500,
|
||||
detail=(
|
||||
f"Couldn't synthesize audio. See Settings → Logs → Backend for the full trace. "
|
||||
f"Underlying error: {e}"
|
||||
f"Underlying error: {_safe_exc_text(e)}"
|
||||
),
|
||||
)
|
||||
finally:
|
||||
|
||||
@@ -189,15 +189,21 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
|
||||
Writes them as `auto=1` rows. Existing terms with the same (source,target)
|
||||
are NOT duplicated. Returns the full current glossary after the pass.
|
||||
"""
|
||||
from services.translator import _llm_client, _llm_model, _llm_timeout # reuse same client
|
||||
# Resolved through the LLM Skills registry so auto-extract can be toggled
|
||||
# or routed to its own provider (Settings → LLM Skills) independently of
|
||||
# the translation pipeline. None == disabled or no provider configured.
|
||||
from services import llm_skills
|
||||
|
||||
client = _llm_client()
|
||||
if client is None:
|
||||
handle = llm_skills.resolve_skill_client("glossary_extract")
|
||||
if handle is None:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"Auto-extract needs an LLM. Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
|
||||
"(Ollama works locally: base_url=http://localhost:11434/v1) and try again."
|
||||
"Auto-extract needs an LLM. Set one up in Settings → LLM Providers "
|
||||
"(pick a provider, add its key, choose a model, Test) — or use local "
|
||||
"Ollama / LM Studio for a fully offline setup — and make sure the "
|
||||
"Glossary auto-extract skill is enabled in Settings → LLM Skills, "
|
||||
"then try again."
|
||||
),
|
||||
)
|
||||
|
||||
@@ -220,9 +226,9 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
|
||||
)
|
||||
|
||||
try:
|
||||
res = client.chat.completions.create(
|
||||
model=_llm_model(),
|
||||
timeout=_llm_timeout(),
|
||||
res = handle.client.chat.completions.create(
|
||||
model=handle.model,
|
||||
timeout=handle.timeout,
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
@@ -231,9 +237,18 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
|
||||
body = (res.choices[0].message.content or "").strip()
|
||||
except Exception as e:
|
||||
logger.warning("auto-extract LLM call failed: %s", e)
|
||||
# Scrub the provider error — some OpenAI-compatible providers echo the
|
||||
# API key or a user_id in the body, which must not reach the UI verbatim.
|
||||
from core.scrub import scrub_provider_error
|
||||
from services import llm_providers
|
||||
_p = llm_providers.active_provider()
|
||||
_key = llm_providers.resolve_api_key(_p) if _p else None
|
||||
raise HTTPException(
|
||||
status_code=502,
|
||||
detail=f"LLM didn't respond. Check Settings → Logs → Backend for the trace. Error: {e}",
|
||||
detail=(
|
||||
"LLM didn't respond. Check Settings → Logs → Backend for the trace. "
|
||||
f"Error: {scrub_provider_error(e, _key)}"
|
||||
),
|
||||
)
|
||||
|
||||
# Parse: SOURCE || TARGET || note (lines are allowed to be sloppy — we're forgiving).
|
||||
|
||||
@@ -22,7 +22,6 @@ from __future__ import annotations
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import asyncio
|
||||
import tempfile
|
||||
from typing import Literal, Optional
|
||||
|
||||
@@ -30,7 +29,7 @@ from fastapi import APIRouter, File, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
|
||||
|
||||
logger = logging.getLogger("omnivoice.openai_compat")
|
||||
|
||||
@@ -109,6 +108,23 @@ class SpeechRequest(BaseModel):
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
|
||||
)
|
||||
# #1014: these two were silently DISCARDED before (pydantic ignores
|
||||
# undeclared fields) — a 200 OK that quietly dropped the caller's quality
|
||||
# knobs. Declared now and passed through, matching the native /generate
|
||||
# form fields (defaults there: num_step=16, guidance_scale=2.0; the
|
||||
# model's documented "quality" preset is num_step=32).
|
||||
num_step: Optional[int] = Field(
|
||||
default=None,
|
||||
ge=1,
|
||||
le=128,
|
||||
description="OmniVoice extension: iterative unmasking steps (app default 16; 32 = the model's documented quality preset).",
|
||||
)
|
||||
guidance_scale: Optional[float] = Field(
|
||||
default=None,
|
||||
gt=0,
|
||||
le=20,
|
||||
description="OmniVoice extension: classifier-free guidance scale (app default 2.0).",
|
||||
)
|
||||
|
||||
|
||||
class TranscriptionResponse(BaseModel):
|
||||
@@ -238,10 +254,10 @@ def _run_tts(backend, text: str, kw: dict):
|
||||
sr = backend.sample_rate
|
||||
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
|
||||
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
|
||||
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
|
||||
# output; applied to a studio engine it adds an audible level pump and a
|
||||
# reverb tail that degrade the very output we want clean. Loudness
|
||||
# normalisation still runs — it's a benign peak scale, not dynamics.
|
||||
# That chain's highpass + Compressor is tuned for OmniVoice's 24 kHz clone
|
||||
# output; applied to a studio engine it adds an audible level pump that
|
||||
# degrades the very output we want clean. Loudness normalisation still
|
||||
# runs — it's a benign peak scale, not dynamics.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
@@ -275,6 +291,10 @@ async def create_speech(req: SpeechRequest):
|
||||
kw["chunk_duration"] = req.chunk_duration
|
||||
if req.chunk_threshold is not None:
|
||||
kw["chunk_threshold"] = req.chunk_threshold
|
||||
if req.num_step is not None:
|
||||
kw["num_step"] = req.num_step
|
||||
if req.guidance_scale is not None:
|
||||
kw["guidance_scale"] = req.guidance_scale
|
||||
if req.language:
|
||||
kw["language"] = req.language
|
||||
if req.instruct:
|
||||
@@ -312,9 +332,44 @@ async def create_speech(req: SpeechRequest):
|
||||
# Not a profile ID — might be a KittenTTS preset or similar
|
||||
kw["voice"] = voice
|
||||
|
||||
# ── #1033/#1037/#1014: warm the engine under the LOAD budget before the
|
||||
# generate clock starts. The T4 verification (#1014) measured a fresh
|
||||
# install's first /v1/audio/speech burning its whole 300s generate budget
|
||||
# on the multi-GB checkpoint download (0% GPU util throughout) and dying
|
||||
# with a misleading "too heavy for the available compute" error. Model
|
||||
# loading gets OMNIVOICE_MODEL_LOAD_TIMEOUT (default 1200s); once warm
|
||||
# this is a per-request no-op.
|
||||
from services.model_manager import _model_load_timeout
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
wav, sr = await loop.run_in_executor(_gpu_pool, _run_tts, backend, req.input, kw)
|
||||
await run_on_gpu_pool_guarded(
|
||||
backend.ensure_ready,
|
||||
what=f"TTS engine '{backend.id}' model load",
|
||||
timeout=_model_load_timeout(),
|
||||
)
|
||||
# Catch the BUILTIN TimeoutError base, not GpuJobTimeoutError by name:
|
||||
# several tests reload services.model_manager mid-suite, so a class
|
||||
# imported at call time can differ in identity from the one the guard
|
||||
# (bound at this module's import) actually raises — the except would
|
||||
# silently miss. The builtin base has one identity forever. (Caught by
|
||||
# this exact test failing CI-only, in full-suite order.)
|
||||
except TimeoutError as e:
|
||||
logger.warning("engine load exceeded the model-load budget: %s", e)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
f"TTS engine '{backend.id}' did not finish loading within its "
|
||||
f"model-load budget — on a first run this usually means the weight "
|
||||
f"download is slow or stalled (check Settings → Models for "
|
||||
f"progress), not that generation failed. Retry once the model "
|
||||
f"shows as installed."
|
||||
),
|
||||
) from e
|
||||
|
||||
try:
|
||||
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
|
||||
# GPU pool and brick the backend (#730 class).
|
||||
wav, sr = await run_on_gpu_pool_guarded(
|
||||
lambda: _run_tts(backend, req.input, kw), what="OpenAI TTS generate")
|
||||
except Exception as e:
|
||||
logger.exception("OpenAI TTS failed: %s", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
@@ -384,12 +439,15 @@ async def create_transcription(
|
||||
try:
|
||||
backend = get_active_asr_backend()
|
||||
|
||||
# Run transcription in the thread pool to avoid blocking the event loop
|
||||
loop = asyncio.get_running_loop()
|
||||
# Run transcription in the thread pool to avoid blocking the event loop,
|
||||
# bounded so a stuck/starved ASR returns a 504 with guidance instead of
|
||||
# hanging the request forever (see run_transcribe_guarded).
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
word_ts = response_format == "verbose_json"
|
||||
result = await loop.run_in_executor(
|
||||
result = await run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
lambda: backend.transcribe(tmp_path, word_timestamps=word_ts),
|
||||
what="OpenAI",
|
||||
)
|
||||
|
||||
# Extract the full text from segments
|
||||
@@ -456,6 +514,12 @@ async def create_transcription(
|
||||
# Default: json
|
||||
return TranscriptionResponse(text=full_text)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except TimeoutError as e:
|
||||
# ASRTimeoutError (subclass): backend alive, ASR too heavy for compute.
|
||||
logger.warning("OpenAI transcription timed out: %s", e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.exception("OpenAI transcription failed: %s", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@@ -60,6 +60,11 @@ async def export_persona(
|
||||
profile = dict(row)
|
||||
|
||||
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
|
||||
# #693: if OMNIVOICE_MODEL is set, record the *resolved* checkpoint in the
|
||||
# exported bundle so a leaked engine id (e.g. "omnivoice") can't be baked in;
|
||||
# keep "" when unset (the bundle's "engine unspecified" marker).
|
||||
from services.model_manager import resolve_omnivoice_checkpoint
|
||||
engine_id = resolve_omnivoice_checkpoint() if os.environ.get("OMNIVOICE_MODEL", "").strip() else ""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
content = await loop.run_in_executor(
|
||||
@@ -70,7 +75,7 @@ async def export_persona(
|
||||
license_spdx=license_spdx,
|
||||
tags=tag_list,
|
||||
include_reference=include_reference,
|
||||
engine_id=os.environ.get("OMNIVOICE_MODEL", ""),
|
||||
engine_id=engine_id,
|
||||
omnivoice_version=APP_VERSION,
|
||||
),
|
||||
)
|
||||
|
||||
@@ -12,6 +12,7 @@ from core.db import db_conn
|
||||
from core.config import VOICES_DIR, OUTPUTS_DIR
|
||||
from core import event_bus
|
||||
from core.personalities import get_personalities
|
||||
from omnivoice.utils.voice_design import heal_design_instruct, sanitize_instruct
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -72,10 +73,28 @@ async def create_profile(
|
||||
raise ValueError("not an object")
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
|
||||
# Root-cause close for #983: a design profile must never be PERSISTED
|
||||
# with a partial vd_states shape, regardless of which client (older
|
||||
# frontend build, hand-edited payload, third-party API caller) created
|
||||
# it — a missing category key crashes DesignMethodPanel's render on
|
||||
# every future client that selects this profile. CATEGORY_ORDER is the
|
||||
# same single source of truth the frontend's CATEGORIES keys mirror
|
||||
# (core/describe_voice.py), so this can't drift from the picker.
|
||||
from core.describe_voice import CATEGORY_ORDER
|
||||
for _cat in CATEGORY_ORDER:
|
||||
parsed.setdefault(_cat, "Auto")
|
||||
vd_states = _json.dumps(parsed)
|
||||
# An all-Auto design (every category left on "Auto") yields an empty
|
||||
# instruct — that's still a valid, saveable voice: synthesis falls back
|
||||
# to neutral instruct-only conditioning (see generation.py design path).
|
||||
# Don't gate save on a non-empty instruct.
|
||||
#
|
||||
# Defence-in-depth against the "[object Object]" / freeform-prose poison
|
||||
# (#550 #571 #594 #596): never persist an instruct the engine validator
|
||||
# would reject. Sanitize the submitted instruct and, if it's unusable,
|
||||
# rebuild the tags from vd_states — so the row is always generation-safe
|
||||
# regardless of which frontend build saved it.
|
||||
instruct = heal_design_instruct(instruct, parsed)
|
||||
|
||||
profile_id = str(uuid.uuid4())[:8]
|
||||
|
||||
@@ -167,6 +186,10 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
|
||||
continue
|
||||
if col == "name" and not val.strip():
|
||||
raise HTTPException(status_code=400, detail="A voice profile needs a name.")
|
||||
if col == "instruct":
|
||||
# Never let an edit persist a validator-rejecting instruct (prose /
|
||||
# "[object Object]"); keep only whitelist tags (#550 #571 #594 #596).
|
||||
val = sanitize_instruct(val)
|
||||
fields.append(f"{col} = ?")
|
||||
params.append(val.strip() if col in ("name", "language") else val)
|
||||
if not fields:
|
||||
|
||||
@@ -0,0 +1,306 @@
|
||||
"""
|
||||
Pronunciation dictionary router — Expressive-TTS Spec 01 Phase 1.
|
||||
|
||||
CRUD for the DB-backed, per-language pronunciation dictionary the
|
||||
``PronunciationPanel`` (Settings → Pronunciation) edits, plus a model-free
|
||||
``/pronunciation/test`` dry-run. Entries are applied as pure text substitution
|
||||
before synthesis (see ``services/pronunciation.apply_pronunciation`` and the
|
||||
generate path), so a saved entry actually changes the audio on every engine.
|
||||
|
||||
Endpoints (loopback-only, like the dictation router):
|
||||
GET /pronunciation → list every entry
|
||||
POST /pronunciation → create one entry
|
||||
PUT /pronunciation/{entry_id} → update an entry (partial)
|
||||
DELETE /pronunciation/{entry_id} → remove an entry
|
||||
POST /pronunciation/test → dry-run substitution (no model)
|
||||
GET /pronunciation/export → all entries as JSON (round-trips import)
|
||||
POST /pronunciation/import → bulk add entries from JSON
|
||||
|
||||
Scope: ``language='*'`` is global (applies to every request); a 2-letter code
|
||||
(``'en'``, ``'de'``) applies only when the request language matches.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from core.db import db_conn
|
||||
from services.pronunciation import apply_pronunciation, entries_for_language
|
||||
|
||||
logger = logging.getLogger("omnivoice.pronunciation")
|
||||
router = APIRouter()
|
||||
|
||||
_VALID_TYPES = ("respelling", "ipa", "cmu")
|
||||
_ALL_LANG = "*"
|
||||
|
||||
# IPA: the input is validated as a non-empty string of Unicode letters / IPA
|
||||
# extension codepoints + the usual suprasegmental marks; we reject ASCII control
|
||||
# and the bracket/pipe chars that would collide with the inline grammar. This is
|
||||
# a charset gate (catches obvious garbage early), not a full IPA grammar.
|
||||
_IPA_BAD = re.compile(r"[\[\]\|\x00-\x1f]")
|
||||
# CMU / ARPABET: space-separated phoneme tokens (letters + an optional 0-2 stress
|
||||
# digit), e.g. "N AH0 V AE1 D AH0". Reject anything else.
|
||||
_CMU_TOKEN = re.compile(r"^[A-Za-z]{1,3}[0-2]?$")
|
||||
|
||||
|
||||
def _validate_type_replacement(etype: str, replacement: str) -> None:
|
||||
"""Raise 400 on a phoneme replacement that's obviously malformed.
|
||||
|
||||
Respelling rows accept any text. IPA rows must be a non-empty string free of
|
||||
bracket/pipe/control chars. CMU rows must be space-separated ARPABET tokens.
|
||||
Validating on save (not at synth) means a model never sees garbage phonemes
|
||||
(Spec 01 §R3 — never pass unvalidated phoneme strings to a model).
|
||||
"""
|
||||
if etype == "respelling":
|
||||
return
|
||||
rep = (replacement or "").strip()
|
||||
if not rep:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"A {etype.upper()} entry needs a phoneme string in 'replacement'.",
|
||||
)
|
||||
if etype == "ipa":
|
||||
if _IPA_BAD.search(rep):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="That IPA string contains brackets, a pipe, or control characters. "
|
||||
"Use plain IPA symbols, e.g. ˈnɛvʌdə.",
|
||||
)
|
||||
elif etype == "cmu":
|
||||
tokens = rep.split()
|
||||
if not tokens or any(not _CMU_TOKEN.match(tok) for tok in tokens):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="That doesn't look like CMU/ARPABET. Use space-separated tokens with "
|
||||
"optional stress digits, e.g. N AH0 V AE1 D AH0.",
|
||||
)
|
||||
|
||||
|
||||
def _norm_language(language: Optional[str]) -> str:
|
||||
"""Normalize a scope to '*' (global) or a lowercase 2-letter code."""
|
||||
if not language:
|
||||
return _ALL_LANG
|
||||
s = str(language).strip()
|
||||
if not s or s == _ALL_LANG or s.lower() == "auto":
|
||||
return _ALL_LANG
|
||||
return s.lower()[:2]
|
||||
|
||||
|
||||
def _row_to_dict(r) -> dict:
|
||||
d = dict(r)
|
||||
d["enabled"] = bool(d.get("enabled"))
|
||||
# ``scope`` is the UI-facing alias for ``language`` ('*' shows as Global).
|
||||
d["scope"] = d.get("language") or _ALL_LANG
|
||||
return d
|
||||
|
||||
|
||||
# ── Schemas ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class PronEntry(BaseModel):
|
||||
term: str
|
||||
replacement: str = ""
|
||||
type: str = "respelling"
|
||||
language: str = _ALL_LANG
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class PronEntryUpdate(BaseModel):
|
||||
term: Optional[str] = None
|
||||
replacement: Optional[str] = None
|
||||
type: Optional[str] = None
|
||||
language: Optional[str] = None
|
||||
enabled: Optional[bool] = None
|
||||
|
||||
|
||||
class PronTestRequest(BaseModel):
|
||||
text: str
|
||||
language: Optional[str] = None
|
||||
|
||||
|
||||
class PronImportRequest(BaseModel):
|
||||
entries: List[PronEntry]
|
||||
replace: bool = False # True → clear existing rows first
|
||||
|
||||
|
||||
# ── CRUD ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/pronunciation", dependencies=[Depends(require_loopback)])
|
||||
def list_entries():
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
|
||||
).fetchall()
|
||||
return [_row_to_dict(r) for r in rows]
|
||||
|
||||
|
||||
@router.post("/pronunciation", dependencies=[Depends(require_loopback)])
|
||||
def create_entry(entry: PronEntry):
|
||||
term = entry.term.strip()
|
||||
if not term:
|
||||
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
|
||||
etype = (entry.type or "respelling").strip().lower()
|
||||
if etype not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unknown entry type {entry.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
|
||||
)
|
||||
_validate_type_replacement(etype, entry.replacement)
|
||||
eid = str(uuid.uuid4())[:12]
|
||||
now = time.time()
|
||||
lang = _norm_language(entry.language)
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
(eid, term, entry.replacement, etype, lang, 1 if entry.enabled else 0, now),
|
||||
)
|
||||
row = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (eid,)
|
||||
).fetchone()
|
||||
return _row_to_dict(row)
|
||||
|
||||
|
||||
@router.put("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
|
||||
def update_entry(entry_id: str, patch: PronEntryUpdate):
|
||||
with db_conn() as conn:
|
||||
existing = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
|
||||
).fetchone()
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="No such pronunciation entry.")
|
||||
|
||||
# Resolve the post-update type + replacement so phoneme validation runs
|
||||
# against the final state (e.g. switching type without changing text).
|
||||
new_type = (patch.type.strip().lower() if patch.type is not None else existing["type"]) or "respelling"
|
||||
if new_type not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unknown entry type {patch.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
|
||||
)
|
||||
new_replacement = patch.replacement if patch.replacement is not None else existing["replacement"]
|
||||
_validate_type_replacement(new_type, new_replacement)
|
||||
|
||||
fields, params = [], []
|
||||
if patch.term is not None:
|
||||
term = patch.term.strip()
|
||||
if not term:
|
||||
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
|
||||
fields.append("term = ?"); params.append(term)
|
||||
if patch.replacement is not None:
|
||||
fields.append("replacement = ?"); params.append(patch.replacement)
|
||||
if patch.type is not None:
|
||||
fields.append("type = ?"); params.append(new_type)
|
||||
if patch.language is not None:
|
||||
fields.append("language = ?"); params.append(_norm_language(patch.language))
|
||||
if patch.enabled is not None:
|
||||
fields.append("enabled = ?"); params.append(1 if patch.enabled else 0)
|
||||
if not fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="PUT body was empty. Include at least one field to change, or DELETE the entry.",
|
||||
)
|
||||
params.append(entry_id)
|
||||
# nosec B608 - `fields` are fixed literal assignments ("term = ?", …) from
|
||||
# the allowlist above; every user value is a bound `?` parameter, never
|
||||
# interpolated. The f-string only joins constant column fragments.
|
||||
conn.execute(
|
||||
f"UPDATE pronunciation_entries SET {', '.join(fields)} WHERE id = ?", # nosec B608
|
||||
params,
|
||||
)
|
||||
row = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
|
||||
).fetchone()
|
||||
return _row_to_dict(row)
|
||||
|
||||
|
||||
@router.delete("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
|
||||
def delete_entry(entry_id: str):
|
||||
with db_conn() as conn:
|
||||
cur = conn.execute("DELETE FROM pronunciation_entries WHERE id = ?", (entry_id,))
|
||||
return {"deleted": cur.rowcount > 0}
|
||||
|
||||
|
||||
# ── Dry-run + import/export ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/pronunciation/test", dependencies=[Depends(require_loopback)])
|
||||
def test_substitution(req: PronTestRequest):
|
||||
"""Show the post-substitution text for ``req.text`` — no model call.
|
||||
|
||||
Applies the same dictionary + inline ``[[…]]`` resolution the synth path
|
||||
runs, so the user sees exactly what the engine will be handed.
|
||||
"""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries"
|
||||
).fetchall()
|
||||
substituted = apply_pronunciation(req.text, rows, req.language)
|
||||
applied = entries_for_language(rows, req.language)
|
||||
return {
|
||||
"input": req.text,
|
||||
"substituted": substituted,
|
||||
"changed": substituted != req.text,
|
||||
"applied_terms": sorted(applied.keys(), key=len, reverse=True),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/pronunciation/export", dependencies=[Depends(require_loopback)])
|
||||
def export_entries():
|
||||
"""Every entry as a JSON-serializable list (round-trips ``/import``)."""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT term, replacement, type, language, enabled "
|
||||
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
|
||||
).fetchall()
|
||||
return {"entries": [
|
||||
{"term": r["term"], "replacement": r["replacement"], "type": r["type"],
|
||||
"language": r["language"], "enabled": bool(r["enabled"])}
|
||||
for r in rows
|
||||
]}
|
||||
|
||||
|
||||
@router.post("/pronunciation/import", dependencies=[Depends(require_loopback)])
|
||||
def import_entries(req: PronImportRequest):
|
||||
"""Bulk-add entries. ``replace=true`` clears the table first.
|
||||
|
||||
Each entry is validated like ``POST /pronunciation``; one bad row fails the
|
||||
whole import (400) so the table is never left half-applied.
|
||||
"""
|
||||
now = time.time()
|
||||
cleaned = []
|
||||
for e in req.entries:
|
||||
term = e.term.strip()
|
||||
if not term:
|
||||
continue # silently skip blank terms — they're a no-op anyway
|
||||
etype = (e.type or "respelling").strip().lower()
|
||||
if etype not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Entry {term!r}: unknown type {e.type!r}.",
|
||||
)
|
||||
_validate_type_replacement(etype, e.replacement)
|
||||
cleaned.append((str(uuid.uuid4())[:12], term, e.replacement, etype,
|
||||
_norm_language(e.language), 1 if e.enabled else 0, now))
|
||||
with db_conn() as conn:
|
||||
if req.replace:
|
||||
conn.execute("DELETE FROM pronunciation_entries")
|
||||
conn.executemany(
|
||||
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
cleaned,
|
||||
)
|
||||
return {"imported": len(cleaned), "replaced": req.replace}
|
||||
@@ -12,6 +12,7 @@ The state endpoint duplicates `/system/hf-token/state` (which lives on
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import asdict
|
||||
@@ -134,12 +135,22 @@ class _RefinementBody(BaseModel):
|
||||
|
||||
|
||||
def _refinement_state():
|
||||
from services.refinement import get_refinement_config
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
from services.refinement import (
|
||||
_skill_llm,
|
||||
get_last_refine_status,
|
||||
get_refinement_config,
|
||||
)
|
||||
|
||||
cfg = get_refinement_config()
|
||||
# The UI shows whether refinement can actually run (needs an LLM).
|
||||
cfg["llm_ready"] = get_active_llm_backend().id != "off"
|
||||
# `llm_ready` only means "an endpoint is CONFIGURED" — a placeholder/dead
|
||||
# endpoint still reads ready. It's resolved through the LLM Skills registry
|
||||
# so a disabled dictation_refinement skill / per-skill provider override
|
||||
# reads the same here as on the actual refine path. The honesty layer is
|
||||
# `last_refine_status`: {ok, reason, at} from the most recent final, so the
|
||||
# panel can flag a configured-but-failing LLM (the real safety is the hard
|
||||
# refine timeout, which keeps a dead endpoint from ever stalling the final).
|
||||
cfg["llm_ready"] = _skill_llm().id != "off"
|
||||
cfg["last_refine_status"] = get_last_refine_status()
|
||||
return cfg
|
||||
|
||||
|
||||
@@ -234,6 +245,248 @@ def set_llm_endpoint(body: _LLMEndpointBody):
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
# ── Multi-provider LLM registry (Settings → LLM Providers) ────────────────
|
||||
# Keys persist ENCRYPTED via settings_store.set_secret (never .env, never
|
||||
# returned). base_url/model/account overrides are non-secret. Loopback-gated
|
||||
# by the router dep, so LAN peers can't read masks or write keys.
|
||||
|
||||
class _LLMProviderBody(BaseModel):
|
||||
api_key: str | None = Field(None, description="API key; '' clears it, None leaves unchanged")
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
account_id: str | None = Field(None, description="Cloudflare account id")
|
||||
make_active: bool = False
|
||||
|
||||
|
||||
class _LLMActiveBody(BaseModel):
|
||||
provider: str = Field(..., description="provider id to activate")
|
||||
|
||||
|
||||
@router.get("/llm-providers")
|
||||
def list_llm_providers():
|
||||
"""All providers with resolved base_url/model + whether a key is configured.
|
||||
|
||||
Never returns key material — only `has_key`/`key_from_env` booleans.
|
||||
"""
|
||||
from services import llm_providers
|
||||
return {
|
||||
"active": llm_providers.active_provider_id(),
|
||||
"providers": [llm_providers.describe(p) for p in llm_providers.all_providers()],
|
||||
}
|
||||
|
||||
|
||||
@router.put("/llm-providers/{provider_id}")
|
||||
def save_llm_provider(provider_id: str, body: _LLMProviderBody):
|
||||
"""Save a provider's key (encrypted) + optional base_url/model/account.
|
||||
|
||||
A None field is left unchanged; an empty api_key clears the stored key.
|
||||
"""
|
||||
from services import llm_providers
|
||||
p = llm_providers.get_provider(provider_id)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
|
||||
if body.api_key is not None:
|
||||
llm_providers.save_key(provider_id, body.api_key.strip())
|
||||
llm_providers.save_overrides(
|
||||
provider_id, base_url=body.base_url, model=body.model,
|
||||
account_id=body.account_id,
|
||||
)
|
||||
# An explicit save also claims the active slot when the user has never
|
||||
# chosen a provider (#963). Without this, a saved-and-tested local
|
||||
# provider (Ollama/LM Studio) evaporates on restart: active_provider_id()
|
||||
# deliberately excludes local providers from auto-select, so the plain
|
||||
# "Save" left nothing persisted to resolve. Gated on the STORED selection
|
||||
# only — an explicit prior choice is never stolen by a plain save, and an
|
||||
# unconfigured provider can't claim the slot.
|
||||
if body.make_active or (
|
||||
llm_providers.stored_active_provider_id() is None
|
||||
and llm_providers.is_configured(p)
|
||||
):
|
||||
llm_providers.set_active_provider(provider_id)
|
||||
return list_llm_providers()
|
||||
|
||||
|
||||
@router.post("/llm-providers/active")
|
||||
def set_active_llm_provider(body: _LLMActiveBody):
|
||||
from services import llm_providers
|
||||
if llm_providers.get_provider(body.provider) is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {body.provider!r}")
|
||||
llm_providers.set_active_provider(body.provider)
|
||||
return list_llm_providers()
|
||||
|
||||
|
||||
def _scrub_llm_detail(e: Exception, api_key: str | None) -> str:
|
||||
"""Scrubbed, UI-safe failure text. scrub_text() covers env secrets and
|
||||
home paths — but a STORE-persisted key isn't in the env, and some
|
||||
providers echo the key in error bodies, so redact the exact resolved key
|
||||
explicitly before the generic pass."""
|
||||
from core.scrub import scrub_text
|
||||
detail = f"{type(e).__name__}: {e}"
|
||||
if api_key and api_key != "local" and len(api_key) >= 8:
|
||||
detail = detail.replace(api_key, "•••")
|
||||
return scrub_text(detail)
|
||||
|
||||
|
||||
def _classify_llm_error(e: Exception) -> str:
|
||||
"""Map a provider-call failure to an actionable kind the UI can localize.
|
||||
|
||||
Kinds: auth (bad/missing key), not_found (model or endpoint path),
|
||||
rate_limit, network (DNS/conn/timeout), error (everything else).
|
||||
Status codes win when the OpenAI SDK provides one; exception-family
|
||||
names catch the non-HTTP failures (DNS, refused, TLS, timeout).
|
||||
"""
|
||||
status = getattr(e, "status_code", None)
|
||||
if status in (401, 403):
|
||||
return "auth"
|
||||
if status == 404:
|
||||
return "not_found"
|
||||
if status == 429:
|
||||
return "rate_limit"
|
||||
name = type(e).__name__
|
||||
if name in ("APIConnectionError", "APITimeoutError", "ConnectError",
|
||||
"ConnectTimeout", "TimeoutError"):
|
||||
return "network"
|
||||
if name == "AuthenticationError":
|
||||
return "auth"
|
||||
if name == "NotFoundError":
|
||||
return "not_found"
|
||||
if name == "RateLimitError":
|
||||
return "rate_limit"
|
||||
return "error"
|
||||
|
||||
|
||||
@router.post("/llm-providers/{provider_id}/test")
|
||||
def test_llm_provider(provider_id: str):
|
||||
"""One cheap round-trip against a provider to prove the key/URL work.
|
||||
|
||||
Temporarily activates the provider for the probe by resolving its config
|
||||
directly (does not change the persisted active selection). Returns
|
||||
latency_ms plus, on failure, a classified ``kind`` (config / auth /
|
||||
not_found / rate_limit / network / error) so the UI shows an actionable,
|
||||
localizable message instead of a raw exception string.
|
||||
"""
|
||||
import time as _time
|
||||
|
||||
from services import llm_providers
|
||||
p = llm_providers.get_provider(provider_id)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
|
||||
base_url = llm_providers.resolve_base_url(p)
|
||||
api_key = llm_providers.resolve_api_key(p)
|
||||
if not base_url:
|
||||
return {"ok": False, "kind": "config", "detail": "No Base URL set for this provider."}
|
||||
if not api_key:
|
||||
return {"ok": False, "kind": "config", "detail": "No API key configured for this provider."}
|
||||
t0 = _time.monotonic()
|
||||
try:
|
||||
from openai import OpenAI
|
||||
# max_retries=0: this is an interactive probe with a live spinner — the
|
||||
# SDK's default 2 automatic retries turn a 429/timeout into a ~34s hang.
|
||||
# Surface the first failure immediately instead.
|
||||
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
|
||||
res = client.chat.completions.create(
|
||||
model=llm_providers.resolve_model(p),
|
||||
messages=[{"role": "user", "content": "Reply with the single word: ok"}],
|
||||
timeout=20,
|
||||
)
|
||||
reply = (res.choices[0].message.content or "").strip()
|
||||
return {
|
||||
"ok": True,
|
||||
"model": llm_providers.resolve_model(p),
|
||||
"reply": reply[:80],
|
||||
"latency_ms": int((_time.monotonic() - t0) * 1000),
|
||||
}
|
||||
except Exception as e: # noqa: BLE001 — surface a clean, scrubbed error to the UI
|
||||
return {
|
||||
"ok": False,
|
||||
"kind": _classify_llm_error(e),
|
||||
"detail": _scrub_llm_detail(e, api_key),
|
||||
"latency_ms": int((_time.monotonic() - t0) * 1000),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/llm-providers/{provider_id}/models")
|
||||
def list_llm_provider_models(provider_id: str):
|
||||
"""List model ids the provider's key can access (OpenAI-compat /models).
|
||||
|
||||
Powers the model-picker datalist in Settings → LLM Providers so users
|
||||
don't have to guess model names. Read-only; failures return the same
|
||||
classified shape as /test; capped so a huge catalog can't bloat the UI.
|
||||
"""
|
||||
from services import llm_providers
|
||||
p = llm_providers.get_provider(provider_id)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
|
||||
base_url = llm_providers.resolve_base_url(p)
|
||||
api_key = llm_providers.resolve_api_key(p)
|
||||
if not base_url or not api_key:
|
||||
return {"ok": False, "kind": "config", "models": []}
|
||||
try:
|
||||
from openai import OpenAI
|
||||
# max_retries=0: interactive probe — fail fast, don't burn ~34s on the
|
||||
# SDK's default retry ladder when the key/URL is wrong (matches /test).
|
||||
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
|
||||
ids = sorted(m.id for m in client.models.list(timeout=10))
|
||||
# Cap so a huge catalog can't bloat the datalist; flag the cap so the UI
|
||||
# can say "first 200 shown" rather than implying it's the full list.
|
||||
return {"ok": True, "models": ids[:200], "truncated": len(ids) > 200}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {
|
||||
"ok": False,
|
||||
"kind": _classify_llm_error(e),
|
||||
"detail": _scrub_llm_detail(e, api_key),
|
||||
"models": [],
|
||||
}
|
||||
|
||||
|
||||
# ── LLM Skills (Settings → LLM Skills) ─────────────────────────────────────
|
||||
# Per-feature enable/route control for every LLM consumption point. Each
|
||||
# skill can be toggled off (degrades exactly like "no LLM configured") or
|
||||
# routed to a specific provider (local Ollama/LM Studio vs a remote key)
|
||||
# instead of the one global active provider. Loopback-gated (router dep).
|
||||
|
||||
|
||||
class _LLMSkillBody(BaseModel):
|
||||
enabled: bool | None = Field(None, description="None leaves the toggle unchanged")
|
||||
provider_override: str | None = Field(
|
||||
None,
|
||||
description="provider id to route this skill to; '' or null clears "
|
||||
"it (skill follows the active provider). Omit to leave "
|
||||
"unchanged.",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/llm-skills")
|
||||
def list_llm_skills():
|
||||
"""Every LLM skill with its toggle, routing, and resolved ready status."""
|
||||
from services import llm_skills
|
||||
return {"skills": [llm_skills.describe(s.id) for s in llm_skills.all_skills()]}
|
||||
|
||||
|
||||
@router.put("/llm-skills/{skill_id}")
|
||||
def set_llm_skill(skill_id: str, body: _LLMSkillBody):
|
||||
"""Toggle a skill and/or set its provider routing.
|
||||
|
||||
Field semantics match the providers PUT: an omitted field is left
|
||||
unchanged; ``provider_override: ""``/``null`` clears the override.
|
||||
404 for an unknown skill or an unknown provider id.
|
||||
"""
|
||||
from services import llm_skills
|
||||
if llm_skills.get_skill(skill_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown LLM skill {skill_id!r}")
|
||||
kwargs = {}
|
||||
if body.enabled is not None:
|
||||
kwargs["enabled"] = body.enabled
|
||||
if "provider_override" in body.model_fields_set:
|
||||
kwargs["provider_override"] = body.provider_override
|
||||
try:
|
||||
if kwargs:
|
||||
llm_skills.configure_skill(skill_id, **kwargs)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
return list_llm_skills()
|
||||
|
||||
|
||||
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
|
||||
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
|
||||
# endpoint. The handler is loopback-gated (router-level dep) and the
|
||||
@@ -397,6 +650,41 @@ def set_models_dir(body: _ModelsDirBody):
|
||||
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
|
||||
|
||||
|
||||
# ── Storage report (Settings → Storage) ────────────────────────────────────
|
||||
# Per-volume disk totals + du-style sizes for everything the app owns (HF
|
||||
# model cache, app data subtotals, engine venvs, temp files) with server-side
|
||||
# warnings. Heavy directory walks run in a worker thread with per-category
|
||||
# deadlines and a 5-minute in-process cache (services.storage_report), so the
|
||||
# endpoint stays cheap on repeat Settings visits. Loopback-gated via the
|
||||
# router-level dep like every sibling.
|
||||
|
||||
|
||||
@router.get("/storage")
|
||||
async def get_storage_report(refresh: bool = Query(False)):
|
||||
"""Disk + per-category storage usage for the Settings → Storage panel.
|
||||
|
||||
`refresh=1` bypasses the 5-minute cache and rescans. `min_free_gb`
|
||||
reuses the setup wizard's constant so both surfaces warn at the same
|
||||
threshold.
|
||||
"""
|
||||
from api.routers.setup.wizard import MIN_FREE_GB
|
||||
from core.config import DATA_DIR
|
||||
from services import storage_report
|
||||
|
||||
try:
|
||||
return await asyncio.to_thread(
|
||||
storage_report.get_report,
|
||||
data_dir=DATA_DIR,
|
||||
hf_cache_dir=_effective_models_dir(),
|
||||
app_venv=storage_report.default_app_venv(),
|
||||
min_free_gb=MIN_FREE_GB,
|
||||
refresh=refresh,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("storage report failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to compute storage report")
|
||||
|
||||
|
||||
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
|
||||
# Restricted-network users (e.g. behind the Great Firewall) need to point
|
||||
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
|
||||
@@ -439,6 +727,11 @@ def set_hf_mirror(body: _HFMirrorBody):
|
||||
url = (body.url or "").strip().rstrip("/")
|
||||
if url and not url.startswith(("http://", "https://")):
|
||||
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
|
||||
# Compare against the currently-persisted value (normalised the same way) so
|
||||
# a no-op save doesn't nag the user to restart. Only a real change to the
|
||||
# persisted endpoint can require a restart.
|
||||
previous = (user_env.get_user_env(_HF_ENDPOINT_ENV) or "").strip().rstrip("/")
|
||||
changed = url != previous
|
||||
try:
|
||||
if url:
|
||||
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
|
||||
@@ -449,6 +742,98 @@ def set_hf_mirror(body: _HFMirrorBody):
|
||||
except Exception:
|
||||
logger.exception("set_hf_mirror failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
|
||||
# HF endpoint is read at import time by huggingface_hub, so the override
|
||||
# is only guaranteed once the backend restarts.
|
||||
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
|
||||
# Model Store downloads pick up the new mirror immediately — the download
|
||||
# path resolves the endpoint per-call and we updated os.environ above. Only
|
||||
# transformers-side model *loads* (which read HF_ENDPOINT at import time)
|
||||
# need a restart, so restart_required is True ONLY when the value actually
|
||||
# changed — a no-op re-save never asks for a restart.
|
||||
return {"configured": url, "restart_required": changed, "presets": _HF_MIRROR_PRESETS}
|
||||
|
||||
|
||||
# ── OpenAI-compatible remote ASR (#877) ─────────────────────────────────────
|
||||
# A path to Qwen3-ASR/FunASR/SenseVoice — or OpenAI's own Whisper API — today,
|
||||
# without waiting on transformers to ship a direct Qwen3-ASR integration.
|
||||
# base_url/model are plain settings_store text rows; the key is encrypted via
|
||||
# settings_store.set_secret — same convention as /llm-providers, never
|
||||
# returned to the client, '' clears it, omitted/None leaves it unchanged.
|
||||
|
||||
|
||||
class _ASROpenAICompatBody(BaseModel):
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
api_key: str | None = Field(None, description="'' clears it, None leaves unchanged")
|
||||
|
||||
|
||||
@router.get("/asr-openai-compat")
|
||||
def get_asr_openai_compat():
|
||||
from services import asr_backend
|
||||
|
||||
return {
|
||||
"base_url": asr_backend.resolve_openai_compat_asr_base_url(),
|
||||
"model": asr_backend.resolve_openai_compat_asr_model(),
|
||||
"has_key": asr_backend.openai_compat_asr_has_key(),
|
||||
}
|
||||
|
||||
|
||||
@router.put("/asr-openai-compat")
|
||||
def set_asr_openai_compat(body: _ASROpenAICompatBody):
|
||||
from services import asr_backend, settings_store
|
||||
|
||||
if body.base_url is not None:
|
||||
url = body.base_url.strip().rstrip("/")
|
||||
if url and not url.startswith(("http://", "https://")):
|
||||
raise HTTPException(status_code=400, detail="Base URL must start with http(s)://")
|
||||
settings_store.set_text(asr_backend._ASR_OPENAI_COMPAT_BASE_URL_KEY, url)
|
||||
if body.model is not None:
|
||||
settings_store.set_text(
|
||||
asr_backend._ASR_OPENAI_COMPAT_MODEL_KEY, body.model.strip() or "whisper-1"
|
||||
)
|
||||
if body.api_key is not None:
|
||||
settings_store.set_secret(
|
||||
asr_backend._ASR_OPENAI_COMPAT_SECRET_NAME, body.api_key.strip()
|
||||
)
|
||||
return get_asr_openai_compat()
|
||||
|
||||
|
||||
# ── Updates panel: shipped changelog + pre-migration DB backup state ────────
|
||||
# (feat/safe-updates). Both are read-only, local-first surfaces for
|
||||
# Settings → Updates: the "What's new" viewer reads the CHANGELOG.md that
|
||||
# ships with the app, and the backup line shows the newest pre-migration
|
||||
# snapshot written by core.db_backup before `alembic upgrade head` runs.
|
||||
|
||||
|
||||
@router.get("/changelog")
|
||||
def get_changelog(limit_versions: int = Query(5, ge=1, le=50)):
|
||||
"""Structured release notes from the shipped CHANGELOG.md (newest first).
|
||||
|
||||
Bullets are raw markdown-lite (bold leads, `code`, (#NNN) refs) — the
|
||||
frontend renders them safely without HTML. `available: false` when this
|
||||
install has no changelog (never an error: the viewer just hides)."""
|
||||
from core import changelog
|
||||
|
||||
path = changelog.changelog_path()
|
||||
if not path:
|
||||
return {"available": False, "releases": []}
|
||||
try:
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
releases = changelog.parse_changelog(fh.read(), limit_versions)
|
||||
except Exception:
|
||||
logger.exception("changelog parse failed")
|
||||
return {"available": False, "releases": []}
|
||||
return {"available": bool(releases), "releases": releases}
|
||||
|
||||
|
||||
@router.get("/db-backup")
|
||||
def get_db_backup_state():
|
||||
"""Newest pre-migration database backup (or none yet). Feeds the
|
||||
"your data is backed up before every update" line in Settings → Updates."""
|
||||
from core import db_backup
|
||||
from core.config import DB_PATH
|
||||
|
||||
latest = db_backup.latest_backup(DB_PATH)
|
||||
return {
|
||||
"available": latest is not None,
|
||||
"latest": latest,
|
||||
"count": len(db_backup.list_backups(DB_PATH)),
|
||||
"keep": db_backup.KEEP_BACKUPS,
|
||||
}
|
||||
|
||||
@@ -21,7 +21,18 @@ from pydantic import BaseModel
|
||||
from core import prefs
|
||||
from utils import hf_progress
|
||||
from utils import download_aggregator
|
||||
from .models import KNOWN_MODELS, invalidate_cache
|
||||
# Weight-floor scan (MM2-07 / #352) lives in ``models.py`` — the lowest module in
|
||||
# the setup import graph — so install-time validation here, the first-run
|
||||
# install-state detector (#622), and load-time repair share one set of floors and
|
||||
# can't drift apart. ``_MIN_WEIGHT_BYTES``/``_WEIGHT_FLOORS`` re-exported for tests.
|
||||
from .models import ( # noqa: F401
|
||||
KNOWN_MODELS,
|
||||
invalidate_cache,
|
||||
snapshot_has_weights,
|
||||
disk_space_error,
|
||||
_MIN_WEIGHT_BYTES,
|
||||
_WEIGHT_FLOORS,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.download")
|
||||
router = APIRouter()
|
||||
@@ -120,11 +131,15 @@ def compute_plan(plan_files) -> dict:
|
||||
|
||||
|
||||
def _segmented_enabled() -> bool:
|
||||
"""Opt-in IDM-style accelerator (FDL-09), default OFF. Most useful when Xet
|
||||
is inactive (the app's default): the legacy-LFS path is single-stream, so
|
||||
this restores parallel speed AND gives real live byte progress."""
|
||||
"""IDM-style multi-connection accelerator (FDL-09), default **ON**. The app
|
||||
forces the legacy-LFS path (HF_HUB_DISABLE_XET=1) for clear progress, but that
|
||||
path is single-stream and slow — this restores parallel byte-range speed AND
|
||||
real live progress, and falls back to snapshot_download on any error so it
|
||||
can never compromise a correct install. Default-on so first-run downloads are
|
||||
fast out of the box (pairs with an HF token for higher rate limits); set
|
||||
OMNIVOICE_SEGMENTED_DOWNLOAD=0 to force the single-stream path."""
|
||||
return _truthy(prefs.resolve(
|
||||
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False,
|
||||
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=True,
|
||||
))
|
||||
|
||||
|
||||
@@ -223,51 +238,26 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
|
||||
# model.safetensors" (#352). 5 MB clears every weight format we ship
|
||||
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
|
||||
# config-only aux repos.
|
||||
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024
|
||||
|
||||
# Per-role weight-file floors (MM2-07). A valid model has at least one
|
||||
# recognized weight file at or above its extension's floor. ONNX graphs are
|
||||
# legitimately small (a complete model can be well under 5 MB), so a single
|
||||
# 5 MB rule false-positives on them as "truncated" (#352 over-trigger); give
|
||||
# .onnx a lower floor while still rejecting a 0/KB partial. Tensor formats keep
|
||||
# the original 5 MB floor.
|
||||
_WEIGHT_FLOORS = {
|
||||
".safetensors": _MIN_WEIGHT_BYTES,
|
||||
".bin": _MIN_WEIGHT_BYTES,
|
||||
".ckpt": _MIN_WEIGHT_BYTES,
|
||||
".pt": _MIN_WEIGHT_BYTES,
|
||||
".pth": _MIN_WEIGHT_BYTES,
|
||||
".gguf": _MIN_WEIGHT_BYTES,
|
||||
".onnx": 64 * 1024, # a real ONNX graph is ≥ tens of KB; a truncated one is bytes
|
||||
}
|
||||
|
||||
|
||||
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
|
||||
"""Raise OSError when a finished snapshot has no plausible weight file —
|
||||
surfaces the truncated-download class (#352) at install time, where the
|
||||
retry loop and the UI's re-download path can deal with it, instead of at
|
||||
first synthesis with an opaque transformers error.
|
||||
|
||||
A snapshot is valid if it contains a recognized weight file meeting its
|
||||
per-extension floor (MM2-07) OR any file ≥ the global 5 MB floor (the
|
||||
original lenient catch — kept so this is never stricter than before)."""
|
||||
Delegates the weight check to ``models.snapshot_has_weights`` (single source of
|
||||
the floors); only the install-time error message lives here."""
|
||||
if snapshot_has_weights(snapshot_path):
|
||||
return
|
||||
biggest = 0
|
||||
try:
|
||||
biggest = 0
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
size = os.path.getsize(os.path.join(root, f))
|
||||
biggest = max(biggest, os.path.getsize(os.path.join(root, f)))
|
||||
except OSError:
|
||||
continue
|
||||
biggest = max(biggest, size)
|
||||
ext = os.path.splitext(f)[1].lower()
|
||||
floor = _WEIGHT_FLOORS.get(ext)
|
||||
if floor is not None and size >= floor:
|
||||
return # a recognized weight file of plausible size
|
||||
if size >= _MIN_WEIGHT_BYTES:
|
||||
return # original lenient catch (non-standard weight names)
|
||||
except OSError:
|
||||
return # can't inspect — don't block the install on the checker itself
|
||||
pass
|
||||
raise OSError(
|
||||
f"{repo_id}: download finished but no model weights were found in the "
|
||||
"snapshot (largest file "
|
||||
@@ -415,6 +405,26 @@ async def install_model(req: InstallModelRequest):
|
||||
try:
|
||||
_plan = snapshot_download(**_preflight_kwargs)
|
||||
_summary = compute_plan(_plan)
|
||||
# Disk-space guard (before a single byte flows): the preflight
|
||||
# gives an exact "to download" size, so reject an install that
|
||||
# would overrun the cache volume — with the numbers named —
|
||||
# instead of failing mid-download with a cryptic OSError. No-op
|
||||
# when it fits or the size is unknown. Same on every platform.
|
||||
_disk_err = disk_space_error(_summary["to_download_bytes"])
|
||||
if _disk_err:
|
||||
logger.info("model install %s: rejected — %s", req.repo_id, _disk_err)
|
||||
_resolving.set() # stop the heartbeat thread before we bail
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_error",
|
||||
"error": _disk_err,
|
||||
})
|
||||
# A disk-full is not a transient network failure — don't set
|
||||
# a cooldown (freeing space, not waiting, is the fix). The
|
||||
# outer finally still cleans up the aggregator + context.
|
||||
return
|
||||
download_aggregator.start(
|
||||
req.repo_id,
|
||||
total_bytes=_summary["to_download_bytes"],
|
||||
@@ -449,10 +459,11 @@ async def install_model(req: InstallModelRequest):
|
||||
raise _InstallCancelled()
|
||||
_attempt += 1
|
||||
try:
|
||||
# Opt-in segmented accelerator (FDL-09): parallel byte-range
|
||||
# fetch with real live progress, for the legacy-LFS path.
|
||||
# Any failure falls through to snapshot_download — the
|
||||
# accelerator can never compromise a correct install.
|
||||
# Segmented accelerator (FDL-09, default ON): parallel
|
||||
# byte-range fetch with real live progress, for the
|
||||
# legacy-LFS path. Any failure falls through to
|
||||
# snapshot_download — the accelerator can never compromise a
|
||||
# correct install.
|
||||
_snapshot_path = None
|
||||
if _attempt == 1 and _segmented_enabled() and not _xet_active():
|
||||
try:
|
||||
@@ -517,12 +528,18 @@ async def install_model(req: InstallModelRequest):
|
||||
logger.info("model install failed for %s: %s", req.repo_id, e)
|
||||
import time as _time_fail
|
||||
_install_cooldowns[req.repo_id] = _time_fail.time()
|
||||
# #874: when the install failed because the configured HF mirror is
|
||||
# unreachable, name the mirror + the setting instead of leaking the
|
||||
# raw connectivity error. #959: likewise for the SOCKS-proxy class
|
||||
# (missing socksio fails the download's session construction).
|
||||
# No-op for every other failure.
|
||||
from core.failure import append_hint
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_error",
|
||||
"error": str(e),
|
||||
"error": append_hint(str(e)),
|
||||
})
|
||||
finally:
|
||||
_cancelled.discard(req.repo_id)
|
||||
|
||||
@@ -123,6 +123,66 @@ def hf_cache_dir() -> str:
|
||||
)
|
||||
|
||||
|
||||
# ── Disk-space guard (shared, single-sourced) ──────────────────────────────
|
||||
# MIN_FREE_GB is the headroom we insist on keeping free on the model-cache
|
||||
# volume — the wizard's absolute pre-install floor AND the extra buffer the
|
||||
# per-install check demands on top of the download itself, so an "Install all"
|
||||
# can't fill the disk to the brim (setup/download.py). Lives here — the lowest
|
||||
# module in the setup import graph — so the wizard, the /models header, and the
|
||||
# install endpoint can't drift apart (mirrors the weight-floor single-sourcing).
|
||||
_GIB = 1024 ** 3
|
||||
MIN_FREE_GB = 10
|
||||
|
||||
|
||||
def disk_free_bytes(path: "str | None" = None) -> int:
|
||||
"""Free bytes on the volume backing *path* (defaults to the HF cache).
|
||||
|
||||
Walks up to the nearest existing ancestor so a not-yet-created cache dir
|
||||
still probes the correct mount point. ``shutil.disk_usage`` is cross-platform
|
||||
(macOS/Windows/Linux) so this behaves identically everywhere. Never raises.
|
||||
"""
|
||||
import shutil
|
||||
try:
|
||||
p = Path(path or hf_cache_dir()).resolve()
|
||||
while not p.exists():
|
||||
parent = p.parent
|
||||
if parent == p: # reached the volume root
|
||||
break
|
||||
p = parent
|
||||
return int(shutil.disk_usage(str(p)).free)
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def disk_space_error(to_download_bytes: "int | None", *, cache_dir: "str | None" = None) -> "str | None":
|
||||
"""Actionable message when *to_download_bytes* (+ MIN_FREE_GB headroom) won't
|
||||
fit on the cache volume; ``None`` when it fits, the size is unknown, or the
|
||||
volume can't be probed (never block on missing information).
|
||||
|
||||
Names the three numbers a user needs to act — needs X, headroom Y, have Z —
|
||||
so "Install all" can't silently overrun the disk (issue: no pre-install disk
|
||||
check). Platform-agnostic; applied identically on macOS/Windows/Linux.
|
||||
"""
|
||||
if not to_download_bytes or to_download_bytes <= 0:
|
||||
return None # unknown plan (older/gated repo, mirror without dry-run) → don't block
|
||||
cache = cache_dir or hf_cache_dir()
|
||||
free = disk_free_bytes(cache)
|
||||
if free <= 0:
|
||||
return None # couldn't probe the volume → don't block on missing info
|
||||
required = int(to_download_bytes) + MIN_FREE_GB * _GIB
|
||||
if free >= required:
|
||||
return None
|
||||
|
||||
def _gb(n: int) -> str:
|
||||
return f"{n / _GIB:.1f} GB"
|
||||
|
||||
return (
|
||||
f"Not enough disk space to install: this download needs {_gb(int(to_download_bytes))} "
|
||||
f"plus {MIN_FREE_GB} GB free headroom ({_gb(required)} total), but only {_gb(free)} "
|
||||
f"is free at {cache}. Free up space (or move the model cache to a bigger volume) and retry."
|
||||
)
|
||||
|
||||
|
||||
def _repo_dir_name(repo_id: str) -> str:
|
||||
"""HF cache dir name for a repo: 'k2-fsa/OmniVoice' → 'models--k2-fsa--OmniVoice'."""
|
||||
return "models--" + repo_id.replace("/", "--")
|
||||
@@ -146,6 +206,94 @@ def _hub_cache_roots() -> list[str]:
|
||||
return roots
|
||||
|
||||
|
||||
# ── Weight-presence (truncated-cache) detection ─────────────────────────────
|
||||
# A cache that downloaded config/tokenizer files but not the weight shard still
|
||||
# occupies bytes on disk, so a size-only "installed" check (#352/#581/#606) reads
|
||||
# it as installed and the first-run wizard hides the re-download button, stranding
|
||||
# the user (#622). These helpers tell a *complete* snapshot from a truncated one by
|
||||
# checking for a plausible weight file — the same class `download.py` guards at
|
||||
# install time and `model_manager.py` repairs at load time. Shared here (the lowest
|
||||
# module in the setup import graph; `download.py` imports from this module) so the
|
||||
# floors live in exactly one place and can't drift between the three call sites.
|
||||
|
||||
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024 # tensor formats: a real shard is ≥ a few MB
|
||||
|
||||
# Per-extension floors. ONNX graphs are legitimately small (a complete model can be
|
||||
# well under 5 MB), so they get a lower floor that still rejects a bytes-only partial.
|
||||
_WEIGHT_FLOORS = {
|
||||
".safetensors": _MIN_WEIGHT_BYTES,
|
||||
".bin": _MIN_WEIGHT_BYTES,
|
||||
".ckpt": _MIN_WEIGHT_BYTES,
|
||||
".pt": _MIN_WEIGHT_BYTES,
|
||||
".pth": _MIN_WEIGHT_BYTES,
|
||||
".gguf": _MIN_WEIGHT_BYTES,
|
||||
".onnx": 64 * 1024,
|
||||
}
|
||||
|
||||
|
||||
def snapshot_has_weights(snapshot_path: str) -> bool:
|
||||
"""True when a finished snapshot dir holds a plausible weight file.
|
||||
|
||||
A snapshot is complete if it contains a recognized weight file meeting its
|
||||
per-extension floor OR any file ≥ the global 5 MB floor (the lenient catch for
|
||||
non-standard weight names). Returns True when the path can't be inspected — an
|
||||
un-walkable dir must never be reported as truncated, only a confirmed weight-less
|
||||
one. `getsize` follows symlinks, so HF's snapshot→blob links resolve correctly;
|
||||
a broken link (missing blob) raises OSError and is skipped, i.e. counts as absent.
|
||||
"""
|
||||
try:
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
size = os.path.getsize(os.path.join(root, f))
|
||||
except OSError:
|
||||
continue
|
||||
ext = os.path.splitext(f)[1].lower()
|
||||
floor = _WEIGHT_FLOORS.get(ext)
|
||||
if floor is not None and size >= floor:
|
||||
return True
|
||||
if size >= _MIN_WEIGHT_BYTES:
|
||||
return True
|
||||
except OSError:
|
||||
return True # can't inspect — don't mislabel as truncated
|
||||
return False
|
||||
|
||||
|
||||
def _snapshot_dirs(repo_id: str) -> list[str]:
|
||||
"""Existing snapshot revision dirs for a repo across the candidate cache roots."""
|
||||
name = _repo_dir_name(repo_id)
|
||||
dirs: list[str] = []
|
||||
for root in _hub_cache_roots():
|
||||
snaps = os.path.join(root, name, "snapshots")
|
||||
try:
|
||||
for rev in os.listdir(snaps):
|
||||
rev_dir = os.path.join(snaps, rev)
|
||||
if os.path.isdir(rev_dir):
|
||||
dirs.append(rev_dir)
|
||||
except OSError:
|
||||
continue
|
||||
return dirs
|
||||
|
||||
|
||||
def cache_is_complete(model: dict) -> bool:
|
||||
"""True when this model's on-disk cache is usable (not a truncated download).
|
||||
|
||||
Config-only repos (``config_only: true`` in models.yaml — e.g. pyannote's
|
||||
diarisation pipeline, whose real weights live in referenced sub-repos) carry no
|
||||
weight file of their own, so the weight check would false-positive them as
|
||||
incomplete (#622 caveat). They're exempt: cache presence alone means complete.
|
||||
A weight-bearing repo is complete only if at least one of its snapshots has
|
||||
weights; if no snapshot dir is found on disk we can't prove truncation, so we
|
||||
don't downgrade (the size-based caller already decided it's cached).
|
||||
"""
|
||||
if model.get("config_only"):
|
||||
return True
|
||||
dirs = _snapshot_dirs(model["repo_id"])
|
||||
if not dirs:
|
||||
return True
|
||||
return any(snapshot_has_weights(d) for d in dirs)
|
||||
|
||||
|
||||
def _is_cached_on_disk(repo_id: str) -> bool:
|
||||
"""Direct-filesystem fallback for is_cached when scan_cache_dir is unavailable.
|
||||
|
||||
@@ -286,9 +434,15 @@ def list_models():
|
||||
out = []
|
||||
for m in KNOWN_MODELS:
|
||||
cached = cached_by_repo.get(m["repo_id"])
|
||||
on_disk = cached is not None and cached["size_on_disk"] > 0
|
||||
# A size-positive cache can still be a truncated download (config landed,
|
||||
# weight shard didn't). Treat that as not-installed + incomplete so the
|
||||
# wizard re-offers the download instead of stranding the user (#622).
|
||||
incomplete = on_disk and not cache_is_complete(m)
|
||||
out.append({
|
||||
**m,
|
||||
"installed": cached is not None and cached["size_on_disk"] > 0,
|
||||
"installed": on_disk and not incomplete,
|
||||
"incomplete": incomplete,
|
||||
"size_on_disk_bytes": cached["size_on_disk"] if cached else 0,
|
||||
"nb_files": cached["nb_files"] if cached else 0,
|
||||
"supported": _model_supported(m),
|
||||
@@ -297,6 +451,10 @@ def list_models():
|
||||
"models": out,
|
||||
"total_installed_bytes": sum(m["size_on_disk_bytes"] for m in out),
|
||||
"hf_cache_dir": hf_cache_dir(),
|
||||
# Free space on the cache volume, so the Model Store header can warn
|
||||
# BEFORE an "Install all" overruns the disk (pairs with the per-install
|
||||
# disk_space_error guard in setup/download.py).
|
||||
"disk_free_gb": round(disk_free_bytes() / _GIB, 1),
|
||||
"platform_tags": _current_platform_tags(),
|
||||
}
|
||||
_set_cache("models", response)
|
||||
@@ -383,6 +541,9 @@ def recommendations():
|
||||
entries = []
|
||||
for rid in recommended_ids:
|
||||
meta = known_by_id.get(rid, {})
|
||||
# Mirror /models: a truncated cache (weights missing) is not installed, so
|
||||
# the wizard counts it toward the remaining download instead of "all set".
|
||||
installed = rid in cached_ids and cache_is_complete(meta or {"repo_id": rid})
|
||||
entries.append({
|
||||
"repo_id": rid,
|
||||
"label": meta.get("label", rid),
|
||||
@@ -390,7 +551,7 @@ def recommendations():
|
||||
"size_gb": meta.get("size_gb", 0),
|
||||
"required": bool(meta.get("required", False)),
|
||||
"note": meta.get("note"),
|
||||
"installed": rid in cached_ids,
|
||||
"installed": installed,
|
||||
})
|
||||
|
||||
to_download_gb = sum(e["size_gb"] for e in entries if not e["installed"])
|
||||
|
||||
@@ -18,33 +18,20 @@ import sys
|
||||
from fastapi import APIRouter
|
||||
|
||||
from api.schemas import SetupStatusResponse, PreflightResponse
|
||||
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached
|
||||
# MIN_FREE_GB + disk_free_bytes are single-sourced in ``.models`` (the lowest
|
||||
# module in the setup import graph) so the wizard gate, the /models header, and
|
||||
# the per-install disk guard can't drift apart.
|
||||
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached, MIN_FREE_GB, disk_free_bytes
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.wizard")
|
||||
router = APIRouter()
|
||||
|
||||
MIN_FREE_GB = 10
|
||||
|
||||
|
||||
def _disk_free_gb(path: str) -> float:
|
||||
"""Return free GB on the volume containing *path*.
|
||||
|
||||
If *path* doesn't exist yet (e.g. after a fresh wipe), walk up to the
|
||||
nearest existing ancestor so ``shutil.disk_usage`` can still probe the
|
||||
correct mount point.
|
||||
"""
|
||||
try:
|
||||
from pathlib import Path
|
||||
p = Path(path).resolve()
|
||||
# Walk up until we find a directory that exists
|
||||
while not p.exists():
|
||||
parent = p.parent
|
||||
if parent == p: # root
|
||||
break
|
||||
p = parent
|
||||
return _shutil.disk_usage(str(p)).free / (1024 ** 3)
|
||||
except Exception:
|
||||
return 0.0
|
||||
"""Free GB on the volume containing *path* (thin GB wrapper over the shared
|
||||
``models.disk_free_bytes``, which walks up to the nearest existing ancestor
|
||||
for a not-yet-created path)."""
|
||||
return disk_free_bytes(path) / (1024 ** 3)
|
||||
|
||||
|
||||
# ── Setup Status ───────────────────────────────────────────────────────────
|
||||
@@ -181,16 +168,40 @@ def _detect_gpu() -> dict:
|
||||
return info
|
||||
|
||||
|
||||
def _probe_network(host: str = "huggingface.co", timeout: float = 2.0) -> bool:
|
||||
def _probe_network(host: str = "huggingface.co", port: int = 443, timeout: float = 2.0) -> bool:
|
||||
"""Tiny TCP connect test."""
|
||||
import socket
|
||||
try:
|
||||
with socket.create_connection((host, 443), timeout=timeout):
|
||||
with socket.create_connection((host, port), timeout=timeout):
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _hf_endpoint_host() -> tuple[str, int]:
|
||||
"""Host/port of the Hugging Face endpoint actually in effect.
|
||||
|
||||
Mirror-aware: restricted-network users (e.g. behind the Great Firewall)
|
||||
point HF_ENDPOINT at a mirror via Settings → Models → Hugging Face
|
||||
mirror. Probing hardcoded huggingface.co would fail them even when their
|
||||
configured mirror works fine.
|
||||
"""
|
||||
try:
|
||||
from core.failure import configured_hf_mirror
|
||||
mirror = configured_hf_mirror()
|
||||
except Exception:
|
||||
mirror = ""
|
||||
if mirror:
|
||||
try:
|
||||
from urllib.parse import urlsplit
|
||||
u = urlsplit(mirror)
|
||||
if u.hostname:
|
||||
return u.hostname, u.port or (80 if u.scheme == "http" else 443)
|
||||
except Exception:
|
||||
pass
|
||||
return "huggingface.co", 443
|
||||
|
||||
|
||||
def _ram_gb() -> float:
|
||||
try:
|
||||
import psutil
|
||||
@@ -413,15 +424,49 @@ def preflight():
|
||||
"status": r_status, "detail": r_detail, "fix": r_fix,
|
||||
})
|
||||
|
||||
# ── Network
|
||||
net_ok = _probe_network()
|
||||
# ── Network — probes the HF endpoint actually in effect (mirror-aware),
|
||||
# and a dead network is a WARNING, not a blocker. The app is local-first:
|
||||
# already-downloaded models work offline, and a hard fail here dead-ends
|
||||
# restricted-network users (e.g. China, where huggingface.co is blocked)
|
||||
# on the very first screen — before they can reach the mirror setting
|
||||
# that fixes it. Model downloads surface their own actionable errors.
|
||||
net_host, net_port = _hf_endpoint_host()
|
||||
net_ok = _probe_network(net_host, net_port)
|
||||
mirror_reachable = False
|
||||
if not net_ok and net_host == "huggingface.co":
|
||||
# Official endpoint blocked — if the community mirror is reachable,
|
||||
# tell the user exactly which switch unblocks them.
|
||||
mirror_reachable = _probe_network("hf-mirror.com")
|
||||
if net_ok:
|
||||
net_fix = None
|
||||
elif mirror_reachable:
|
||||
net_fix = (
|
||||
"huggingface.co is blocked on this network, but the hf-mirror.com "
|
||||
"community mirror is reachable — apply it below and re-check. "
|
||||
"Model downloads will use the mirror immediately."
|
||||
)
|
||||
elif net_host != "huggingface.co":
|
||||
net_fix = (
|
||||
f"Your configured Hugging Face mirror ({net_host}) is unreachable "
|
||||
"— it may be down or blocked. Pick another mirror or the official "
|
||||
"endpoint below, or continue offline: models already downloaded "
|
||||
"keep working."
|
||||
)
|
||||
else:
|
||||
net_fix = (
|
||||
"Check internet connection, VPN, or corporate firewall whitelist "
|
||||
"for huggingface.co. You can continue — models already downloaded "
|
||||
"keep working offline; new downloads need a connection or a "
|
||||
"mirror (configurable below)."
|
||||
)
|
||||
checks.append({
|
||||
"id": "network", "label": "Network (huggingface.co)",
|
||||
"status": "pass" if net_ok else "fail",
|
||||
"detail": "Reachable" if net_ok else "Unreachable on port 443",
|
||||
"fix": None if net_ok else
|
||||
"Check internet connection, VPN, or corporate firewall "
|
||||
"whitelist for huggingface.co.",
|
||||
"id": "network", "label": f"Network ({net_host})",
|
||||
"status": "pass" if net_ok else "warn",
|
||||
"detail": "Reachable" if net_ok else f"Unreachable on port {net_port}",
|
||||
"fix": net_fix,
|
||||
# Frontend affordance hint: the wizard offers the mirror quick-pick
|
||||
# when the endpoint is unreachable (PreflightCheck allows extras).
|
||||
"mirror_reachable": mirror_reachable,
|
||||
})
|
||||
|
||||
# Aggregate
|
||||
|
||||
@@ -18,7 +18,7 @@ import shutil
|
||||
|
||||
from core.config import OUTPUTS_DIR, DATA_DIR, CRASH_LOG_PATH, LOG_PATH, IDLE_TIMEOUT_SECONDS
|
||||
from core.version import APP_VERSION
|
||||
from services.model_manager import get_model_status, get_best_device
|
||||
from services.model_manager import get_model_status, get_best_device, resolve_omnivoice_checkpoint
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
|
||||
# Router-level loopback gate. Every route mounted on `router` (GET + POST,
|
||||
@@ -208,7 +208,7 @@ def system_info():
|
||||
"outputs_dir": OUTPUTS_DIR,
|
||||
"crash_log_path": CRASH_LOG_PATH,
|
||||
"idle_timeout_seconds": IDLE_TIMEOUT_SECONDS,
|
||||
"model_checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"model_checkpoint": resolve_omnivoice_checkpoint(), # #693: show the effective checkpoint, not a leaked raw value
|
||||
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
|
||||
"has_hf_token": _has_hf_token(),
|
||||
|
||||
@@ -77,6 +77,10 @@ async def probe(req: ProbeReq):
|
||||
class IncrementalReq(BaseModel):
|
||||
segments: list[dict]
|
||||
stored_hashes: Optional[dict[str, str]] = None
|
||||
# P1.3 — the ACTIVE track's language code. When set, fingerprints are
|
||||
# scoped to that language (pass that language's stored hashes alongside);
|
||||
# omitted → legacy language-agnostic hashing, kept for old callers.
|
||||
lang: Optional[str] = None
|
||||
|
||||
|
||||
@router.post("/tools/incremental")
|
||||
@@ -84,6 +88,7 @@ def plan_incremental(req: IncrementalReq):
|
||||
return incremental.plan_incremental(
|
||||
req.segments,
|
||||
stored_hashes=req.stored_hashes or {},
|
||||
track_lang=req.lang,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -182,8 +182,8 @@ async def ws_tts(websocket: WebSocket):
|
||||
sentences = [text]
|
||||
|
||||
# Run generation in the GPU pool
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
import functools
|
||||
from services.model_manager import run_on_gpu_pool_guarded
|
||||
|
||||
def _generate(sentence_text):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
@@ -204,8 +204,13 @@ async def ws_tts(websocket: WebSocket):
|
||||
started = False
|
||||
|
||||
for sentence in sentences:
|
||||
wav_tensor, sr = await loop.run_in_executor(
|
||||
_gpu_pool, _generate, sentence
|
||||
# Bounded + pool-reset on hang so a wedged generate can't
|
||||
# starve the GPU pool and brick the backend (#730 class). On
|
||||
# timeout GpuJobTimeoutError propagates to the handler below,
|
||||
# which sends an actionable error frame.
|
||||
wav_tensor, sr = await run_on_gpu_pool_guarded(
|
||||
functools.partial(_generate, sentence),
|
||||
what="TTS generate",
|
||||
)
|
||||
|
||||
if not started:
|
||||
|
||||
Binary file not shown.
@@ -14,6 +14,10 @@
|
||||
# required (optional) — true if the app needs this model to function
|
||||
# platforms (optional) — restrict to specific OS+arch tags (e.g. darwin-arm64, cuda)
|
||||
# note (optional) — shown in the UI as a tooltip/footnote
|
||||
# config_only (optional) — true for pipeline repos that ship no weight file of
|
||||
# their own (weights live in referenced sub-repos). Such
|
||||
# a cache is legitimately tiny, so the truncated-download
|
||||
# (weights-missing) detector must NOT flag it incomplete.
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
models:
|
||||
@@ -116,12 +120,83 @@ models:
|
||||
size_gb: 0.05
|
||||
note: "Smallest/fastest Moonshine, sub-200ms latency. Lower accuracy than base. Requires moonshine-onnx."
|
||||
|
||||
# ── sherpa-onnx live dictation (ONNX, CPU, streaming + offline) ────────
|
||||
# Live faster-than-real-time dictation via the k2-fsa/sherpa-onnx runtime.
|
||||
# `engine: sherpa-onnx`, `dictation_id` (backend model id), and `tag`
|
||||
# (offline | streaming) are extra fields the model-store list passes through
|
||||
# so the dictation UI can filter/group these (role=ASR, engine=sherpa-onnx).
|
||||
# Requires `uv add sherpa-onnx` (CPU wheels, all platforms).
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
|
||||
label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)"
|
||||
role: ASR
|
||||
size_gb: 0.18
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-parakeet-tdt-v3
|
||||
tag: offline
|
||||
note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"
|
||||
label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)"
|
||||
role: ASR
|
||||
size_gb: 0.17
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-parakeet-tdt-v2
|
||||
tag: offline
|
||||
note: "English live dictation. CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20"
|
||||
label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)"
|
||||
role: ASR
|
||||
size_gb: 0.13
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-bilingual-zh-en
|
||||
tag: streaming
|
||||
note: "True streaming partials as you speak (zh+en). CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en"
|
||||
label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)"
|
||||
role: ASR
|
||||
size_gb: 0.115
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-paraformer-bilingual-zh-en
|
||||
tag: streaming
|
||||
note: "True streaming partials (zh+en). CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17"
|
||||
label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)"
|
||||
role: ASR
|
||||
size_gb: 0.128
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-en-20m
|
||||
tag: streaming
|
||||
note: "Tiny English streaming model, very low latency. CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23"
|
||||
label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)"
|
||||
role: ASR
|
||||
size_gb: 0.074
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-zh-14m
|
||||
tag: streaming
|
||||
note: "Tiny Chinese streaming model, very low latency. CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-whisper-tiny"
|
||||
label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)"
|
||||
role: ASR
|
||||
size_gb: 0.116
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-whisper-tiny
|
||||
tag: offline
|
||||
note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
# ── Diarisation ───────────────────────────────────────────────────────
|
||||
|
||||
- repo_id: "pyannote/speaker-diarization-3.1"
|
||||
label: "pyannote speaker diarisation (multi-speaker videos)"
|
||||
role: Diarisation
|
||||
size_gb: 0.8
|
||||
config_only: true # pipeline repo; real weights live in referenced sub-repos
|
||||
note: "Needs an HF_TOKEN with license accepted."
|
||||
|
||||
# ── Optional TTS ──────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
"""Parse the shipped CHANGELOG.md into structured release notes.
|
||||
|
||||
Feeds ``GET /api/settings/changelog`` — the Settings → Updates "What's new"
|
||||
viewer. Local-first by design: the changelog ships with the app (repo root in
|
||||
dev; copied into the packaged project dir by the Tauri bootstrap alongside
|
||||
README.md), so the viewer works fully offline.
|
||||
|
||||
The house format (see CHANGELOG.md / the release-notes hard rule):
|
||||
|
||||
## [X.Y.Z] — DATE
|
||||
one-paragraph headline (the "intro")
|
||||
### Added / Fixed / Changed / ...
|
||||
- **Bold one-line lead.** 1-3 lines of plain-English why. (#NNN)
|
||||
|
||||
Bullets may be a single long line (recent sections) *or* hard-wrapped across
|
||||
indented continuation lines (older sections) — the parser normalizes both to
|
||||
one logical line per bullet. Bullets stay raw markdown-lite; the frontend's
|
||||
safe renderer handles **bold** / `code` / (#NNN) refs.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
|
||||
#: ``## [0.3.9] — 2026-07-02`` (em/en dash or hyphen; date optional).
|
||||
_RELEASE_RE = re.compile(r"^##\s+\[(?P<version>[^\]]+)\]\s*(?:[—–-]\s*(?P<date>.+?))?\s*$")
|
||||
_SECTION_RE = re.compile(r"^###\s+(?P<title>.+?)\s*$")
|
||||
_BULLET_RE = re.compile(r"^\s*[-*]\s+(?P<text>.*\S)\s*$")
|
||||
|
||||
|
||||
def changelog_path() -> str | None:
|
||||
"""The shipped CHANGELOG.md, or None when this install doesn't have one.
|
||||
|
||||
``backend/core/changelog.py`` → two levels up is the project root: the
|
||||
repo root in dev, and ``<env>/project`` in packaged installs (where the
|
||||
bootstrap copies CHANGELOG.md next to README.md). ``OMNIVOICE_CHANGELOG``
|
||||
overrides for tests/containers.
|
||||
"""
|
||||
override = os.environ.get("OMNIVOICE_CHANGELOG")
|
||||
if override:
|
||||
return override if os.path.isfile(override) else None
|
||||
here = os.path.dirname(os.path.abspath(__file__))
|
||||
candidate = os.path.join(os.path.dirname(os.path.dirname(here)), "CHANGELOG.md")
|
||||
return candidate if os.path.isfile(candidate) else None
|
||||
|
||||
|
||||
def _looks_like_release_version(version: str) -> bool:
|
||||
"""Only released ``X.Y.Z...`` sections (skip ``[Unreleased]`` etc.)."""
|
||||
return bool(re.match(r"^v?\d", version.strip()))
|
||||
|
||||
|
||||
def parse_changelog(text: str, limit_versions: int = 5) -> list[dict]:
|
||||
"""CHANGELOG.md text → newest-first list of releases::
|
||||
|
||||
{"version": "0.3.9", "date": "2026-07-02", "intro": "…",
|
||||
"sections": [{"title": "Fixed", "bullets": ["…", …]}, …]}
|
||||
|
||||
Tolerates both single-line bullets and older hard-wrapped bullets
|
||||
(continuation lines are joined with a space). Content between the version
|
||||
heading and the first ``###`` becomes ``intro`` (paragraphs joined by
|
||||
blank lines).
|
||||
"""
|
||||
releases: list[dict] = []
|
||||
release: dict | None = None
|
||||
section: dict | None = None
|
||||
intro_parts: list[str] = []
|
||||
bullet_open = False # last bullet may still absorb continuation lines
|
||||
intro_new_para = True
|
||||
|
||||
def close_release():
|
||||
nonlocal release, section, intro_parts, bullet_open, intro_new_para
|
||||
if release is not None:
|
||||
release["intro"] = "\n\n".join(p for p in intro_parts if p)
|
||||
release["sections"] = [s for s in release["sections"] if s["bullets"]]
|
||||
releases.append(release)
|
||||
release = None
|
||||
section = None
|
||||
intro_parts = []
|
||||
bullet_open = False
|
||||
intro_new_para = True
|
||||
|
||||
for raw in text.splitlines():
|
||||
m = _RELEASE_RE.match(raw)
|
||||
if m:
|
||||
close_release()
|
||||
if len(releases) >= limit_versions:
|
||||
break
|
||||
version = m.group("version").strip().lstrip("v")
|
||||
if not _looks_like_release_version(version):
|
||||
continue # e.g. [Unreleased] — skip until the next heading
|
||||
release = {
|
||||
"version": version,
|
||||
"date": (m.group("date") or "").strip(),
|
||||
"intro": "",
|
||||
"sections": [],
|
||||
}
|
||||
continue
|
||||
if release is None:
|
||||
continue
|
||||
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
bullet_open = False
|
||||
intro_new_para = True
|
||||
continue
|
||||
|
||||
sm = _SECTION_RE.match(raw)
|
||||
if sm:
|
||||
section = {"title": sm.group("title"), "bullets": []}
|
||||
release["sections"].append(section)
|
||||
bullet_open = False
|
||||
continue
|
||||
|
||||
bm = _BULLET_RE.match(raw)
|
||||
if bm:
|
||||
if section is None:
|
||||
# Rare: a bullet before any ### heading — group it untitled.
|
||||
section = {"title": "", "bullets": []}
|
||||
release["sections"].append(section)
|
||||
section["bullets"].append(bm.group("text"))
|
||||
bullet_open = True
|
||||
continue
|
||||
|
||||
if section is not None:
|
||||
if bullet_open and section["bullets"]:
|
||||
# Hard-wrapped bullet continuation (older sections) → join.
|
||||
section["bullets"][-1] += " " + line
|
||||
continue
|
||||
|
||||
# Headline paragraph(s) before the first ### section.
|
||||
if intro_new_para or not intro_parts:
|
||||
intro_parts.append(line)
|
||||
else:
|
||||
intro_parts[-1] += " " + line
|
||||
intro_new_para = False
|
||||
|
||||
close_release()
|
||||
return releases[:limit_versions]
|
||||
+227
-8
@@ -3,6 +3,8 @@ import sqlite3
|
||||
import logging
|
||||
from contextlib import contextmanager
|
||||
from core.config import DB_PATH
|
||||
from core import db_backup
|
||||
from core.version import APP_VERSION
|
||||
|
||||
logger = logging.getLogger("omnivoice.db")
|
||||
|
||||
@@ -157,6 +159,22 @@ _BASE_SCHEMA = """
|
||||
last_seen_at REAL,
|
||||
created_at REAL
|
||||
);
|
||||
|
||||
-- Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary. A
|
||||
-- per-language word→respelling map applied as pure text substitution
|
||||
-- before synthesis (Settings → Pronunciation). Fresh installs create it
|
||||
-- here; existing DBs get it via alembic 0008_pronunciation_dictionary.
|
||||
-- Both paths converge on this identical schema (dual-path discipline).
|
||||
CREATE TABLE IF NOT EXISTS pronunciation_entries (
|
||||
id TEXT PRIMARY KEY,
|
||||
term TEXT NOT NULL,
|
||||
replacement TEXT NOT NULL DEFAULT '',
|
||||
type TEXT NOT NULL DEFAULT 'respelling',
|
||||
language TEXT NOT NULL DEFAULT '*',
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
created_at REAL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_pron_lang ON pronunciation_entries(language);
|
||||
"""
|
||||
|
||||
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
|
||||
@@ -206,6 +224,72 @@ def _migrate(conn, current: int) -> int:
|
||||
return current
|
||||
|
||||
|
||||
def _reconcile_additive_columns(conn) -> None:
|
||||
"""Make the live schema converge to ``_BASE_SCHEMA`` by ADDing any column the
|
||||
canonical schema declares but an existing table is missing — the belt for
|
||||
when alembic can't run on an upgraded DB.
|
||||
|
||||
``CREATE TABLE IF NOT EXISTS`` (init_db) never adds columns to a table that
|
||||
already exists, the legacy ``_migrate`` only knows pre-0.3 columns, and
|
||||
``_run_alembic_upgrade`` swallows failures. So a DB whose ``alembic_version``
|
||||
is stamped at a removed revision (e.g. after running a preview build), or
|
||||
where alembic isn't importable in the bundled interpreter, would otherwise
|
||||
lose every alembic-era additive column forever — the ``no such column:
|
||||
consent_audio_path`` 500 (#552/#547), and the same class for
|
||||
``kind``/``vd_states``/``is_demo``/.... Additive only: never drops or retypes
|
||||
a column, so it is safe and backward-compatible with existing user data. The
|
||||
canonical names/types/defaults come solely from ``_BASE_SCHEMA`` (developer
|
||||
controlled), so the ALTER is injection-safe.
|
||||
"""
|
||||
canon = sqlite3.connect(":memory:")
|
||||
try:
|
||||
canon.executescript(_BASE_SCHEMA)
|
||||
_tables_sql = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
|
||||
live_tables = {r[0] for r in conn.execute(_tables_sql)}
|
||||
for table in (r[0] for r in canon.execute(_tables_sql)):
|
||||
if table not in live_tables:
|
||||
continue # whole table missing → init_db's CREATE already made it
|
||||
have = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
|
||||
# (cid, name, type, notnull, dflt_value, pk)
|
||||
for _cid, name, ctype, notnull, dflt, _pk in canon.execute(f"PRAGMA table_info({table})"):
|
||||
if name in have or not _IDENT_RE.match(name):
|
||||
continue
|
||||
ddl = f'ALTER TABLE "{table}" ADD COLUMN "{name}" {ctype or "TEXT"}'
|
||||
if dflt is not None:
|
||||
ddl += f" DEFAULT {dflt}"
|
||||
elif notnull:
|
||||
ddl += " DEFAULT ''" # SQLite requires a default to ADD a NOT NULL column
|
||||
try:
|
||||
conn.execute(ddl)
|
||||
logger.info("schema reconcile: added missing column %s.%s", table, name)
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "duplicate column" not in str(exc).lower():
|
||||
logger.warning("schema reconcile ALTER %s.%s failed: %s", table, name, exc)
|
||||
conn.commit()
|
||||
finally:
|
||||
canon.close()
|
||||
|
||||
|
||||
def ensure_schema() -> None:
|
||||
"""Idempotently ensure the base tables + additive columns exist.
|
||||
|
||||
A runtime self-heal for a DB that somehow missed init — e.g. a write hitting
|
||||
``no such table: generation_history`` (#710) because ``init_db()``'s
|
||||
``executescript`` never took on that DB. Safe to call anytime: it's just
|
||||
``CREATE ... IF NOT EXISTS`` plus the additive-only column reconcile, so it
|
||||
never drops or retypes anything and is backward-compatible with user data.
|
||||
Cheaper than ``init_db()`` (skips the legacy ``_migrate`` + alembic), so a
|
||||
write path can call it on a schema error and retry without a 500.
|
||||
"""
|
||||
conn = get_db()
|
||||
try:
|
||||
conn.executescript(_BASE_SCHEMA)
|
||||
_reconcile_additive_columns(conn)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def init_db():
|
||||
conn = get_db()
|
||||
try:
|
||||
@@ -214,6 +298,11 @@ def init_db():
|
||||
new_version = _migrate(conn, version)
|
||||
if new_version != version:
|
||||
conn.execute(f"PRAGMA user_version = {new_version}")
|
||||
# Converge any alembic-era additive columns that CREATE TABLE IF NOT
|
||||
# EXISTS + the legacy _migrate don't add to a pre-existing table
|
||||
# (consent_audio_path, kind, ...). Runs regardless of whether alembic
|
||||
# below succeeds, so an unrunnable alembic can't leave a 500-ing schema.
|
||||
_reconcile_additive_columns(conn)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -225,12 +314,88 @@ def init_db():
|
||||
_run_alembic_upgrade()
|
||||
|
||||
|
||||
class MigrationError(RuntimeError):
|
||||
"""A schema migration failed *while executing*. Startup must NOT continue
|
||||
on a possibly half-migrated database — the caller lets this propagate so
|
||||
the process stops with an actionable message naming the pre-migration
|
||||
backup (see ``core.db_backup``). Restore is deliberately manual: silently
|
||||
auto-restoring the snapshot could itself discard user data."""
|
||||
|
||||
|
||||
def _reconcile_after_alembic_skip() -> None:
|
||||
"""Converge the schema directly when alembic can't run at all (not
|
||||
importable, or stamped at a removed revision — #552/#547) so additive
|
||||
columns still land instead of 500-ing on `no such column`. Only for the
|
||||
"nothing was applied" classes; a mid-migration failure must NOT reach
|
||||
here (see MigrationError)."""
|
||||
try:
|
||||
conn = get_db()
|
||||
try:
|
||||
_reconcile_additive_columns(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("schema reconcile after alembic skip also failed: %s", exc)
|
||||
|
||||
|
||||
def _stamped_revisions(db_path: str) -> set | None:
|
||||
"""Revisions recorded in ``alembic_version`` (empty set = never stamped),
|
||||
or None when the DB can't be read."""
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
try:
|
||||
return {r[0] for r in conn.execute("SELECT version_num FROM alembic_version")}
|
||||
except sqlite3.OperationalError:
|
||||
return set() # table absent — nothing ever stamped
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
|
||||
|
||||
def _plan_alembic(cfg) -> str:
|
||||
"""Decide what an ``upgrade head`` run would actually do:
|
||||
|
||||
- ``up_to_date`` — stamped at head; upgrade is a no-op.
|
||||
- ``pending`` — migrations WILL execute (snapshot the DB first).
|
||||
- ``unknown_revision`` — stamped at a revision this build doesn't ship
|
||||
(preview→stable downgrade, #552/#547); upgrade would fail before
|
||||
applying anything, so skip it and reconcile additively instead.
|
||||
- ``indeterminate`` — can't tell; treat like pending (snapshot, run).
|
||||
"""
|
||||
try:
|
||||
from alembic.script import ScriptDirectory
|
||||
|
||||
script = ScriptDirectory.from_config(cfg)
|
||||
known = {rev.revision for rev in script.walk_revisions()}
|
||||
heads = set(script.get_heads())
|
||||
stamped = _stamped_revisions(DB_PATH)
|
||||
if stamped is None:
|
||||
return "indeterminate"
|
||||
if stamped and not stamped <= known:
|
||||
return "unknown_revision"
|
||||
if stamped == heads:
|
||||
return "up_to_date"
|
||||
return "pending"
|
||||
except Exception: # noqa: BLE001
|
||||
return "indeterminate"
|
||||
|
||||
|
||||
def _run_alembic_upgrade() -> None:
|
||||
"""Best-effort `alembic upgrade head` on startup. Non-fatal: if alembic
|
||||
isn't reachable (e.g. user running a stripped-down install or migrations
|
||||
were already applied out-of-band), log a warning and move on. The
|
||||
_BASE_SCHEMA CREATE TABLE IF NOT EXISTS above guarantees the runtime
|
||||
schema is correct regardless."""
|
||||
"""`alembic upgrade head` on startup, wrapped in the data-safety net.
|
||||
|
||||
Failure classes are handled differently on purpose:
|
||||
|
||||
- alembic unavailable / stamped at an unknown revision → **non-fatal**
|
||||
(nothing was applied; warn + `_reconcile_additive_columns` keeps the
|
||||
schema converged, exactly the pre-existing #552/#547 behavior).
|
||||
- migrations actually pending → the DB is snapshotted first
|
||||
(``omnivoice.db.backup-<version>-<n>``, newest 3 kept), then upgraded.
|
||||
- a migration fails **while executing** → raise :class:`MigrationError`:
|
||||
startup stops with a message naming the backup, instead of silently
|
||||
running the app on a half-migrated DB.
|
||||
"""
|
||||
try:
|
||||
import os
|
||||
from alembic import command
|
||||
@@ -246,8 +411,62 @@ def _run_alembic_upgrade() -> None:
|
||||
return
|
||||
cfg = Config(ini)
|
||||
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{DB_PATH}")
|
||||
except Exception as exc: # noqa: BLE001 — alembic not importable / bad ini
|
||||
logger.warning("alembic upgrade head skipped: %s", exc)
|
||||
_reconcile_after_alembic_skip()
|
||||
return
|
||||
|
||||
plan = _plan_alembic(cfg)
|
||||
if plan == "up_to_date":
|
||||
return
|
||||
if plan == "unknown_revision":
|
||||
logger.warning(
|
||||
"alembic_version is stamped at a revision this build doesn't ship "
|
||||
"(preview/newer build ran on this DB) — skipping alembic and "
|
||||
"reconciling the schema additively (#552/#547)"
|
||||
)
|
||||
_reconcile_after_alembic_skip()
|
||||
return
|
||||
|
||||
# Migrations may actually execute: snapshot the DB first so a failed or
|
||||
# interrupted migration can never cost user data. A backup problem alone
|
||||
# must not brick startup (the >500 MB skip is by design), so log and go on.
|
||||
# ``db_backup``/``APP_VERSION`` are module-level imports (top of file), not
|
||||
# re-imported here: a test that patches ``core.db_backup.MAX_BACKUP_DB_BYTES``
|
||||
# on the object it imported at collection must see the same object this
|
||||
# function uses. A lazy ``from core import db_backup`` would re-resolve
|
||||
# through the (possibly re-imported) ``core`` package and silently miss the
|
||||
# patch after another suite purged ``core.*`` from ``sys.modules``.
|
||||
backup_path = None
|
||||
try:
|
||||
backup_path = db_backup.snapshot_before_migration(DB_PATH, APP_VERSION)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("Pre-migration DB backup failed — continuing without one")
|
||||
|
||||
try:
|
||||
command.upgrade(cfg, "head")
|
||||
except Exception as exc:
|
||||
# Don't block startup on a migration tooling problem. The runtime
|
||||
# schema is already correct via _BASE_SCHEMA.
|
||||
logger.warning("alembic upgrade head skipped: %s", exc)
|
||||
if "Can't locate revision" in str(exc):
|
||||
# Belt for an unknown-revision case _plan_alembic missed: alembic
|
||||
# bails before applying anything, so the old non-fatal path is safe.
|
||||
logger.warning("alembic upgrade head skipped: %s", exc)
|
||||
_reconcile_after_alembic_skip()
|
||||
return
|
||||
backup_note = (
|
||||
f"A backup of your data from just before the migration is at: {backup_path}"
|
||||
if backup_path
|
||||
else "No pre-migration backup was written this run (see the log above)"
|
||||
)
|
||||
msg = (
|
||||
f"Database migration failed while running: {exc}. "
|
||||
f"OmniVoice stopped instead of running on a partially migrated database, "
|
||||
f"and nothing was auto-restored (your database at {DB_PATH} was left "
|
||||
f"exactly as the failed migration left it). "
|
||||
f"{backup_note}. "
|
||||
"What to do: relaunch to retry; if it keeps failing, report it at "
|
||||
"https://github.com/debpalash/OmniVoice-Studio/issues (keep the backup file). "
|
||||
"To roll back manually: quit the app, replace omnivoice.db with the backup "
|
||||
"file, and reinstall the previous version."
|
||||
)
|
||||
logger.error(msg)
|
||||
raise MigrationError(msg) from exc
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
"""Pre-migration SQLite safety net (data-safe updates).
|
||||
|
||||
Before ``alembic upgrade head`` applies *pending* migrations at startup —
|
||||
which is exactly the first launch of a new app version that changed the
|
||||
schema — the live database is snapshotted next to itself as
|
||||
``omnivoice.db.backup-<version>-<n>`` so a failed or interrupted migration
|
||||
can never cost user data (voices, projects, history, settings).
|
||||
|
||||
Design rules (owner intent: "never corrupt/erase user data on update"):
|
||||
|
||||
- Snapshots use the SQLite online-backup API (``sqlite3.Connection.backup``),
|
||||
not a file copy — the live DB runs in WAL mode, so a plain copy could miss
|
||||
everything still sitting in ``omnivoice.db-wal``.
|
||||
- Only the most recent ``KEEP_BACKUPS`` snapshots are kept; older ones are
|
||||
pruned so backups can't grow without bound.
|
||||
- DBs larger than ``MAX_BACKUP_DB_BYTES`` are skipped with a log line (a
|
||||
multi-hundred-MB copy on every schema upgrade is worse than the risk it
|
||||
hedges on those installs).
|
||||
- Restore is NEVER automatic. On migration failure the caller
|
||||
(``core.db._run_alembic_upgrade``) stops startup and names the backup path
|
||||
so the user (or a support thread) decides — a silent auto-restore could
|
||||
itself discard data written after the snapshot.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
logger = logging.getLogger("omnivoice.db.backup")
|
||||
|
||||
#: Keep this many snapshots; older ones are pruned after each new snapshot.
|
||||
KEEP_BACKUPS = 3
|
||||
|
||||
#: Skip the snapshot (with a log line) when the DB exceeds this size.
|
||||
MAX_BACKUP_DB_BYTES = 500 * 1024 * 1024
|
||||
|
||||
#: ``<db name>.backup-<version>-<n>`` — ``<version>`` may itself contain
|
||||
#: dashes (preview builds stamp ``0.3.9-41``), so the counter is the final
|
||||
#: ``-<digits>`` group.
|
||||
_BACKUP_SUFFIX_RE = re.compile(r"\.backup-(?P<version>.+)-(?P<n>\d+)$")
|
||||
|
||||
|
||||
def _sanitize_version(version: str) -> str:
|
||||
"""Version string → filesystem-safe fragment (defense in depth; real
|
||||
versions are semver and already safe)."""
|
||||
safe = re.sub(r"[^A-Za-z0-9._-]", "_", str(version).strip()) or "unknown"
|
||||
return safe[:64]
|
||||
|
||||
|
||||
def list_backups(db_path: str) -> list[str]:
|
||||
"""All backup files for ``db_path``, newest first (mtime desc)."""
|
||||
directory = os.path.dirname(os.path.abspath(db_path)) or "."
|
||||
base = os.path.basename(db_path)
|
||||
try:
|
||||
names = os.listdir(directory)
|
||||
except OSError:
|
||||
return []
|
||||
out = []
|
||||
for name in names:
|
||||
if not name.startswith(base + ".backup-"):
|
||||
continue
|
||||
if not _BACKUP_SUFFIX_RE.search(name[len(base):]):
|
||||
continue
|
||||
out.append(os.path.join(directory, name))
|
||||
out.sort(key=lambda p: (_mtime(p), p), reverse=True)
|
||||
return out
|
||||
|
||||
|
||||
def _mtime(path: str) -> float:
|
||||
try:
|
||||
return os.path.getmtime(path)
|
||||
except OSError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def latest_backup(db_path: str) -> dict | None:
|
||||
"""Newest backup as ``{"path", "created_at", "size_bytes"}`` or None."""
|
||||
backups = list_backups(db_path)
|
||||
if not backups:
|
||||
return None
|
||||
path = backups[0]
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
return None
|
||||
return {"path": path, "created_at": st.st_mtime, "size_bytes": st.st_size}
|
||||
|
||||
|
||||
def _next_counter(db_path: str, safe_version: str) -> int:
|
||||
"""Next free ``<n>`` for this version so a re-run never overwrites an
|
||||
earlier snapshot of the same version."""
|
||||
base = os.path.basename(db_path)
|
||||
prefix = f"{base}.backup-{safe_version}-"
|
||||
highest = 0
|
||||
for path in list_backups(db_path):
|
||||
name = os.path.basename(path)
|
||||
if not name.startswith(prefix):
|
||||
continue
|
||||
tail = name[len(prefix):]
|
||||
if tail.isdigit():
|
||||
highest = max(highest, int(tail))
|
||||
return highest + 1
|
||||
|
||||
|
||||
def prune_backups(db_path: str, keep: int = KEEP_BACKUPS) -> list[str]:
|
||||
"""Delete all but the ``keep`` newest backups. Returns deleted paths."""
|
||||
deleted = []
|
||||
for path in list_backups(db_path)[keep:]:
|
||||
try:
|
||||
os.remove(path)
|
||||
deleted.append(path)
|
||||
logger.info("Pruned old DB backup %s", path)
|
||||
except OSError as exc:
|
||||
logger.warning("Could not prune old DB backup %s: %s", path, exc)
|
||||
return deleted
|
||||
|
||||
|
||||
def snapshot_before_migration(db_path: str, version: str) -> str | None:
|
||||
"""Snapshot ``db_path`` to ``<db>.backup-<version>-<n>``.
|
||||
|
||||
Returns the backup path, or None when skipped (no DB yet, or DB larger
|
||||
than ``MAX_BACKUP_DB_BYTES``). Raises on an actual backup failure so the
|
||||
caller can decide (the caller treats that as "continue without a backup",
|
||||
logged loudly — a backup problem must not brick startup by itself).
|
||||
"""
|
||||
if not os.path.isfile(db_path):
|
||||
logger.debug("No DB at %s yet — nothing to back up", db_path)
|
||||
return None
|
||||
size = os.path.getsize(db_path)
|
||||
if size > MAX_BACKUP_DB_BYTES:
|
||||
logger.info(
|
||||
"Skipping pre-migration DB backup: %s is %.0f MB (> %.0f MB limit)",
|
||||
db_path, size / (1024 * 1024), MAX_BACKUP_DB_BYTES / (1024 * 1024),
|
||||
)
|
||||
return None
|
||||
|
||||
safe_version = _sanitize_version(version)
|
||||
target = f"{db_path}.backup-{safe_version}-{_next_counter(db_path, safe_version)}"
|
||||
tmp = f"{target}.part-{os.getpid()}"
|
||||
src = sqlite3.connect(db_path)
|
||||
try:
|
||||
dst = sqlite3.connect(tmp)
|
||||
try:
|
||||
# Online backup: consistent snapshot including WAL contents.
|
||||
src.backup(dst)
|
||||
dst.commit()
|
||||
finally:
|
||||
dst.close()
|
||||
except BaseException:
|
||||
try:
|
||||
os.remove(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
finally:
|
||||
src.close()
|
||||
os.replace(tmp, target)
|
||||
# A same-second rotation must still rank the new file newest.
|
||||
try:
|
||||
now = time.time()
|
||||
os.utime(target, (now, now))
|
||||
except OSError:
|
||||
pass
|
||||
logger.info("Pre-migration DB backup written: %s (%.1f MB)", target, size / (1024 * 1024))
|
||||
prune_backups(db_path)
|
||||
return target
|
||||
@@ -76,6 +76,12 @@ _CLASS_RULES: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
"connection refused",
|
||||
"connection reset",
|
||||
"connection aborted",
|
||||
# transformers' download-failure wording ("We couldn't connect to
|
||||
# '<endpoint>' to load the files") — the #874 mirror-down class was
|
||||
# journaled as UNKNOWN without these.
|
||||
"couldn't connect to",
|
||||
"could not connect to",
|
||||
"max retries exceeded",
|
||||
"timed out",
|
||||
"timeout",
|
||||
"name or service not known",
|
||||
|
||||
+252
-1
@@ -21,6 +21,7 @@ import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from core import error_docs_map
|
||||
from core.logging_filter import REDACTED, _HF_TOKEN_RE
|
||||
@@ -37,9 +38,166 @@ _HINTS: dict[str, str] = {
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
|
||||
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
|
||||
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
|
||||
"OS_INVALID_ARGUMENT": "The OS rejected a file operation (Errno 22 / invalid argument) — in the transcribe path this is the temporary WAV write before ASR. It's almost always the temp directory: missing, read-only, on a full or removed drive, or blocked by antivirus. Check that your system TEMP/TMP folder exists and is writable and the drive has free space (add an OmniVoice antivirus exclusion if you use one), then retry.",
|
||||
"SOCKS_PROXY_SUPPORT_MISSING": "A SOCKS proxy is configured in your environment (ALL_PROXY/HTTPS_PROXY=socks5://…) and the backend's HTTP client is missing SOCKS support. Newer OmniVoice builds ship SOCKS support (the socksio package) — update the app. If you still see this, unset ALL_PROXY/HTTPS_PROXY for OmniVoice, or run `uv pip install 'httpx[socks]'` in the backend venv, then restart.",
|
||||
"SSL_HANDSHAKE_FAILURE": "A corporate or antivirus proxy is intercepting HTTPS traffic and re-signing certificates with its own CA — your OS trusts that CA, but Python's bundled certifi CA list doesn't, so the TLS handshake fails even though the connection reached the server. Newer OmniVoice builds trust the OS certificate store at startup (the truststore package), which should already fix this — update the app and retry. If you still see this, add an HTTPS-scanning exclusion for OmniVoice/Python in your antivirus, or ask IT for the proxy's CA bundle and set SSL_CERT_FILE to it, then restart.",
|
||||
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
|
||||
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
|
||||
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
|
||||
# HF_MIRROR_UNREACHABLE has a DYNAMIC hint (it names the configured mirror)
|
||||
# — see hf_mirror_hint(); build_failure special-cases it.
|
||||
}
|
||||
|
||||
|
||||
# ── HF mirror connectivity (#874) ────────────────────────────────────────────
|
||||
# When a non-default HF_ENDPOINT (a mirror, e.g. hf-mirror.com — set via
|
||||
# Settings → Models → Hugging Face mirror) is configured and a model
|
||||
# download/load fails with a connectivity error, the raw transformers/hf_hub
|
||||
# message ("We couldn't connect to 'https://hf-mirror.com' to load the files…")
|
||||
# gives the user no next step. This is the single classifier for that class,
|
||||
# shared by every surface: build_failure() (model status, dub/task events),
|
||||
# the global 500 handler (main.py — covers /generate and every other route
|
||||
# that can leak a model-load error), and the model-install SSE
|
||||
# (setup/download.py).
|
||||
|
||||
_OFFICIAL_HF_ENDPOINTS = {"https://huggingface.co", "https://hf.co"}
|
||||
|
||||
# Connectivity signatures across the layers an HF download failure surfaces
|
||||
# from: transformers' wording, huggingface_hub errors, requests/urllib3, and
|
||||
# raw socket/DNS failures (Linux/macOS/Windows variants).
|
||||
_HF_CONNECTIVITY_SIGNATURES = (
|
||||
"couldn't connect to", # transformers: "We couldn't connect to '<endpoint>' …"
|
||||
"could not connect to",
|
||||
"connection error", # huggingface_hub / requests
|
||||
"connection refused",
|
||||
"connection reset",
|
||||
"connection aborted",
|
||||
"max retries exceeded", # urllib3 via requests
|
||||
"failed to establish a new connection",
|
||||
"name or service not known", # Linux DNS
|
||||
"temporary failure in name resolution",
|
||||
"nodename nor servname provided", # macOS DNS
|
||||
"getaddrinfo failed", # Windows DNS
|
||||
"timed out",
|
||||
"an error happened while trying to locate the file on the hub", # LocalEntryNotFoundError
|
||||
"we cannot find the requested files", # LocalEntryNotFoundError
|
||||
)
|
||||
|
||||
# The failure must also be Hugging-Face-shaped — the configured endpoint/host
|
||||
# named in the message, or HF-download wording — so a random socket error
|
||||
# (e.g. a local LLM provider being down) doesn't get the mirror hint just
|
||||
# because a mirror happens to be configured.
|
||||
_HF_CONTEXT_MARKERS = (
|
||||
"huggingface",
|
||||
"hf_hub",
|
||||
"hf-hub",
|
||||
"load the files", # transformers
|
||||
"cached files", # transformers
|
||||
"the requested files", # LocalEntryNotFoundError
|
||||
"locate the file on the hub",
|
||||
"snapshot_download",
|
||||
)
|
||||
|
||||
|
||||
def configured_hf_mirror() -> str:
|
||||
"""The non-default Hugging Face endpoint (mirror) in effect, or "".
|
||||
|
||||
Same resolution the download paths use: ``HF_ENDPOINT`` env (what
|
||||
Settings → Models → Hugging Face mirror persists via user_env, and what
|
||||
the HF libraries read) with the ``hf_endpoint`` pref as fallback
|
||||
(mirrors setup/download.py's ``prefs.resolve``). Never raises.
|
||||
"""
|
||||
ep = (os.environ.get("HF_ENDPOINT") or "").strip()
|
||||
if not ep:
|
||||
try:
|
||||
from core import prefs
|
||||
|
||||
ep = str(prefs.get("hf_endpoint", "") or "").strip()
|
||||
except Exception:
|
||||
ep = ""
|
||||
ep = ep.rstrip("/")
|
||||
if not ep or ep.lower() in _OFFICIAL_HF_ENDPOINTS:
|
||||
return ""
|
||||
return ep
|
||||
|
||||
|
||||
def hf_mirror_hint(reason: Optional[str]) -> str:
|
||||
"""Actionable hint when ``reason`` is an HF-download connectivity failure
|
||||
and a non-default mirror endpoint is configured; "" otherwise.
|
||||
|
||||
The hint names the configured mirror, says it may be down, points at the
|
||||
setting (Settings → Models → Hugging Face mirror), suggests the official
|
||||
endpoint when the model isn't cached yet, and notes the restart
|
||||
requirement (HF reads HF_ENDPOINT at import time — see the hf-mirror
|
||||
endpoints in api/routers/settings.py). Never raises.
|
||||
"""
|
||||
mirror = configured_hf_mirror()
|
||||
if not mirror:
|
||||
return ""
|
||||
low = (reason or "").lower()
|
||||
if not any(sig in low for sig in _HF_CONNECTIVITY_SIGNATURES):
|
||||
return ""
|
||||
try:
|
||||
host = (urlsplit(mirror).netloc or "").lower()
|
||||
except Exception:
|
||||
host = ""
|
||||
if not (
|
||||
mirror.lower() in low
|
||||
or (host and host in low)
|
||||
or any(m in low for m in _HF_CONTEXT_MARKERS)
|
||||
):
|
||||
return ""
|
||||
return (
|
||||
f"Your Hugging Face mirror is set to {mirror}, which couldn't be "
|
||||
"reached — the mirror may be down or blocked on your network. If the "
|
||||
'model isn\'t in your local cache yet, switch to "Hugging Face '
|
||||
'(official)" in Settings → Models → Hugging Face mirror (or wait for '
|
||||
"the mirror to recover), then restart OmniVoice — the mirror setting "
|
||||
"is applied when the app starts."
|
||||
)
|
||||
|
||||
|
||||
def append_hf_mirror_hint(text: str) -> str:
|
||||
"""``"{text} — {hint}"`` when the mirror-connectivity class applies;
|
||||
``text`` unchanged otherwise. For surfaces that hand a raw error string to
|
||||
the UI (the global 500 handler, the model-install SSE). Never raises."""
|
||||
try:
|
||||
hint = hf_mirror_hint(text)
|
||||
except Exception:
|
||||
return text
|
||||
return f"{text} — {hint}" if hint else text
|
||||
|
||||
|
||||
# Classes whose hint is safe to attach on the CONTEXT-FREE surfaces (the
|
||||
# global 500 handler in main.py, the model-install SSE in setup/download.py),
|
||||
# where all we have is a raw error string with no stage. Only classes whose
|
||||
# classify() trigger is unmistakable belong here — e.g. VIDEO_DOWNLOAD_NETWORK
|
||||
# must NOT be added: its bare "timed out" trigger would stamp a "video server"
|
||||
# hint on a model-load timeout that leaks through the 500 handler.
|
||||
_CONTEXT_FREE_HINT_CLASSES = frozenset({
|
||||
"SOCKS_PROXY_SUPPORT_MISSING",
|
||||
"SSL_HANDSHAKE_FAILURE",
|
||||
})
|
||||
|
||||
|
||||
def append_hint(text: str) -> str:
|
||||
"""``"{text} — {hint}"`` for raw-string surfaces (the global 500 handler,
|
||||
the model-install SSE): the dynamic mirror hint (#874) when that class
|
||||
applies, else a context-free static class hint (#959). ``text`` unchanged
|
||||
otherwise — a no-op for every other error. Never raises."""
|
||||
try:
|
||||
hint = hf_mirror_hint(text)
|
||||
if not hint:
|
||||
topic = classify(text)
|
||||
if topic in _CONTEXT_FREE_HINT_CLASSES:
|
||||
hint = _HINTS.get(topic, "")
|
||||
except Exception:
|
||||
return text
|
||||
return f"{text} — {hint}" if hint else text
|
||||
|
||||
|
||||
def classify(reason: str) -> str:
|
||||
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
|
||||
|
||||
@@ -55,10 +213,100 @@ def classify(reason: str) -> str:
|
||||
return "APPIMAGE_WEBKIT_WHITESCREEN"
|
||||
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
return "PYANNOTE_LICENSE_REQUIRED"
|
||||
# ASR robustness (#551 / #549): name the class so the no-segments toast is
|
||||
# actionable. Place before the generic returns so a compute-type/transformers
|
||||
# failure gets its hint rather than falling through to "".
|
||||
if "compute type" in low or "efficient float16" in low:
|
||||
return "COMPUTE_TYPE_UNSUPPORTED"
|
||||
# #763: a bare OS-level EINVAL ("[Errno 22] Invalid argument") while writing
|
||||
# the per-chunk temp WAV for transcription (tempfile.NamedTemporaryFile /
|
||||
# soundfile.write on the system temp dir) used to collapse into a dead-end
|
||||
# "produced no segments. [Errno 22] Invalid argument" toast with no next
|
||||
# step. errno 22 is EINVAL on every platform; in this path it's almost always
|
||||
# a temp dir that's missing, read-only, on a full/removed drive, or blocked
|
||||
# by antivirus. Name the class so build_failure attaches an actionable hint
|
||||
# instead of a raw errno. Matching the errno (not the generic "invalid
|
||||
# argument" wording) keeps this from mislabelling unrelated failures; the
|
||||
# transformers "errno 2" rule below is unaffected — it also requires the
|
||||
# transformers + site-packages markers, which this signature lacks.
|
||||
if "errno 22" in low:
|
||||
return "OS_INVALID_ARGUMENT"
|
||||
if (
|
||||
"could not import module" in low
|
||||
or "autofeatureextractor" in low
|
||||
# A corrupted/incomplete transformers install: a model load lazily
|
||||
# resolves a module file that's MISSING from site-packages (an
|
||||
# interrupted `uv sync`, antivirus removal, or a partial update), e.g.
|
||||
# `[Errno 2] No such file or directory:
|
||||
# '.../site-packages/transformers/models/qwen3/modeling_qwen3.py'`.
|
||||
# That's a FileNotFoundError, not an ImportError, so the matches above
|
||||
# miss it and the user got a useless "try restarting". Substring-match
|
||||
# the package + the missing-file signal (separately, so it works on both
|
||||
# POSIX `/` and Windows `\` paths).
|
||||
or (
|
||||
("no such file" in low or "errno 2" in low)
|
||||
and "transformers" in low
|
||||
and "site-packages" in low
|
||||
)
|
||||
):
|
||||
return "TRANSFORMERS_IMPORT"
|
||||
# #959: httpx raises ImportError AT CLIENT CONSTRUCTION ("Using SOCKS
|
||||
# proxy, but the 'socksio' package is not installed. Make sure to install
|
||||
# httpx using `pip install httpx[socks]`.") when ALL_PROXY/HTTPS_PROXY is
|
||||
# socks5:// and socksio isn't importable. It surfaced from
|
||||
# huggingface_hub's get_session() inside model load — a bare 500 on
|
||||
# /generate with no next step. Checked BEFORE the HF-auth/mirror rules so
|
||||
# a message that also carries HF wording still names this class.
|
||||
if "socks proxy" in low or "socksio" in low:
|
||||
return "SOCKS_PROXY_SUPPORT_MISSING"
|
||||
# #976: a TLS handshake failing AFTER the TCP connection succeeds — the
|
||||
# signature of a corporate/antivirus proxy that TLS-inspects traffic and
|
||||
# re-signs certificates with a CA the OS trusts but Python's bundled
|
||||
# certifi list doesn't (a different failure mode from #984's TCP-level
|
||||
# "can't reach the host at all"). Requires "ssl" plus a handshake/cert-
|
||||
# verify marker so a generic connection error isn't mislabelled.
|
||||
if "ssl" in low and (
|
||||
"handshake" in low
|
||||
or "certificate verify failed" in low
|
||||
or "sslv3_alert" in low
|
||||
or "sslcertverificationerror" in low
|
||||
):
|
||||
return "SSL_HANDSHAKE_FAILURE"
|
||||
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
|
||||
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
|
||||
):
|
||||
return "HF_AUTH_FAILED"
|
||||
# #874: a model download that failed because the CONFIGURED HF mirror is
|
||||
# unreachable. Env-aware by design — the class only exists when a
|
||||
# non-default HF_ENDPOINT is configured. Checked BEFORE the video-download
|
||||
# network class so a model download's "timed out"/"connection reset"
|
||||
# names the mirror instead of the "video server".
|
||||
if hf_mirror_hint(reason):
|
||||
return "HF_MIRROR_UNREACHABLE"
|
||||
# Video download (#554/#536): a non-downloadable URL shape vs a transient
|
||||
# network drop — both previously surfaced as a bare yt-dlp string with no
|
||||
# next step. UNSUPPORTED first (more specific) so "Unable to download video:
|
||||
# Broken pipe" still classifies as a network blip.
|
||||
if "unsupported url" in low or "no video formats" in low or "is not a valid url" in low:
|
||||
return "UNSUPPORTED_VIDEO_URL"
|
||||
if (
|
||||
"broken pipe" in low
|
||||
or "connection reset" in low
|
||||
or "unable to download video" in low
|
||||
or "remote end closed" in low
|
||||
or "timed out" in low
|
||||
):
|
||||
return "VIDEO_DOWNLOAD_NETWORK"
|
||||
# A relocated/corrupted venv whose interpreter can't bootstrap its stdlib —
|
||||
# the Rust self-heal rebuilds it; this names the class for the toast.
|
||||
if "no module named 'encodings'" in low:
|
||||
return "BROKEN_VENV"
|
||||
# #564: the interpreter starts fine but the backend can't import its OWN
|
||||
# `omnivoice` package (a venv missing the editable install). Same self-heal
|
||||
# class — Clean & Retry / the bootstrap repair rebuilds it. The trailing
|
||||
# quote keeps a legitimately-named `omnivoice_*` helper from matching.
|
||||
if "no module named 'omnivoice'" in low:
|
||||
return "BROKEN_VENV"
|
||||
return ""
|
||||
|
||||
|
||||
@@ -154,12 +402,15 @@ def build_failure(
|
||||
|
||||
reason = sanitize(raw) or error_class
|
||||
docs_topic = classify(raw)
|
||||
# HF_MIRROR_UNREACHABLE's hint is dynamic (it names the configured mirror)
|
||||
# so it can't live in the static _HINTS table.
|
||||
hint = hf_mirror_hint(raw) if docs_topic == "HF_MIRROR_UNREACHABLE" else _HINTS.get(docs_topic, "")
|
||||
fields: dict[str, Any] = {
|
||||
"reason": reason,
|
||||
"error": reason, # backward-compat mirror for older frontends
|
||||
"error_class": error_class,
|
||||
"stage": stage,
|
||||
"hint": _HINTS.get(docs_topic, ""),
|
||||
"hint": hint,
|
||||
"docs_topic": docs_topic,
|
||||
"docs_url": error_docs_map.ERROR_DOCS.get(docs_topic, ""),
|
||||
"detail": sanitize(raw),
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
"""Resolve the project's own ``omnivoice`` package from source when the venv's
|
||||
editable install is missing (#564).
|
||||
|
||||
``omnivoice`` is normally an editable install in the backend venv. An interrupted
|
||||
or offline ``uv sync`` can install dependencies yet never lay the editable record
|
||||
(``_editable_impl_omnivoice.pth``), or an antivirus quarantine can remove it —
|
||||
leaving a venv that starts uvicorn but cannot ``import omnivoice``, so it boots
|
||||
fine and only fails at the first model call (``No module named 'omnivoice'``).
|
||||
|
||||
The desktop layout always copies ``omnivoice/`` next to ``backend/``, so we fall
|
||||
back to importing it from there. The bootstrap now also gates on omnivoice being
|
||||
importable (re-syncing to re-lay the editable install), but this keeps the
|
||||
backend resilient even when that repair hasn't run yet.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def find_omnivoice_source_root(candidates):
|
||||
"""Return the first candidate dir holding ``omnivoice/__init__.py``, else None."""
|
||||
for root in candidates:
|
||||
if root and os.path.isfile(os.path.join(root, "omnivoice", "__init__.py")):
|
||||
return root
|
||||
return None
|
||||
|
||||
|
||||
def _candidate_roots(backend_dir):
|
||||
"""Source roots to probe, most-specific first.
|
||||
|
||||
``OMNIVOICE_PROJECT_ROOT`` lets the launcher point at the staged project dir
|
||||
explicitly; otherwise the desktop layout puts ``omnivoice/`` beside
|
||||
``backend/`` (parent of ``backend_dir``).
|
||||
"""
|
||||
roots = []
|
||||
env = os.environ.get("OMNIVOICE_PROJECT_ROOT")
|
||||
if env:
|
||||
roots.append(env)
|
||||
roots.append(os.path.dirname(os.path.abspath(backend_dir)))
|
||||
return roots
|
||||
|
||||
|
||||
def _already_importable():
|
||||
import importlib.util
|
||||
try:
|
||||
return importlib.util.find_spec("omnivoice") is not None
|
||||
except (ImportError, ValueError):
|
||||
# A half-laid spec (e.g. a stale .pth pointing at a deleted dir) raises
|
||||
# rather than returning None — treat it as "not importable" so we fall
|
||||
# back to the on-disk source.
|
||||
return False
|
||||
|
||||
|
||||
def ensure_omnivoice_importable(backend_dir, logger=None):
|
||||
"""Make ``import omnivoice`` work, falling back to the sibling source tree.
|
||||
|
||||
No-op when the editable/site-packages install already resolves it. Otherwise
|
||||
appends the first source root containing ``omnivoice/`` to ``sys.path``
|
||||
(appended, never inserted, so a real install keeps precedence). Returns the
|
||||
root that was added, or ``None`` if none was needed or found.
|
||||
"""
|
||||
if _already_importable():
|
||||
return None
|
||||
root = find_omnivoice_source_root(_candidate_roots(backend_dir))
|
||||
if root and root not in sys.path:
|
||||
sys.path.append(root)
|
||||
if logger:
|
||||
logger.warning(
|
||||
"omnivoice not importable from the venv (missing/broken editable "
|
||||
"install) — resolving it from source at %s (#564)", root,
|
||||
)
|
||||
elif logger and root is None:
|
||||
logger.error(
|
||||
"omnivoice is not importable and no source tree was found next to "
|
||||
"%s — the install is incomplete; relaunch to let the bootstrap "
|
||||
"repair the venv (#564)", backend_dir,
|
||||
)
|
||||
return root
|
||||
@@ -61,9 +61,15 @@ def seed_sample_project():
|
||||
if count > 0:
|
||||
return # Not first run — skip
|
||||
|
||||
# Check if demo audio exists
|
||||
# The demo clip is committed at backend/assets/samples/demo_voice.wav and
|
||||
# bundled with the app (#621). If it's somehow absent (e.g. a partial
|
||||
# checkout), skip the seed gracefully rather than seeding a profile that
|
||||
# points at a missing file — run scripts/build_demos.sh to regenerate it.
|
||||
if not os.path.isfile(_DEMO_AUDIO):
|
||||
logger.warning("Demo audio not found at %s — skipping onboarding seed", _DEMO_AUDIO)
|
||||
logger.warning(
|
||||
"Demo audio not found at %s — skipping onboarding seed "
|
||||
"(regenerate with scripts/build_demos.sh)", _DEMO_AUDIO,
|
||||
)
|
||||
return
|
||||
|
||||
# Copy demo audio to voices directory
|
||||
|
||||
+55
-7
@@ -36,18 +36,39 @@ _TOKEN_PATTERNS = (
|
||||
re.compile(r"github_pat_[A-Za-z0-9_]{20,}"), # GitHub fine-grained PAT
|
||||
re.compile(r"gh[pousr]_[A-Za-z0-9]{30,}"), # GitHub classic tokens
|
||||
re.compile(r"sk-[A-Za-z0-9_\-]{20,}"), # OpenAI-style API keys
|
||||
# A backend error can carry a secret from *any* provider (the LLM-providers
|
||||
# feature ships a dozen), so match the common credential shapes too, not
|
||||
# just the four vendors above — a leaked key in a public issue is real harm.
|
||||
re.compile(r"eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{6,}"), # JWT (Bearer)
|
||||
re.compile(r"AIza[0-9A-Za-z_\-]{35}"), # Google API key
|
||||
re.compile(r"xox[baprs]-[A-Za-z0-9\-]{10,}"), # Slack token
|
||||
re.compile(r"AKIA[0-9A-Z]{16}"), # AWS access key id
|
||||
re.compile(r"(?i)bearer\s+[A-Za-z0-9._\-]{16,}"), # opaque bearer tokens
|
||||
)
|
||||
|
||||
# Secrets carried in a URL query string (`?token=…`, `&api_key=…`). Redact the
|
||||
# VALUE while keeping the param name + separator so the URL stays legible. Bare
|
||||
# `key=` is intentionally excluded — too common in non-secret text; shaped keys
|
||||
# are already caught above and named env vars by the sweep below.
|
||||
_URL_SECRET_RE = re.compile(
|
||||
r"((?:access[_-]?token|api[_-]?key|apikey|auth[_-]?token|token|secret|password|passwd|pwd)=)"
|
||||
r"([^&\s\"'#]{6,})",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Home-directory shapes for all three supported platforms. Matched
|
||||
# pattern-wise (not just this machine's $HOME) so paths quoted from a
|
||||
# user's pasted log on another OS get cleaned too.
|
||||
# IGNORECASE because Windows is case-insensitive and tools routinely emit the
|
||||
# lowercase `c:\users\<name>` form, which the CLAUDE.md redaction spec still
|
||||
# requires to become `~`. `Users`/`users`, `Home`/`home` all match.
|
||||
_HOME_PATTERNS = (
|
||||
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
|
||||
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
|
||||
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+"), # Windows, forward slashes (file URLs, normalized traces)
|
||||
re.compile(r"/Users/[^/\s\"']+"), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+"), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows, backslashes
|
||||
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+", re.IGNORECASE), # Windows, forward slashes
|
||||
re.compile(r"/Users/[^/\s\"']+", re.IGNORECASE), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+", re.IGNORECASE), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+", re.IGNORECASE), # Windows, backslashes
|
||||
)
|
||||
|
||||
# Values shorter than this are too entropy-poor to be real secrets and too
|
||||
@@ -85,19 +106,25 @@ def scrub_text(text: str | None) -> str:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 2. Credential-shaped substrings.
|
||||
# 2. Credential-shaped substrings + URL query secrets.
|
||||
for pat in _TOKEN_PATTERNS:
|
||||
try:
|
||||
s = pat.sub(REDACTED, s)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
s = _URL_SECRET_RE.sub(lambda m: m.group(1) + REDACTED, s)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3. This process's real home dir (covers symlinked/nonstandard homes
|
||||
# the generic patterns miss), then the per-OS shapes.
|
||||
# the generic patterns miss), then the per-OS shapes. Boundary-aware so
|
||||
# a home of `/Users/john` doesn't rewrite `/Users/johnny` to `~ny`
|
||||
# (leaking the fragment + mangling the path).
|
||||
try:
|
||||
home = os.path.expanduser("~")
|
||||
if home and home not in ("/", "~"):
|
||||
s = s.replace(home, "~")
|
||||
s = re.sub(re.escape(home) + r"(?=[/\\\s\"']|$)", "~", s)
|
||||
except Exception:
|
||||
pass
|
||||
for pat in _HOME_PATTERNS:
|
||||
@@ -107,3 +134,24 @@ def scrub_text(text: str | None) -> str:
|
||||
pass
|
||||
|
||||
return s
|
||||
|
||||
|
||||
def scrub_provider_error(detail: object, api_key: str | None = None) -> str:
|
||||
"""UI-safe text for an LLM/translation provider failure.
|
||||
|
||||
Some OpenAI-compatible providers echo the caller's key or a stable
|
||||
``user_id`` back inside their error bodies, and a raw ``str(exc)`` on the
|
||||
translate / glossary paths would surface that verbatim. This redacts the
|
||||
exact resolved ``api_key`` first (in the provider-registry case it isn't a
|
||||
shaped/known-env secret, so ``scrub_text`` alone can miss it) then runs the
|
||||
generic secret + home-path scrub. Never raises — scrubbing must not mask a
|
||||
failure with a new one. Mirrors ``settings._scrub_llm_detail`` so every
|
||||
surface redacts identically.
|
||||
"""
|
||||
s = str(detail if detail is not None else "")
|
||||
try:
|
||||
if api_key and api_key != "local" and len(api_key) >= _MIN_SECRET_LEN:
|
||||
s = s.replace(api_key, REDACTED)
|
||||
except Exception:
|
||||
pass
|
||||
return scrub_text(s)
|
||||
|
||||
+33
-4
@@ -2,14 +2,43 @@
|
||||
|
||||
Read from the installed package metadata (driven by ``pyproject.toml``) so the
|
||||
FastAPI/API version and exported-bundle metadata never drift to a stale literal
|
||||
again (the prior "0.4.0" / "0.2.7" bug). Falls back to a literal only when
|
||||
running from a raw source checkout that was never ``uv sync``'d.
|
||||
— the prior "0.4.0" / "0.2.7" bug, and the v0.3.6 desktop build that reported
|
||||
"0.3.5" because the *frozen* backend couldn't read its own metadata.
|
||||
|
||||
Resolution order:
|
||||
1. installed package metadata — correct in any ``uv sync``'d env and, thanks
|
||||
to ``copy_metadata('omnivoice')`` in ``backend.spec``, in the frozen build;
|
||||
2. ``pyproject.toml`` walked up from this file — correct for a raw source
|
||||
checkout that was never installed;
|
||||
3. ``_FALLBACK_VERSION`` — a last resort, kept in lockstep with the four
|
||||
version files by ``tests/test_app_version.py`` so it can never silently
|
||||
drift again.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
from pathlib import Path
|
||||
|
||||
# Last-resort literal. Guarded by
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.3.15"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
"""Version for contexts where package metadata is unavailable."""
|
||||
for parent in Path(__file__).resolve().parents:
|
||||
pyproject = parent / "pyproject.toml"
|
||||
if pyproject.is_file():
|
||||
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject.read_text())
|
||||
if match:
|
||||
return match.group(1)
|
||||
return _FALLBACK_VERSION
|
||||
|
||||
|
||||
try:
|
||||
APP_VERSION = version("omnivoice")
|
||||
except PackageNotFoundError: # non-installed source checkout
|
||||
APP_VERSION = "0.3.5"
|
||||
except PackageNotFoundError: # frozen build w/o metadata, or non-installed checkout
|
||||
APP_VERSION = _fallback_version()
|
||||
|
||||
@@ -50,6 +50,22 @@ def _recv(stream):
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
# NOTE: keep this compute_type fallback in lockstep with
|
||||
# services/asr_backend.py:_compute_type_candidates / _is_compute_type_error.
|
||||
# This sidecar runs in a child proc with a clean import path, so we duplicate a
|
||||
# tiny copy rather than cross-importing the heavy services package (#551).
|
||||
def _ct_candidates(device):
|
||||
override = os.environ.get("ASR_COMPUTE_TYPE")
|
||||
if override:
|
||||
return [override]
|
||||
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
|
||||
|
||||
|
||||
def _is_ct_error(msg):
|
||||
low = msg.lower()
|
||||
return "compute type" in low or "efficient float16" in low
|
||||
|
||||
|
||||
def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
@@ -60,8 +76,20 @@ def _get_model():
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
except Exception:
|
||||
device = "cpu"
|
||||
compute = "float16" if device == "cuda" else "int8"
|
||||
_model = WhisperModel(name, device=device, compute_type=compute)
|
||||
# Degrade fp16 → int8 rather than crash on GPUs without efficient fp16
|
||||
# (older Maxwell/Pascal, GTX 16xx, CTranslate2/cuDNN mismatch) (#551).
|
||||
last_err = None
|
||||
for compute in _ct_candidates(device):
|
||||
try:
|
||||
_model = WhisperModel(name, device=device, compute_type=compute)
|
||||
break
|
||||
except (ValueError, RuntimeError) as e:
|
||||
last_err = e
|
||||
if _is_ct_error(str(e)):
|
||||
continue
|
||||
raise
|
||||
else:
|
||||
raise last_err
|
||||
return _model
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
"""Confucius4-TTS sidecar package (issue #590).
|
||||
|
||||
Confucius4-TTS (netease-youdao) is an LLM-based multilingual / cross-lingual
|
||||
zero-shot voice-cloning TTS: 14 languages, **no reference transcript required**,
|
||||
cross-lingual voice transfer, Apache-2.0 (https://github.com/netease-youdao/Confucius4-TTS).
|
||||
|
||||
Like IndexTTS / MOSS-TTS-v1.5 / dots.tts it runs in its **own subprocess venv**
|
||||
(upstream: Python 3.10 + CUDA 12.6 + its own deps), isolated from the OmniVoice
|
||||
parent. It is **opt-in** — selected in the engine picker and enabled only when
|
||||
the user points ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` at a clone — so it can never
|
||||
become a broken default on any platform (the strict default-parity rule).
|
||||
|
||||
Status (#590): **validated end-to-end** (2026-07-02, Apple Silicon, CPU) — the
|
||||
synthesis API (``confuciustts.cli.inference.ConfuciusTTS`` →
|
||||
``.generate(text, lang, prompt_wav)`` → tensor, ``model.sample_rate``) produced
|
||||
audible speech at 22 050 Hz; the sidecar's pure logic is unit-tested
|
||||
(``tests/test_confucius4_sidecar.py``). CPU inference is slow (~17× realtime),
|
||||
so CUDA is the recommended path. Gated off by default, so this affects no one
|
||||
until they opt in.
|
||||
|
||||
Three entry points: ``Confucius4Backend`` (this module), ``main.py`` (the sidecar,
|
||||
runs under the Confucius4 venv — never imported by the parent), and
|
||||
``bootstrap.py`` (venv probe + lazy bootstrap).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.confucius4")
|
||||
|
||||
|
||||
class Confucius4Backend(SubprocessBackend):
|
||||
"""Confucius4-TTS (netease-youdao) — LLM-based, 14 langs, zero-shot clone.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv. First synthesize cold-loads the checkpoint; subsequent calls
|
||||
reuse the process.
|
||||
|
||||
Installation::
|
||||
|
||||
git clone https://github.com/netease-youdao/Confucius4-TTS.git
|
||||
cd Confucius4-TTS
|
||||
uv venv --python 3.10 && uv pip install -r requirements.txt
|
||||
|
||||
(Upstream ships no pyproject.toml/setup.py, so there is nothing to
|
||||
``pip install -e`` — the sidecar sys.path-inserts the clone instead.)
|
||||
Then set ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` to the clone root and restart.
|
||||
License: Apache-2.0. CUDA recommended; CPU validated but ~17× realtime.
|
||||
"""
|
||||
|
||||
id = "confucius4-tts"
|
||||
display_name = (
|
||||
"Confucius4-TTS (LLM, 14 langs, cross-lingual zero-shot clone, CUDA/CPU, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # timbre comes from a reference clip
|
||||
# Upstream vocoder rate (config target_sample_rate) — confirmed 22 050 Hz by
|
||||
# a live run (2026-07-02); still re-read from the sidecar's ready/audio frames.
|
||||
_DEFAULT_SAMPLE_RATE = 22050
|
||||
# CUDA fast path + CPU fallback, both exercised (CPU end-to-end validated).
|
||||
# No MPS claim — upstream has no Metal path.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# Verify the venv on disk only — do NOT import the engine here (separate
|
||||
# interpreter). A real health-check runs on the user's "Test engine"
|
||||
# action in Settings.
|
||||
from engines.confucius4.bootstrap import (
|
||||
CONFUCIUS4_SIDECAR_SCRIPT,
|
||||
is_confucius4_installed,
|
||||
)
|
||||
if not is_confucius4_installed():
|
||||
return False, (
|
||||
"Confucius4-TTS venv not found. Set OMNIVOICE_CONFUCIUS4_TTS_DIR "
|
||||
"to your Confucius4-TTS clone (the directory containing "
|
||||
"requirements.txt) and restart OmniVoice. CUDA GPU recommended "
|
||||
"(CPU works but is slow). See docs/engines/confucius4-tts.md."
|
||||
)
|
||||
if not CONFUCIUS4_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"Confucius4-TTS sidecar script missing at "
|
||||
f"{CONFUCIUS4_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.confucius4.bootstrap import resolve_confucius4_venv
|
||||
return resolve_confucius4_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.confucius4.bootstrap import CONFUCIUS4_SIDECAR_SCRIPT
|
||||
return CONFUCIUS4_SIDECAR_SCRIPT
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 14 languages with the caller's language passed through at synthesize
|
||||
# time; "multi" on the protocol surface.
|
||||
return ["multi"]
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the Confucius4 sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — reference clip path → ``prompt_wav`` (zero-shot
|
||||
cloning). Optional but recommended for a specific voice.
|
||||
* ``language`` — ISO code / name → ``lang`` (cross-lingual transfer).
|
||||
* ``ref_text`` is intentionally ignored — Confucius4 is unconstrained
|
||||
cloning (no reference transcript needed).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["Confucius4Backend"]
|
||||
@@ -0,0 +1,217 @@
|
||||
"""Confucius4-TTS venv probe + lazy bootstrap (issue #590).
|
||||
|
||||
Confucius4-TTS (netease-youdao) is an LLM-based multilingual zero-shot cloning
|
||||
TTS — 14 languages, no reference transcript required, Apache-2.0. Like the other
|
||||
heavyweight opt-in engines (IndexTTS / MOSS-TTS-v1.5 / dots.tts) it runs in its
|
||||
**own subprocess venv**: upstream targets Python 3.10 + CUDA 12.6 with its own
|
||||
dependency set, which we keep off the parent interpreter.
|
||||
|
||||
Probe order (existing power-user installs win — zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv/`` — the user's clone-level venv.
|
||||
2. ``backend/engines/confucius4/.venv/`` — this package's own venv.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -r <clone>/requirements.txt``
|
||||
(+ ``uv pip install -e <clone>`` only if upstream ever ships packaging).
|
||||
|
||||
Validated end-to-end 2026-07-02 (Apple Silicon, CPU): upstream ships **no
|
||||
pyproject.toml/setup.py**, so ``confuciustts`` is importable only with the
|
||||
clone root on ``sys.path`` — the import probe and the sidecar both handle
|
||||
that. The engine is opt-in (env-dir gated) and never touched unless
|
||||
``OMNIVOICE_CONFUCIUS4_TTS_DIR`` is set, so this can't affect the default
|
||||
install on any platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.confucius4.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint.
|
||||
CONFUCIUS4_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2).
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's Confucius4-TTS clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_CONFUCIUS4_TTS_DIR"
|
||||
|
||||
#: The package importable from the clone (verify against upstream).
|
||||
_IMPORT_PROBE = "confuciustts"
|
||||
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_confucius4_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable venv (no subprocess spawn)."""
|
||||
return any(cand.is_file() for cand in _probe_paths())
|
||||
|
||||
|
||||
def resolve_confucius4_venv() -> Path:
|
||||
"""Resolve the sidecar's Python interpreter (probe order in the docstring).
|
||||
Memoised. Raises :exc:`RuntimeError` if none can be located and bootstrap
|
||||
is unavailable."""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import(cand):
|
||||
logger.info("Confucius4 venv resolved from %s: %s", _CLONE_DIR_ENV, cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import(cand):
|
||||
logger.info("Confucius4 venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"Confucius4-TTS is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your Confucius4-TTS clone "
|
||||
"(the directory that contains requirements.txt), then restart "
|
||||
"OmniVoice. See docs/engines/confucius4-tts.md."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _import_probe_code() -> str:
|
||||
"""Probe snippet mirroring the sidecar's import semantics: upstream is not
|
||||
pip-installable, so ``confuciustts`` resolves via the clone on sys.path."""
|
||||
clone = os.environ.get(_CLONE_DIR_ENV, "")
|
||||
if clone:
|
||||
return f"import sys; sys.path.insert(0, {clone!r}); import {_IMPORT_PROBE}"
|
||||
return f"import {_IMPORT_PROBE}"
|
||||
|
||||
|
||||
def _venv_can_import(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify ``import confuciustts`` works."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", _import_probe_code()],
|
||||
capture_output=True, timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("Confucius4 import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"Confucius4 import probe non-zero for %s: %s",
|
||||
python_path, proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
return shutil.which("uv")
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/confucius4/.venv and install the user's clone."""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the Confucius4-TTS venv but was not "
|
||||
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
|
||||
"from https://docs.astral.sh/uv/ and re-launch OmniVoice."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping Confucius4 venv at %s from %s (several minutes on first "
|
||||
"launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", "--python", "3.10", str(_ENGINES_VENV_DIR)],
|
||||
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for Confucius4 bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
requirements = clone_dir / "requirements.txt"
|
||||
try:
|
||||
if requirements.is_file():
|
||||
subprocess.run(
|
||||
[uv, "pip", "install", "--python", str(python_path),
|
||||
"-r", str(requirements)],
|
||||
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
# Editable install only if upstream ever ships packaging metadata —
|
||||
# as of 2026-07 there is none, and `uv pip install -e` on a bare clone
|
||||
# fails outright. Import resolution is handled via sys.path instead.
|
||||
if (clone_dir / "pyproject.toml").is_file() or (clone_dir / "setup.py").is_file():
|
||||
subprocess.run(
|
||||
[uv, "pip", "install", "--python", str(python_path), "-e", str(clone_dir)],
|
||||
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install failed during Confucius4 bootstrap "
|
||||
f"({clone_dir}): "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
|
||||
"See docs/engines/confucius4-tts.md."
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import(python_path):
|
||||
raise RuntimeError(
|
||||
f"Confucius4 bootstrap completed but `import {_IMPORT_PROBE}` still "
|
||||
f"fails from {python_path}. Verify {clone_dir} is a valid clone. "
|
||||
"See docs/engines/confucius4-tts.md."
|
||||
)
|
||||
|
||||
logger.info("Confucius4 venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CONFUCIUS4_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_confucius4_installed",
|
||||
"resolve_confucius4_venv",
|
||||
]
|
||||
@@ -0,0 +1,216 @@
|
||||
"""Confucius4-TTS sidecar entry point (issue #590).
|
||||
|
||||
Runs inside ``engines/confucius4/.venv`` (or the user's
|
||||
``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv``), isolated from the OmniVoice parent.
|
||||
Same isolation rationale as the IndexTTS / MOSS-TTS-v1.5 / dots.tts sidecars.
|
||||
|
||||
Stdlib-only at import time; ``confuciustts`` + torch are imported lazily on the
|
||||
first synthesize op so the ``ready`` frame fits inside the parent's 30 s spawn
|
||||
handshake.
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow: ready → ping/pong → synthesize (→ progress, → audio) → shutdown.
|
||||
|
||||
Status (#590): the model API below
|
||||
(``confuciustts.cli.inference.ConfuciusTTS(config_path=…, device=…)`` and
|
||||
``model.generate(text=, lang=, prompt_wav=)`` → audio tensor, ``model.sample_rate``)
|
||||
is **validated end-to-end** (2026-07-02, Apple Silicon, CPU): live generate()
|
||||
produced audible speech at 22 050 Hz. This sidecar's pure logic is unit-tested
|
||||
in ``tests/test_confucius4_sidecar.py``. Opt-in, so it affects no one until
|
||||
enabled.
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code. NO logging of os.environ.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: Upstream BigVGAN vocoder rate — ``target_sample_rate: 22050`` in
|
||||
#: ``config/inference_config.yaml``, confirmed by a live end-to-end run
|
||||
#: (2026-07-02). The real value is still re-read from ``model.sample_rate``
|
||||
#: on each generate() so a future upstream change can't corrupt audio.
|
||||
CONFUCIUS_SAMPLE_RATE = 22050
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
_model = None
|
||||
|
||||
|
||||
def _config_path() -> str:
|
||||
"""Locate Confucius4's inference config (``config/inference_config.yaml``)
|
||||
under the clone, or an explicit override."""
|
||||
explicit = os.environ.get("OMNIVOICE_CONFUCIUS4_CONFIG")
|
||||
if explicit:
|
||||
return explicit
|
||||
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
|
||||
return os.path.join(clone, "config", "inference_config.yaml")
|
||||
|
||||
|
||||
def _ensure_clone_on_sys_path() -> None:
|
||||
"""Make ``import confuciustts`` resolve from the user's clone.
|
||||
|
||||
Upstream Confucius4-TTS is **not pip-installable** (no pyproject.toml /
|
||||
setup.py as of 2026-07); its own ``example.py`` sys.path-inserts the repo
|
||||
root instead. Mirror that here so the sidecar works from a plain
|
||||
``uv pip install -r requirements.txt`` venv. Inserted at position 0 so the
|
||||
clone the user pointed at always wins over any stale installed copy.
|
||||
"""
|
||||
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
|
||||
if clone and clone not in sys.path:
|
||||
sys.path.insert(0, clone)
|
||||
|
||||
|
||||
def _load_model(stdout):
|
||||
"""Cold-construct the Confucius4 model (CUDA, else CPU — both validated)."""
|
||||
global _model
|
||||
if _model is not None:
|
||||
return _model
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
_ensure_clone_on_sys_path()
|
||||
import torch
|
||||
from confuciustts.cli.inference import ConfuciusTTS # type: ignore[import-not-found]
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
_model = ConfuciusTTS(config_path=_config_path(), device=device)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _model
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
import numpy as np
|
||||
arr = audio.detach().to("cpu").float().numpy() if hasattr(audio, "detach") else np.asarray(audio)
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0)
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _normalize_language(raw):
|
||||
"""Confucius4 expects an ISO-ish language code (e.g. 'en', 'zh'). Empty /
|
||||
'auto' → 'en' as a safe default (the API requires a lang)."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return "en"
|
||||
s = raw.strip().lower()
|
||||
if not s or s == "auto":
|
||||
return "en"
|
||||
return s[:2] if (len(s) >= 2 and s[:2].isalpha()) else s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
model = _load_model(stdout)
|
||||
|
||||
gen_kwargs: dict = {"text": text, "lang": _normalize_language(msg.get("language"))}
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
gen_kwargs["prompt_wav"] = ref_audio
|
||||
|
||||
audio = model.generate(**gen_kwargs)
|
||||
sample_rate = int(getattr(model, "sample_rate", CONFUCIUS_SAMPLE_RATE))
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "confucius4-tts",
|
||||
"sample_rate": CONFUCIUS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error", "stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {"op": "error", "stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}"})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error", "stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,187 @@
|
||||
"""dots.tts sidecar package (issue #498).
|
||||
|
||||
dots.tts is rednote-hilab's 2B fully-continuous autoregressive TTS — widely
|
||||
cited as among the strongest open zero-shot voice-cloning models. 24
|
||||
languages, 48 kHz output, Apache-2.0 (code + checkpoints).
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent, for the same ``transformers`` reason as IndexTTS and
|
||||
MOSS-TTS-v1.5: dots.tts pins ``transformers==4.57.0`` (verified against
|
||||
``constraints/recommended.txt``), while OmniVoice pins
|
||||
``transformers>=5.3.0``. The two cannot share one interpreter.
|
||||
|
||||
Cross-platform honesty (the strict default-parity rule): dots.tts's
|
||||
upstream package declares **Linux + macOS** classifiers only — **no
|
||||
Windows** — and its device code is **CUDA-or-CPU with no MPS branch**
|
||||
(verified in ``runtime.py``). So:
|
||||
|
||||
* It is **opt-in** (engine-picker selection + a user-provided clone),
|
||||
never a default — so it never becomes a broken default on any platform.
|
||||
* ``is_available()`` returns ``False`` with a clear reason on **Windows**
|
||||
rather than offering an engine that can't run there. Windows users are
|
||||
pointed at WSL2 / a Linux or macOS host.
|
||||
* ``gpu_compat = ("cuda", "cpu")`` — no MPS claim. On Apple Silicon the
|
||||
upstream package runs on CPU (slow but correct); the faster MLX path is
|
||||
a community port we deliberately don't auto-wire here.
|
||||
|
||||
Three public entry points: ``DotsTTSBackend`` (this module), ``main.py``
|
||||
(sidecar, runs under dots.tts's ``transformers==4.57`` venv — never imported
|
||||
by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts")
|
||||
|
||||
|
||||
class DotsTTSBackend(SubprocessBackend):
|
||||
"""dots.tts (rednote-hilab) — 2B, 24 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==4.57.0``). First synthesize cold-loads
|
||||
the ~9 GB checkpoint (bf16 on CUDA); subsequent calls reuse the process.
|
||||
|
||||
Installation (OmniVoice prefers a user's existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/rednote-hilab/dots.tts.git
|
||||
cd dots.tts
|
||||
uv venv && uv pip install -e . -c constraints/recommended.txt
|
||||
|
||||
Set ``OMNIVOICE_DOTS_TTS_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/dots_tts/.venv`` lazily on first launch if no venv
|
||||
exists yet; the user's existing ``${DIR}/.venv`` is preferred if present.
|
||||
|
||||
Best cloning quality uses the ``dots.tts-soar`` checkpoint (the default)
|
||||
and BOTH a reference clip and its exact transcript (continuation
|
||||
cloning). License: Apache-2.0.
|
||||
"""
|
||||
|
||||
id = "dots-tts"
|
||||
display_name = (
|
||||
"dots.tts (2B, 24 langs, zero-shot clone, CUDA/CPU, 48 kHz, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
# dots.tts emits 48 kHz (verified via checkpoint vocoder.sample_rate).
|
||||
_DEFAULT_SAMPLE_RATE = 48000
|
||||
# CUDA + CPU only; no MPS branch in upstream runtime.py.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# Cross-platform parity: dots.tts upstream is Linux/macOS-only (no
|
||||
# Windows classifier, no Windows install path). Refuse cleanly on
|
||||
# Windows instead of advertising an engine that can't run.
|
||||
if sys.platform == "win32":
|
||||
return False, (
|
||||
"dots.tts is not supported on Windows — upstream targets "
|
||||
"Linux and macOS only. Run OmniVoice under WSL2, or use a "
|
||||
"Linux/macOS host. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
# Do NOT import dots_tts here: it pins transformers==4.57, which can't
|
||||
# coexist with the parent's transformers>=5.3 in one interpreter —
|
||||
# the reason for the subprocess isolation. Verify the venv on disk
|
||||
# only; a real health-check is gated on the user's "Test engine"
|
||||
# action in Settings.
|
||||
from engines.dots_tts.bootstrap import (
|
||||
DOTS_TTS_SIDECAR_SCRIPT,
|
||||
is_dots_tts_installed,
|
||||
)
|
||||
if not is_dots_tts_installed():
|
||||
return False, (
|
||||
"dots.tts venv not found. Set OMNIVOICE_DOTS_TTS_DIR to your "
|
||||
"dots.tts clone (the directory containing pyproject.toml) and "
|
||||
"restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/dots-tts.md for the full install walk-through."
|
||||
)
|
||||
if not DOTS_TTS_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"dots.tts sidecar script missing at "
|
||||
f"{DOTS_TTS_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.dots_tts.bootstrap import resolve_dots_tts_venv
|
||||
return resolve_dots_tts_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
|
||||
return DOTS_TTS_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 24 languages with auto-detect; expose "multi" on the protocol
|
||||
# surface and translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the dots.tts sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — reference clip path → ``prompt_audio_path``
|
||||
(zero-shot cloning). Optional.
|
||||
* ``ref_text`` — the reference transcript → ``prompt_text``.
|
||||
Best cloning fidelity ("continuation"). Upstream
|
||||
REQUIRES ``prompt_audio_path`` when ``prompt_text``
|
||||
is set, so we drop a stray ref_text with no
|
||||
ref_audio rather than let the sidecar raise.
|
||||
* ``language`` — ISO code / name / None (auto-detect).
|
||||
* ``num_step`` — flow-matching steps → ``num_steps`` (default 10;
|
||||
use 4 for the ``dots.tts-mf`` checkpoint).
|
||||
* ``guidance_scale`` — CFG (default 1.2; >2 amplifies energy).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — only valid alongside ref_audio.
|
||||
forwarded["ref_text"] = ref_text
|
||||
elif kw.get("ref_text"):
|
||||
logger.info(
|
||||
"dots-tts: ref_text supplied without ref_audio; ignoring "
|
||||
"(upstream requires prompt_audio_path when prompt_text is set)."
|
||||
)
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
# OmniVoice's generic num_step default is 16; dots.tts's own default
|
||||
# is 10. Honor an explicit value, else use the dots-appropriate 10.
|
||||
num_step = kw.get("num_step")
|
||||
forwarded["num_steps"] = int(num_step) if num_step is not None else 10
|
||||
|
||||
# dots.tts's own CFG default is 1.2 (the generic 2.0 over-energises).
|
||||
guidance = kw.get("guidance_scale")
|
||||
forwarded["guidance_scale"] = float(guidance) if guidance is not None else 1.2
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["DotsTTSBackend"]
|
||||
@@ -0,0 +1,226 @@
|
||||
"""dots.tts venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
Resolves which Python interpreter runs the dots.tts sidecar. Mirrors
|
||||
``engines.indextts.bootstrap`` / ``engines.moss_tts_v15.bootstrap`` because
|
||||
dots.tts has the same shape of problem: a hard ``transformers==4.57.0`` pin
|
||||
that conflicts with the parent's ``transformers>=5.3`` — so it runs in its
|
||||
own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_DOTS_TTS_DIR}/.venv/`` — the user's clone-level venv.
|
||||
2. ``backend/engines/dots_tts/.venv/`` — this package's own venv.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e <clone> -c
|
||||
<clone>/constraints/recommended.txt`` (the upstream-pinned stack:
|
||||
torch==2.8.0, transformers==4.57.0, …).
|
||||
|
||||
Caching: memoised after first success. Tests reset via :func:`invalidate`.
|
||||
|
||||
Security: same posture as IndexTTS — bootstrap never touches HF_TOKEN; the
|
||||
sidecar's stderr is redacted by the parent's ``HFTokenRedactor``; the
|
||||
editable install comes from a user-controlled clone they already trust.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint.
|
||||
DOTS_TTS_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2).
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's dots.tts clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_DOTS_TTS_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_dots_tts_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable dots.tts venv. Does NOT spawn
|
||||
the venv Python — that's saved for :func:`resolve_dots_tts_venv`."""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_dots_tts_venv() -> Path:
|
||||
"""Resolve the sidecar's Python interpreter (probe order in the module
|
||||
docstring). Memoised. Raises :exc:`RuntimeError` if none can be located
|
||||
and the bootstrap path is unavailable."""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv.
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info(
|
||||
"dots.tts venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info("dots.tts venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"dots.tts is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your dots.tts clone "
|
||||
"(the directory that contains pyproject.toml and constraints/), "
|
||||
"then restart OmniVoice. See docs/engines/dots-tts.md for the "
|
||||
"full install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_dots(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify ``import dots_tts.runtime`` works.
|
||||
Bounded by ``_IMPORT_PROBE_TIMEOUT_S``. False on any failure."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import dots_tts.runtime"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("dots.tts import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"dots.tts import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
return shutil.which("uv")
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/dots_tts/.venv and editable-install the user's clone
|
||||
with the upstream constraints file."""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the dots.tts venv but was not found "
|
||||
"on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv from "
|
||||
"https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping dots.tts venv at %s from %s (this can take several "
|
||||
"minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for dots.tts bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
install_cmd = [
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", str(clone_dir),
|
||||
]
|
||||
# Apply the upstream pin set when it ships with the clone.
|
||||
constraints = clone_dir / "constraints" / "recommended.txt"
|
||||
if constraints.is_file():
|
||||
install_cmd += ["-c", str(constraints)]
|
||||
try:
|
||||
subprocess.run(
|
||||
install_cmd, check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during dots.tts bootstrap "
|
||||
f"({clone_dir}): "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
|
||||
"See docs/engines/dots-tts.md."
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_dots(python_path):
|
||||
raise RuntimeError(
|
||||
"dots.tts bootstrap completed but `import dots_tts.runtime` still "
|
||||
f"fails from {python_path}. Verify that {clone_dir} is a valid "
|
||||
"dots.tts clone. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
logger.info("dots.tts venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DOTS_TTS_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_dots_tts_installed",
|
||||
"resolve_dots_tts_venv",
|
||||
]
|
||||
@@ -0,0 +1,255 @@
|
||||
"""dots.tts sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/dots_tts/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_DOTS_TTS_DIR}/.venv``) with ``transformers==4.57.0``, isolated
|
||||
from the OmniVoice parent (``transformers>=5.3``). Same isolation rationale
|
||||
as the IndexTTS / MOSS-TTS-v1.5 sidecars.
|
||||
|
||||
Stdlib-only at import time; ``dots_tts`` + torch are imported lazily on the
|
||||
first synthesize op so the ``ready`` frame fits inside the parent's 30 s
|
||||
spawn handshake even on a cold filesystem.
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "dots-tts",
|
||||
"sample_rate": 48000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/ref.wav",
|
||||
"ref_text": "transcript", "language": "EN",
|
||||
"num_steps": 10, "guidance_scale": 1.2}
|
||||
-> {"op": "progress", ...} (cold load) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 48000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: dots.tts emits 48 kHz (checkpoint vocoder.sample_rate). Advertised in the
|
||||
#: ready frame; the real value is re-read from each generate() result.
|
||||
DOTS_SAMPLE_RATE = 48000
|
||||
|
||||
#: Default checkpoint. ``-soar`` is the best-cloning variant; ``-mf`` is the
|
||||
#: fastest (use num_steps=4). Overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "rednote-hilab/dots.tts-soar"
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — (runtime,). Device is auto-selected inside the
|
||||
# dots.tts runtime (cuda-or-cpu, no MPS); we don't pass a device.
|
||||
_runtime = None
|
||||
|
||||
|
||||
def _load_runtime(stdout):
|
||||
"""Cold-construct the dots.tts runtime.
|
||||
|
||||
``DotsTtsRuntime.from_pretrained`` auto-selects cuda-or-cpu internally
|
||||
(no MPS path). precision is bf16 on CUDA; on CPU we fall back to fp32
|
||||
(bf16 CPU kernels are spotty). Both overridable via env.
|
||||
"""
|
||||
global _runtime
|
||||
if _runtime is not None:
|
||||
return _runtime
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from dots_tts.runtime import DotsTtsRuntime # type: ignore[import-not-found]
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_DOTS_TTS_MODEL", _DEFAULT_REPO)
|
||||
default_precision = "bfloat16" if torch.cuda.is_available() else "float32"
|
||||
precision = os.environ.get("OMNIVOICE_DOTS_TTS_PRECISION", default_precision)
|
||||
optimize = os.environ.get("OMNIVOICE_DOTS_TTS_OPTIMIZE", "0") == "1"
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
_runtime = DotsTtsRuntime.from_pretrained(
|
||||
repo,
|
||||
precision=precision,
|
||||
optimize=optimize,
|
||||
)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _runtime
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor (1, N) in [-1, 1] to base64 int16 PCM."""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _normalize_language(raw):
|
||||
"""Map OmniVoice's language value to what dots.tts accepts, or None.
|
||||
|
||||
dots.tts accepts None/"auto_detect", ISO codes upper-cased ("EN"/"ZH"),
|
||||
or names ("english"). A 2-letter ISO code is upper-cased; anything else
|
||||
is passed through; empty / "auto" → None (auto-detect)."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if len(s) == 2 and s.isalpha():
|
||||
return s.upper()
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
runtime = _load_runtime(stdout)
|
||||
|
||||
gen_kwargs: dict = {
|
||||
"text": text,
|
||||
"num_steps": int(msg.get("num_steps", 10)),
|
||||
"guidance_scale": float(msg.get("guidance_scale", 1.2)),
|
||||
}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
gen_kwargs["prompt_audio_path"] = ref_audio
|
||||
ref_text = msg.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — upstream requires prompt_audio_path when
|
||||
# prompt_text is set (the parent already enforces this).
|
||||
gen_kwargs["prompt_text"] = ref_text
|
||||
|
||||
language = _normalize_language(msg.get("language"))
|
||||
if language:
|
||||
gen_kwargs["language"] = language
|
||||
|
||||
result = runtime.generate(**gen_kwargs)
|
||||
audio = result["audio"]
|
||||
sample_rate = int(result.get("sample_rate", DOTS_SAMPLE_RATE))
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "dots-tts",
|
||||
"sample_rate": DOTS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,194 @@
|
||||
"""MOSS-TTS-v1.5 sidecar package (issue #498).
|
||||
|
||||
MOSS-TTS-v1.5 is OpenMOSS's 8B flagship TTS — a Qwen3-8B language backbone
|
||||
plus a 1.6B audio codec, 31 languages, zero-shot voice cloning, token-level
|
||||
duration control and inline ``[pause Ns]`` markers. Apache-2.0.
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent process, for the *same* reason IndexTTS does: a hard
|
||||
``transformers`` version conflict. MOSS-TTS-v1.5's ``torch-runtime`` extra
|
||||
pins ``transformers==5.0.0`` (verified against the upstream
|
||||
``pyproject.toml``), while OmniVoice pins ``transformers>=5.3.0``. The two
|
||||
cannot share one interpreter — so MOSS lives behind ``SubprocessBackend``
|
||||
with a dedicated venv, exactly like ``engines.indextts``.
|
||||
|
||||
Three public entry points live in this package:
|
||||
|
||||
* ``MossTTSV15Backend`` (this module) — the SubprocessBackend subclass
|
||||
that ``services.tts_backend._LAZY_REGISTRY`` resolves on first access.
|
||||
Defined HERE (not in ``services.tts_backend``) to break the import
|
||||
cycle: ``services.subprocess_backend`` imports ``TTSBackend`` from
|
||||
``services.tts_backend``, so the backend class must live downstream of
|
||||
that module finishing its import. Same indirection as IndexTTS /
|
||||
Supertonic-3.
|
||||
* ``main.py`` — the sidecar entrypoint (runs under MOSS's venv with
|
||||
``transformers==5.0.0``; never imported by the parent).
|
||||
* ``bootstrap.py`` — the venv-probe + lazy-bootstrap helper.
|
||||
|
||||
Do NOT import ``main.py`` from the parent process — it runs under a
|
||||
different venv (``transformers==5.0.0``) and importing it in-process would
|
||||
re-introduce the exact conflict this isolation exists to avoid.
|
||||
|
||||
Hardware honesty (cross-platform rule): MOSS-TTS-v1.5's upstream documents
|
||||
only CUDA and CPU. There is **no documented or tested MPS path** — the
|
||||
custom ``trust_remote_code`` modelling code and the separate audio
|
||||
tokenizer are unverified on Apple Silicon. We therefore advertise
|
||||
``gpu_compat = ("cuda", "cpu")`` and the sidecar selects ``cuda`` when
|
||||
present else ``cpu`` — it never silently routes to MPS where it might
|
||||
crash. On Apple Silicon the engine honestly resolves to CPU (slow but
|
||||
correct), and the engine is opt-in regardless, so it never becomes a
|
||||
broken default on any platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15")
|
||||
|
||||
#: 1 second of audio ≈ 12.5 codec tokens (MOSS-TTS-v1.5 model card). Used to
|
||||
#: translate OmniVoice's ``duration`` (seconds) into the model's ``tokens``
|
||||
#: duration-control argument.
|
||||
TOKENS_PER_SECOND: float = 12.5
|
||||
|
||||
|
||||
class MossTTSV15Backend(SubprocessBackend):
|
||||
"""MOSS-TTS-v1.5 (OpenMOSS) — 8B, 31 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==5.0.0``). The first synthesize cold-loads
|
||||
~16 GB of bf16 weights (CUDA) / fp32 (CPU); subsequent calls reuse the
|
||||
process and the in-memory model.
|
||||
|
||||
Installation (transparent to power users who already cloned MOSS-TTS —
|
||||
OmniVoice prefers their existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/OpenMOSS/MOSS-TTS.git
|
||||
cd MOSS-TTS
|
||||
# CUDA host:
|
||||
uv venv && uv pip install -e ".[torch-runtime]"
|
||||
# non-CUDA host (CPU): install plain torch/transformers instead of +cu128
|
||||
|
||||
Set ``OMNIVOICE_MOSS_TTS_V15_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/moss_tts_v15/.venv`` lazily on first launch if no venv
|
||||
exists yet (CUDA hosts only — the upstream ``torch-runtime`` extra is
|
||||
``+cu128``); the user's existing ``${DIR}/.venv`` is preferred if
|
||||
present, so no re-install is needed.
|
||||
|
||||
License: Apache-2.0 (code + weights) — no acceptance gate needed.
|
||||
"""
|
||||
|
||||
id = "moss-tts-v15"
|
||||
display_name = (
|
||||
"MOSS-TTS-v1.5 (8B, 31 langs, zero-shot clone, CUDA/CPU, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Honest hardware surface: upstream documents CUDA + CPU only. MPS is
|
||||
# undocumented / untested, so we do NOT claim it (cross-platform rule).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# IMPORTANT: do NOT attempt to import MOSS / its transformers==5.0.0
|
||||
# here. The parent pins transformers>=5.3 — co-importing the two in
|
||||
# one interpreter is exactly the conflict this subprocess isolation
|
||||
# exists to avoid. We only verify the venv exists on disk; a real
|
||||
# health-check (spawn + ping) is gated on the user's "Test engine"
|
||||
# action in Settings, same as IndexTTS.
|
||||
from engines.moss_tts_v15.bootstrap import (
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT,
|
||||
is_moss_tts_v15_installed,
|
||||
)
|
||||
if not is_moss_tts_v15_installed():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 venv not found. Set OMNIVOICE_MOSS_TTS_V15_DIR "
|
||||
"to your MOSS-TTS clone (the directory containing pyproject.toml) "
|
||||
"and restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/moss-tts-v15.md for the full install walk-through."
|
||||
)
|
||||
if not MOSS_TTS_V15_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 sidecar script missing at "
|
||||
f"{MOSS_TTS_V15_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.moss_tts_v15.bootstrap import resolve_moss_tts_v15_venv
|
||||
return resolve_moss_tts_v15_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
return MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 31 languages with multilingual handling; expose "multi" on the
|
||||
# protocol surface (same as OmniVoice / CosyVoice / Supertonic-3) and
|
||||
# translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the MOSS-TTS-v1.5 sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — path to a reference clip. When present, MOSS
|
||||
runs zero-shot voice cloning (``reference=``).
|
||||
Optional: without it the model uses its own
|
||||
default voice.
|
||||
* ``ref_text`` — accepted but unused in clone mode (MOSS's
|
||||
zero-shot path needs only the audio); kept in
|
||||
the signature so the common call-site doesn't
|
||||
need engine-specific knowledge.
|
||||
* ``language`` — ISO code or name; mapped to a MOSS language name
|
||||
in the sidecar, omitted (auto-detect) if unknown.
|
||||
* ``duration`` — target seconds → ``tokens`` (1 s ≈ 12.5 tokens).
|
||||
* ``max_new_tokens`` — generation cap (default 4096).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
forwarded["ref_text"] = ref_text
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
duration = kw.get("duration")
|
||||
if duration is not None:
|
||||
target_tokens = int(float(duration) * TOKENS_PER_SECOND)
|
||||
if target_tokens > 0:
|
||||
forwarded["tokens"] = target_tokens
|
||||
|
||||
max_new_tokens = kw.get("max_new_tokens")
|
||||
if max_new_tokens is not None:
|
||||
forwarded["max_new_tokens"] = int(max_new_tokens)
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["MossTTSV15Backend", "TOKENS_PER_SECOND"]
|
||||
@@ -0,0 +1,272 @@
|
||||
"""MOSS-TTS-v1.5 venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
The parent process needs to know *which Python interpreter* to spawn the
|
||||
MOSS-TTS-v1.5 sidecar under. This module owns that resolution. It mirrors
|
||||
``engines.indextts.bootstrap`` because MOSS has the same shape of problem:
|
||||
a hard ``transformers`` pin (``==5.0.0``) that conflicts with the parent's
|
||||
``transformers>=5.3`` — so MOSS runs in its own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv/`` — the user's clone-level
|
||||
venv. Highest priority: a user who already cloned MOSS-TTS and ran
|
||||
``uv pip install -e ".[torch-runtime]"`` (per upstream docs) gets
|
||||
reused verbatim, no re-download of the ~16 GB model.
|
||||
2. ``backend/engines/moss_tts_v15/.venv/`` — this package's own venv,
|
||||
created by step 3 if needed.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e
|
||||
"${DIR}[torch-runtime]"``. Requires ``OMNIVOICE_MOSS_TTS_V15_DIR``.
|
||||
The upstream ``torch-runtime`` extra is CUDA (``+cu128``), so the
|
||||
auto-bootstrap targets CUDA hosts; non-CUDA (CPU/Mac) users set up
|
||||
their own venv per docs/engines/moss-tts-v15.md (Probe 1).
|
||||
|
||||
Caching: resolution is memoised after the first successful call. Tests
|
||||
reset via :func:`invalidate`.
|
||||
|
||||
Security: bootstrap never touches HF_TOKEN; the sidecar's stderr is drained
|
||||
by SubprocessBackend through the parent root logger where Phase 1's
|
||||
``HFTokenRedactor`` strips token bytes. ``uv pip install -e`` installs from
|
||||
a user-controlled clone the user already trusts (same posture as IndexTTS).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint. ``MossTTSV15Backend.sidecar_script``
|
||||
#: returns this; SubprocessBackend spawns it with the resolved venv python.
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2). The MOSS-TTS clone, when bootstrapped,
|
||||
#: is installed into this venv via ``uv pip install -e``.
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's MOSS-TTS clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_MOSS_TTS_V15_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
# Timeouts — bounded so a wedged venv never hangs the parent. The bootstrap
|
||||
# install can take many minutes on a cold cache (MOSS pulls a CUDA torch
|
||||
# build + transformers + an audio codec stack).
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_moss_tts_v15_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable MOSS-TTS-v1.5 venv.
|
||||
|
||||
Returns True if either Probe 1 or Probe 2 has a Python executable on
|
||||
disk. Does NOT spawn the venv Python — that's saved for
|
||||
:func:`resolve_moss_tts_v15_venv`, which is only invoked on the first
|
||||
generate() / health_check(). This fires on every Settings render via
|
||||
``MossTTSV15Backend.is_available()``, so it stays cheap.
|
||||
"""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_moss_tts_v15_venv() -> Path:
|
||||
"""Resolve the path to the Python interpreter that runs the sidecar.
|
||||
|
||||
Probe order described in the module docstring. Memoised. Raises
|
||||
:exc:`RuntimeError` if no working venv can be located AND the bootstrap
|
||||
path is unavailable.
|
||||
"""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv (highest priority for back-compat).
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info(
|
||||
"MOSS-TTS-v1.5 venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info("MOSS-TTS-v1.5 venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your MOSS-TTS clone "
|
||||
"(the directory that contains pyproject.toml), then restart "
|
||||
"OmniVoice. See docs/engines/moss-tts-v15.md for the full "
|
||||
"install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
"""Return the python executable path inside a venv directory.
|
||||
|
||||
Handles the Unix (``bin/python``) vs Windows (``Scripts/python.exe``)
|
||||
layout. No filesystem access — caller checks .is_file().
|
||||
"""
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
"""Ordered list of candidate venv-python paths (no .is_file() check)."""
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_moss(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify the MOSS stack imports.
|
||||
|
||||
MOSS-TTS-v1.5 loads via ``transformers`` + ``trust_remote_code`` (no
|
||||
fixed top-level package to import), so the readiness signal is that the
|
||||
venv has a working ``transformers`` + ``torch`` — which only the
|
||||
``[torch-runtime]`` install provides. Bounded by
|
||||
``_IMPORT_PROBE_TIMEOUT_S`` so a wedged venv never hangs the parent.
|
||||
Returns False on any failure (non-zero exit, timeout, OSError).
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import transformers, torch"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("moss-tts-v15 import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"moss-tts-v15 import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
"""Find the uv binary — bundled first (Tauri-set env var), else PATH."""
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
sys_uv = shutil.which("uv")
|
||||
if sys_uv:
|
||||
return sys_uv
|
||||
return None
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/moss_tts_v15/.venv and install the user's clone into it.
|
||||
|
||||
Runs ``uv venv <engines_venv>`` then ``uv pip install --python
|
||||
<engines_venv>/bin/python -e "<clone>[torch-runtime]"``. Verifies the
|
||||
result by re-probing the import — a successful uv invocation that still
|
||||
can't import the stack indicates a deeper environment problem (e.g. the
|
||||
``+cu128`` torch-runtime extra can't resolve on a non-CUDA host) and we
|
||||
raise with whatever stderr we captured plus a docs pointer.
|
||||
"""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the MOSS-TTS-v1.5 venv but was not "
|
||||
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
|
||||
"from https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping MOSS-TTS-v1.5 venv at %s from %s (this can take "
|
||||
"several minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True,
|
||||
timeout=_UV_VENV_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for MOSS-TTS-v1.5 bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
try:
|
||||
subprocess.run(
|
||||
[
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", f"{clone_dir}[torch-runtime]",
|
||||
],
|
||||
check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during MOSS-TTS-v1.5 bootstrap "
|
||||
f"({clone_dir}). On a non-CUDA host the upstream '[torch-runtime]' "
|
||||
"extra (cu128) cannot resolve — set up the venv manually per "
|
||||
"docs/engines/moss-tts-v15.md. Error: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_moss(python_path):
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 bootstrap completed but the transformers/torch "
|
||||
f"import still fails from {python_path}. Verify that {clone_dir} "
|
||||
"is a valid MOSS-TTS clone. See docs/engines/moss-tts-v15.md."
|
||||
)
|
||||
|
||||
logger.info("MOSS-TTS-v1.5 venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MOSS_TTS_V15_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_moss_tts_v15_installed",
|
||||
"resolve_moss_tts_v15_venv",
|
||||
]
|
||||
@@ -0,0 +1,303 @@
|
||||
"""MOSS-TTS-v1.5 sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/moss_tts_v15/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv``) with ``transformers==5.0.0``,
|
||||
isolated from the OmniVoice parent process which pins ``transformers>=5.3``.
|
||||
Same isolation rationale as the IndexTTS sidecar.
|
||||
|
||||
Stdlib-only at import time. The model + transformers + torch are imported
|
||||
lazily on the first synthesize op so the sidecar emits its ``ready`` frame
|
||||
inside the parent's 30 s spawn handshake even on a cold filesystem (an 8B
|
||||
model takes well over 30 s to cold-load).
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "moss-tts-v15",
|
||||
"sample_rate": 24000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/spk.wav", "language": "fr",
|
||||
"tokens": 325, "max_new_tokens": 4096}
|
||||
-> {"op": "progress", ...} (cold load only) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 24000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: Native sample rate MOSS-TTS-v1.5 emits. Advertised in the ready frame so
|
||||
#: the parent doesn't have to import MOSS just to learn the rate. Confirmed
|
||||
#: via ``processor.model_config.sampling_rate`` (the real value is read from
|
||||
#: the loaded model at synthesize time; this is the handshake default).
|
||||
MOSS_SAMPLE_RATE = 24000
|
||||
|
||||
#: HF repo id for the weights, overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "OpenMOSS-Team/MOSS-TTS-v1.5"
|
||||
|
||||
#: ISO-639-1 → MOSS language name. MOSS's ``build_user_message`` takes a
|
||||
#: language *name* ("French"), not a code. Unknown codes are omitted so the
|
||||
#: model auto-detects. Covers the high-traffic subset of MOSS's 31 langs.
|
||||
_ISO_TO_NAME = {
|
||||
"en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean",
|
||||
"fr": "French", "de": "German", "es": "Spanish", "it": "Italian",
|
||||
"pt": "Portuguese", "ru": "Russian", "ar": "Arabic", "hi": "Hindi",
|
||||
"nl": "Dutch", "pl": "Polish", "tr": "Turkish", "vi": "Vietnamese",
|
||||
"th": "Thai", "id": "Indonesian", "cs": "Czech", "el": "Greek",
|
||||
"he": "Hebrew", "fa": "Persian", "uk": "Ukrainian", "sv": "Swedish",
|
||||
}
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). The parent can't see a
|
||||
child's VRAM, so we self-report it in the pong. 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — populated on the first synthesize op and reused.
|
||||
# Holds (processor, model, device, sample_rate).
|
||||
_state = None
|
||||
|
||||
|
||||
def _load_model(stdout):
|
||||
"""Cold-construct the MOSS-TTS-v1.5 processor + model.
|
||||
|
||||
Device selection is CUDA-or-CPU only — MOSS's upstream documents no MPS
|
||||
path and the custom ``trust_remote_code`` modelling code is untested on
|
||||
Apple Silicon, so we never route to MPS where it might crash. dtype is
|
||||
bf16 on CUDA, fp32 on CPU (bf16 CPU ops are spotty). Emits progress
|
||||
frames so the parent can surface the multi-GB cold-load latency.
|
||||
"""
|
||||
global _state
|
||||
if _state is not None:
|
||||
return _state
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_MOSS_TTS_V15_MODEL", _DEFAULT_REPO)
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
# "sdpa" works on CUDA + CPU and needs no extra dep. flash_attention_2
|
||||
# (Ampere+ CUDA, optional flash-attn) is opt-in via env.
|
||||
attn = os.environ.get("OMNIVOICE_MOSS_TTS_V15_ATTN", "sdpa")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
|
||||
# The audio tokenizer is a separate sub-module that must be moved to the
|
||||
# device independently (easy to miss — see upstream README).
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
repo,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn,
|
||||
torch_dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
sample_rate = int(getattr(processor.model_config, "sampling_rate", MOSS_SAMPLE_RATE))
|
||||
_state = (processor, model, device, sample_rate)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _state
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor to base64 int16 PCM.
|
||||
|
||||
MOSS returns a float tensor in [-1, 1] (1-D or (1, N)); we squeeze to
|
||||
mono, clip, scale to int16, and base64 so the wire frame stays JSON-safe.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _resolve_language(raw):
|
||||
"""Map OmniVoice's language value to a MOSS language name, or None.
|
||||
|
||||
Accepts an ISO-639-1 code or a full name. Unknown / empty / "auto"
|
||||
values return None so MOSS auto-detects."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if s.lower() in _ISO_TO_NAME:
|
||||
return _ISO_TO_NAME[s.lower()]
|
||||
# Already a language name (or an unknown code) — pass it through; MOSS
|
||||
# ignores a language it doesn't recognise.
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
import torch
|
||||
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
processor, model, device, sample_rate = _load_model(stdout)
|
||||
|
||||
user_kwargs: dict = {"text": text}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
# Zero-shot voice cloning: the reference audio alone is enough in
|
||||
# MOSS's clone mode (ref_text is not consumed here). The processor's
|
||||
# audio tokenizer encodes the reference into the prompt.
|
||||
user_kwargs["reference"] = [ref_audio]
|
||||
|
||||
language = _resolve_language(msg.get("language"))
|
||||
if language:
|
||||
user_kwargs["language"] = language
|
||||
|
||||
tokens = msg.get("tokens")
|
||||
if tokens is not None:
|
||||
user_kwargs["tokens"] = int(tokens)
|
||||
|
||||
max_new_tokens = int(msg.get("max_new_tokens", 4096))
|
||||
|
||||
conversations = [[processor.build_user_message(**user_kwargs)]]
|
||||
|
||||
with torch.no_grad():
|
||||
batch = processor(conversations, mode="generation")
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=max_new_tokens,
|
||||
)
|
||||
|
||||
decoded = processor.decode(outputs)
|
||||
audio = decoded[0].audio_codes_list[0]
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import — nothing above this line
|
||||
# touches transformers/torch, so we make the 30 s spawn window even cold.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "moss-tts-v15",
|
||||
"sample_rate": MOSS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
# Per-op failure is recoverable — emit the error frame and stay
|
||||
# alive so the parent can retry without paying the respawn +
|
||||
# multi-GB model-load cost again.
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -70,6 +70,7 @@ class Supertonic3Backend(SubprocessBackend):
|
||||
id = "supertonic3"
|
||||
display_name = "Supertonic-3 (31 langs, CPU ONNX, 7 preset voices, OpenRAIL-M)"
|
||||
supports_voice_design = False # preset voices only
|
||||
supports_cloning = False # preset voices only; generate() never reads ref_audio
|
||||
# TTS-04: honest hardware reporting. Supertonic-3 has no CUDA / MPS
|
||||
# path in the SDK ‑‑ ONNX Runtime CPU EP only.
|
||||
gpu_compat: tuple[str, ...] = ("cpu",)
|
||||
|
||||
+274
-26
@@ -9,6 +9,16 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
if _backend_dir not in sys.path:
|
||||
sys.path.insert(0, _backend_dir)
|
||||
|
||||
# #564: also make the project's OWN `omnivoice` package importable from source
|
||||
# when the venv's editable install is missing/broken (interrupted/offline
|
||||
# `uv sync`, antivirus-quarantined `_editable_impl_omnivoice.pth`, …). Without
|
||||
# this the backend boots fine and only fails at the first model call with
|
||||
# `No module named 'omnivoice'`. The bootstrap now gates on omnivoice being
|
||||
# importable too (re-syncing to re-lay the editable install); this is the
|
||||
# runtime safety net. See core/omnivoice_path.py for the full rationale.
|
||||
from core.omnivoice_path import ensure_omnivoice_importable
|
||||
ensure_omnivoice_importable(_backend_dir)
|
||||
|
||||
# Triton is unavailable on Windows — disable torch.compile / dynamo / inductor
|
||||
# to prevent TritonMissing errors at inference time. Must be set before torch
|
||||
# is imported (it is lazily imported in services/model_manager.py). Uses
|
||||
@@ -129,6 +139,24 @@ os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
|
||||
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "15")
|
||||
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "30")
|
||||
|
||||
# ── OS trust store for TLS (#976) ───────────────────────────────────────────
|
||||
# Users behind a corporate/antivirus proxy that TLS-inspects HTTPS traffic get
|
||||
# a raw "[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure"
|
||||
# on every model install — the TCP connection succeeds (a different failure
|
||||
# mode from #984's TCP-level blocked-host case), but the proxy re-signs the
|
||||
# certificate with its own root CA, which the OS trusts (Windows CryptoAPI/
|
||||
# SChannel) and Python's bundled `certifi` CA list does not. `inject_into_ssl`
|
||||
# patches `ssl.SSLContext` process-wide to verify against the OS trust store
|
||||
# instead, which is the actual fix (not just a nicer error message). Must run
|
||||
# here — at MODULE level, before huggingface_hub/requests/httpx do any network
|
||||
# I/O — not inside lifespan(), which runs too late. Not platform-gated: it's a
|
||||
# correctness improvement everywhere. Best-effort: never block startup.
|
||||
try:
|
||||
import truststore
|
||||
|
||||
truststore.inject_into_ssl()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Prevent torchaudio from lazy-importing torchcodec (broken on some installs).
|
||||
# Proper fix = exclude torchcodec in pyproject.toml; this is a belt-and-braces guard.
|
||||
@@ -145,6 +173,16 @@ from logging.handlers import RotatingFileHandler
|
||||
# written to prefs.json so they survive backend restarts. Read them back
|
||||
# here — before any user code reads os.environ — so the values are available
|
||||
# from startup.
|
||||
#
|
||||
# Legacy (≤v0.3.7) Translation-LLM rows (env.TRANSLATE_*) must migrate into
|
||||
# the custom LLM provider's settings store BEFORE the re-import below — once
|
||||
# TRANSLATE_BASE_URL lands in os.environ it hijacks the LLM provider
|
||||
# selection for the whole session (#963). Real env vars are untouched.
|
||||
try:
|
||||
from services.llm_providers import migrate_legacy_translate_prefs
|
||||
migrate_legacy_translate_prefs()
|
||||
except Exception:
|
||||
pass # never block startup on the migration; it retries next launch
|
||||
_PERSISTED_ENV_PREFIX = "env."
|
||||
try:
|
||||
from core.prefs import _load as _load_all_prefs
|
||||
@@ -327,6 +365,7 @@ from api.routers import (
|
||||
events,
|
||||
capture,
|
||||
capture_ws,
|
||||
dictation,
|
||||
openai_compat,
|
||||
tts_stream,
|
||||
marketplace,
|
||||
@@ -334,6 +373,7 @@ from api.routers import (
|
||||
sonitranslate,
|
||||
audiobook,
|
||||
longform_jobs,
|
||||
pronunciation, # Expressive-TTS Spec 01: user pronunciation dictionary
|
||||
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
|
||||
)
|
||||
from utils import hf_progress
|
||||
@@ -375,8 +415,145 @@ def _env_flag(name: str, default: bool = False) -> bool:
|
||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _capture_preload_delay_s() -> float:
|
||||
"""Seconds after boot before the dictation (capture ASR) model warms.
|
||||
|
||||
Late enough that it never competes with startup I/O or the TTS preload;
|
||||
overridable via OMNIVOICE_CAPTURE_PRELOAD_DELAY (mostly for tests)."""
|
||||
raw = os.environ.get("OMNIVOICE_CAPTURE_PRELOAD_DELAY", "")
|
||||
try:
|
||||
v = float(raw)
|
||||
if v >= 0:
|
||||
return v
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return 30.0
|
||||
|
||||
|
||||
def _capture_preload_ram_ok(min_free_bytes: int = 4 * 1024**3) -> bool:
|
||||
"""RAM guard for the dictation warm-up: skip below 4 GB free so the
|
||||
background load never pushes a small machine into swap. If free memory
|
||||
can't be measured, warm anyway (the load path has its own error handling)."""
|
||||
try:
|
||||
import psutil
|
||||
return psutil.virtual_memory().available >= min_free_bytes
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
|
||||
def _mcp_start_timeout_s() -> float:
|
||||
"""Seconds to wait for the MCP session manager to start before giving up
|
||||
and serving without it (#632). Overridable via OMNIVOICE_MCP_START_TIMEOUT_S."""
|
||||
raw = os.environ.get("OMNIVOICE_MCP_START_TIMEOUT_S", "")
|
||||
try:
|
||||
v = float(raw)
|
||||
if v > 0:
|
||||
return v
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return 30.0
|
||||
|
||||
|
||||
async def _serve_mcp(session_manager, ready: "asyncio.Event", stop: "asyncio.Event") -> None:
|
||||
"""Own the MCP session manager's full enter→exit lifecycle in ONE task.
|
||||
|
||||
FastMCP's ``run()`` opens an anyio task group, and anyio requires the cancel
|
||||
scope to be exited in the *same task* that entered it. So we must NOT enter
|
||||
it via ``wait_for`` (which runs the enter in a throwaway sub-task) or on the
|
||||
lifespan task and exit it elsewhere — either raises "Attempted to exit cancel
|
||||
scope in a different task". This coroutine enters and exits the context
|
||||
itself: it signals ``ready`` once mounted, then idles until ``stop``.
|
||||
"""
|
||||
try:
|
||||
async with session_manager.run():
|
||||
ready.set()
|
||||
await stop.wait()
|
||||
except Exception as e:
|
||||
logger.warning("MCP session manager stopped: %s", e)
|
||||
finally:
|
||||
ready.set() # never leave startup blocked on the readiness wait
|
||||
|
||||
|
||||
async def _start_mcp_session_manager(session_manager, *, timeout: float):
|
||||
"""Start MCP off the startup critical path; wait up to ``timeout`` for it to
|
||||
signal ready. Returns ``(task, stop_event, mounted)``.
|
||||
|
||||
The MCP layer is best-effort and must never wedge backend startup. On some
|
||||
platforms (observed: Apple-Silicon M1, #632) ``run()`` can *hang* on its
|
||||
anyio task group; the old code awaited the enter before serving, so the hang
|
||||
meant "Application startup complete" never fired and the whole backend was
|
||||
unreachable with no error. Now the enter lives in its own task and we only
|
||||
*optionally* wait on a ready signal — a hang becomes a logged warning + a
|
||||
backend that serves normally without MCP.
|
||||
"""
|
||||
stop = asyncio.Event()
|
||||
if session_manager is None:
|
||||
return None, stop, False
|
||||
ready = asyncio.Event()
|
||||
task = asyncio.create_task(_serve_mcp(session_manager, ready, stop))
|
||||
try:
|
||||
await asyncio.wait_for(ready.wait(), timeout=timeout)
|
||||
mounted = not task.done() # ready is also set on failure → not mounted
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(
|
||||
"MCP session manager did not signal ready within %.0fs (#632); "
|
||||
"serving without waiting. Set OMNIVOICE_MCP_START_TIMEOUT_S to adjust.",
|
||||
timeout,
|
||||
)
|
||||
mounted = False
|
||||
return task, stop, mounted
|
||||
|
||||
|
||||
async def _cancel_and_await_tasks(*tasks, timeout: float = 3.0) -> None:
|
||||
"""Cancel each background task and give it a bounded chance to actually
|
||||
finish before shutdown proceeds — ``None`` entries are skipped (a task
|
||||
that's conditionally created, e.g. ``capture_preload_task``, may not
|
||||
exist).
|
||||
|
||||
``task.cancel()`` alone is not enough for a task awaiting
|
||||
``run_in_executor()``: once the underlying OS thread is inside blocking
|
||||
native/import work, cancellation can't stop it, so cancel-and-move-on lets
|
||||
shutdown finish while that thread is still running — invisible to
|
||||
asyncio, but very much alive when the interpreter starts tearing down
|
||||
module state under it (#1000 class). Awaiting with a bound (instead of
|
||||
just cancelling) gives an early-stage task a real chance to exit cleanly
|
||||
first; a task that's genuinely still deep in blocking work times out here
|
||||
same as before, and the caller's own GPU-pool reset handles that case.
|
||||
"""
|
||||
for t in tasks:
|
||||
if t is None:
|
||||
continue
|
||||
t.cancel()
|
||||
for t in tasks:
|
||||
if t is None:
|
||||
continue
|
||||
try:
|
||||
await asyncio.wait_for(t, timeout=timeout)
|
||||
except (asyncio.CancelledError, asyncio.TimeoutError):
|
||||
pass
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup watchdog (#632): a silent hang during startup (e.g. a model-load /
|
||||
# MCP deadlock on some platforms) means "Application startup complete" never
|
||||
# logs and the app sits forever with no error. If startup hasn't finished
|
||||
# within the window, dump every thread's stack to stderr (→ backend_err.log)
|
||||
# so the hang point is captured instead of invisible. Cancelled the instant
|
||||
# startup completes, so a normal (even slow-download) boot never trips it.
|
||||
# Tune with OMNIVOICE_STARTUP_WATCHDOG_S (seconds; 0 disables). Best-effort —
|
||||
# never let the diagnostic itself break startup.
|
||||
_watchdog_armed = False
|
||||
try:
|
||||
import faulthandler
|
||||
_wd = float(os.environ.get("OMNIVOICE_STARTUP_WATCHDOG_S", "300"))
|
||||
if _wd > 0 and hasattr(faulthandler, "dump_traceback_later"):
|
||||
faulthandler.dump_traceback_later(_wd, repeat=False, exit=False)
|
||||
_watchdog_armed = True
|
||||
logger.info("Startup watchdog armed: thread dump if startup exceeds %.0fs (#632).", _wd)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
init_db()
|
||||
# Network sharing is loopback-only by default; the PIN middleware stays
|
||||
# inert until enable() sets a PIN. Seed the (disabled) state so the
|
||||
@@ -425,11 +602,20 @@ async def lifespan(app: FastAPI):
|
||||
worker_task = asyncio.create_task(task_manager.worker())
|
||||
# Warm the TTS model in the background so first /generate is instant.
|
||||
preload_task = asyncio.create_task(preload_model())
|
||||
# Capture ASR is useful to keep warm, but it is another large model in
|
||||
# unified memory on Apple Silicon. Keep launch lean by default; users who
|
||||
# prefer instant dictation can opt in with OMNIVOICE_PRELOAD_CAPTURE_ASR=1.
|
||||
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR"):
|
||||
# Dictation v2: the capture ASR warms in the background BY DEFAULT — a
|
||||
# deferred (~30s post-boot) load off the event loop, so startup stays
|
||||
# lean and the first dictation is instant instead of a cold model load.
|
||||
# OMNIVOICE_PRELOAD_CAPTURE_ASR=0 opts out; the warm-up is also skipped
|
||||
# under 4 GB free RAM (checked at warm time, not boot time).
|
||||
capture_preload_task = None # only assigned when the preload actually runs (#1000 class)
|
||||
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR", default=True):
|
||||
async def _preload_capture_asr():
|
||||
await asyncio.sleep(_capture_preload_delay_s())
|
||||
if not _capture_preload_ram_ok():
|
||||
logger.info(
|
||||
"Capture ASR preload skipped: <4GB free RAM; "
|
||||
"dictation ASR will load on first use.")
|
||||
return
|
||||
loading_detail = None
|
||||
prev_loading_detail = None
|
||||
try:
|
||||
@@ -459,31 +645,64 @@ async def lifespan(app: FastAPI):
|
||||
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
|
||||
|
||||
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP transport needs its session manager running
|
||||
# for the lifetime of the app. It's created lazily by streamable_http_app()
|
||||
# (called in mount_mcp below), so we stack its `run()` context into ours
|
||||
# via AsyncExitStack rather than replacing this lifespan. Best-effort: a
|
||||
# missing/broken MCP layer must never stop the rest of the backend.
|
||||
from contextlib import AsyncExitStack
|
||||
async with AsyncExitStack() as _mcp_stack:
|
||||
_sm = getattr(app.state, "mcp_session_manager", None)
|
||||
if _sm is not None:
|
||||
try:
|
||||
await _mcp_stack.enter_async_context(_sm.run())
|
||||
logger.info("MCP server mounted at /mcp")
|
||||
except Exception as e:
|
||||
logger.warning("MCP session manager failed to start: %s", e)
|
||||
yield
|
||||
# FastMCP's Streamable-HTTP transport needs its session manager running for
|
||||
# the lifetime of the app. Run it in its OWN task that owns the full
|
||||
# enter→exit lifecycle (anyio task-affinity, see _serve_mcp) and only wait,
|
||||
# with a timeout, for it to signal ready — so a hang on its anyio group
|
||||
# (observed on M1, #632) can never wedge "Application startup complete".
|
||||
_sm = getattr(app.state, "mcp_session_manager", None)
|
||||
mcp_task, mcp_stop, mcp_mounted = await _start_mcp_session_manager(
|
||||
_sm, timeout=_mcp_start_timeout_s()
|
||||
)
|
||||
if mcp_mounted:
|
||||
logger.info("MCP server mounted at /mcp")
|
||||
# Startup finished — disarm the hang watchdog before serving (#632).
|
||||
if _watchdog_armed:
|
||||
try:
|
||||
import faulthandler
|
||||
faulthandler.cancel_dump_traceback_later()
|
||||
except Exception:
|
||||
pass
|
||||
yield
|
||||
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
|
||||
logger.info("Shutdown: cleaning up…")
|
||||
idle_task.cancel()
|
||||
worker_task.cancel()
|
||||
# Wait for tasks to finish their current iteration
|
||||
for t in (idle_task, worker_task):
|
||||
# Stop MCP first — signal its task to exit its own anyio context (correct
|
||||
# task-affinity), then bound the wait so a wedged manager can't hang exit.
|
||||
mcp_stop.set()
|
||||
if mcp_task is not None:
|
||||
try:
|
||||
await asyncio.wait_for(t, timeout=3.0)
|
||||
except (asyncio.CancelledError, asyncio.TimeoutError):
|
||||
await asyncio.wait_for(mcp_task, timeout=5.0)
|
||||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
# preload_task/capture_preload_task matter most here (#1000 class): a quit
|
||||
# mid-preload used to fall straight through to "Shutdown: done." while the
|
||||
# model load was still running on a GPU-pool thread — cancel() can't stop
|
||||
# a thread already inside blocking import/load work, so the process
|
||||
# reported a clean exit while that background thread was still mid-
|
||||
# `import transformers`, and got torn down by interpreter finalization
|
||||
# instead. That surfaced as a misleading "Could not import module
|
||||
# 'AutoFeatureExtractor'" — transformers' own generic lazy-import wrapper,
|
||||
# not a real dependency problem. Awaiting here lets an early-stage load
|
||||
# (still importing, not yet mid weight-download) finish cleanly before we
|
||||
# report done; a load that's genuinely deep into a multi-GB download still
|
||||
# times out — _reset_gpu_pool() below abandons it either way.
|
||||
#
|
||||
# 20s, not the original 3s (code-review finding post-merge): a cold
|
||||
# transformers import alone can take longer than 3s on a slow disk or a
|
||||
# first-ever launch, so the original bound left a real residual window —
|
||||
# cancellation detaches the asyncio task, but the underlying OS thread
|
||||
# keeps running past it, and shutdown could still report "done" while
|
||||
# that thread was alive. Python cannot forcibly kill a running thread, so
|
||||
# no finite bound eliminates this outright — 20s just shrinks the window
|
||||
# from "any preload" to "an unusually slow cold-import," which is the
|
||||
# practical ceiling before a longer shutdown itself becomes the
|
||||
# complaint. A thread that's still running past 20s was never going to
|
||||
# finish in a shutdown-appropriate timeframe regardless.
|
||||
await _cancel_and_await_tasks(
|
||||
idle_task, worker_task, preload_task, capture_preload_task, timeout=20.0,
|
||||
)
|
||||
# Unload the model and free GPU memory
|
||||
try:
|
||||
import services.model_manager as mm
|
||||
@@ -491,6 +710,10 @@ async def lifespan(app: FastAPI):
|
||||
mm.model = None
|
||||
logger.info("Shutdown: model unloaded.")
|
||||
mm.free_vram()
|
||||
# Abandon a still-running preload's GPU-pool thread (Python can't kill
|
||||
# a thread mid blocking call) so it can't outlive this shutdown block
|
||||
# holding a reference into module state that's about to be torn down.
|
||||
mm._reset_gpu_pool()
|
||||
except Exception:
|
||||
pass
|
||||
# Run GC to release any remaining references
|
||||
@@ -573,8 +796,17 @@ async def global_exception_handler(request: Request, exc: Exception):
|
||||
headers["Access-Control-Allow-Origin"] = origin
|
||||
headers["Access-Control-Allow-Credentials"] = "true"
|
||||
headers["Vary"] = "Origin"
|
||||
# #874: a model download that failed because the CONFIGURED Hugging Face
|
||||
# mirror (HF_ENDPOINT) is unreachable used to leak the raw transformers
|
||||
# message ("We couldn't connect to 'https://hf-mirror.com' …") as the 500
|
||||
# detail with no next step. #959: same story for the SOCKS-proxy class
|
||||
# ("Using SOCKS proxy, but the 'socksio' package is not installed").
|
||||
# Appending the shared hints HERE covers every route that can leak a
|
||||
# model-load/download error (generate, dub, archetypes, …), not just TTS
|
||||
# generate. append_hint is a no-op for every other error and never raises.
|
||||
from core.failure import append_hint
|
||||
return JSONResponse(
|
||||
{"detail": str(exc), "error_class": _entry.get("error_class")},
|
||||
{"detail": append_hint(str(exc)), "error_class": _entry.get("error_class")},
|
||||
status_code=500,
|
||||
headers=headers,
|
||||
)
|
||||
@@ -740,6 +972,20 @@ app.add_middleware(NetworkAccessMiddleware)
|
||||
# keyed non-loopback client must reach them.
|
||||
app.add_middleware(BearerKeyMiddleware)
|
||||
|
||||
# Register canonical audio MIME types before any StaticFiles mount.
|
||||
# Python's `mimetypes.guess_type()` returns `audio/x-wav` for `.wav` and
|
||||
# `audio/x-flac` for `.flac` on most platforms — these are vendor-experimental
|
||||
# (x- prefix, never IANA-registered). macOS Chrome/Safari MIME-sniff leniently
|
||||
# via CoreAudio so playback works there, but Linux Chrome/Firefox (FFmpeg) and
|
||||
# Android Chrome (ExoPlayer) strictly honor the declared type and treat the
|
||||
# x- variants as download-only — manifesting as the play button silently
|
||||
# doing nothing in the browser app while working in the Tauri desktop shell.
|
||||
# `audio/wav` / `audio/flac` are the IANA-canonical types.
|
||||
# Ref: https://www.iana.org/assignments/media-types/media-types.xhtml#audio
|
||||
import mimetypes as _mimetypes
|
||||
_mimetypes.add_type("audio/wav", ".wav")
|
||||
_mimetypes.add_type("audio/flac", ".flac")
|
||||
|
||||
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
|
||||
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
|
||||
|
||||
@@ -789,6 +1035,7 @@ app.include_router(watermark.router)
|
||||
app.include_router(events.router)
|
||||
app.include_router(capture.router)
|
||||
app.include_router(capture_ws.router)
|
||||
app.include_router(dictation.router)
|
||||
app.include_router(openai_compat.router)
|
||||
app.include_router(tts_stream.router)
|
||||
app.include_router(marketplace.router)
|
||||
@@ -796,6 +1043,7 @@ app.include_router(personas.router)
|
||||
app.include_router(sonitranslate.router)
|
||||
app.include_router(audiobook.router)
|
||||
app.include_router(longform_jobs.router)
|
||||
app.include_router(pronunciation.router) # Expressive-TTS Spec 01: pronunciation dictionary
|
||||
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
|
||||
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
|
||||
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
|
||||
|
||||
@@ -17,7 +17,13 @@ from core.config import DB_PATH # noqa: E402 — backend/ is on sys.path via al
|
||||
|
||||
config = context.config
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
# `disable_existing_loggers=False` is deliberate: this env runs *inside* the
|
||||
# live app (startup `alembic upgrade head`), so the default (True) would
|
||||
# disable every already-created application logger — e.g. silence
|
||||
# `omnivoice.db.backup`'s "Skipping pre-migration DB backup" line and the
|
||||
# rest of the app's logging for the remainder of the process. A migration
|
||||
# must never mute the app (or leak that mute across a test session).
|
||||
fileConfig(config.config_file_name, disable_existing_loggers=False)
|
||||
|
||||
# SQLite file URL. Honour an externally-set URL (tests pass one via
|
||||
# `cfg.set_main_option("sqlalchemy.url", ...)` to point at a fixture DB),
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Heal voice_profiles.instruct poisoned with the "[object Object]" sentinel.
|
||||
|
||||
Revision ID: 0006_strip_object_object_instruct
|
||||
Revises: 0005_unified_profiles
|
||||
Create Date: 2026-06-20 00:00:00.000000
|
||||
|
||||
A pre-fix Voice Studio build ("Save design as profile") passed the
|
||||
``buildDesignInstruct()`` *object* straight to FormData, which string-coerced it
|
||||
to the literal ``"[object Object]"`` and persisted that into
|
||||
``voice_profiles.instruct`` (#550 #545 #542 #537 #530 #525). On first
|
||||
preview/use that value fails the engine instruct validator with a 400. The
|
||||
frontend + backend fixes stop any NEW poisoned rows; this migration heals the
|
||||
ones already saved on the buggy build (the local-first backward-compat rule —
|
||||
existing project data must keep working without manual migration).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
revision: str = "0006_strip_object_object_instruct"
|
||||
down_revision: Union[str, None] = "0005_unified_profiles"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if "voice_profiles" in inspect(bind).get_table_names():
|
||||
# Idempotent: only touches rows whose instruct is literally the sentinel.
|
||||
op.execute("UPDATE voice_profiles SET instruct='' WHERE instruct='[object Object]'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible heal — the original garbage sentinel is not worth restoring.
|
||||
pass
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Rebuild design-profile instructs poisoned with prose / "[object Object]".
|
||||
|
||||
Revision ID: 0007_rebuild_poisoned_design_instruct
|
||||
Revises: 0006_strip_object_object_instruct
|
||||
Create Date: 2026-06-22 00:00:00.000000
|
||||
|
||||
Migration 0006 *blanked* the literal ``"[object Object]"`` sentinel. That stops
|
||||
the 400 on use, but it also throws away the designed voice: a row that read
|
||||
``"[object Object]"`` (or freeform prose like "A gentle, quiet male voice…")
|
||||
becomes ``instruct=''`` and then renders with the engine's neutral default —
|
||||
which is why an Indonesian *female* designed voice came out *male* (#594), and
|
||||
why prose-poisoned designs still 400 (#571 #596).
|
||||
|
||||
This migration heals it properly: for every design profile it recomputes a
|
||||
validator-safe instruct, preferring any whitelist tags already in the stored
|
||||
value and otherwise rebuilding the tags from ``vd_states`` (the authoritative
|
||||
category→pick map the Voice Design picker persists). Non-design rows simply get
|
||||
their instruct sanitized (poison dropped). Idempotent — a healthy row is left
|
||||
byte-for-byte unchanged, so re-running is a no-op.
|
||||
|
||||
Self-contained by design: alembic migrations must not import evolving app code
|
||||
(``omnivoice`` would also drag in torch at startup), so the tag whitelist is a
|
||||
frozen snapshot of ``omnivoice.utils.voice_design._INSTRUCT_ALL_VALID``.
|
||||
``tests/test_migration_0007_instruct_rebuild.py`` asserts the snapshot stays in
|
||||
sync with the canonical set.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
revision: str = "0007_rebuild_poisoned_design_instruct"
|
||||
down_revision: Union[str, None] = "0006_strip_object_object_instruct"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
# Frozen snapshot of the design-instruct whitelist + mutually-exclusive
|
||||
# categories (omnivoice/utils/voice_design.py). Kept self-contained so the
|
||||
# migration's behaviour is pinned to the data it heals, not to future vocab
|
||||
# edits. Parity is guarded by the migration test.
|
||||
_CATEGORIES = [
|
||||
{"male", "女", "female", "男"},
|
||||
{"child", "teenager", "young adult", "middle-aged", "elderly",
|
||||
"儿童", "少年", "青年", "中年", "老年"},
|
||||
{"very low pitch", "low pitch", "moderate pitch", "high pitch", "very high pitch",
|
||||
"极低音调", "低音调", "中音调", "高音调", "极高音调"},
|
||||
{"whisper", "耳语"},
|
||||
{"american accent", "british accent", "australian accent", "chinese accent",
|
||||
"canadian accent", "indian accent", "korean accent", "portuguese accent",
|
||||
"russian accent", "japanese accent"},
|
||||
{"河南话", "陕西话", "四川话", "贵州话", "云南话", "桂林话",
|
||||
"济南话", "石家庄话", "甘肃话", "宁夏话", "青岛话", "东北话"},
|
||||
]
|
||||
_ALL_VALID = set().union(*_CATEGORIES)
|
||||
|
||||
|
||||
def _valid_from_items(items) -> str:
|
||||
"""One whitelist tag per category, first-seen order; everything else dropped."""
|
||||
seen = set()
|
||||
out = []
|
||||
for raw in items:
|
||||
tag = str(raw if raw is not None else "").strip().lower()
|
||||
if not tag or tag not in _ALL_VALID:
|
||||
continue
|
||||
ci = next((i for i, c in enumerate(_CATEGORIES) if tag in c), -1)
|
||||
if ci in seen:
|
||||
continue
|
||||
seen.add(ci)
|
||||
out.append(tag)
|
||||
return ", ".join(out)
|
||||
|
||||
|
||||
def _heal(instruct, vd_states, is_design) -> str:
|
||||
healed = _valid_from_items(re.split(r"\s*[,,]\s*", str(instruct or "").strip()))
|
||||
if healed or not is_design:
|
||||
return healed
|
||||
# Stored instruct was all-poison — recover the design from vd_states.
|
||||
if not vd_states:
|
||||
return ""
|
||||
try:
|
||||
vd = json.loads(vd_states)
|
||||
except (ValueError, TypeError):
|
||||
return ""
|
||||
return _valid_from_items(vd.values()) if isinstance(vd, dict) else ""
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
insp = inspect(bind)
|
||||
if "voice_profiles" not in insp.get_table_names():
|
||||
return
|
||||
cols = {c["name"] for c in insp.get_columns("voice_profiles")}
|
||||
has_kind = "kind" in cols
|
||||
has_vd = "vd_states" in cols
|
||||
|
||||
select = "SELECT id, instruct"
|
||||
select += ", kind" if has_kind else ""
|
||||
select += ", vd_states" if has_vd else ""
|
||||
select += " FROM voice_profiles"
|
||||
|
||||
for row in bind.exec_driver_sql(select).mappings().all():
|
||||
instruct = row["instruct"] or ""
|
||||
is_design = (row["kind"] == "design") if has_kind else bool(instruct)
|
||||
vd = row["vd_states"] if has_vd else None
|
||||
healed = _heal(instruct, vd, is_design)
|
||||
if healed != instruct:
|
||||
bind.exec_driver_sql(
|
||||
"UPDATE voice_profiles SET instruct = ? WHERE id = ?",
|
||||
(healed, row["id"]),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible heal — the original poisoned value isn't worth restoring.
|
||||
pass
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary
|
||||
|
||||
Revision ID: 0008_pronunciation_dictionary
|
||||
Revises: 0007_rebuild_poisoned_design_instruct
|
||||
Create Date: 2026-06-25 00:00:00.000000
|
||||
|
||||
Adds the ``pronunciation_entries`` table backing the user-editable, per-language
|
||||
pronunciation dictionary (Settings → Pronunciation). Each row maps a ``term`` to
|
||||
a ``replacement`` the engine pronounces correctly, scoped global (``language='*'``)
|
||||
or to a 2-letter language. Applied as pure text substitution before synthesis, so
|
||||
every engine honors it.
|
||||
|
||||
* ``id`` TEXT PRIMARY KEY — stable row id.
|
||||
* ``term`` TEXT — the word/phrase to match (whole-word, case-insensitive).
|
||||
* ``replacement`` TEXT — the respelling (or, for phoneme rows, the markup).
|
||||
* ``type`` TEXT — 'respelling' | 'ipa' | 'cmu'.
|
||||
* ``language`` TEXT — '*' = global, else a language code (e.g. 'en', 'de').
|
||||
* ``enabled`` INTEGER — 1 = applied, 0 = parked.
|
||||
* ``created_at`` REAL.
|
||||
|
||||
Additive + idempotent (guarded by sqlite_master), matching 0002/0003/0004, so
|
||||
re-running on a fresh-install DB where ``_BASE_SCHEMA`` already created the table
|
||||
is a no-op (Backward-compatible project data constraint). The same table is
|
||||
mirrored into ``core/db.py::_BASE_SCHEMA`` so fresh installs and migrated DBs
|
||||
converge on an identical end-state (the dual-path discipline).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0008_pronunciation_dictionary"
|
||||
down_revision: Union[str, None] = "0007_rebuild_poisoned_design_instruct"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_table(name: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
row = bind.execute(
|
||||
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
|
||||
{"n": name},
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("pronunciation_entries"):
|
||||
return
|
||||
op.create_table(
|
||||
"pronunciation_entries",
|
||||
sa.Column("id", sa.Text(), primary_key=True),
|
||||
sa.Column("term", sa.Text(), nullable=False),
|
||||
sa.Column("replacement", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("type", sa.Text(), nullable=False, server_default="respelling"),
|
||||
sa.Column("language", sa.Text(), nullable=False, server_default="*"),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default="1"),
|
||||
sa.Column("created_at", sa.Float(), nullable=True),
|
||||
)
|
||||
op.create_index("idx_pron_lang", "pronunciation_entries", ["language"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("pronunciation_entries"):
|
||||
op.drop_index("idx_pron_lang", table_name="pronunciation_entries")
|
||||
op.drop_table("pronunciation_entries")
|
||||
@@ -129,7 +129,7 @@ class TranslateRequest(BaseModel):
|
||||
provider: Optional[str] = None
|
||||
source_lang: Optional[str] = None # ISO 639-1; overrides job detection
|
||||
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect→adapt) | "autofit" (cinematic + strict fit-to-slot)
|
||||
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
|
||||
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
|
||||
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
|
||||
|
||||
+1011
-54
File diff suppressed because it is too large
Load Diff
@@ -1,9 +1,12 @@
|
||||
"""
|
||||
Audio DSP pipeline — broadcast-grade mastering + configurable effects chain.
|
||||
|
||||
The default `apply_mastering()` is the same chain shipped since v0.1.0
|
||||
(highpass + compressor + light reverb). The new `apply_effects_chain()`
|
||||
lets callers build custom pipelines from a list of named effects.
|
||||
`apply_mastering()` is the shared pre-stage that runs before the user's
|
||||
effect preset: highpass + gentle compression only (see `MASTERING_CHAIN`).
|
||||
Reverb is deliberately NOT part of it — it is preset-declared only (e.g.
|
||||
cinematic, warm); a hidden reverb here used to bake echo into every non-raw
|
||||
synthesis, which field reports flagged. `apply_effects_chain()` lets callers
|
||||
build custom pipelines from a list of named effects.
|
||||
|
||||
All effects use Spotify's `pedalboard` library. When pedalboard isn't
|
||||
installed, every function degrades gracefully (returns audio unmodified).
|
||||
@@ -97,24 +100,26 @@ def get_effect_chain(preset_id: str) -> list[dict]:
|
||||
|
||||
# ── Core DSP functions ──────────────────────────────────────────────────
|
||||
|
||||
#: Shared pre-preset mastering stage: highpass + gentle compression ONLY.
|
||||
#: Reverb must never live here — a hidden Reverb in this chain baked echo
|
||||
#: into every non-raw synthesis regardless of the chosen preset (field
|
||||
#: reports of echoey voices; the podcast preset even promises "no reverb").
|
||||
#: Reverb is preset-declared only (see EFFECT_PRESETS: cinematic, warm).
|
||||
MASTERING_CHAIN = [
|
||||
{"type": "highpass", "cutoff_hz": 60},
|
||||
{"type": "compressor", "threshold_db": -15, "ratio": 1.5, "attack_ms": 2.0, "release_ms": 100},
|
||||
]
|
||||
|
||||
|
||||
def apply_mastering(audio_tensor, sample_rate=24000):
|
||||
"""Applies professional Broadcast-grade DSP (EQ, Compressor, light Reverb) to the clone voice."""
|
||||
"""Applies the broadcast pre-stage (highpass + gentle compression) to the clone voice.
|
||||
|
||||
Reverb is intentionally absent — only user-chosen effect presets declare
|
||||
it. Degrades gracefully: pedalboard missing or any DSP error returns the
|
||||
input unmodified.
|
||||
"""
|
||||
try:
|
||||
from pedalboard import Pedalboard, Compressor, Reverb, HighpassFilter
|
||||
import numpy as np
|
||||
board = Pedalboard([
|
||||
HighpassFilter(cutoff_frequency_hz=60),
|
||||
Compressor(threshold_db=-15, ratio=1.5, attack_ms=2.0, release_ms=100),
|
||||
Reverb(room_size=0.10, wet_level=0.08, dry_level=0.95)
|
||||
])
|
||||
audio_np = audio_tensor.cpu().numpy()
|
||||
if audio_np.ndim == 1:
|
||||
audio_np = audio_np[np.newaxis, :]
|
||||
effected = board(audio_np, sample_rate, reset=False)
|
||||
return torch.from_numpy(effected).to(audio_tensor.device)
|
||||
except ImportError:
|
||||
return audio_tensor # Fail gracefully if pedalboard isn't installed
|
||||
return apply_effects_chain(audio_tensor, sample_rate, MASTERING_CHAIN)
|
||||
except Exception as e:
|
||||
logger.warning("Mastering DSP Error: %s", e)
|
||||
return audio_tensor
|
||||
|
||||
@@ -65,10 +65,14 @@ class AudiobookPlan:
|
||||
def char_count(self) -> int:
|
||||
return sum(c.char_count for c in self.chapters)
|
||||
|
||||
@property
|
||||
def chapter_count(self) -> int:
|
||||
return len(self.chapters)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"chapters": [c.to_dict() for c in self.chapters],
|
||||
"chapter_count": len(self.chapters),
|
||||
"chapter_count": self.chapter_count,
|
||||
"char_count": self.char_count,
|
||||
}
|
||||
|
||||
@@ -100,9 +104,10 @@ def synthesize_chapter(
|
||||
):
|
||||
"""Render a chapter's spans to one waveform via an injected ``synth``.
|
||||
|
||||
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
|
||||
span of text in the given voice (``speed`` may be ``None`` for the engine
|
||||
default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
``synth(text, voice_id, speed)`` returns a float32 audio tensor — 1-D
|
||||
``(samples,)`` or ``(channels, samples)``; real engines emit ``(1, samples)``
|
||||
per the ``TTSBackend`` contract (#897) — for a span of text in the given
|
||||
voice (``speed`` may be ``None`` for the engine default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
|
||||
given) respells each span's text before chunking so the engine pronounces
|
||||
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
|
||||
@@ -114,23 +119,35 @@ def synthesize_chapter(
|
||||
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
|
||||
from services.pronunciation import apply_lexicon
|
||||
|
||||
parts: list = []
|
||||
items: list = [] # ("a", tensor) for audio, ("s", n_samples) for silence
|
||||
for span in spans:
|
||||
if span.text:
|
||||
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
|
||||
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
|
||||
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
|
||||
if len(rendered) == 1:
|
||||
parts.append(rendered[0])
|
||||
items.append(("a", rendered[0]))
|
||||
elif rendered:
|
||||
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
|
||||
items.append(("a", concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms)))
|
||||
if span.pause_ms_after > 0:
|
||||
n = int(sample_rate * span.pause_ms_after / 1000.0)
|
||||
if n > 0:
|
||||
parts.append(torch.zeros(n, dtype=torch.float32))
|
||||
items.append(("s", n))
|
||||
|
||||
if not parts:
|
||||
if not items:
|
||||
return torch.zeros(0, dtype=torch.float32), 0.0
|
||||
# Engines return (1, samples) per the TTSBackend contract while a bare
|
||||
# zeros(n) is 1-D — mixing the two crashed the final concat (#897). So
|
||||
# materialize inter-span silence AFTER the loop, matching the rendered
|
||||
# audio's channel dims / dtype / device (same pattern as generation.py's
|
||||
# _render_with_pauses). A silence-only chapter stays 1-D float32 as before.
|
||||
ref = next((t for kind, t in items if kind == "a"), None)
|
||||
parts: list = [
|
||||
val if kind == "a"
|
||||
else (torch.zeros(val, dtype=torch.float32) if ref is None
|
||||
else torch.zeros(*ref.shape[:-1], val, dtype=ref.dtype, device=ref.device))
|
||||
for kind, val in items
|
||||
]
|
||||
# Hard-concat spans + silences (crossfading silence would bleed the gap).
|
||||
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
|
||||
return audio, audio.shape[-1] / float(sample_rate)
|
||||
|
||||
@@ -42,6 +42,46 @@ _ABBREVIATIONS = frozenset({
|
||||
# [pause 300ms] markers). The splitter must never cut inside one.
|
||||
_BRACKET_TAG_RE = re.compile(r"\[[^\]]*\]")
|
||||
|
||||
# Dense scripts (CJK ideographs, kana, Hangul) where ~1 character = 1 syllable,
|
||||
# so an N-char chunk is far more *speech* than N Latin chars. Counted by code
|
||||
# point (see _dense_char_count) so there are no literal CJK chars in source.
|
||||
def _dense_char_count(text: str) -> int:
|
||||
"""Number of CJK / kana / Hangul characters in *text* (dense scripts)."""
|
||||
n = 0
|
||||
for ch in text:
|
||||
o = ord(ch)
|
||||
if (0x3040 <= o <= 0x30FF or 0x3400 <= o <= 0x4DBF
|
||||
or 0x4E00 <= o <= 0x9FFF or 0xAC00 <= o <= 0xD7AF
|
||||
or 0xF900 <= o <= 0xFAFF):
|
||||
n += 1
|
||||
return n
|
||||
|
||||
# A chunk that is predominantly dense-script (>= this fraction) gets the smaller
|
||||
# limit; below it, the text is mostly spaced/Latin and the full limit applies.
|
||||
_DENSE_FRACTION_THRESHOLD = 0.3
|
||||
# Speech-per-char multiplier for dense scripts vs Latin (~1 ideograph ≈ 2.5
|
||||
# Latin chars of audio). Used to scale the char limit down.
|
||||
_DENSE_SPEECH_FACTOR = 2.5
|
||||
|
||||
|
||||
def _effective_max_chars(text: str, max_chars: int) -> int:
|
||||
"""Scale *max_chars* down for dense-script text (#505).
|
||||
|
||||
Long-form (5+ min) generation degrades — repeated / skipped / mispronounced
|
||||
words — when a single chunk's acoustic sequence gets too long. With CJK /
|
||||
kana / Hangul, ~1 char = 1 syllable, so an 800-char chunk is ~4-5 minutes of
|
||||
audio in one shot, well past the model's reliable range. When a chunk is
|
||||
predominantly dense-script, cap it to ``max_chars / _DENSE_SPEECH_FACTOR``
|
||||
(floored) so each chunk's spoken length stays bounded. Latin / spaced text
|
||||
is unchanged. ``max_chars <= 0`` (chunking disabled) is left untouched.
|
||||
"""
|
||||
if max_chars <= 0 or not text:
|
||||
return max_chars
|
||||
dense = _dense_char_count(text)
|
||||
if dense and dense / len(text) >= _DENSE_FRACTION_THRESHOLD:
|
||||
return max(120, min(max_chars, round(max_chars / _DENSE_SPEECH_FACTOR)))
|
||||
return max_chars
|
||||
|
||||
|
||||
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
|
||||
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
|
||||
@@ -54,6 +94,9 @@ def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS)
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return []
|
||||
# #505: dense-script text packs far more speech per char, so cap the chunk
|
||||
# smaller to keep each chunk's spoken length in the model's reliable range.
|
||||
max_chars = _effective_max_chars(text, max_chars)
|
||||
if max_chars <= 0 or len(text) <= max_chars:
|
||||
return [text]
|
||||
|
||||
@@ -140,14 +183,43 @@ def _safe_hard_cut(segment: str, max_chars: int) -> int:
|
||||
return cut
|
||||
|
||||
|
||||
def _normalize_chunk_shapes(chunks: list) -> list:
|
||||
"""Coerce mixed-rank / mixed-channel chunks to one concat-compatible shape.
|
||||
|
||||
Engines return ``(1, samples)`` per the ``TTSBackend.generate`` contract,
|
||||
but silence buffers and some model paths hand over bare ``(samples,)``
|
||||
tensors — ``torch.cat`` then dies with "Tensors must have same number of
|
||||
dimensions" (#897). Promote lower-rank chunks with leading singleton dims
|
||||
to the highest rank present, then broadcast singleton channel dims up to
|
||||
the widest channel count (mono follows stereo). Rank-homogeneous,
|
||||
channel-homogeneous input is returned untouched, so all-1-D / all-2-D
|
||||
callers keep their exact output shape; a genuine channel conflict
|
||||
(e.g. 2 vs 3 channels) still raises, which is the honest outcome.
|
||||
"""
|
||||
target = max(c.dim() for c in chunks)
|
||||
if any(c.dim() != target for c in chunks):
|
||||
promoted = []
|
||||
for c in chunks:
|
||||
while c.dim() < target:
|
||||
c = c.unsqueeze(0)
|
||||
promoted.append(c)
|
||||
chunks = promoted
|
||||
if target > 1:
|
||||
lead = tuple(max(c.shape[i] for c in chunks) for i in range(target - 1))
|
||||
chunks = [c if tuple(c.shape[:-1]) == lead else c.expand(*lead, -1)
|
||||
for c in chunks]
|
||||
return chunks
|
||||
|
||||
|
||||
def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
|
||||
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
|
||||
|
||||
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
|
||||
samples on the last axis — matching what ``_render_with_pauses`` handles).
|
||||
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
|
||||
is a hard concat.
|
||||
Mixed ranks / mono-vs-multichannel chunks are normalized to one shape
|
||||
first (#897), so no producer can crash the concat. Crossfade overlap is
|
||||
clamped to the shorter neighbor; ``crossfade_ms=0`` is a hard concat.
|
||||
"""
|
||||
import torch
|
||||
|
||||
@@ -156,6 +228,7 @@ def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
return torch.zeros(1, dtype=torch.float32)
|
||||
if len(chunks) == 1:
|
||||
return chunks[0]
|
||||
chunks = _normalize_chunk_shapes(chunks)
|
||||
|
||||
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
|
||||
result = chunks[0]
|
||||
|
||||
@@ -26,6 +26,10 @@ from services.llm_backend import get_active_llm_backend, OffBackend
|
||||
|
||||
logger = logging.getLogger("omnivoice.director")
|
||||
|
||||
# LLM Skills registry id — Settings → LLM Skills can disable the LLM parse
|
||||
# or route it to a specific provider. Disabled == the heuristic parser.
|
||||
_SKILL_ID = "direction_parse"
|
||||
|
||||
|
||||
# ── Taxonomy (stable contract) ──────────────────────────────────────────────
|
||||
# Additive per dimension — multiple values allowed. Unknown tokens are ignored
|
||||
@@ -147,7 +151,10 @@ def parse(text: str) -> Direction:
|
||||
if not text or not text.strip():
|
||||
return Direction(source=text or "")
|
||||
|
||||
llm = get_active_llm_backend()
|
||||
from services import llm_skills
|
||||
# `active=` forwards this module's (monkeypatch-able) name so the
|
||||
# no-override path is byte-identical to the pre-skills behavior.
|
||||
llm = llm_skills.skill_backend(_SKILL_ID, active=lambda: get_active_llm_backend())
|
||||
if isinstance(llm, OffBackend):
|
||||
return _heuristic_parse(text)
|
||||
|
||||
|
||||
@@ -187,6 +187,14 @@ def put_job(job_id: str, job: dict) -> None:
|
||||
def save_job(job_id: str, job: dict, filename: str = "", duration: float = 0.0, content_hash: str = "") -> None:
|
||||
"""Persist dub job state to SQLite so it survives restarts. Uses UPSERT
|
||||
on `id` so repeated saves in a session keep the latest snapshot.
|
||||
|
||||
language / language_code / content_hash only update when the incoming
|
||||
value is non-empty: the ingest-time insert runs before the target
|
||||
language is known (both columns ""), generation sets them on the job
|
||||
dict, and a later save from a job that lost them (e.g. hydrated from an
|
||||
old row) must not clobber the healed columns back to "". The frontend
|
||||
keys history restore off language_code, so a frozen "" hid finished
|
||||
tracks until the user re-picked a language.
|
||||
"""
|
||||
try:
|
||||
segments = job.get("segments") or []
|
||||
@@ -200,6 +208,8 @@ def save_job(job_id: str, job: dict, filename: str = "", duration: float = 0.0,
|
||||
filename=excluded.filename,
|
||||
duration=excluded.duration,
|
||||
segments_count=excluded.segments_count,
|
||||
language=CASE WHEN excluded.language != '' THEN excluded.language ELSE dub_history.language END,
|
||||
language_code=CASE WHEN excluded.language_code != '' THEN excluded.language_code ELSE dub_history.language_code END,
|
||||
tracks=excluded.tracks,
|
||||
job_data=excluded.job_data,
|
||||
content_hash=CASE WHEN excluded.content_hash != '' THEN excluded.content_hash ELSE dub_history.content_hash END""",
|
||||
@@ -435,6 +445,60 @@ def _ensure_browser_playable_mp4(video_path: str) -> str:
|
||||
return video_path
|
||||
|
||||
|
||||
# Bounded retry for transient download failures (#579/#598). yt-dlp's own
|
||||
# `retries`/`fragment_retries` cover per-fragment HTTP flakes, but a broken
|
||||
# pipe ([Errno 32]) raised while the write side of a pipe closes mid-stream
|
||||
# (a killed ffmpeg merge child, a CDN reset during muxing) aborts the whole
|
||||
# `extract_info` call and is NOT covered by them — so a single transient blip
|
||||
# failed the entire ingest with a raw "Broken pipe". We add a small download-
|
||||
# level retry on top, cleaning up the partial download between attempts so a
|
||||
# half-written `original.*` can't poison the next try.
|
||||
_YT_DOWNLOAD_RETRIES = 2 # total attempts = 1 + retries = 3
|
||||
|
||||
|
||||
def _is_transient_download_error(exc: BaseException) -> bool:
|
||||
"""True when a download failure is worth retrying (broken pipe / net drop).
|
||||
|
||||
Reuses the single failure taxonomy (`VIDEO_DOWNLOAD_NETWORK`) rather than a
|
||||
parallel keyword list, so "what counts as transient" stays single-sourced
|
||||
with the error-hint classification. ``BrokenPipeError``/``ConnectionError``
|
||||
are matched by class too, since a bare instance may be wrapped or re-raised
|
||||
with a stripped message that no longer contains "broken pipe".
|
||||
"""
|
||||
if isinstance(exc, (BrokenPipeError, ConnectionError)):
|
||||
return True
|
||||
return failure.classify(str(exc)) == "VIDEO_DOWNLOAD_NETWORK"
|
||||
|
||||
|
||||
# YouTube serves some videos' high-quality formats signature-protected to the
|
||||
# default player client, so the media download 403s even though extraction
|
||||
# worked. Forcing an alternate client commonly bypasses it; on a 403 we escalate
|
||||
# through these (in order) before giving up (#625).
|
||||
_YT_PLAYER_CLIENTS = ["tv", "android", "web_safari"]
|
||||
|
||||
|
||||
def _is_forbidden_download_error(exc: BaseException) -> bool:
|
||||
"""True for an HTTP 403 — not transient (the same client keeps 403ing), but
|
||||
often fixable by switching the YouTube player client."""
|
||||
s = str(exc)
|
||||
return "403" in s or "Forbidden" in s
|
||||
|
||||
|
||||
def _cleanup_partial_download(job_dir: str) -> None:
|
||||
"""Remove any half-written `original.*` files before a retry.
|
||||
|
||||
A partial download left on disk would otherwise be picked up as a "finished"
|
||||
file by the post-download codec probe, or collide with the next attempt's
|
||||
output. Best-effort — never raises on the failure path.
|
||||
"""
|
||||
import glob
|
||||
for stale in glob.glob(os.path.join(job_dir, "original.*")):
|
||||
try:
|
||||
os.remove(stale)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def yt_download_sync(
|
||||
url: str,
|
||||
job_dir: str,
|
||||
@@ -480,6 +544,12 @@ def yt_download_sync(
|
||||
"quiet": True,
|
||||
"no_warnings": True,
|
||||
"restrictfilenames": True,
|
||||
# Don't stamp the downloaded file's mtime with the video's upload date
|
||||
# (#642): on Windows an out-of-range/invalid timestamp makes the os.utime
|
||||
# call raise `[Errno 22] Invalid argument`, failing the whole ingest. We
|
||||
# download to a throwaway `original.*` and never use its mtime, so skip
|
||||
# it entirely (equivalent to yt-dlp's --no-mtime).
|
||||
"updatetime": False,
|
||||
"socket_timeout": 30,
|
||||
# Resilience against YouTube CDN flakes: a single empty fragment
|
||||
# (commonly the very last one — "Did not get any data blocks")
|
||||
@@ -491,17 +561,64 @@ def yt_download_sync(
|
||||
"extractor_retries": 5,
|
||||
"skip_unavailable_fragments": True,
|
||||
}
|
||||
# #712: the format selector above pulls separate video+audio streams, so
|
||||
# yt-dlp muxes them via ffmpeg (merge_output_format=mp4). yt-dlp only looks
|
||||
# for ffmpeg on PATH and aborts with "you have requested merging of multiple
|
||||
# formats but ffmpeg is not installed" — but OmniVoice's ffmpeg is often a
|
||||
# bundled Tauri sidecar / imageio-ffmpeg binary that isn't on PATH (common on
|
||||
# Windows). Point yt-dlp at the exact ffmpeg we resolve so the merge works.
|
||||
_ffmpeg_bin = find_ffmpeg()
|
||||
if _ffmpeg_bin:
|
||||
ydl_opts["ffmpeg_location"] = _ffmpeg_bin
|
||||
if progress_hook is not None:
|
||||
ydl_opts["progress_hooks"] = [progress_hook]
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
info = ydl.extract_info(url, download=True)
|
||||
path = ydl.prepare_filename(info)
|
||||
root, _ = os.path.splitext(path)
|
||||
mp4 = root + ".mp4"
|
||||
if os.path.exists(mp4):
|
||||
video_path = mp4
|
||||
else:
|
||||
video_path = path
|
||||
|
||||
# Download with a bounded retry on transient/broken-pipe-class failures
|
||||
# (#579/#598). A broken pipe mid-mux isn't recoverable inside yt-dlp's own
|
||||
# fragment retries, but a fresh `extract_info` usually succeeds. Between
|
||||
# attempts we wipe the partial `original.*` so a half-written file can't be
|
||||
# mistaken for a finished download.
|
||||
info = None
|
||||
path = None
|
||||
transient_used = 0
|
||||
client_idx = 0
|
||||
while True:
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
info = ydl.extract_info(url, download=True)
|
||||
path = ydl.prepare_filename(info)
|
||||
break
|
||||
except Exception as exc:
|
||||
_cleanup_partial_download(job_dir)
|
||||
# 403 Forbidden: not transient — escalate the YouTube player client,
|
||||
# which commonly bypasses a signature-protected format set (#625).
|
||||
if _is_forbidden_download_error(exc) and client_idx < len(_YT_PLAYER_CLIENTS):
|
||||
client = _YT_PLAYER_CLIENTS[client_idx]
|
||||
client_idx += 1
|
||||
ydl_opts = {**ydl_opts, "extractor_args": {"youtube": {"player_client": [client]}}}
|
||||
logger.warning(
|
||||
"Download 403 for %s — retrying with player_client=%s (#625)", url, client,
|
||||
)
|
||||
continue
|
||||
# Transient/broken-pipe: a fresh extract_info usually succeeds
|
||||
# (#579/#598). A 403 never counts here — it's escalated above.
|
||||
if (transient_used < _YT_DOWNLOAD_RETRIES
|
||||
and _is_transient_download_error(exc)
|
||||
and not _is_forbidden_download_error(exc)):
|
||||
transient_used += 1
|
||||
logger.warning(
|
||||
"Transient download failure for %s (attempt %d/%d): %s — retrying",
|
||||
url, transient_used, _YT_DOWNLOAD_RETRIES, exc,
|
||||
)
|
||||
time.sleep(2 * transient_used) # brief, increasing backoff
|
||||
continue
|
||||
raise
|
||||
root, _ = os.path.splitext(path)
|
||||
mp4 = root + ".mp4"
|
||||
if os.path.exists(mp4):
|
||||
video_path = mp4
|
||||
else:
|
||||
video_path = path
|
||||
# Browser-playability guard: WKWebView (Tauri on macOS) refuses to
|
||||
# decode VP9/AV1 video and Opus audio even when they're wrapped in an
|
||||
# mp4 container, and refuses .webm/.mkv outright. We probe the actual
|
||||
|
||||
@@ -63,7 +63,21 @@ def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResul
|
||||
"routing_reason": _caveat(caps),
|
||||
}
|
||||
|
||||
# 3. Host has an accelerator the engine lacks, but engine supports cpu
|
||||
# 3. CPU-native engine (declares ONLY cpu) has nothing to fall back FROM,
|
||||
# so on ANY accelerator host it is benign cpu_only (neutral), never a
|
||||
# warn-tone "CPU fallback". This must precede the fallback rule below —
|
||||
# a ("cpu",) engine matches `"cpu" in targets` too, and would otherwise
|
||||
# be mis-classed cpu_fallback on a GPU/MPS host. (A cpu host reaches
|
||||
# rule 5 unchanged, keeping its DirectML note.) Engines that *could*
|
||||
# accelerate elsewhere (e.g. ("cuda", "cpu")) are untouched.
|
||||
if fam != "cpu" and targets == ("cpu",):
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_only",
|
||||
"routing_reason": None,
|
||||
}
|
||||
|
||||
# 4. Host has an accelerator the engine lacks, but engine supports cpu
|
||||
# → the no-silent-fallback signal.
|
||||
if fam != "cpu" and "cpu" in targets:
|
||||
if fam == "rocm" and "cuda" in targets and "rocm" not in targets:
|
||||
@@ -76,7 +90,7 @@ def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResul
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 4. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
|
||||
# 5. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
|
||||
# and engine supports cpu → benign; must not warn or block.
|
||||
if fam == "cpu" and "cpu" in targets:
|
||||
reason = None
|
||||
@@ -93,7 +107,7 @@ def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResul
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 5. Engine needs an accelerator this host lacks and has no cpu path.
|
||||
# 6. Engine needs an accelerator this host lacks and has no cpu path.
|
||||
first = targets[0]
|
||||
return {
|
||||
"effective_device": first,
|
||||
|
||||
@@ -49,7 +49,7 @@ def _canon_value(field: str, value):
|
||||
return value
|
||||
|
||||
|
||||
def segment_fingerprint(seg: dict) -> str:
|
||||
def segment_fingerprint(seg: dict, track_lang: str | None = None) -> str:
|
||||
"""Deterministic hash of the inputs that actually affect TTS output.
|
||||
|
||||
Any change to `_GEN_INPUT_FIELDS` flips the hash and the segment becomes
|
||||
@@ -61,8 +61,20 @@ def segment_fingerprint(seg: dict) -> str:
|
||||
so a fingerprint computed from the generate request (server defaults
|
||||
filled in) matches one recomputed later from the client's raw segment
|
||||
state — the root cause of #281's "1 edit re-dubs all N lines".
|
||||
|
||||
``track_lang`` (P1.3) is the TRACK's language code (`req.language_code`,
|
||||
e.g. "es"). It is part of the fingerprint because the same segment text
|
||||
renders different audio per language — without it, a bn hash could
|
||||
vouch for an es WAV on a multi-track job. It is only mixed in when
|
||||
provided, so hashes computed by legacy callers (and hashes stored by
|
||||
previous builds, which never carried a language) keep their old values;
|
||||
a legacy hash therefore never matches a lang-scoped fingerprint and the
|
||||
segment reads as stale — the safe direction (one clean regen, never a
|
||||
wrong-language splice).
|
||||
"""
|
||||
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
|
||||
if track_lang:
|
||||
payload["track_lang"] = str(track_lang)
|
||||
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
|
||||
|
||||
@@ -120,6 +132,7 @@ def plan_incremental(
|
||||
segments: list[dict],
|
||||
*,
|
||||
stored_hashes: dict[str, str] | None = None,
|
||||
track_lang: str | None = None,
|
||||
) -> dict:
|
||||
"""Return `{stale, fresh, total, fingerprints}` where:
|
||||
|
||||
@@ -133,6 +146,13 @@ def plan_incremental(
|
||||
`stored_hashes` may come from the caller's own bookkeeping (e.g. the
|
||||
`dub_history.job_data["seg_hashes"]` we'll start writing in Phase 4.5).
|
||||
When missing, every segment is considered stale (first run).
|
||||
|
||||
`track_lang` (P1.3) scopes the plan to ONE dub track: pass the track's
|
||||
language code together with THAT language's stored hashes
|
||||
(`job_data["seg_hashes_by_lang"][lang]`) so staleness is judged against
|
||||
the active track, never against whatever language was generated last.
|
||||
Must match the language the generate run hashed with, or every segment
|
||||
reads stale (#281 parity class).
|
||||
"""
|
||||
stored = stored_hashes or {}
|
||||
stale: list[str] = []
|
||||
@@ -142,7 +162,7 @@ def plan_incremental(
|
||||
sid = str(seg.get("id", ""))
|
||||
if not sid:
|
||||
continue
|
||||
fp = segment_fingerprint(seg)
|
||||
fp = segment_fingerprint(seg, track_lang=track_lang)
|
||||
fingerprints[sid] = fp
|
||||
prev = stored.get(sid)
|
||||
if prev == fp:
|
||||
|
||||
@@ -54,9 +54,12 @@ class LLMBackend(ABC):
|
||||
def model_name(self) -> str: ...
|
||||
|
||||
@abstractmethod
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
"""One-shot chat completion. Returns the assistant content string.
|
||||
Raises on failure — callers decide whether to fallback gracefully.
|
||||
``temperature`` is only sent to the provider when set — callers that
|
||||
leave it None keep the provider default (existing behavior).
|
||||
"""
|
||||
|
||||
|
||||
@@ -67,8 +70,18 @@ class OpenAICompatBackend(LLMBackend):
|
||||
id = "openai-compat"
|
||||
display_name = "OpenAI-compatible (real OpenAI, Ollama, LM Studio, …)"
|
||||
|
||||
def __init__(self):
|
||||
def __init__(self, provider=None):
|
||||
"""``provider``: optional ``llm_providers.Provider`` to bind this
|
||||
instance to (LLM Skills per-skill routing). None keeps the historical
|
||||
behavior — resolve the ACTIVE provider at call time."""
|
||||
self._client = None
|
||||
self._provider = provider
|
||||
|
||||
def _resolve_provider(self):
|
||||
if self._provider is not None:
|
||||
return self._provider
|
||||
from services import llm_providers
|
||||
return llm_providers.active_provider()
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
@@ -76,67 +89,89 @@ class OpenAICompatBackend(LLMBackend):
|
||||
import openai # noqa: F401
|
||||
except ImportError:
|
||||
return False, "openai package missing (install with `pip install openai`)."
|
||||
base_url = os.environ.get("TRANSLATE_BASE_URL")
|
||||
api_key = (
|
||||
os.environ.get("TRANSLATE_API_KEY")
|
||||
or os.environ.get("OPENAI_API_KEY")
|
||||
or ("local" if base_url else None)
|
||||
)
|
||||
if not api_key:
|
||||
# Resolve through the provider registry — the active provider carries
|
||||
# its own base_url/key/model. Legacy single-endpoint setups (a lone
|
||||
# TRANSLATE_BASE_URL) resolve to the "custom" provider, so this stays
|
||||
# backward-compatible with pre-registry configs.
|
||||
from services import llm_providers
|
||||
p = llm_providers.active_provider()
|
||||
if p is None:
|
||||
return False, (
|
||||
"No LLM configured. Set TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY) to "
|
||||
"point at OpenAI, Ollama (http://localhost:11434/v1), or any compatible host."
|
||||
"No LLM configured. Add a provider key in Settings → LLM Providers "
|
||||
"(OpenAI/OpenRouter/Groq/… or a local Ollama), or set "
|
||||
"TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY)."
|
||||
)
|
||||
return True, "ready"
|
||||
if not llm_providers.resolve_base_url(p):
|
||||
return False, f"{p.display_name}: set a Base URL in Settings → LLM Providers."
|
||||
if not llm_providers.has_key(p):
|
||||
return False, f"{p.display_name}: add an API key in Settings → LLM Providers."
|
||||
return True, f"ready ({p.display_name})"
|
||||
|
||||
@property
|
||||
def model_name(self) -> str:
|
||||
from services import llm_providers
|
||||
p = self._resolve_provider()
|
||||
if p is not None:
|
||||
return llm_providers.resolve_model(p)
|
||||
return os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
|
||||
|
||||
def _get_client(self):
|
||||
if self._client is not None:
|
||||
return self._client
|
||||
from openai import OpenAI
|
||||
base_url = os.environ.get("TRANSLATE_BASE_URL")
|
||||
api_key = (
|
||||
os.environ.get("TRANSLATE_API_KEY")
|
||||
or os.environ.get("OPENAI_API_KEY")
|
||||
or ("local" if base_url else None)
|
||||
)
|
||||
from services import llm_providers
|
||||
p = self._resolve_provider()
|
||||
if p is None:
|
||||
raise RuntimeError("LLM not configured. See `is_available()` for the hint.")
|
||||
base_url = llm_providers.resolve_base_url(p)
|
||||
api_key = llm_providers.resolve_api_key(p)
|
||||
if not api_key:
|
||||
raise RuntimeError("LLM not configured. See `is_available()` for the hint.")
|
||||
kw = {"api_key": api_key}
|
||||
if base_url:
|
||||
kw["base_url"] = base_url
|
||||
self._client = OpenAI(**kw)
|
||||
# max_retries=0 so a 429 + Retry-After can't make one chat() sleep
|
||||
# through the Autofit fit-pass wall-clock budget (speech_rate).
|
||||
self._client = OpenAI(max_retries=0, **kw)
|
||||
return self._client
|
||||
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
return self.chat_messages(
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
timeout=timeout,
|
||||
temperature=temperature,
|
||||
)
|
||||
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
"""One-shot completion over a full message list.
|
||||
|
||||
Additive surface for callers that need structured few-shot turns
|
||||
(dictation refinement, Wave 2.1) — small local models pattern-match
|
||||
and echo inline examples, so examples must arrive as prior chat
|
||||
turns, not inside the system prompt.
|
||||
|
||||
``temperature`` is only forwarded when set (Cinematic/Autofit pin 0.2
|
||||
— the provider default of 1.0 makes local models drift and invent);
|
||||
every other caller leaves it None and keeps the provider default.
|
||||
"""
|
||||
if timeout is None:
|
||||
try:
|
||||
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
|
||||
except ValueError:
|
||||
timeout = 45.0
|
||||
kw = {}
|
||||
if temperature is not None:
|
||||
kw["temperature"] = temperature
|
||||
res = self._get_client().chat.completions.create(
|
||||
model=self.model_name,
|
||||
timeout=timeout,
|
||||
messages=messages,
|
||||
**kw,
|
||||
)
|
||||
return (res.choices[0].message.content or "").strip()
|
||||
|
||||
|
||||
@@ -0,0 +1,437 @@
|
||||
"""LLM provider registry — the OpenAI-compatible providers OmniVoice can use
|
||||
for Cinematic / Autofit translation (and any future LLM feature).
|
||||
|
||||
Every provider here speaks the OpenAI chat-completions shape, so a single
|
||||
client (`llm_backend.OpenAICompatBackend`) drives all of them — the only
|
||||
per-provider differences are ``base_url``, ``model``, and the API key. This
|
||||
module is the one place that knows those defaults and resolves the live value
|
||||
for the *active* provider.
|
||||
|
||||
Resolution precedence for every field (key / base_url / model), highest first:
|
||||
1. Environment variable — power-user / `.env` override, wins always.
|
||||
2. Encrypted settings store (UI-entered) — `settings_store.get_secret` for
|
||||
keys, `get_text` for base_url/model overrides.
|
||||
3. Built-in default from the table below.
|
||||
|
||||
Local providers (Ollama, LM Studio) need no key — a "local" sentinel is used
|
||||
so the OpenAI client is happy. This keeps the local-first path fully offline:
|
||||
nothing is sent anywhere unless the user picks a remote provider *and* a
|
||||
feature gate (quality="cinematic"/"autofit") fires.
|
||||
|
||||
Keys entered in the UI are stored **encrypted** (never in `.env`, never
|
||||
returned to the client). `.env` keys remain a valid override for CI / power
|
||||
users.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.llm_providers")
|
||||
|
||||
# Settings-store row names (non-secret overrides live in the plaintext table;
|
||||
# keys live in the encrypted secret table under ``llm_key.<id>``).
|
||||
_ACTIVE_PROVIDER_KEY = "llm.active_provider"
|
||||
_BASE_URL_KEY = "llm.base_url." # + provider id
|
||||
_MODEL_KEY = "llm.model." # + provider id
|
||||
SECRET_PREFIX = "llm_key." # + provider id → settings_store secret name
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Provider:
|
||||
id: str
|
||||
display_name: str
|
||||
default_base_url: str
|
||||
default_model: str
|
||||
# Env var names checked (in order) for the API key. First one set wins.
|
||||
key_envs: tuple[str, ...] = ()
|
||||
base_url_env: Optional[str] = None
|
||||
model_env: Optional[str] = None
|
||||
local: bool = False # runs on the user's machine → no key, offline
|
||||
# Key optional when a base_url is set (self-hosted OpenAI-compatible servers
|
||||
# — vLLM, LM Studio behind a custom URL — often ignore the key). Preserves
|
||||
# the pre-registry behaviour where a lone TRANSLATE_BASE_URL was usable
|
||||
# keyless.
|
||||
key_optional: bool = False
|
||||
needs_account: bool = False # Cloudflare: base_url needs an account id
|
||||
account_env: Optional[str] = None
|
||||
signup_url: str = ""
|
||||
notes: str = ""
|
||||
|
||||
|
||||
# Order here is the display order in the settings page. OpenAI first (the
|
||||
# canonical), then the free/fast cloud providers from the shipped .env, then
|
||||
# the local engines, then Custom.
|
||||
_PROVIDERS: tuple[Provider, ...] = (
|
||||
Provider("openai", "OpenAI", "https://api.openai.com/v1", "gpt-4o-mini",
|
||||
key_envs=("OPENAI_API_KEY", "TRANSLATE_API_KEY"),
|
||||
base_url_env="OPENAI_BASE_URL", model_env="OPENAI_MODEL",
|
||||
signup_url="https://platform.openai.com/api-keys",
|
||||
notes="GPT-4o / o-series. Highest quality; paid."),
|
||||
Provider("openrouter", "OpenRouter", "https://openrouter.ai/api/v1",
|
||||
"openai/gpt-4o-mini",
|
||||
key_envs=("OPENROUTER_API_KEY",), base_url_env="OPENROUTER_BASE_URL",
|
||||
model_env="OPENROUTER_MODEL",
|
||||
signup_url="https://openrouter.ai/keys",
|
||||
notes="One key, hundreds of models incl. free tiers."),
|
||||
Provider("groq", "Groq", "https://api.groq.com/openai/v1",
|
||||
"llama-3.3-70b-versatile",
|
||||
key_envs=("GROQ_API_KEY",), base_url_env="GROQ_BASE_URL",
|
||||
model_env="GROQ_MODEL", signup_url="https://console.groq.com/keys",
|
||||
notes="Very fast Llama/Mixtral inference. Generous free tier."),
|
||||
Provider("cerebras", "Cerebras", "https://api.cerebras.ai/v1",
|
||||
"llama-3.3-70b",
|
||||
key_envs=("CEREBRAS_API_KEY",), base_url_env="CEREBRAS_BASE_URL",
|
||||
model_env="CEREBRAS_MODEL", signup_url="https://cloud.cerebras.ai",
|
||||
notes="Fastest Llama inference. Free tier."),
|
||||
Provider("google-ai", "Google AI (Gemini)",
|
||||
"https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
"gemini-2.0-flash",
|
||||
key_envs=("GOOGLE_AI_API_KEY",), base_url_env="GOOGLE_AI_BASE_URL",
|
||||
model_env="GOOGLE_AI_MODEL",
|
||||
signup_url="https://aistudio.google.com/app/apikey",
|
||||
notes="Gemini via OpenAI-compatible endpoint. Free tier."),
|
||||
Provider("mistral", "Mistral", "https://api.mistral.ai/v1",
|
||||
"mistral-small-latest",
|
||||
key_envs=("MISTRAL_API_KEY",), base_url_env="MISTRAL_BASE_URL",
|
||||
model_env="MISTRAL_MODEL", signup_url="https://console.mistral.ai/api-keys",
|
||||
notes="Strong multilingual models. Free tier."),
|
||||
Provider("cohere", "Cohere", "https://api.cohere.ai/compatibility/v1",
|
||||
"command-r-08-2024",
|
||||
key_envs=("COHERE_API_KEY",), base_url_env="COHERE_BASE_URL",
|
||||
model_env="COHERE_MODEL", signup_url="https://dashboard.cohere.com/api-keys",
|
||||
notes="Command models; good for RAG/translation. Free trial keys."),
|
||||
Provider("nvidia", "NVIDIA NIM", "https://integrate.api.nvidia.com/v1",
|
||||
"meta/llama-3.3-70b-instruct",
|
||||
key_envs=("NVIDIA_API_KEY",), base_url_env="NVIDIA_BASE_URL",
|
||||
model_env="NVIDIA_MODEL", signup_url="https://build.nvidia.com",
|
||||
notes="NIM-hosted open models. Free credits."),
|
||||
Provider("github-models", "GitHub Models",
|
||||
"https://models.github.ai/inference", "openai/gpt-4o-mini",
|
||||
key_envs=("GITHUB_MODELS_API_KEY",), base_url_env="GITHUB_MODELS_BASE_URL",
|
||||
model_env="GITHUB_MODELS_MODEL",
|
||||
signup_url="https://github.com/settings/tokens",
|
||||
notes="Uses a GitHub PAT. Free for dev, rate-limited."),
|
||||
Provider("cloudflare", "Cloudflare Workers AI",
|
||||
"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1",
|
||||
"@cf/meta/llama-3.3-70b-instruct-fp8-fast",
|
||||
key_envs=("CLOUDFLARE_API_KEY",), base_url_env="CLOUDFLARE_BASE_URL",
|
||||
model_env="CLOUDFLARE_MODEL", needs_account=True,
|
||||
account_env="CLOUDFLARE_ACCOUNT_ID",
|
||||
signup_url="https://dash.cloudflare.com/profile/api-tokens",
|
||||
notes="Needs an Account ID. Free tier."),
|
||||
Provider("huggingface", "Hugging Face", "https://router.huggingface.co/v1",
|
||||
"meta-llama/Llama-3.3-70B-Instruct",
|
||||
key_envs=("HUGGINGFACE_API_KEY", "HF_TOKEN"),
|
||||
base_url_env="HUGGINGFACE_BASE_URL", model_env="HUGGINGFACE_MODEL",
|
||||
signup_url="https://huggingface.co/settings/tokens",
|
||||
notes="HF Inference router. Reuses your HF token."),
|
||||
Provider("sambanova", "SambaNova", "https://api.sambanova.ai/v1",
|
||||
"Meta-Llama-3.3-70B-Instruct",
|
||||
key_envs=("SAMBANOVA_API_KEY",), base_url_env="SAMBANOVA_BASE_URL",
|
||||
model_env="SAMBANOVA_MODEL", signup_url="https://cloud.sambanova.ai",
|
||||
notes="Fast open models. Free tier."),
|
||||
Provider("siliconflow", "SiliconFlow", "https://api.siliconflow.com/v1",
|
||||
"Qwen/Qwen2.5-7B-Instruct",
|
||||
key_envs=("SILICONFLOW_API_KEY",), base_url_env="SILICONFLOW_BASE_URL",
|
||||
model_env="SILICONFLOW_MODEL", signup_url="https://siliconflow.com",
|
||||
notes="Qwen/DeepSeek and more. Strong for CJK."),
|
||||
Provider("ollama", "Ollama (local)", "http://localhost:11434/v1",
|
||||
"llama3.1", local=True,
|
||||
base_url_env="OLLAMA_BASE_URL", model_env="OLLAMA_MODEL",
|
||||
signup_url="https://ollama.com",
|
||||
notes="Fully offline. Run `ollama pull llama3.1` first."),
|
||||
Provider("lmstudio", "LM Studio (local)", "http://localhost:1234/v1",
|
||||
"local-model", local=True,
|
||||
base_url_env="LMSTUDIO_BASE_URL", model_env="LMSTUDIO_MODEL",
|
||||
signup_url="https://lmstudio.ai",
|
||||
notes="Fully offline. Start the LM Studio local server."),
|
||||
Provider("custom", "Custom (OpenAI-compatible)", "", "",
|
||||
key_envs=("TRANSLATE_API_KEY",), base_url_env="TRANSLATE_BASE_URL",
|
||||
model_env="TRANSLATE_MODEL", key_optional=True,
|
||||
notes="Any OpenAI-compatible host. Set Base URL + Model (+ key)."),
|
||||
)
|
||||
|
||||
_BY_ID: dict[str, Provider] = {p.id: p for p in _PROVIDERS}
|
||||
|
||||
|
||||
def all_providers() -> tuple[Provider, ...]:
|
||||
return _PROVIDERS
|
||||
|
||||
|
||||
def get_provider(pid: str) -> Optional[Provider]:
|
||||
return _BY_ID.get(pid)
|
||||
|
||||
|
||||
# ── Field resolution (env → store → default) ──────────────────────────────
|
||||
|
||||
def _env_first(names: tuple[str, ...]) -> Optional[str]:
|
||||
for n in names:
|
||||
v = os.environ.get(n)
|
||||
if v:
|
||||
return v
|
||||
return None
|
||||
|
||||
|
||||
def resolve_account_id(p: Provider) -> str:
|
||||
"""The Cloudflare-style account id: env override → stored → empty."""
|
||||
from services import settings_store
|
||||
return (
|
||||
(p.account_env and os.environ.get(p.account_env))
|
||||
or settings_store.get_text(f"llm.account.{p.id}")
|
||||
or ""
|
||||
)
|
||||
|
||||
|
||||
def resolve_base_url(p: Provider, *, substitute: bool = True) -> str:
|
||||
"""Resolve a provider's base URL (env → stored override → default).
|
||||
|
||||
``substitute`` interpolates ``{account_id}`` for account-scoped providers
|
||||
(Cloudflare) so the *client* gets a working URL. The UI passes
|
||||
``substitute=False`` so the field shows/saves the raw template — baking the
|
||||
substituted value back into a stored override would freeze the URL and make
|
||||
later account-id changes silently no-op (the bug this guards against).
|
||||
"""
|
||||
from services import settings_store
|
||||
val = (
|
||||
(p.base_url_env and os.environ.get(p.base_url_env))
|
||||
or settings_store.get_text(_BASE_URL_KEY + p.id)
|
||||
or p.default_base_url
|
||||
)
|
||||
if substitute and p.needs_account and val and "{account_id}" in val:
|
||||
val = val.replace("{account_id}", resolve_account_id(p))
|
||||
return val or ""
|
||||
|
||||
|
||||
def resolve_model(p: Provider) -> str:
|
||||
from services import settings_store
|
||||
return (
|
||||
(p.model_env and os.environ.get(p.model_env))
|
||||
or settings_store.get_text(_MODEL_KEY + p.id)
|
||||
or p.default_model
|
||||
)
|
||||
|
||||
|
||||
def resolve_api_key(p: Provider) -> Optional[str]:
|
||||
"""Env key → encrypted stored key → 'local' sentinel for local/keyless."""
|
||||
from services import settings_store
|
||||
env_key = _env_first(p.key_envs)
|
||||
if env_key:
|
||||
return env_key
|
||||
stored = settings_store.get_secret(SECRET_PREFIX + p.id)
|
||||
if stored:
|
||||
return stored
|
||||
if p.local or (p.key_optional and resolve_base_url(p)):
|
||||
return "local" # self-hosted OpenAI-compatible servers ignore the key
|
||||
return None
|
||||
|
||||
|
||||
def has_key(p: Provider) -> bool:
|
||||
"""True if a usable key is resolvable (local, or keyless-with-base_url)."""
|
||||
if p.local:
|
||||
return True
|
||||
if _env_first(p.key_envs) or _key_in_store(p.id):
|
||||
return True
|
||||
return bool(p.key_optional and resolve_base_url(p))
|
||||
|
||||
|
||||
def _key_in_store(pid: str) -> bool:
|
||||
from services import settings_store
|
||||
return (SECRET_PREFIX + pid) in settings_store.list_secret_names()
|
||||
|
||||
|
||||
def is_configured(p: Provider) -> bool:
|
||||
"""Usable end-to-end: has a base_url (custom needs one set) and a key."""
|
||||
if not resolve_base_url(p):
|
||||
return False
|
||||
return has_key(p)
|
||||
|
||||
|
||||
# ── Active provider selection ─────────────────────────────────────────────
|
||||
|
||||
def stored_active_provider_id() -> Optional[str]:
|
||||
"""The user's explicitly-persisted selection ONLY — no env pin, no legacy
|
||||
TRANSLATE_* fallback, no auto-detect.
|
||||
|
||||
``None`` means the user has never chosen a provider. This is what gates
|
||||
save-activates in the settings router (#963): an explicit save may claim
|
||||
the *empty* slot, but must never steal it from a made choice.
|
||||
"""
|
||||
from services import settings_store
|
||||
stored = settings_store.get_text(_ACTIVE_PROVIDER_KEY)
|
||||
return stored if stored and stored in _BY_ID else None
|
||||
|
||||
|
||||
def active_provider_id() -> Optional[str]:
|
||||
"""The provider Cinematic/Autofit should use.
|
||||
|
||||
Precedence: env ``LLM_DEFAULT_PROVIDER`` → stored selection → first
|
||||
configured provider → None. Legacy ``TRANSLATE_BASE_URL`` users with no
|
||||
explicit selection resolve to ``custom`` (its envs are TRANSLATE_*).
|
||||
"""
|
||||
env_pick = os.environ.get("LLM_DEFAULT_PROVIDER")
|
||||
if env_pick and env_pick in _BY_ID:
|
||||
return env_pick
|
||||
stored = stored_active_provider_id()
|
||||
if stored:
|
||||
return stored
|
||||
# Legacy: a lone TRANSLATE_BASE_URL means the old single-endpoint setup.
|
||||
if os.environ.get("TRANSLATE_BASE_URL"):
|
||||
return "custom"
|
||||
# Auto-select only a provider with a real key. Local providers (Ollama/
|
||||
# LM Studio) are *always* "configured" (no key needed) but we must NOT
|
||||
# assume their server is running — they require an explicit selection.
|
||||
for p in _PROVIDERS:
|
||||
if not p.local and is_configured(p):
|
||||
return p.id
|
||||
return None
|
||||
|
||||
|
||||
def set_active_provider(pid: str) -> None:
|
||||
from services import settings_store
|
||||
if pid not in _BY_ID:
|
||||
raise ValueError(f"unknown provider {pid!r}")
|
||||
settings_store.set_text(_ACTIVE_PROVIDER_KEY, pid)
|
||||
|
||||
|
||||
def active_provider() -> Optional[Provider]:
|
||||
pid = active_provider_id()
|
||||
return _BY_ID.get(pid) if pid else None
|
||||
|
||||
|
||||
# ── UI + persistence helpers ──────────────────────────────────────────────
|
||||
|
||||
def save_key(pid: str, api_key: str) -> None:
|
||||
"""Persist (encrypted) or clear an API key for a provider."""
|
||||
from services import settings_store
|
||||
if pid not in _BY_ID:
|
||||
raise ValueError(f"unknown provider {pid!r}")
|
||||
settings_store.set_secret(SECRET_PREFIX + pid, api_key or "")
|
||||
|
||||
|
||||
def save_overrides(pid: str, *, base_url: Optional[str] = None,
|
||||
model: Optional[str] = None,
|
||||
account_id: Optional[str] = None) -> None:
|
||||
from services import settings_store
|
||||
if pid not in _BY_ID:
|
||||
raise ValueError(f"unknown provider {pid!r}")
|
||||
p = _BY_ID[pid]
|
||||
if base_url is not None:
|
||||
bu = base_url.strip()
|
||||
# Never freeze an override that equals the built-in default. Critical
|
||||
# for account-templated URLs (Cloudflare): persisting the shown value
|
||||
# would pin the base_url and stop later account-id edits from taking
|
||||
# effect. Clearing (→ empty) falls the resolver back to the default
|
||||
# template so substitution stays live. Also self-heals a stale override
|
||||
# if a provider's default URL changes in a future release.
|
||||
settings_store.set_text(_BASE_URL_KEY + pid, "" if bu == p.default_base_url else bu)
|
||||
if model is not None:
|
||||
settings_store.set_text(_MODEL_KEY + pid, model.strip())
|
||||
if account_id is not None:
|
||||
settings_store.set_text(f"llm.account.{pid}", account_id.strip())
|
||||
|
||||
|
||||
def _active_env_pin() -> Optional[str]:
|
||||
"""The provider id pinned by ``LLM_DEFAULT_PROVIDER`` (if set + valid)."""
|
||||
pick = os.environ.get("LLM_DEFAULT_PROVIDER")
|
||||
return pick if pick and pick in _BY_ID else None
|
||||
|
||||
|
||||
def describe(p: Provider) -> dict:
|
||||
"""Client-safe provider descriptor — NEVER includes the key material.
|
||||
|
||||
The ``*_from_env`` booleans mirror ``key_from_env`` so the UI can disable an
|
||||
env-pinned field (and the make-active button) with an explainer instead of
|
||||
letting the user edit a value the resolver will silently override. ``base_url``
|
||||
is the RAW template (``substitute=False``) so an account-scoped default shows
|
||||
``{account_id}`` rather than a baked-in value; ``account_id`` is returned
|
||||
separately for account-scoped providers so the field can round-trip.
|
||||
"""
|
||||
d = {
|
||||
"id": p.id,
|
||||
"display_name": p.display_name,
|
||||
"local": p.local,
|
||||
"needs_account": p.needs_account,
|
||||
"signup_url": p.signup_url,
|
||||
"notes": p.notes,
|
||||
"base_url": resolve_base_url(p, substitute=False),
|
||||
"model": resolve_model(p),
|
||||
"has_key": has_key(p),
|
||||
"key_from_env": bool(_env_first(p.key_envs)),
|
||||
"base_url_from_env": bool(p.base_url_env and os.environ.get(p.base_url_env)),
|
||||
"model_from_env": bool(p.model_env and os.environ.get(p.model_env)),
|
||||
"active_from_env": _active_env_pin() is not None,
|
||||
"configured": is_configured(p),
|
||||
}
|
||||
if p.needs_account:
|
||||
d["account_id"] = resolve_account_id(p)
|
||||
d["account_from_env"] = bool(p.account_env and os.environ.get(p.account_env))
|
||||
return d
|
||||
|
||||
|
||||
# ── Legacy TRANSLATE_* prefs migration (#963) ──────────────────────────────
|
||||
|
||||
# prefs.json row → the custom-provider field it becomes.
|
||||
_LEGACY_TRANSLATE_PREFS: tuple[tuple[str, str], ...] = (
|
||||
("env.TRANSLATE_BASE_URL", "base_url"),
|
||||
("env.TRANSLATE_MODEL", "model"),
|
||||
("env.TRANSLATE_API_KEY", "api_key"),
|
||||
)
|
||||
|
||||
|
||||
def migrate_legacy_translate_prefs() -> bool:
|
||||
"""Move the retired (≤v0.3.7) Translation-LLM panel's prefs rows into the
|
||||
``custom`` provider's own settings-store rows, then delete them.
|
||||
|
||||
Those ``env.TRANSLATE_*`` rows in prefs.json are re-imported into
|
||||
``os.environ`` on every launch (main.py), and a live ``TRANSLATE_BASE_URL``
|
||||
makes :func:`active_provider_id` resolve to ``custom`` ahead of the stored
|
||||
selection fallbacks — silently hijacking the active slot on every restart
|
||||
(issue #963, "Ollama works until I restart"). Must run BEFORE main.py's
|
||||
prefs→env import so the rows never reach the environment.
|
||||
|
||||
Semantics:
|
||||
* Each value is copied only where the store has no value yet — a user's
|
||||
later edit of the custom provider always wins over legacy leftovers.
|
||||
* The prefs row is deleted afterwards either way, so it can never be
|
||||
re-imported as env again (the migration is one-shot per row).
|
||||
* Real process env vars are NEVER touched — a shell/.env
|
||||
``TRANSLATE_BASE_URL`` keeps its documented override behavior.
|
||||
* A row whose store write fails is kept in prefs (it still works via the
|
||||
env import this launch and the migration retries next launch).
|
||||
|
||||
Returns True if any prefs row was migrated/removed.
|
||||
"""
|
||||
from core import prefs
|
||||
from services import settings_store
|
||||
|
||||
changed = False
|
||||
for prefs_key, field in _LEGACY_TRANSLATE_PREFS:
|
||||
try:
|
||||
raw = prefs.get(prefs_key)
|
||||
except Exception:
|
||||
logger.exception("legacy TRANSLATE prefs read failed (%s)", prefs_key)
|
||||
return changed
|
||||
if raw is None:
|
||||
continue
|
||||
val = str(raw).strip()
|
||||
try:
|
||||
if val:
|
||||
if field == "base_url":
|
||||
if not settings_store.get_text(_BASE_URL_KEY + "custom"):
|
||||
save_overrides("custom", base_url=val)
|
||||
elif field == "model":
|
||||
if not settings_store.get_text(_MODEL_KEY + "custom"):
|
||||
save_overrides("custom", model=val)
|
||||
else: # api_key — encrypted store, never overwrite an existing one
|
||||
if not _key_in_store("custom"):
|
||||
save_key("custom", val)
|
||||
prefs.delete(prefs_key)
|
||||
changed = True
|
||||
except Exception:
|
||||
# Store not ready (e.g. settings table missing) — keep the prefs
|
||||
# row so the legacy env import still works and we retry next boot.
|
||||
logger.exception("legacy TRANSLATE prefs migration failed (%s)", prefs_key)
|
||||
return changed
|
||||
@@ -0,0 +1,323 @@
|
||||
"""LLM Skills registry — per-feature enable/route control for every LLM call.
|
||||
|
||||
Every LLM-powered capability ("skill") in the backend is registered here, so
|
||||
the Settings → LLM Skills panel can (a) toggle it and (b) route it to a
|
||||
specific provider (a local Ollama/LM Studio vs a remote key) instead of
|
||||
everything riding the one global active provider.
|
||||
|
||||
The six consumption points today:
|
||||
|
||||
dub_translation — api/routers/dub_translate.py (the Dub tab's direct
|
||||
"LLM" translation engine; provider=openai branch)
|
||||
cinematic_translation — services/translator.py (Cinematic + Autofit
|
||||
REFLECT/ADAPT rewrite; dub_translate quality gate)
|
||||
slot_fitting — services/speech_rate.py (trim/expand a line to its
|
||||
time slot; Autofit strict pass + /tools/rate-fit)
|
||||
glossary_extract — api/routers/glossary.py auto-extract
|
||||
direction_parse — services/director.py (natural-language direction →
|
||||
taxonomy tokens; /tools/direction + dub generate)
|
||||
dictation_refinement — services/refinement.py (dictation transcript
|
||||
cleanup on finals)
|
||||
|
||||
Design rules:
|
||||
|
||||
* **Disabled == unconfigured.** A disabled skill degrades through the exact
|
||||
same path the feature takes today when no LLM is configured (Fast
|
||||
translation fallback, refinement pass-through, heuristic direction parse,
|
||||
no-llm slot fit, 503 on glossary auto-extract). No new degradation modes.
|
||||
* **Override > active > none.** A per-skill provider override (persisted in
|
||||
settings_store) wins over the global active provider. No override → the
|
||||
active provider, resolved exactly as before (so existing setups see zero
|
||||
behavior change; all skills default to enabled with no override).
|
||||
* **Persistence** is two plaintext settings rows per skill:
|
||||
``llm_skill.<id>.enabled`` ("1"/"0", absent = enabled) and
|
||||
``llm_skill.<id>.provider`` (provider id, absent/empty = active provider).
|
||||
Keys stay in the provider registry (encrypted) — nothing secret here.
|
||||
* ``OMNIVOICE_LLM_BACKEND=off`` remains the global kill switch: it also
|
||||
silences skills routed through a per-skill override.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.llm_skills")
|
||||
|
||||
_ENABLED_KEY = "llm_skill.{sid}.enabled"
|
||||
_PROVIDER_KEY = "llm_skill.{sid}.provider"
|
||||
|
||||
_UNSET = object()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LLMSkill:
|
||||
"""A registered LLM consumption point. name/description resolve via the
|
||||
frontend i18n layer (localization hard rule — no hardcoded UI text)."""
|
||||
|
||||
id: str
|
||||
name_key: str
|
||||
description_key: str
|
||||
|
||||
|
||||
def _skill(sid: str) -> LLMSkill:
|
||||
return LLMSkill(
|
||||
id=sid,
|
||||
name_key=f"settings.llmskills_{sid}_name",
|
||||
description_key=f"settings.llmskills_{sid}_desc",
|
||||
)
|
||||
|
||||
|
||||
# Display order in the settings panel: the dub pipeline first (translation →
|
||||
# refine → fit → glossary → direction), then dictation.
|
||||
_SKILLS: tuple[LLMSkill, ...] = (
|
||||
_skill("dub_translation"),
|
||||
_skill("cinematic_translation"),
|
||||
_skill("slot_fitting"),
|
||||
_skill("glossary_extract"),
|
||||
_skill("direction_parse"),
|
||||
_skill("dictation_refinement"),
|
||||
)
|
||||
|
||||
_BY_ID: dict[str, LLMSkill] = {s.id: s for s in _SKILLS}
|
||||
|
||||
|
||||
def all_skills() -> tuple[LLMSkill, ...]:
|
||||
return _SKILLS
|
||||
|
||||
|
||||
def get_skill(skill_id: str) -> Optional[LLMSkill]:
|
||||
return _BY_ID.get(skill_id)
|
||||
|
||||
|
||||
# ── Persistence (settings_store text rows) ─────────────────────────────────
|
||||
|
||||
|
||||
def is_enabled(skill_id: str) -> bool:
|
||||
"""Skill toggle. Absent row = enabled (all skills default on)."""
|
||||
from services import settings_store
|
||||
|
||||
raw = settings_store.get_text(_ENABLED_KEY.format(sid=skill_id))
|
||||
return raw != "0"
|
||||
|
||||
|
||||
def provider_override(skill_id: str) -> Optional[str]:
|
||||
"""The per-skill provider id, or None when the skill follows the active
|
||||
provider. A stored id that no longer exists in the registry reads as None
|
||||
(stale override — resolution falls back to the active provider)."""
|
||||
from services import llm_providers, settings_store
|
||||
|
||||
raw = (settings_store.get_text(_PROVIDER_KEY.format(sid=skill_id)) or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
if llm_providers.get_provider(raw) is None:
|
||||
logger.warning("llm_skills: stale provider override %r on %s — ignoring",
|
||||
raw, skill_id)
|
||||
return None
|
||||
return raw
|
||||
|
||||
|
||||
def configure_skill(skill_id: str, *, enabled: Optional[bool] = None,
|
||||
provider_override: Any = _UNSET) -> None:
|
||||
"""Persist a skill's toggle and/or provider routing.
|
||||
|
||||
``provider_override``: omit to leave unchanged; ``None``/``""`` clears it
|
||||
(skill follows the active provider); a provider id routes the skill there.
|
||||
Raises KeyError for an unknown skill, ValueError for an unknown provider.
|
||||
"""
|
||||
if skill_id not in _BY_ID:
|
||||
raise KeyError(f"unknown LLM skill {skill_id!r}. Known: {sorted(_BY_ID)}")
|
||||
from services import llm_providers, settings_store
|
||||
|
||||
if enabled is not None:
|
||||
settings_store.set_text(_ENABLED_KEY.format(sid=skill_id),
|
||||
"1" if enabled else "0")
|
||||
if provider_override is not _UNSET:
|
||||
pid = (provider_override or "").strip()
|
||||
if pid and llm_providers.get_provider(pid) is None:
|
||||
raise ValueError(f"unknown provider {pid!r}")
|
||||
settings_store.set_text(_PROVIDER_KEY.format(sid=skill_id), pid)
|
||||
|
||||
|
||||
# ── Resolution (override > active > none) ──────────────────────────────────
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkillResolution:
|
||||
skill: LLMSkill
|
||||
enabled: bool
|
||||
provider: Optional[Any] # llm_providers.Provider or None
|
||||
source: str # "override" | "active" | "none"
|
||||
ready: bool
|
||||
reason: Optional[str] # None | "disabled" | "no_provider" | "unconfigured"
|
||||
|
||||
|
||||
def resolve_skill(skill_id: str) -> SkillResolution:
|
||||
"""Resolve a skill's effective provider + ready status.
|
||||
|
||||
Precedence: per-skill override → global active provider → none. Ready
|
||||
means enabled AND the effective provider is configured end-to-end.
|
||||
Raises KeyError for an unknown skill.
|
||||
"""
|
||||
skill = _BY_ID.get(skill_id)
|
||||
if skill is None:
|
||||
raise KeyError(f"unknown LLM skill {skill_id!r}. Known: {sorted(_BY_ID)}")
|
||||
from services import llm_providers
|
||||
|
||||
enabled = is_enabled(skill_id)
|
||||
override = provider_override(skill_id)
|
||||
if override:
|
||||
provider = llm_providers.get_provider(override)
|
||||
source = "override"
|
||||
else:
|
||||
provider = llm_providers.active_provider()
|
||||
source = "active" if provider is not None else "none"
|
||||
|
||||
if not enabled:
|
||||
ready, reason = False, "disabled"
|
||||
elif provider is None:
|
||||
ready, reason = False, "no_provider"
|
||||
elif not llm_providers.is_configured(provider):
|
||||
ready, reason = False, "unconfigured"
|
||||
else:
|
||||
ready, reason = True, None
|
||||
return SkillResolution(skill=skill, enabled=enabled, provider=provider,
|
||||
source=source, ready=ready, reason=reason)
|
||||
|
||||
|
||||
def effective_provider(skill_id: str) -> Optional[Any]:
|
||||
"""The provider a skill would call (override or active), or None."""
|
||||
return resolve_skill(skill_id).provider
|
||||
|
||||
|
||||
# ── Client / backend construction ───────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SkillClient:
|
||||
"""A ready-to-call OpenAI-compatible client bound to the skill's provider."""
|
||||
|
||||
client: Any # openai.OpenAI
|
||||
model: str
|
||||
provider_id: str
|
||||
timeout: float
|
||||
|
||||
|
||||
def _default_timeout() -> float:
|
||||
try:
|
||||
return float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
|
||||
except ValueError:
|
||||
return 45.0
|
||||
|
||||
|
||||
def resolve_skill_client(skill_id: str) -> Optional[SkillClient]:
|
||||
"""OpenAI-compat client + model for a skill, or None.
|
||||
|
||||
None when the skill is disabled, no provider resolves, the provider is
|
||||
unconfigured, or the openai package is missing — callers treat None
|
||||
exactly like "no LLM configured" (their existing degradation path).
|
||||
Raises KeyError for an unknown skill (programming error, not user state).
|
||||
"""
|
||||
res = resolve_skill(skill_id)
|
||||
if not res.ready:
|
||||
return None
|
||||
try:
|
||||
from openai import OpenAI
|
||||
except ImportError:
|
||||
logger.warning("openai package not installed — LLM skill %s unavailable.",
|
||||
skill_id)
|
||||
return None
|
||||
from services import llm_providers
|
||||
|
||||
api_key = llm_providers.resolve_api_key(res.provider)
|
||||
if not api_key:
|
||||
return None
|
||||
kw: dict[str, Any] = {"api_key": api_key}
|
||||
base_url = llm_providers.resolve_base_url(res.provider)
|
||||
if base_url:
|
||||
kw["base_url"] = base_url
|
||||
# max_retries=0: a rate-limited provider returning 429 + a long Retry-After
|
||||
# would otherwise let the SDK sleep+retry inside a single call, blowing the
|
||||
# skill's wall-clock budget (the cinematic pass budget, the glossary call
|
||||
# timeout) from inside one request. Fail fast — the per-call timeout and the
|
||||
# pass-level budget are the only bounds we want. Mirrors OpenAICompatBackend.
|
||||
#
|
||||
# #959 class guard: OpenAI() eagerly builds its httpx client, which can
|
||||
# raise AT CONSTRUCTION for environment-shaped reasons — the reported one
|
||||
# is httpx's ImportError under ALL_PROXY/HTTPS_PROXY=socks5:// without
|
||||
# socksio; a malformed proxy URL or broken cert bundle fails the same way.
|
||||
# The contract here is already "None == LLM unavailable, degrade" — a bad
|
||||
# proxy env must degrade the skill, never 500 the calling feature.
|
||||
try:
|
||||
client = OpenAI(max_retries=0, **kw)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"LLM client construction failed for skill %s (provider %s): %s — "
|
||||
"treating the skill as unavailable.",
|
||||
skill_id, res.provider.id, exc,
|
||||
)
|
||||
return None
|
||||
return SkillClient(
|
||||
client=client,
|
||||
model=llm_providers.resolve_model(res.provider),
|
||||
provider_id=res.provider.id,
|
||||
timeout=_default_timeout(),
|
||||
)
|
||||
|
||||
|
||||
def skill_backend(skill_id: str, active: Optional[Callable[[], Any]] = None):
|
||||
"""LLMBackend for a skill — the drop-in for ``get_active_llm_backend()``.
|
||||
|
||||
* disabled skill → OffBackend (same object the no-LLM path returns today,
|
||||
so every caller's ``id == "off"`` / ``isinstance(…, OffBackend)`` check
|
||||
degrades identically);
|
||||
* no override → the ``active`` callable (callers pass their module-local
|
||||
``get_active_llm_backend`` so existing monkeypatch seams keep working),
|
||||
defaulting to ``llm_backend.get_active_llm_backend`` — the exact legacy
|
||||
path, env/prefs overrides included;
|
||||
* override → an OpenAICompatBackend bound to that provider, or OffBackend
|
||||
when the provider is unconfigured, openai is missing, or the global
|
||||
``OMNIVOICE_LLM_BACKEND=off`` kill switch is set.
|
||||
"""
|
||||
from services.llm_backend import OffBackend, OpenAICompatBackend
|
||||
|
||||
res = resolve_skill(skill_id)
|
||||
if not res.enabled:
|
||||
return OffBackend()
|
||||
if res.source != "override":
|
||||
if active is not None:
|
||||
return active()
|
||||
from services import llm_backend
|
||||
return llm_backend.get_active_llm_backend()
|
||||
if os.environ.get("OMNIVOICE_LLM_BACKEND") == "off":
|
||||
return OffBackend()
|
||||
if not res.ready:
|
||||
return OffBackend()
|
||||
try:
|
||||
import openai # noqa: F401
|
||||
except ImportError:
|
||||
return OffBackend()
|
||||
return OpenAICompatBackend(provider=res.provider)
|
||||
|
||||
|
||||
# ── API descriptor ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def describe(skill_id: str) -> dict:
|
||||
"""Client-safe skill descriptor for GET /api/settings/llm-skills."""
|
||||
res = resolve_skill(skill_id)
|
||||
p = res.provider
|
||||
return {
|
||||
"id": res.skill.id,
|
||||
"name_key": res.skill.name_key,
|
||||
"description_key": res.skill.description_key,
|
||||
"enabled": res.enabled,
|
||||
"provider_override": provider_override(skill_id),
|
||||
"provider": p.id if p is not None else None,
|
||||
"provider_display_name": p.display_name if p is not None else None,
|
||||
"provider_local": p.local if p is not None else None,
|
||||
"provider_source": res.source,
|
||||
"ready": res.ready,
|
||||
"reason": res.reason,
|
||||
}
|
||||
@@ -45,11 +45,33 @@ def _asr_device() -> str:
|
||||
return "cpu"
|
||||
|
||||
|
||||
def _active_tts_id() -> Optional[str]:
|
||||
"""Configured TTS engine id, or None if it can't be resolved. Attribution
|
||||
is advisory — a prefs/import hiccup must never break /model/loaded."""
|
||||
try:
|
||||
from services.tts_backend import active_backend_id
|
||||
return active_backend_id()
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _tts_attribution(engine_id: str, active: Optional[str]) -> dict:
|
||||
"""Per-entry engine attribution for TTS-family models. A model can stay
|
||||
resident in VRAM after the user switches engines (freed only by unload/
|
||||
idle-evict), so the panel needs to know which entry synthesis actually
|
||||
routes to. ``is_active_engine`` is None when the active id is unknown."""
|
||||
return {
|
||||
"engine_id": engine_id,
|
||||
"is_active_engine": (engine_id == active) if active is not None else None,
|
||||
}
|
||||
|
||||
|
||||
def list_loaded() -> dict:
|
||||
"""Enumerate every currently-loaded model. Shape: ``{"models": [...],
|
||||
"count": n}`` with per-model id/name/checkpoint/device/vram_mb/unloadable
|
||||
(+ optional ``note``)."""
|
||||
models: list[dict] = []
|
||||
active_tts = _active_tts_id()
|
||||
|
||||
# 1. In-process TTS model (OmniVoice)
|
||||
if mm.model is not None:
|
||||
@@ -60,10 +82,11 @@ def list_loaded() -> dict:
|
||||
models.append({
|
||||
"id": "tts",
|
||||
"name": "OmniVoice TTS",
|
||||
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"checkpoint": mm.resolve_omnivoice_checkpoint(), # #693: effective checkpoint, not a leaked raw value
|
||||
"device": device,
|
||||
"vram_mb": round(_tts_vram_mb(), 1),
|
||||
"unloadable": True,
|
||||
**_tts_attribution("omnivoice", active_tts),
|
||||
})
|
||||
|
||||
# 2. ASR (WhisperX) — co-loaded with and released alongside the TTS model.
|
||||
@@ -105,6 +128,7 @@ def list_loaded() -> dict:
|
||||
"device": get_best_device(),
|
||||
"vram_mb": round(float(s.get("vram_mb") or 0), 1),
|
||||
"unloadable": True,
|
||||
**_tts_attribution(s["id"], active_tts),
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -3,7 +3,7 @@ import time
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from concurrent.futures import ThreadPoolExecutor, Executor
|
||||
|
||||
# ── Lazy imports ─────────────────────────────────────────────────────
|
||||
# torch and OmniVoice are heavy (~2-3s import on Apple Silicon).
|
||||
@@ -25,7 +25,16 @@ def _lazy_torch():
|
||||
def _lazy_omnivoice():
|
||||
global _OmniVoice
|
||||
if _OmniVoice is None:
|
||||
from omnivoice.models.omnivoice import OmniVoice as _OV
|
||||
try:
|
||||
from omnivoice.models.omnivoice import OmniVoice as _OV
|
||||
except ModuleNotFoundError:
|
||||
# The venv's editable install is missing/broken (#564). main.py wires
|
||||
# the source fallback at startup, but resolve it here too so the
|
||||
# model-load path self-heals and logs the paths it searched.
|
||||
from core.omnivoice_path import ensure_omnivoice_importable
|
||||
_backend_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ensure_omnivoice_importable(_backend_dir, logger)
|
||||
from omnivoice.models.omnivoice import OmniVoice as _OV
|
||||
_OmniVoice = _OV
|
||||
return _OmniVoice
|
||||
|
||||
@@ -35,17 +44,30 @@ from core.config import IDLE_TIMEOUT_SECONDS, CPU_POOL_WORKERS
|
||||
logger = logging.getLogger("omnivoice.model")
|
||||
|
||||
# Per-TTS-job VRAM headroom estimate. OmniVoice's forward + autoregressive
|
||||
# decode peaks around 1.6 GB on a 24 kHz 8-second utterance; we budget 2.5 GB
|
||||
# to leave room for the ASR/diarization pipelines that run concurrently in
|
||||
# the same process. Tuned empirically — bumps to 3 GB if anyone reports OOM
|
||||
# at 16 GB on a multi-segment dub.
|
||||
_GPU_VRAM_PER_JOB_GB = 2.5
|
||||
# decode peaks around 1.6 GB, but the interactive clone path co-loads WhisperX
|
||||
# large-v3 ASR (~3 GB) to transcribe the reference, so a *concurrent* clone job
|
||||
# is realistically ~5 GB. The old 2.5 GB budget over-committed: an 8 GB card
|
||||
# (~7 GB free) got 2 workers, and two concurrent clone jobs blew past VRAM into
|
||||
# a sticky CUDA "illegal memory access" that aborts the whole backend process —
|
||||
# the wave of "Can't reach the local backend" crash reports on 8 GB GPUs
|
||||
# (#567/#570/#571/#580/#582/#583/#584). Budgeting 5 GB serializes to 1 worker on
|
||||
# ≤10 GB cards (no contention → no crash) while 16/24 GB cards still parallelize.
|
||||
# Power users override with OMNIVOICE_GPU_WORKERS.
|
||||
_GPU_VRAM_PER_JOB_GB = 5.0
|
||||
_GPU_WORKER_CAP = 4
|
||||
|
||||
_gpu_pool_singleton: "ThreadPoolExecutor | None" = None
|
||||
_gpu_pool_singleton: "_ResilientGpuPool | None" = None
|
||||
_cpu_pool = ThreadPoolExecutor(max_workers=CPU_POOL_WORKERS)
|
||||
|
||||
|
||||
def _workers_for_free_vram(free_gb: float) -> int:
|
||||
"""GPU worker count for a given free-VRAM figure: free // per-job budget,
|
||||
floored at 1 and capped at _GPU_WORKER_CAP. Pure so the sizing policy is
|
||||
unit-tested without a GPU (the #567 crash hinged on this returning >1 on
|
||||
8 GB cards)."""
|
||||
return max(1, min(_GPU_WORKER_CAP, int(free_gb // _GPU_VRAM_PER_JOB_GB)))
|
||||
|
||||
|
||||
def _pick_gpu_workers() -> int:
|
||||
"""Pick a sensible GPU worker count from the runtime environment.
|
||||
|
||||
@@ -68,7 +90,7 @@ def _pick_gpu_workers() -> int:
|
||||
if hasattr(torch, "cuda") and torch.cuda.is_available():
|
||||
free_bytes, _total = torch.cuda.mem_get_info()
|
||||
free_gb = free_bytes / (1024 ** 3)
|
||||
workers = max(1, min(_GPU_WORKER_CAP, int(free_gb // _GPU_VRAM_PER_JOB_GB)))
|
||||
workers = _workers_for_free_vram(free_gb)
|
||||
logger.info(
|
||||
"GPU pool sized to %d worker(s) — %.1f GB free / %.1f GB per job (cap %d)",
|
||||
workers, free_gb, _GPU_VRAM_PER_JOB_GB, _GPU_WORKER_CAP,
|
||||
@@ -87,14 +109,82 @@ def _build_gpu_pool() -> ThreadPoolExecutor:
|
||||
return ThreadPoolExecutor(max_workers=workers, thread_name_prefix="gpu-pool")
|
||||
|
||||
|
||||
def _get_gpu_pool() -> ThreadPoolExecutor:
|
||||
"""Internal accessor. Same singleton as the module-level `_gpu_pool`
|
||||
attribute, but resolvable from inside this module (Python's module
|
||||
`__getattr__` only fires for unresolved lookups from *outside*).
|
||||
class _ResilientGpuPool(Executor):
|
||||
"""A stable, self-healing wrapper around the GPU `ThreadPoolExecutor`.
|
||||
|
||||
The crash this fixes (#589 #599): `_reset_gpu_pool()` shuts the pool down on
|
||||
a model-load timeout, but consumers that captured the executor *object* at
|
||||
import time (`from services.model_manager import _gpu_pool` at module level —
|
||||
generation, dub_generate, dub_core, dub_translate, openai_compat) kept
|
||||
submitting to the dead pool and got `RuntimeError: cannot schedule new
|
||||
futures after shutdown` on the next generate/dub/translate.
|
||||
|
||||
Making `_gpu_pool` a single long-lived wrapper whose *inner* pool is swapped
|
||||
means those references never go stale: every `submit()` resolves the live
|
||||
pool, and a submit that races a shutdown rebuilds once and retries. Building
|
||||
the inner pool stays lazy so we still size workers after torch's device
|
||||
probe (the reason for the original `__getattr__` indirection).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._pool: "ThreadPoolExecutor | None" = None
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _live_pool(self) -> ThreadPoolExecutor:
|
||||
pool = self._pool
|
||||
if pool is None:
|
||||
with self._lock:
|
||||
if self._pool is None:
|
||||
self._pool = _build_gpu_pool()
|
||||
pool = self._pool
|
||||
return pool
|
||||
|
||||
def submit(self, fn, /, *args, **kwargs):
|
||||
try:
|
||||
return self._live_pool().submit(fn, *args, **kwargs)
|
||||
except RuntimeError as e:
|
||||
# "cannot schedule new futures after shutdown": the inner pool was
|
||||
# reset (or torn down) under us. Rebuild once and retry so a stale
|
||||
# caller self-heals instead of 500-ing. (Interpreter-shutdown races
|
||||
# re-raise on the retry — we don't loop.)
|
||||
if "shutdown" not in str(e).lower():
|
||||
raise
|
||||
with self._lock:
|
||||
self._pool = _build_gpu_pool()
|
||||
pool = self._pool
|
||||
return pool.submit(fn, *args, **kwargs)
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Abandon the current worker pool; the next submit builds a fresh one.
|
||||
|
||||
Python can't kill a thread wedged in a timed-out load, but dropping the
|
||||
poisoned pool means a retry gets a clean worker instead of queueing
|
||||
behind the wedged one. The wrapper identity is preserved, so references
|
||||
held by importers stay valid.
|
||||
"""
|
||||
with self._lock:
|
||||
pool, self._pool = self._pool, None
|
||||
if pool is not None:
|
||||
try:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def shutdown(self, wait=True, *, cancel_futures=False):
|
||||
with self._lock:
|
||||
pool, self._pool = self._pool, None
|
||||
if pool is not None:
|
||||
pool.shutdown(wait=wait, cancel_futures=cancel_futures)
|
||||
|
||||
|
||||
def _get_gpu_pool() -> "_ResilientGpuPool":
|
||||
"""Internal accessor for the GPU pool singleton. Same object as the
|
||||
module-level `_gpu_pool` attribute, but resolvable from inside this module
|
||||
(Python's module `__getattr__` only fires for lookups from *outside*).
|
||||
"""
|
||||
global _gpu_pool_singleton
|
||||
if _gpu_pool_singleton is None:
|
||||
_gpu_pool_singleton = _build_gpu_pool()
|
||||
_gpu_pool_singleton = _ResilientGpuPool()
|
||||
return _gpu_pool_singleton
|
||||
|
||||
|
||||
@@ -108,6 +198,93 @@ def __getattr__(name: str):
|
||||
return _get_gpu_pool()
|
||||
raise AttributeError(f"module 'services.model_manager' has no attribute {name!r}")
|
||||
|
||||
|
||||
# ── GPU-job timeout guard (#730 class; residual #850/#802/#755 …) ─────
|
||||
# A blocking GPU job that wedges on a Windows+CUDA hang keeps occupying its
|
||||
# worker forever — run_in_executor can't cancel the thread. With a 1–2 worker
|
||||
# pool that starves *every* other request, so the next user action surfaces as
|
||||
# the misleading "Can't reach the local backend" even though the process is
|
||||
# alive. ASR/dub/model-load already bound+reset on hang (run_transcribe_guarded,
|
||||
# _reset_pool_on_wedge, _load_model_with_timeout); the TTS **generate** paths
|
||||
# (generation.py, tts_stream.py) were the last unguarded dispatch — and the
|
||||
# residual on-main reports all fail on generate:start (audio). This is the same
|
||||
# guard generalised so every GPU dispatch shares one recovery path.
|
||||
GPU_JOB_TIMEOUT_S = float(os.environ.get("OMNIVOICE_GENERATE_TIMEOUT_S", "300.0"))
|
||||
|
||||
|
||||
class GpuJobTimeoutError(TimeoutError):
|
||||
"""A GPU-pool job exceeded its wall-clock bound and was abandoned.
|
||||
|
||||
The backend is alive — the job was too heavy for the available compute
|
||||
(most often a VRAM-starved GPU). Pool capacity is restored automatically by
|
||||
resetting the pool; the message carries the durable fix.
|
||||
"""
|
||||
|
||||
|
||||
async def run_on_gpu_pool_guarded(fn, *, what: str = "GPU job",
|
||||
timeout: float = GPU_JOB_TIMEOUT_S,
|
||||
executor=None):
|
||||
"""Run blocking ``fn`` on the GPU pool with a hard wall-clock bound.
|
||||
|
||||
On timeout, ``reset()`` the pool (abandon the wedged worker so the next
|
||||
submit gets a fresh one) and raise :class:`GpuJobTimeoutError`. ``fn`` must
|
||||
be a zero-arg callable — wrap args with ``functools.partial`` at the call
|
||||
site. Deliberately mirrors ``asr_backend.run_transcribe_guarded`` so every
|
||||
GPU dispatch shares one bound+recover path (#730 class). Executors without
|
||||
``reset`` (a plain ThreadPoolExecutor in tests) still get the bound + error.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
ex = executor if executor is not None else _get_gpu_pool()
|
||||
fut = loop.run_in_executor(ex, fn)
|
||||
try:
|
||||
return await asyncio.wait_for(fut, timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
_reset = getattr(ex, "reset", None)
|
||||
if callable(_reset):
|
||||
try:
|
||||
_reset()
|
||||
logger.warning(
|
||||
"%s exceeded %.0fs — abandoned the GPU-pool worker to "
|
||||
"restore capacity (#730).", what, timeout,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("GPU pool reset after %s timeout failed", what)
|
||||
raise GpuJobTimeoutError(_timeout_guidance(what, timeout))
|
||||
|
||||
|
||||
def _timeout_guidance(what: str, timeout: float) -> str:
|
||||
"""Device-aware timeout message (#896): a CPU-only host must never be told
|
||||
to "set the engine to CPU" or blamed on VRAM — on CPU the job is simply
|
||||
compute-bound. GPU hosts keep the VRAM-contention guidance."""
|
||||
family = "cuda" # conservative default: GPU wording if the probe fails
|
||||
try:
|
||||
from core.device_caps import detect_host_caps
|
||||
family = detect_host_caps().family
|
||||
except Exception: # noqa: BLE001 — guidance must never mask the timeout
|
||||
pass
|
||||
common = (
|
||||
f"{what} exceeded {timeout:.0f}s and was abandoned — the backend is "
|
||||
"running, but the job was too heavy for the available compute. "
|
||||
"Capacity was restored automatically; "
|
||||
)
|
||||
if family == "cpu":
|
||||
return common + (
|
||||
"this machine renders on CPU, where long generations are "
|
||||
"compute-bound. For a durable fix try shorter text or a lighter "
|
||||
"engine (OmniVoice GGUF and Supertonic-3 are CPU-tuned). If you "
|
||||
"expect very long single generations, raise "
|
||||
"OMNIVOICE_GENERATE_TIMEOUT_S."
|
||||
)
|
||||
return common + (
|
||||
"most often the GPU is VRAM-starved (a resident model and this job "
|
||||
"contend for memory). For a durable fix, Flush caches / Unload the "
|
||||
"resident model (top toolbar or Settings → Models) before retrying, "
|
||||
"try shorter text, a lighter engine, or set the engine to CPU in "
|
||||
"Settings → Models. (Raise OMNIVOICE_GENERATE_TIMEOUT_S for very "
|
||||
"long single generations.)"
|
||||
)
|
||||
|
||||
|
||||
model = None # type: ignore
|
||||
_model_lock = asyncio.Lock()
|
||||
_last_used = time.time()
|
||||
@@ -218,6 +395,19 @@ def get_best_device():
|
||||
compatible, warning = check_device_compatibility()
|
||||
if not compatible:
|
||||
logger.warning(warning)
|
||||
# #756: the GPU's compute capability isn't in this torch build's arch
|
||||
# list, so CUDA kernels can't launch ("no kernel image is available
|
||||
# for execution") — every generate would 500. Too-old (Pascal sm_61)
|
||||
# and too-new (Blackwell sm_120 on pre-cu128 wheels) both land here.
|
||||
# Fall back to CPU so the app WORKS (slowly) instead of dead-ending;
|
||||
# OMNIVOICE_FORCE_CUDA=1 overrides for users who installed a matching
|
||||
# torch and know the arch_list probe is wrong for their setup.
|
||||
if not _env_flag("OMNIVOICE_FORCE_CUDA"):
|
||||
logger.warning(
|
||||
"Falling back to CPU: this GPU is unsupported by the installed "
|
||||
"PyTorch build (set OMNIVOICE_FORCE_CUDA=1 to force CUDA anyway)."
|
||||
)
|
||||
return "cpu"
|
||||
return "cuda"
|
||||
|
||||
# ── Intel Arc / discrete GPU via IPEX ────────────────────────────
|
||||
@@ -449,6 +639,168 @@ def should_preload_tts_asr() -> bool:
|
||||
return _env_flag("OMNIVOICE_PRELOAD_TTS_ASR")
|
||||
|
||||
|
||||
def _is_incomplete_cache_error(exc: BaseException) -> bool:
|
||||
"""True when `exc` is the truncated-HF-cache class (#352 / #581).
|
||||
|
||||
transformers raises an OSError whose message contains "does not appear to
|
||||
have a file named …" when the on-disk snapshot has config/tokenizer files
|
||||
but no weight shard — the signature of an interrupted download. We match on
|
||||
that phrase (stable across transformers 4.x/5.x) rather than the error type,
|
||||
since the same OSError type covers unrelated I/O failures."""
|
||||
return "does not appear to have a file named" in str(exc)
|
||||
|
||||
|
||||
def _hf_offline() -> bool:
|
||||
"""Respect HF's offline switches so repair never makes a network call the
|
||||
user opted out of. `snapshot_download` would itself raise offline, but
|
||||
checking up front lets us skip straight to the actionable message."""
|
||||
return _env_flag("HF_HUB_OFFLINE") or _env_flag("TRANSFORMERS_OFFLINE")
|
||||
|
||||
|
||||
# Why the LAST _repair_model_cache run failed ("" when it succeeded / hasn't
|
||||
# run). #886: the "could not be auto-repaired" message used to drop the cause
|
||||
# entirely, so a mirror outage, offline mode, or a full disk all read the same.
|
||||
_last_repair_error: str = ""
|
||||
|
||||
|
||||
def _repair_failure_detail() -> str:
|
||||
"""One sanitized clause naming why auto-repair failed, or "" (#886).
|
||||
|
||||
Feeds user-facing messages (the generate 500 detail / model status), so it
|
||||
goes through core.failure.sanitize — and because the cause text is now part
|
||||
of the surfaced error, the shared HF-mirror hint (#874) fires on it when
|
||||
the repair failed against an unreachable configured mirror."""
|
||||
if not _last_repair_error:
|
||||
return ""
|
||||
try:
|
||||
from core.failure import sanitize
|
||||
cause = sanitize(_last_repair_error)
|
||||
except Exception:
|
||||
cause = _last_repair_error
|
||||
return f" Auto-repair failed with: {cause}."
|
||||
|
||||
|
||||
def _repair_model_cache(checkpoint: str, *, force: bool = False) -> bool:
|
||||
"""Re-fetch a checkpoint's missing files in place and report success.
|
||||
|
||||
An interrupted download leaves the cache missing only some files;
|
||||
`snapshot_download` resumes/fills exactly those (already-present, correctly
|
||||
sized blobs are skipped by hash, so a near-complete cache repairs in
|
||||
seconds and a complete one would no-op). Returns False — leaving the caller
|
||||
to surface the actionable delete-and-reinstall message — when repair is
|
||||
impossible (offline) or the re-fetch itself fails (no network, gated repo,
|
||||
full disk). Never raises; repair is best-effort.
|
||||
|
||||
``force=True`` passes ``force_download`` so the re-fetch replaces files that
|
||||
are *present but corrupt* — a truncated/garbled blob that still has the right
|
||||
size won't be re-fetched by the default resume (#739). It re-downloads the
|
||||
whole snapshot, so it's the last resort the load path only reaches after a
|
||||
plain resume-repair didn't fix the cache."""
|
||||
global _last_repair_error
|
||||
_last_repair_error = ""
|
||||
if _hf_offline():
|
||||
logger.warning(
|
||||
"Model cache for %s is incomplete but HF offline mode is set — "
|
||||
"cannot auto-repair.", checkpoint,
|
||||
)
|
||||
_last_repair_error = (
|
||||
"Hugging Face offline mode is enabled (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE)"
|
||||
)
|
||||
return False
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
except Exception as imp_err: # pragma: no cover - huggingface_hub is a hard dep
|
||||
logger.warning("Cannot import snapshot_download to repair cache: %s", imp_err)
|
||||
_last_repair_error = f"{type(imp_err).__name__}: {imp_err}"
|
||||
return False
|
||||
dl_kwargs: dict = {"repo_id": checkpoint}
|
||||
endpoint = os.environ.get("HF_ENDPOINT")
|
||||
if endpoint:
|
||||
dl_kwargs["endpoint"] = endpoint
|
||||
if force:
|
||||
# Replace present-but-corrupt blobs that resume would trust by size.
|
||||
dl_kwargs["force_download"] = True
|
||||
if os.name == "nt":
|
||||
# Match the install path (download.py): avoid symlinks on Windows.
|
||||
dl_kwargs["local_dir_use_symlinks"] = False
|
||||
|
||||
def _attempt() -> None:
|
||||
"""One snapshot_download, tolerating an hf_hub that rejects the optional
|
||||
symlink knob. Lets real failures (network, gated repo, disk) propagate."""
|
||||
try:
|
||||
snapshot_download(**dl_kwargs)
|
||||
except TypeError:
|
||||
# Older/newer huggingface_hub may not accept local_dir_use_symlinks
|
||||
# on a cache-only call — retry without the optional knob.
|
||||
dl_kwargs.pop("local_dir_use_symlinks", None)
|
||||
snapshot_download(**dl_kwargs)
|
||||
|
||||
# Bounded retries (#739): an incomplete cache *is* an interrupted download, so
|
||||
# a single transient blip mid-repair shouldn't drop the user back to a manual
|
||||
# delete-and-reinstall. snapshot_download resumes between attempts (present,
|
||||
# correctly-sized blobs are skipped by hash), so each retry continues where
|
||||
# the last left off — cheap and idempotent. Counts/backoff are env-tunable
|
||||
# for restricted networks and kept fast (backoff=0) in tests.
|
||||
try:
|
||||
retries = max(1, int(os.environ.get("OMNIVOICE_MODEL_REPAIR_RETRIES", "3")))
|
||||
except ValueError:
|
||||
retries = 3
|
||||
try:
|
||||
backoff = max(0.0, float(os.environ.get("OMNIVOICE_MODEL_REPAIR_BACKOFF_S", "2")))
|
||||
except ValueError:
|
||||
backoff = 2.0
|
||||
|
||||
logger.info(
|
||||
"Auto-repairing incomplete model cache for %s (up to %d attempt(s)) …",
|
||||
checkpoint, retries,
|
||||
)
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
_attempt()
|
||||
logger.info("Auto-repair of %s completed; retrying model load.", checkpoint)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Auto-repair of %s attempt %d/%d failed: %s",
|
||||
checkpoint, attempt, retries, e,
|
||||
)
|
||||
_last_repair_error = f"{type(e).__name__}: {e}"
|
||||
if attempt < retries and backoff:
|
||||
time.sleep(backoff * attempt)
|
||||
return False
|
||||
|
||||
|
||||
_DEFAULT_OMNIVOICE_CHECKPOINT = "k2-fsa/OmniVoice"
|
||||
|
||||
|
||||
def resolve_omnivoice_checkpoint() -> str:
|
||||
"""Resolve the OmniVoice TTS checkpoint from ``OMNIVOICE_MODEL``, self-healing
|
||||
a misconfigured value.
|
||||
|
||||
A valid checkpoint is either a HuggingFace repo id (``org/repo`` — contains a
|
||||
``/``) or an existing local directory. A bare token like ``"omnivoice"`` — a
|
||||
TTS *engine id* that leaked into ``OMNIVOICE_MODEL`` (e.g. a stale pref/env) —
|
||||
is neither, and would crash model load with *"omnivoice is not a local folder
|
||||
and is not a valid model identifier listed on huggingface.co/models"* (#693).
|
||||
Fall back to the default rather than 500 on every launch.
|
||||
"""
|
||||
checkpoint = os.environ.get("OMNIVOICE_MODEL", _DEFAULT_OMNIVOICE_CHECKPOINT).strip()
|
||||
if not checkpoint:
|
||||
return _DEFAULT_OMNIVOICE_CHECKPOINT
|
||||
# Honor a HF repo id (org/repo) or an EXPLICIT local path (absolute, or with
|
||||
# a path separator). A bare token like "omnivoice" must NOT be treated as a
|
||||
# local dir even if a cwd-relative folder happens to share its name — that
|
||||
# is exactly the engine-id leak (#693), so self-heal to the default.
|
||||
if "/" in checkpoint or "\\" in checkpoint or os.path.isabs(checkpoint):
|
||||
return checkpoint
|
||||
logger.warning(
|
||||
"OMNIVOICE_MODEL=%r is not a HuggingFace repo id (org/repo) or a local "
|
||||
"path — falling back to %s (#693).",
|
||||
checkpoint, _DEFAULT_OMNIVOICE_CHECKPOINT,
|
||||
)
|
||||
return _DEFAULT_OMNIVOICE_CHECKPOINT
|
||||
|
||||
|
||||
def _load_model_sync():
|
||||
global model
|
||||
from utils.hf_progress import register_listener, unregister_listener
|
||||
@@ -475,7 +827,7 @@ def _load_model_sync():
|
||||
OmniVoice = _lazy_omnivoice()
|
||||
device = get_best_device()
|
||||
|
||||
checkpoint = os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice")
|
||||
checkpoint = resolve_omnivoice_checkpoint()
|
||||
_set_loading("loading_weights", f"Loading TTS weights on {device}…")
|
||||
logger.info("Loading OmniVoice model on device: %s", device)
|
||||
preload_asr = should_preload_tts_asr()
|
||||
@@ -483,23 +835,67 @@ def _load_model_sync():
|
||||
logger.info("Preloading PyTorch Whisper with TTS model.")
|
||||
else:
|
||||
logger.info("Skipping PyTorch Whisper preload; ASR will load on demand.")
|
||||
try:
|
||||
_model = OmniVoice.from_pretrained(
|
||||
def _load():
|
||||
return OmniVoice.from_pretrained(
|
||||
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
|
||||
)
|
||||
|
||||
try:
|
||||
_model = _load()
|
||||
except OSError as e:
|
||||
# #352: a truncated HF cache surfaces here as "does not appear to
|
||||
# have a file named pytorch_model.bin or model.safetensors".
|
||||
# Translate to an actionable message instead of the raw
|
||||
# transformers error.
|
||||
if "does not appear to have a file named" in str(e):
|
||||
# #352 / #581: a truncated HF cache surfaces here as "does not
|
||||
# appear to have a file named pytorch_model.bin or
|
||||
# model.safetensors". Instead of dead-ending the user with a
|
||||
# manual delete-and-reinstall instruction, try to self-repair: an
|
||||
# interrupted download leaves the cache missing only some files,
|
||||
# and snapshot_download() resumes/fills exactly those (a complete
|
||||
# cache never reaches this branch, so the fast path is untouched).
|
||||
if not _is_incomplete_cache_error(e):
|
||||
raise
|
||||
_set_loading("loading_weights", "Repairing incomplete model cache…")
|
||||
if not _repair_model_cache(checkpoint):
|
||||
raise RuntimeError(
|
||||
f"The TTS model cache for {checkpoint} is incomplete "
|
||||
"(weights missing — usually an interrupted download). "
|
||||
"(weights missing — usually an interrupted download)."
|
||||
f"{_repair_failure_detail()} "
|
||||
"Open Settings → Models, delete the OmniVoice TTS model, "
|
||||
"and install it again."
|
||||
) from e
|
||||
raise
|
||||
_set_loading("loading_weights", f"Loading TTS weights on {device}…")
|
||||
try:
|
||||
_model = _load()
|
||||
except OSError as e2:
|
||||
# Resume-repair ran but the cache is still unusable. The usual
|
||||
# cause beyond "repo genuinely lacks weights" is a blob that's
|
||||
# present with the right size but corrupt — snapshot_download's
|
||||
# resume trusts it and never re-fetches it (#739). Force a full
|
||||
# re-download (replaces corrupt blobs) and retry once more before
|
||||
# falling back to the manual delete-and-reinstall message.
|
||||
if _is_incomplete_cache_error(e2):
|
||||
_set_loading("loading_weights", "Re-downloading model files…")
|
||||
if _repair_model_cache(checkpoint, force=True):
|
||||
try:
|
||||
_model = _load()
|
||||
except OSError as e3:
|
||||
raise RuntimeError(
|
||||
f"The TTS model cache for {checkpoint} is incomplete "
|
||||
"and could not be auto-repaired. Open Settings → "
|
||||
"Models, delete the OmniVoice TTS model, and install "
|
||||
"it again."
|
||||
) from e3
|
||||
else:
|
||||
raise RuntimeError(
|
||||
f"The TTS model cache for {checkpoint} is incomplete and "
|
||||
f"could not be auto-repaired.{_repair_failure_detail()} "
|
||||
"Open Settings → Models, delete the OmniVoice TTS model, "
|
||||
"and install it again."
|
||||
) from e2
|
||||
else:
|
||||
raise RuntimeError(
|
||||
f"The TTS model cache for {checkpoint} is incomplete and "
|
||||
"could not be auto-repaired. Open Settings → Models, delete "
|
||||
"the OmniVoice TTS model, and install it again."
|
||||
) from e2
|
||||
|
||||
try:
|
||||
# plan-02 (#65): gate on Triton availability (+ user setting), not
|
||||
@@ -549,9 +945,25 @@ def _load_model_sync():
|
||||
logger.info("OmniVoice model loaded successfully.")
|
||||
return _model
|
||||
except Exception as exc:
|
||||
err_msg = str(exc)
|
||||
# Surface an ACTIONABLE, sanitized error in /model/status (it's shown in
|
||||
# the first-run System Check). build_failure classifies the cause and
|
||||
# attaches a fix hint — e.g. a corrupted transformers install
|
||||
# ([Errno 2] … modeling_*.py) now says "reinstall transformers" instead
|
||||
# of an unhelpful raw path + "try restarting" — and strips the home dir.
|
||||
try:
|
||||
from core.failure import build_failure
|
||||
_f = build_failure(exc, stage="model-load", include_diagnostic=False)
|
||||
err_msg = _f["reason"] + (f" — {_f['hint']}" if _f.get("hint") else "")
|
||||
except Exception: # never let failure-formatting mask the real error
|
||||
err_msg = str(exc)
|
||||
_set_loading("error", "Model loading failed", error=err_msg)
|
||||
logger.error("Model loading failed: %s", err_msg)
|
||||
# #1000 class: transformers' lazy-import machinery wraps ANY disruption
|
||||
# to an inner import (including one interrupted by process teardown)
|
||||
# in a generic "Could not import module X. Are this object's
|
||||
# requirements defined correctly?" — logging only str(exc) discarded
|
||||
# the real cause in __cause__/__context__ and made a shutdown race
|
||||
# look like a broken install. exc_info surfaces the full chain.
|
||||
logger.error("Model loading failed: %s", str(exc), exc_info=exc)
|
||||
raise
|
||||
finally:
|
||||
unregister_listener(lid)
|
||||
@@ -571,19 +983,15 @@ def _model_load_timeout() -> float:
|
||||
|
||||
|
||||
def _reset_gpu_pool() -> None:
|
||||
"""Drop the GPU pool singleton so the next access builds a fresh one.
|
||||
"""Recover from a wedged/timed-out load by abandoning the GPU worker pool.
|
||||
|
||||
Python can't kill the thread stuck in a timed-out load, but abandoning the
|
||||
poisoned single-worker pool means a *retry* gets a clean worker instead of
|
||||
queueing forever behind the wedged one.
|
||||
The resilient wrapper is kept (its identity is shared by every importer);
|
||||
only its inner `ThreadPoolExecutor` is dropped, so the next submit builds a
|
||||
fresh worker. This is what stops stale references from raising "cannot
|
||||
schedule new futures after shutdown" after a reset (#589 #599).
|
||||
"""
|
||||
global _gpu_pool_singleton
|
||||
pool, _gpu_pool_singleton = _gpu_pool_singleton, None
|
||||
if pool is not None:
|
||||
try:
|
||||
pool.shutdown(wait=False, cancel_futures=True)
|
||||
except Exception:
|
||||
pass
|
||||
if _gpu_pool_singleton is not None:
|
||||
_gpu_pool_singleton.reset()
|
||||
|
||||
|
||||
async def _load_model_with_timeout():
|
||||
@@ -622,6 +1030,22 @@ async def get_model():
|
||||
return model
|
||||
|
||||
|
||||
def _checkpoint_in_local_cache(checkpoint: str) -> bool:
|
||||
"""True when ``checkpoint`` is loadable with NO network: an existing local
|
||||
directory, or a COMPLETE HF cache snapshot. ``snapshot_download(...,
|
||||
local_files_only=True)`` never constructs an HTTP session, so a broken
|
||||
proxy env (#959: ``ALL_PROXY``/``HTTPS_PROXY=socks5://`` without socksio)
|
||||
can't false-negative this probe. Never raises."""
|
||||
if os.path.isdir(checkpoint):
|
||||
return True
|
||||
try:
|
||||
from huggingface_hub import snapshot_download
|
||||
snapshot_download(checkpoint, local_files_only=True)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
async def preload_model():
|
||||
"""Background model warm-up — call from lifespan startup.
|
||||
|
||||
@@ -634,15 +1058,35 @@ async def preload_model():
|
||||
return # already loaded
|
||||
try:
|
||||
# Check if the required model checkpoint exists before attempting
|
||||
# a heavy load that would fail and pollute startup logs.
|
||||
checkpoint = os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice")
|
||||
# a heavy load that would fail and pollute startup logs. Use the same
|
||||
# resolver as the load path (#693) so a leaked engine id in
|
||||
# OMNIVOICE_MODEL can't make this model_info() probe fail and silently
|
||||
# disable warm-up (then the first /generate eats the full load).
|
||||
checkpoint = resolve_omnivoice_checkpoint()
|
||||
try:
|
||||
from huggingface_hub import model_info
|
||||
model_info(checkpoint, timeout=5)
|
||||
except Exception:
|
||||
# Model not downloaded yet — skip preload
|
||||
logger.info("Preload skipped: %s not available locally.", checkpoint)
|
||||
return
|
||||
except Exception as probe_err:
|
||||
# The probe failing does NOT mean the model isn't installed — it
|
||||
# means the Hub API wasn't reachable from this process. The #959
|
||||
# class: under ALL_PROXY/HTTPS_PROXY=socks5:// without socksio,
|
||||
# hf_hub's get_session() raises ImportError AT CLIENT CONSTRUCTION;
|
||||
# same story for offline mode, DNS, or firewall failures. Fall back
|
||||
# to a cache-only probe (no HTTP session involved) and warm up
|
||||
# anyway when the model is locally present, instead of silently
|
||||
# skipping and letting the first /generate eat the full load.
|
||||
if not _checkpoint_in_local_cache(checkpoint):
|
||||
logger.info(
|
||||
"Preload skipped: %s not available locally (network probe "
|
||||
"failed: %s: %s).",
|
||||
checkpoint, type(probe_err).__name__, probe_err,
|
||||
)
|
||||
return
|
||||
logger.warning(
|
||||
"Network probe for %s failed (%s: %s) — model found in the "
|
||||
"local cache; warming up from cache.",
|
||||
checkpoint, type(probe_err).__name__, probe_err,
|
||||
)
|
||||
|
||||
logger.info("Preloading TTS model in background…")
|
||||
_last_used = time.time()
|
||||
@@ -651,7 +1095,11 @@ async def preload_model():
|
||||
model = await _load_model_with_timeout()
|
||||
logger.info("Preload complete — model ready.")
|
||||
except Exception as e:
|
||||
logger.warning("Model preload failed (non-fatal): %s", e)
|
||||
# See the matching exc_info note on the _load_model_sync handler above
|
||||
# (#1000 class) — the full chain, not just str(e), is what actually
|
||||
# distinguishes a real dependency problem from a shutdown-interrupted
|
||||
# import.
|
||||
logger.warning("Model preload failed (non-fatal): %s", e, exc_info=e)
|
||||
|
||||
def get_model_status():
|
||||
is_loaded = model is not None
|
||||
|
||||
@@ -10,8 +10,8 @@ plays the moment the video begins and everything feels desynchronised.
|
||||
``snap_segment_starts`` post-processes segments against the actual audio
|
||||
(ideally the Demucs-isolated vocals track, which the dub pipeline already
|
||||
produces): for each segment it scans the waveform inside ``[start, end]``
|
||||
for the first frame whose RMS rises above an adaptive threshold and moves
|
||||
``start`` forward to just before that onset.
|
||||
for the first *sustained* rise of frame RMS above an adaptive threshold
|
||||
and moves ``start`` forward to just before that onset.
|
||||
|
||||
Design constraints:
|
||||
|
||||
@@ -23,6 +23,27 @@ Design constraints:
|
||||
(no frame above the absolute floor) are left untouched.
|
||||
* **Pure NumPy.** No model, no platform-specific code — identical
|
||||
behaviour on macOS / Windows / Linux, trivially unit-testable.
|
||||
|
||||
Robustness against non-speech onsets (#963): a field report showed dubbed
|
||||
lines starting seconds off because "when a noise is heard (a sigh or
|
||||
footsteps), it's interpreted as the start of the conversation". Three
|
||||
guards address that class of failure:
|
||||
|
||||
* **Sustained energy.** A frame only counts as an onset when the energy
|
||||
stays up for a speech-like duration (``SUSTAIN_MIN_S`` within the
|
||||
following ``SUSTAIN_WINDOW_S``). Footsteps/door thuds/clicks light up
|
||||
one or two 20 ms frames and die; syllables keep the energy up.
|
||||
* **Bounded snap distance.** Shifts beyond ``MAX_SNAP_S`` are only
|
||||
trusted when everything being skipped is (near-)silence — the genuine
|
||||
#280 whisper start-stretch, where Demucs removed the leading music and
|
||||
left real silence on the vocals track. Jumping far over *audible*
|
||||
content (e.g. quiet speech sitting under the relative threshold) would
|
||||
play the dub seconds late, so it is refused.
|
||||
* **Source-aware.** Snapping only runs on a separated vocals track
|
||||
(``separated_vocals=True``). On mixed/original audio — Demucs skipped
|
||||
or failed — music, ambience and room tone are all legitimate sustained
|
||||
energy, so any detected "onset" is as likely the score as the speaker;
|
||||
whisper's own timestamps beat a confidently wrong snap.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -51,6 +72,23 @@ RELATIVE_THRESHOLD = 0.10
|
||||
# the whole window is treated as silence and left alone (we'd only be
|
||||
# snapping to noise).
|
||||
ABS_RMS_FLOOR = 1e-3
|
||||
# An onset must be *sustained* to count as speech (#963): within the
|
||||
# SUSTAIN_WINDOW_S that follows a candidate frame, at least SUSTAIN_MIN_S
|
||||
# worth of frames must also sit above the threshold. A ~100 ms footstep
|
||||
# burst fails this; real speech (syllables every few hundred ms) passes.
|
||||
SUSTAIN_WINDOW_S = 0.30
|
||||
SUSTAIN_MIN_S = 0.16
|
||||
# Snaps larger than this are only trusted when the skipped span is
|
||||
# (near-)silence — see _region_mostly_silent (#963).
|
||||
MAX_SNAP_S = 1.5
|
||||
# The skipped span counts as "mostly silent" when at most this fraction of
|
||||
# its frames is audible. Non-zero so an isolated transient bleeding through
|
||||
# separation (a footstep) doesn't block a genuine long silence-trim…
|
||||
SKIPPED_AUDIBLE_FRAC = 0.10
|
||||
# …where "audible" = above max(ABS_RMS_FLOOR, this fraction of the span's
|
||||
# own peak); the relative term keeps a slightly raised residual noise floor
|
||||
# from reading as content.
|
||||
SKIPPED_FLOOR_PEAK_FRAC = 0.02
|
||||
|
||||
|
||||
def _frame_rms(x: np.ndarray, frame_len: int) -> np.ndarray:
|
||||
@@ -70,6 +108,15 @@ def detect_speech_onset(
|
||||
) -> float | None:
|
||||
"""Return the absolute time (s) of the first speech-like frame inside
|
||||
``[start_s, end_s]``, or ``None`` when the window is empty / silent.
|
||||
|
||||
"Speech-like" requires *sustained* energy (#963): within the
|
||||
``SUSTAIN_WINDOW_S`` look-ahead after a candidate frame, at least
|
||||
``SUSTAIN_MIN_S`` worth of frames must also exceed the threshold.
|
||||
Short broadband transients — footsteps, door thuds, mouse clicks —
|
||||
light up one or two 20 ms frames and then die, so they no longer read
|
||||
as "the conversation started here"; real speech keeps the energy up
|
||||
across syllables. A candidate too close to the window's end to prove
|
||||
sustain is rejected (conservative: the ASR timestamp stands).
|
||||
"""
|
||||
if sr <= 0 or end_s <= start_s:
|
||||
return None
|
||||
@@ -86,10 +133,21 @@ def detect_speech_onset(
|
||||
if peak < ABS_RMS_FLOOR:
|
||||
return None # whole window is effectively silent
|
||||
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
|
||||
above = np.nonzero(rms >= threshold)[0]
|
||||
if above.size == 0:
|
||||
above = rms >= threshold
|
||||
candidates = np.nonzero(above)[0]
|
||||
if candidates.size == 0:
|
||||
return None
|
||||
return start_s + float(above[0]) * (frame_len / sr)
|
||||
frame_s = frame_len / sr
|
||||
win_frames = max(1, int(round(SUSTAIN_WINDOW_S / frame_s)))
|
||||
need_frames = max(1, int(round(SUSTAIN_MIN_S / frame_s)))
|
||||
# counts[k] = above-threshold frames within rms[c : c + win_frames]
|
||||
# for candidate c — O(n) via a cumulative sum, no per-candidate scan.
|
||||
cum = np.concatenate(([0], np.cumsum(above)))
|
||||
counts = cum[np.minimum(candidates + win_frames, above.size)] - cum[candidates]
|
||||
sustained = candidates[counts >= need_frames]
|
||||
if sustained.size == 0:
|
||||
return None # only transient bursts in this window
|
||||
return start_s + float(sustained[0]) * frame_s
|
||||
|
||||
|
||||
# Hysteresis for full-track onset listing: after a frame crosses the
|
||||
@@ -138,21 +196,60 @@ def detect_speech_onsets(audio: np.ndarray, sr: int) -> list[float]:
|
||||
return onsets
|
||||
|
||||
|
||||
def _region_mostly_silent(
|
||||
audio: np.ndarray,
|
||||
sr: int,
|
||||
start_s: float,
|
||||
end_s: float,
|
||||
) -> bool:
|
||||
"""True when ``[start_s, end_s]`` contains (almost) no audible content.
|
||||
|
||||
Gates long snaps (> ``MAX_SNAP_S``, #963): jumping far forward is only
|
||||
trustworthy when everything being skipped is silence — the genuine
|
||||
whisper start-stretch of #280, where Demucs stripped the leading music
|
||||
and left real silence on the vocals track. A small fraction of audible
|
||||
frames is tolerated so an isolated transient bleeding through
|
||||
separation (a footstep) doesn't block the trim; *sustained* audible
|
||||
content — e.g. quiet speech sitting below the relative onset
|
||||
threshold — does block it, because skipping past it would desync the
|
||||
dub by the full jump.
|
||||
"""
|
||||
i0 = max(0, int(start_s * sr))
|
||||
i1 = min(len(audio), int(end_s * sr))
|
||||
if i1 <= i0:
|
||||
return True
|
||||
rms = _frame_rms(audio[i0:i1], max(1, int(FRAME_S * sr)))
|
||||
if rms.size == 0:
|
||||
return True
|
||||
floor = max(ABS_RMS_FLOOR, SKIPPED_FLOOR_PEAK_FRAC * float(rms.max()))
|
||||
return float((rms >= floor).mean()) <= SKIPPED_AUDIBLE_FRAC
|
||||
|
||||
|
||||
def snap_segment_starts(
|
||||
segments: Sequence[dict],
|
||||
audio: np.ndarray,
|
||||
sr: int,
|
||||
*,
|
||||
min_shift_s: float = MIN_SHIFT_S,
|
||||
separated_vocals: bool = True,
|
||||
) -> int:
|
||||
"""Snap each segment's ``start`` forward to the actual speech onset.
|
||||
|
||||
Mutates the segment dicts in place (the shape the dub pipeline passes
|
||||
around). Returns the number of segments adjusted.
|
||||
|
||||
``audio`` should be mono float; the Demucs vocals track gives the best
|
||||
signal but the mixed track still beats nothing.
|
||||
``audio`` should be the mono-float **separated vocals** track. When the
|
||||
caller only has mixed/original audio (Demucs skipped or failed), pass
|
||||
``separated_vocals=False``: snapping is then disabled entirely (#963) —
|
||||
on a mixed track music, ambience and footsteps are all sustained energy,
|
||||
so a detected "onset" is as likely the score as the speaker, and
|
||||
whisper's own timestamps beat a confidently wrong snap.
|
||||
"""
|
||||
if not separated_vocals:
|
||||
logger.info(
|
||||
"onset-align: skipped — audio is not a separated vocals track "
|
||||
"(Demucs unavailable/failed); keeping ASR timestamps as-is")
|
||||
return 0
|
||||
if sr <= 0 or audio is None or len(audio) == 0:
|
||||
return 0
|
||||
if audio.ndim > 1:
|
||||
@@ -174,6 +271,13 @@ def snap_segment_starts(
|
||||
shift = new_start - start
|
||||
if shift < min_shift_s:
|
||||
continue
|
||||
if shift > MAX_SNAP_S and not _region_mostly_silent(audio, sr, start, onset):
|
||||
# Long jump over audible content (#963): the "onset" is more
|
||||
# likely a louder late event than the true start — quiet speech
|
||||
# under the relative threshold would be skipped wholesale and
|
||||
# the dub would play seconds LATE. Bounded corrections are fine;
|
||||
# unbounded ones only over true silence (the #280 case).
|
||||
continue
|
||||
# Preserve a minimum playable duration.
|
||||
new_start = min(new_start, end - MIN_SEG_DUR_S)
|
||||
if new_start - start < min_shift_s:
|
||||
|
||||
@@ -145,3 +145,179 @@ def save_lexicon(path, lexicon: Optional[dict]) -> dict[str, str]:
|
||||
encoding="utf-8",
|
||||
)
|
||||
return clean
|
||||
|
||||
|
||||
# ── DB-backed global / per-language dictionary (Expressive-TTS Spec 01) ───────
|
||||
#
|
||||
# The JSON ``load_lexicon``/``save_lexicon`` above stay the per-project audiobook
|
||||
# override. THIS layer is the user-editable, DB-persisted, per-language default
|
||||
# dictionary surfaced in Settings → Pronunciation. Rows scoped ``language="*"``
|
||||
# apply to every request; a 2-letter language row applies only when the request
|
||||
# language's prefix matches (case-insensitive), so a German entry never fires on
|
||||
# an English render. Both layers are pure text substitution — they ride the same
|
||||
# ReDoS-safe ``apply_lexicon`` matcher, so every engine honors them.
|
||||
|
||||
_ALL_LANG = "*"
|
||||
|
||||
|
||||
def _lang_prefix(language: Optional[str]) -> Optional[str]:
|
||||
"""Normalize a request language to a lowercase 2-letter prefix.
|
||||
|
||||
``"Auto"``/``None``/``""`` → ``None`` (means "no language pin": only global
|
||||
``*`` rows apply, language-tagged rows are skipped, mirroring how the engines
|
||||
treat an unset language). A value like ``"en-US"`` / ``"English"`` →
|
||||
``"en"`` (first two letters); matching against entries is on this prefix.
|
||||
"""
|
||||
if not language:
|
||||
return None
|
||||
s = str(language).strip().lower()
|
||||
if not s or s == "auto":
|
||||
return None
|
||||
return s[:2]
|
||||
|
||||
|
||||
def entries_for_language(entries, language: Optional[str]) -> dict[str, str]:
|
||||
"""Collapse DB rows into a ``{term: replacement}`` map for ``apply_lexicon``.
|
||||
|
||||
Filters to ``enabled`` rows whose scope is global (``*``) OR whose language
|
||||
prefix matches the request language. Only the **respelling** path produces a
|
||||
plain substitution here (Phase 1); IPA/CMU rows that carry no respelling are
|
||||
skipped at this layer (they're handled — or honestly degraded — by the
|
||||
engine-markup path, never silently mangling text). A language-specific row
|
||||
overrides a global row with the same (case-folded) term, so a per-language
|
||||
pronunciation can refine the global default.
|
||||
|
||||
``entries`` is any iterable of mappings/rows with ``term``, ``replacement``,
|
||||
``type``, ``language``, ``enabled`` keys (a ``sqlite3.Row`` works directly).
|
||||
"""
|
||||
req_prefix = _lang_prefix(language)
|
||||
# Two passes so language rows win over global rows on the same term: collect
|
||||
# global first, then overlay matching-language rows.
|
||||
glob: dict[str, str] = {}
|
||||
lang: dict[str, str] = {}
|
||||
for e in entries:
|
||||
try:
|
||||
if not int(e["enabled"]):
|
||||
continue
|
||||
except (KeyError, IndexError, TypeError, ValueError):
|
||||
continue
|
||||
term = (e["term"] or "").strip()
|
||||
if not term:
|
||||
continue
|
||||
etype = (e["type"] or "respelling").strip().lower()
|
||||
replacement = e["replacement"] if e["replacement"] is not None else ""
|
||||
# Phase 1: only respelling rows substitute text. IPA/CMU rows without a
|
||||
# respelling fall through (Phase 2 lowers them to engine markup); we do
|
||||
# NOT feed a raw IPA string into the grapheme stream.
|
||||
if etype != "respelling":
|
||||
continue
|
||||
scope = (e["language"] or _ALL_LANG).strip() or _ALL_LANG
|
||||
if scope == _ALL_LANG:
|
||||
glob[term] = str(replacement)
|
||||
else:
|
||||
if req_prefix is not None and scope[:2].lower() == req_prefix:
|
||||
lang[term] = str(replacement)
|
||||
merged = dict(glob)
|
||||
merged.update(lang) # language rows override global on the same term
|
||||
return merged
|
||||
|
||||
|
||||
# ── Inline one-off override: [[term|replacement]] / [[replacement]] ─────────
|
||||
#
|
||||
# Double brackets are unambiguous against the single-bracket grammar
|
||||
# (``[voice:]``/``[pause]``/SSML-lite/``[Name]``): ``_VOICE_RE`` is
|
||||
# ``\[voice:([^\]\[]*)\]`` — it forbids inner brackets, so it can't span a
|
||||
# ``[[…]]``; the SSML-lite / pause vocabularies are closed literal sets that
|
||||
# ``[[…]]`` is not a member of. We resolve ``[[…]]`` BEFORE chunking so the
|
||||
# splitter never sees it. ReDoS-safe: ``\[\[[^\]]*\]\]`` is a bounded literal
|
||||
# class, no nested quantifier.
|
||||
#
|
||||
# [[gif|jiff]] → replaces the literal "gif" → "jiff" for this occurrence
|
||||
# [[Nuh-VAD-uh]] → the bracket content itself is spoken (brackets stripped)
|
||||
# Bounded inner repetition ({0,256}) keeps this strictly linear: ``[^\]]`` also
|
||||
# matches ``[``, so an unbounded run of ``[`` with no closing ``]]`` would let the
|
||||
# engine re-scan O(n) content from O(n) start positions (polynomial ReDoS). The
|
||||
# bound caps per-position work; an inline override is a short respelling, so 256
|
||||
# chars is far more than any real ``[[term|replacement]]`` needs.
|
||||
_INLINE_RE = re.compile(r"\[\[([^\]]{0,256})\]\]")
|
||||
|
||||
|
||||
def apply_inline_overrides(text: str) -> str:
|
||||
"""Resolve ``[[…]]`` one-off pronunciation overrides to plain spoken text.
|
||||
|
||||
``[[term|replacement]]`` → ``replacement`` (the ``term`` half is a label for
|
||||
the author; only the replacement is spoken). ``[[replacement]]`` (no pipe) →
|
||||
``replacement`` with the brackets stripped. Empty ``[[]]`` collapses away.
|
||||
Applied once per occurrence; nothing persists. Single ``[…]`` tags are left
|
||||
untouched (the regex requires a double bracket on both sides).
|
||||
"""
|
||||
if not text or "[[" not in text:
|
||||
return text or ""
|
||||
|
||||
def _repl(m: re.Match) -> str:
|
||||
inner = m.group(1)
|
||||
if "|" in inner:
|
||||
inner = inner.split("|", 1)[1]
|
||||
return inner
|
||||
|
||||
return _INLINE_RE.sub(_repl, text)
|
||||
|
||||
|
||||
def apply_pronunciation(
|
||||
text: str,
|
||||
entries=None,
|
||||
language: Optional[str] = None,
|
||||
*,
|
||||
lexicon: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""Apply the pronunciation dictionary + inline overrides to ``text``.
|
||||
|
||||
Order (load-bearing):
|
||||
1. DB dictionary rows (``entries``) filtered to ``language`` + an optional
|
||||
per-project ``lexicon`` JSON overlay (project wins on term conflict,
|
||||
matching the audiobook layering). Both go through one ``apply_lexicon``
|
||||
pass (longest-term-first, word-boundary aware, idempotent).
|
||||
2. Inline ``[[…]]`` one-off overrides resolved last, so an inline override
|
||||
always wins over any dictionary entry for that occurrence.
|
||||
|
||||
A falsy ``text`` / empty dictionary / no inline markers is a pass-through, so
|
||||
legacy plain text is byte-identical.
|
||||
"""
|
||||
if not text:
|
||||
return text or ""
|
||||
merged = entries_for_language(entries or [], language)
|
||||
if lexicon:
|
||||
# Project-local JSON overlays the DB defaults; project wins on conflict.
|
||||
merged.update(normalize_lexicon(lexicon))
|
||||
out = apply_lexicon(text, merged) if merged else text
|
||||
return apply_inline_overrides(out)
|
||||
|
||||
|
||||
# ── DB load/save ──────────────────────────────────────────────────────────────
|
||||
|
||||
def load_entries_from_db() -> list[dict]:
|
||||
"""Return every pronunciation_entries row as a list of plain dicts.
|
||||
|
||||
Import-light: the DB module is imported lazily so the pure-parser path (and
|
||||
the audiobook JSON path) never pull in sqlite/config.
|
||||
"""
|
||||
from core.db import db_conn
|
||||
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def load_dict_for_request(language: Optional[str] = None) -> dict[str, str]:
|
||||
"""Convenience: DB rows → ``{term: replacement}`` for a request language.
|
||||
|
||||
Returns ``{}`` (a no-op for ``apply_pronunciation``) if the table is absent
|
||||
or the DB can't be opened — pronunciation is never allowed to break synth.
|
||||
"""
|
||||
try:
|
||||
return entries_for_language(load_entries_from_db(), language)
|
||||
except Exception: # noqa: BLE001 — table missing / DB locked → no-op
|
||||
return {}
|
||||
|
||||
+139
-16
@@ -19,13 +19,69 @@ Two tiers, both applied only to FINAL transcripts (never partials):
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
logger = logging.getLogger("omnivoice.refinement")
|
||||
|
||||
# Hard wall-clock budget (seconds) for a single dictation refinement LLM call.
|
||||
# The dictation FINAL must never be delayed longer than this by a slow or dead
|
||||
# LLM endpoint — refinement is best-effort and falls back to the unrefined
|
||||
# (but polished) text on timeout. 4s keeps a healthy local model (Ollama /
|
||||
# LM Studio, sub-second on the tiny cleanup prompt) fully usable while turning
|
||||
# the old worst case — a placeholder/dead endpoint blocking the send ~51s until
|
||||
# the widget's 15s fallback fired — into a bounded ~4s at most. Env-tunable so
|
||||
# power users on a slow local LLM can raise it. Guarded by the regression tests
|
||||
# in tests/backend/services/test_refinement_llm.py and tests/test_capture_ws.py.
|
||||
_DEFAULT_REFINE_TIMEOUT_S = 4.0
|
||||
|
||||
|
||||
def _refine_timeout_s() -> float:
|
||||
"""The refinement LLM budget in seconds (OMNIVOICE_REFINE_TIMEOUT_S).
|
||||
|
||||
Falls back to :data:`_DEFAULT_REFINE_TIMEOUT_S` on an unset/invalid/non-
|
||||
positive value so a bad env var can never disable the bound."""
|
||||
raw = os.environ.get("OMNIVOICE_REFINE_TIMEOUT_S", "")
|
||||
try:
|
||||
v = float(raw)
|
||||
if v > 0:
|
||||
return v
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return _DEFAULT_REFINE_TIMEOUT_S
|
||||
|
||||
|
||||
# Most-recent refinement outcome, so the Settings panel can tell the user when a
|
||||
# configured LLM is actually failing/timing out (the honesty layer behind the
|
||||
# `llm_ready` flag, which only means "an endpoint is configured"). Best-effort,
|
||||
# process-local, cleared on success.
|
||||
_last_refine_status: dict | None = None
|
||||
|
||||
|
||||
def _note_refine_status(*, ok: bool, reason: str | None = None) -> None:
|
||||
global _last_refine_status
|
||||
_last_refine_status = {"ok": bool(ok), "reason": reason, "at": time.time()}
|
||||
|
||||
|
||||
def get_last_refine_status() -> dict | None:
|
||||
"""The last refinement outcome as ``{ok, reason, at}`` or None if refinement
|
||||
hasn't run this session. ``ok=False`` with ``reason`` ("timeout" or a short
|
||||
error string) means a configured LLM failed the most recent final."""
|
||||
return dict(_last_refine_status) if _last_refine_status else None
|
||||
|
||||
|
||||
def _short_reason(exc: Exception) -> str:
|
||||
"""A compact, non-leaky label for a refinement failure (for the UI hint)."""
|
||||
name = type(exc).__name__
|
||||
if "Timeout" in name or "timeout" in str(exc).lower():
|
||||
return "timeout"
|
||||
return name
|
||||
|
||||
# A token (or unit) must repeat at least this many times consecutively to be
|
||||
# treated as an STT artifact. Rhetorical repetition ("no, no, no, no, no" —
|
||||
# five repeats) stays below the threshold and survives.
|
||||
@@ -248,6 +304,19 @@ REFINEMENT_EXAMPLES: list[tuple[str, str]] = [
|
||||
# settings_store key holding the user's refinement config (plain JSON).
|
||||
_SETTINGS_KEY = "dictation_refinement"
|
||||
|
||||
# LLM Skills registry id — Settings → LLM Skills can disable refinement's LLM
|
||||
# use or route it to a specific provider. Disabled == identical pass-through
|
||||
# (the same path as "no LLM configured").
|
||||
_SKILL_ID = "dictation_refinement"
|
||||
|
||||
|
||||
def _skill_llm():
|
||||
"""The skill-resolved backend (OffBackend when disabled/unconfigured)."""
|
||||
from services import llm_skills
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
return llm_skills.skill_backend(_SKILL_ID, active=get_active_llm_backend)
|
||||
|
||||
|
||||
def get_refinement_config() -> dict:
|
||||
"""Read the persisted config: {auto, smart_cleanup, self_correction,
|
||||
@@ -274,43 +343,97 @@ def set_refinement_config(cfg: dict) -> dict:
|
||||
return merged
|
||||
|
||||
|
||||
def refine_transcript(transcript: str, flags: RefinementFlags | None = None) -> str:
|
||||
def refine_transcript(
|
||||
transcript: str,
|
||||
flags: RefinementFlags | None = None,
|
||||
*,
|
||||
timeout_s: float | None = None,
|
||||
) -> str:
|
||||
"""Run the transcript through the configured LLM. Raises on failure —
|
||||
callers decide the fallback (maybe_refine swallows into pass-through)."""
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
callers decide the fallback (maybe_refine swallows into pass-through).
|
||||
|
||||
The LLM HTTP call is bounded by ``timeout_s`` (default: the refinement
|
||||
budget) so a dead/slow endpoint can't tie the call up for the client's full
|
||||
45s LLM timeout — the class of stall this whole module guards against."""
|
||||
flags = flags or RefinementFlags()
|
||||
backend = get_active_llm_backend()
|
||||
backend = _skill_llm()
|
||||
messages = [{"role": "system", "content": build_refinement_prompt(flags)}]
|
||||
for user_turn, assistant_turn in REFINEMENT_EXAMPLES:
|
||||
messages.append({"role": "user", "content": user_turn})
|
||||
messages.append({"role": "assistant", "content": assistant_turn})
|
||||
messages.append({"role": "user", "content": transcript})
|
||||
return backend.chat_messages(messages=messages).strip()
|
||||
budget = timeout_s if timeout_s is not None else _refine_timeout_s()
|
||||
return backend.chat_messages(messages=messages, timeout=budget).strip()
|
||||
|
||||
|
||||
def maybe_refine(transcript: str) -> str | None:
|
||||
def maybe_refine(transcript: str, *, timeout_s: float | None = None) -> str | None:
|
||||
"""Best-effort refinement for the dictation final path.
|
||||
|
||||
Returns the refined text, or None when refinement is off, no LLM
|
||||
backend is configured, the result is empty, or anything fails — the
|
||||
raw transcript always stands. Never raises.
|
||||
raw transcript always stands. Never raises. Records the outcome via
|
||||
:func:`get_last_refine_status` so the UI can flag a failing LLM.
|
||||
|
||||
Blocking (network I/O); the WS/REST callers run it off-thread. Prefer
|
||||
:func:`maybe_refine_async` on the live-dictation path — it adds the hard
|
||||
wall-clock bound so a slow endpoint can never delay the ``final`` send.
|
||||
"""
|
||||
if not transcript or not transcript.strip():
|
||||
return None
|
||||
cfg = get_refinement_config()
|
||||
if not cfg.get("auto", True):
|
||||
return None
|
||||
backend = _skill_llm()
|
||||
if backend.id == "off":
|
||||
# No LLM configured — or the dictation_refinement skill is disabled /
|
||||
# routed to an unconfigured provider — is not a failure. Leave the last
|
||||
# status untouched (same pass-through as today).
|
||||
return None
|
||||
try:
|
||||
cfg = get_refinement_config()
|
||||
if not cfg.get("auto", True):
|
||||
return None
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
backend = get_active_llm_backend()
|
||||
if backend.id == "off":
|
||||
return None
|
||||
refined = refine_transcript(transcript, RefinementFlags.from_dict(cfg))
|
||||
refined = refine_transcript(
|
||||
transcript, RefinementFlags.from_dict(cfg), timeout_s=timeout_s
|
||||
)
|
||||
if not refined:
|
||||
return None
|
||||
_note_refine_status(ok=True)
|
||||
return refined
|
||||
except Exception as e: # noqa: BLE001 — pass-through is the contract
|
||||
logger.warning("Dictation refinement skipped: %s", e)
|
||||
_note_refine_status(ok=False, reason=_short_reason(e))
|
||||
return None
|
||||
|
||||
|
||||
async def maybe_refine_async(
|
||||
transcript: str, *, timeout_s: float | None = None
|
||||
) -> str | None:
|
||||
"""Async, hard-time-bounded refinement for the live-dictation final path.
|
||||
|
||||
Runs :func:`maybe_refine` off-thread under a hard ``OMNIVOICE_REFINE_TIMEOUT_S``
|
||||
(~4s) budget so a slow or dead LLM endpoint can NEVER block the caller — and
|
||||
therefore the dictation ``final`` send — longer than the budget. On timeout
|
||||
(or any failure) it returns None and the raw, already-polished transcript
|
||||
stands. Never raises.
|
||||
|
||||
``asyncio.wait_for`` can't cancel the worker thread, but the LLM call it runs
|
||||
is itself bounded to the same budget (see :func:`refine_transcript`), so an
|
||||
orphaned thread unwinds shortly after rather than lingering the full 45s.
|
||||
"""
|
||||
if not transcript or not transcript.strip():
|
||||
return None
|
||||
budget = timeout_s if timeout_s is not None else _refine_timeout_s()
|
||||
try:
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(maybe_refine, transcript, timeout_s=budget),
|
||||
timeout=budget,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(
|
||||
"Dictation refinement exceeded its %.1fs budget — sending the "
|
||||
"unrefined final (set OMNIVOICE_REFINE_TIMEOUT_S to adjust).", budget,
|
||||
)
|
||||
_note_refine_status(ok=False, reason="timeout")
|
||||
return None
|
||||
except Exception as e: # noqa: BLE001 — best-effort; the raw final stands
|
||||
logger.warning("Dictation refinement failed: %s", e)
|
||||
_note_refine_status(ok=False, reason=_short_reason(e))
|
||||
return None
|
||||
|
||||
@@ -532,13 +532,183 @@ def assign_speakers_from_turns(
|
||||
return segments
|
||||
|
||||
|
||||
def assign_speakers_heuristic(segments: List[dict]) -> List[dict]:
|
||||
"""Two-speaker alternation based on silence gaps."""
|
||||
current = 1
|
||||
def assign_speakers_heuristic(
|
||||
segments: List[dict], num_speakers: Optional[int] = None
|
||||
) -> List[dict]:
|
||||
"""Silence-gap speaker assignment (used when no diarization model runs).
|
||||
|
||||
Base signal: a gap > SPEAKER_GAP seconds between consecutive segments is
|
||||
treated as a speaker change. Without a ``num_speakers`` hint this keeps
|
||||
the legacy behavior — alternate between exactly two labels. With a hint:
|
||||
|
||||
* ``num_speakers=1`` → every segment gets ``"Speaker 1"``.
|
||||
* ``num_speakers>=2`` → labels round-robin across N speakers at each
|
||||
gap boundary, so the user's requested count is represented instead of
|
||||
being silently capped at 2.
|
||||
|
||||
Limits (be honest with callers): this honors the *count*, not voice
|
||||
identity. The rotation order is arbitrary (a returning speaker gets the
|
||||
next label in the cycle, not their own), rapid exchanges with no
|
||||
> SPEAKER_GAP pause still collapse into one label, and N is an upper
|
||||
bound — audio with fewer gap boundaries than N yields fewer labels.
|
||||
Real per-speaker attribution needs pyannote (or an inline-diarizing ASR
|
||||
backend); callers should warn the user accordingly (see dub_core).
|
||||
Invalid hints (non-int, < 1) fall back to the legacy two-speaker cycle.
|
||||
"""
|
||||
try:
|
||||
n = int(num_speakers) if num_speakers is not None else 2
|
||||
except (TypeError, ValueError):
|
||||
n = 2
|
||||
if n < 1:
|
||||
n = 2
|
||||
current = 0 # zero-based rotation index; rendered one-based below
|
||||
last_end = 0.0
|
||||
for i, s in enumerate(segments):
|
||||
if i > 0 and (s["start"] - last_end) > SPEAKER_GAP:
|
||||
current = 2 if current == 1 else 1
|
||||
s["speaker_id"] = f"Speaker {current}"
|
||||
if i > 0 and n > 1 and (s["start"] - last_end) > SPEAKER_GAP:
|
||||
current = (current + 1) % n
|
||||
s["speaker_id"] = f"Speaker {current + 1}"
|
||||
last_end = s["end"]
|
||||
return segments
|
||||
|
||||
|
||||
# ── Speaker-aware re-split (#486) ────────────────────────────────────────────
|
||||
#
|
||||
# Segmentation runs BEFORE diarization and groups words by sentence/duration
|
||||
# only, so one segment can span two speakers' turns. assign_speakers_* then only
|
||||
# *relabels* each segment with its majority speaker — the boundary is lost and a
|
||||
# two-speaker exchange reads as one line. This pass re-splits such a segment at
|
||||
# the word-level speaker boundary, after diarization.
|
||||
#
|
||||
# Hard invariant (the single-speaker no-regression guarantee): a segment whose
|
||||
# words all map to ONE speaker is returned byte-for-byte unchanged — same dict,
|
||||
# id, text, start, end — so single-speaker dubs and their timing never move.
|
||||
|
||||
def _word_speaker(w: "Word", turns: Sequence[tuple]) -> Optional[str]:
|
||||
"""Majority-overlap speaker label for a word; midpoint membership as a
|
||||
fallback; ``None`` when the word has no diarization coverage at all."""
|
||||
acc: dict = {}
|
||||
for ts, te, label in turns:
|
||||
left = max(w.start, ts)
|
||||
right = min(w.end, te)
|
||||
if right > left:
|
||||
acc[label] = acc.get(label, 0.0) + (right - left)
|
||||
if acc:
|
||||
return max(acc.items(), key=lambda kv: kv[1])[0]
|
||||
mid = (w.start + w.end) / 2.0
|
||||
for ts, te, label in turns:
|
||||
if ts <= mid <= te:
|
||||
return label
|
||||
return None
|
||||
|
||||
|
||||
def _fill_and_smooth(labels: List[Optional[str]]) -> List[Optional[str]]:
|
||||
"""Forward/back-fill gaps (words with no coverage inherit a neighbor) and
|
||||
smooth single-word flips, so one mis-attributed word inside a speaker's run
|
||||
(diarization noise) doesn't trigger a spurious split."""
|
||||
out = list(labels)
|
||||
n = len(out)
|
||||
last = None
|
||||
for i in range(n):
|
||||
if out[i] is None:
|
||||
out[i] = last
|
||||
else:
|
||||
last = out[i]
|
||||
nxt = None
|
||||
for i in range(n - 1, -1, -1):
|
||||
if out[i] is None:
|
||||
out[i] = nxt
|
||||
else:
|
||||
nxt = out[i]
|
||||
for i in range(1, n - 1):
|
||||
if out[i] != out[i - 1] and out[i - 1] == out[i + 1]:
|
||||
out[i] = out[i - 1]
|
||||
return out
|
||||
|
||||
|
||||
def _resplit_core(
|
||||
segments: List[dict], words: Sequence["Word"], turns: Sequence[tuple],
|
||||
) -> List[dict]:
|
||||
"""Split each segment that spans >1 speaker at the word-level boundary.
|
||||
|
||||
``turns`` is a normalised list of ``(start, end, speaker_label)``. Single-
|
||||
speaker segments are passed through untouched. Pieces keep the segment's
|
||||
outer start/end (preserving any onset-snap) and use word times for interior
|
||||
boundaries, so the pieces exactly cover the original span.
|
||||
"""
|
||||
if not turns or not words:
|
||||
return segments
|
||||
ordered = sorted(words, key=lambda w: (w.start, w.end))
|
||||
out: List[dict] = []
|
||||
for seg in segments:
|
||||
s0, s1 = seg["start"], seg["end"]
|
||||
seg_words = [w for w in ordered if min(w.end, s1) - max(w.start, s0) > 1e-6]
|
||||
if len(seg_words) < 2:
|
||||
out.append(seg)
|
||||
continue
|
||||
labels = _fill_and_smooth([_word_speaker(w, turns) for w in seg_words])
|
||||
if len({l for l in labels if l is not None}) <= 1:
|
||||
out.append(seg) # single speaker (or unknown) → byte-for-byte unchanged
|
||||
continue
|
||||
runs: List[tuple] = []
|
||||
for w, label in zip(seg_words, labels):
|
||||
if runs and runs[-1][0] == label:
|
||||
runs[-1][1].append(w)
|
||||
else:
|
||||
runs.append((label, [w]))
|
||||
n_runs = len(runs)
|
||||
piece_no = 0
|
||||
for k, (label, ws) in enumerate(runs):
|
||||
text = _clean(" ".join(w.text for w in ws))
|
||||
if not text:
|
||||
continue
|
||||
piece = dict(seg)
|
||||
piece["text"] = text
|
||||
piece["start"] = s0 if k == 0 else ws[0].start
|
||||
piece["end"] = s1 if k == n_runs - 1 else ws[-1].end
|
||||
if label:
|
||||
piece["speaker_id"] = label
|
||||
if piece_no > 0:
|
||||
piece["id"] = f"{seg.get('id', 'seg')}-{piece_no}"
|
||||
if "text_original" in piece:
|
||||
piece["text_original"] = text
|
||||
elif "text_original" in piece:
|
||||
piece["text_original"] = text
|
||||
out.append(piece)
|
||||
piece_no += 1
|
||||
return out
|
||||
|
||||
|
||||
def _diar_speaker_label(raw) -> str:
|
||||
"""``SPEAKER_00`` → ``Speaker 1`` (mirrors assign_speakers_from_diarization)."""
|
||||
try:
|
||||
return f"Speaker {int(str(raw).split('_')[-1]) + 1}"
|
||||
except (ValueError, AttributeError):
|
||||
return str(raw)
|
||||
|
||||
|
||||
def resplit_segments_by_diarization(
|
||||
segments: List[dict], words: Sequence["Word"], diarization,
|
||||
) -> List[dict]:
|
||||
"""Speaker-aware re-split using a pyannote diarization result (#486)."""
|
||||
turns = [
|
||||
(turn.start, turn.end, _diar_speaker_label(spk))
|
||||
for turn, _, spk in diarization.itertracks(yield_label=True)
|
||||
]
|
||||
return _resplit_core(segments, words, turns)
|
||||
|
||||
|
||||
def resplit_segments_by_turns(
|
||||
segments: List[dict], words: Sequence["Word"], turns: Sequence[dict],
|
||||
) -> List[dict]:
|
||||
"""Speaker-aware re-split using inline ASR speaker turns (FunASR cam++).
|
||||
|
||||
``speaker`` is used verbatim (FunASR already labels ``"Speaker N"``), matching
|
||||
:func:`assign_speakers_from_turns`."""
|
||||
norm = [
|
||||
(t["start"], t["end"], t["speaker"])
|
||||
for t in (turns or [])
|
||||
if t.get("speaker") is not None
|
||||
and t.get("start") is not None
|
||||
and t.get("end") is not None
|
||||
]
|
||||
return _resplit_core(segments, words, norm)
|
||||
|
||||
@@ -108,6 +108,107 @@ def clear_hf_token() -> None:
|
||||
conn.execute("DELETE FROM settings WHERE key = ?", (_TOKEN_KEY,))
|
||||
|
||||
|
||||
# ── Generic encrypted secrets (LLM provider API keys, future tokens) ───────
|
||||
# The HF token got the first bespoke encrypted row; the LLM-providers feature
|
||||
# needs the *same* at-rest protection for a dozen provider keys. Rather than
|
||||
# copy the Fernet dance per provider, expose generic secret helpers. Rows are
|
||||
# namespaced with the ``secret.`` prefix so a misrouted ``get_text`` on a
|
||||
# secret key returns opaque ciphertext (defence in depth), and so plaintext
|
||||
# ``settings`` rows can never collide with a secret. Same InvalidToken →
|
||||
# None degrade as the HF path (install moved across machines → fall back to
|
||||
# env), same per-install key.
|
||||
_SECRET_PREFIX = "secret."
|
||||
|
||||
|
||||
def _secret_key_name(name: str) -> str:
|
||||
if not name or not isinstance(name, str):
|
||||
raise ValueError(f"secret name must be a non-empty string, got {name!r}")
|
||||
if name == _TOKEN_KEY or name.startswith(_SECRET_PREFIX):
|
||||
raise ValueError(f"invalid secret name {name!r}")
|
||||
return f"{_SECRET_PREFIX}{name}"
|
||||
|
||||
|
||||
def get_secret(name: str) -> Optional[str]:
|
||||
"""Return a decrypted secret (e.g. an LLM provider API key), or None.
|
||||
|
||||
Mirrors :func:`get_hf_token`: on decrypt failure (install migrated across
|
||||
machines) or any SQLite error, log and return None so callers fall back to
|
||||
env / provider defaults instead of crashing.
|
||||
"""
|
||||
from core.db import db_conn
|
||||
|
||||
key = _secret_key_name(name)
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT value FROM settings WHERE key = ?", (key,)
|
||||
).fetchone()
|
||||
if row is None or not row[0]:
|
||||
return None
|
||||
try:
|
||||
from cryptography.fernet import InvalidToken
|
||||
except ImportError: # pragma: no cover — dep should always be present
|
||||
logger.error("cryptography unavailable; cannot decrypt secret %s", name)
|
||||
return None
|
||||
try:
|
||||
return _fernet().decrypt(row[0].encode("ascii")).decode("utf-8")
|
||||
except InvalidToken:
|
||||
logger.warning(
|
||||
"Stored secret %r failed to decrypt (install moved across "
|
||||
"machines or salt tampered) — falling back to env/default.", name,
|
||||
)
|
||||
return None
|
||||
except Exception:
|
||||
logger.exception("settings_store.get_secret(%s): SQLite read failed", name)
|
||||
return None
|
||||
|
||||
|
||||
def set_secret(name: str, value: str) -> None:
|
||||
"""Persist an encrypted secret. Empty value clears the row."""
|
||||
if not value:
|
||||
clear_secret(name)
|
||||
return
|
||||
from core.db import db_conn
|
||||
|
||||
key = _secret_key_name(name)
|
||||
blob = _fernet().encrypt(value.encode("utf-8")).decode("ascii")
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO settings(key, value, updated_at) "
|
||||
"VALUES (?, ?, ?)",
|
||||
(key, blob, time.time()),
|
||||
)
|
||||
|
||||
|
||||
def clear_secret(name: str) -> None:
|
||||
"""Remove a secret row (salt row preserved, like clear_hf_token)."""
|
||||
from core.db import db_conn
|
||||
|
||||
key = _secret_key_name(name)
|
||||
with db_conn() as conn:
|
||||
conn.execute("DELETE FROM settings WHERE key = ?", (key,))
|
||||
|
||||
|
||||
def list_secret_names() -> list[str]:
|
||||
"""Return the bare names of all stored secrets (no values, no ciphertext).
|
||||
|
||||
Lets the LLM-providers settings API report *which* providers have a key
|
||||
configured without ever decrypting or returning the key material.
|
||||
"""
|
||||
from core.db import db_conn
|
||||
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT key FROM settings WHERE key LIKE ?",
|
||||
(f"{_SECRET_PREFIX}%",),
|
||||
).fetchall()
|
||||
return [r[0][len(_SECRET_PREFIX):] for r in rows if r and r[0]]
|
||||
except Exception:
|
||||
logger.exception("settings_store.list_secret_names: SQLite read failed")
|
||||
return []
|
||||
|
||||
|
||||
# ── Non-secret text settings ──────────────────────────────────────────────
|
||||
# Plan 01-02 Task 4 (INST-12): the Performance panel needs to persist a
|
||||
# boolean toggle (`perf.torch_compile_disabled`). It is NOT a secret — no
|
||||
@@ -128,7 +229,8 @@ def get_text(key: str, default: Optional[str] = None) -> Optional[str]:
|
||||
looking like opaque bytes — callers MUST use `get_hf_token()` for
|
||||
secrets and only ever pass non-secret keys to `get_text()`.
|
||||
"""
|
||||
if key == _TOKEN_KEY: # defence in depth — never let a misrouted call leak ciphertext
|
||||
if key == _TOKEN_KEY or key.startswith(_SECRET_PREFIX):
|
||||
# defence in depth — never let a misrouted call leak ciphertext
|
||||
return default
|
||||
from core.db import db_conn
|
||||
|
||||
@@ -150,10 +252,10 @@ def set_text(key: str, value: str) -> None:
|
||||
|
||||
Use for non-secret config only. For tokens, use `set_hf_token()`.
|
||||
"""
|
||||
if key == _TOKEN_KEY:
|
||||
if key == _TOKEN_KEY or key.startswith(_SECRET_PREFIX):
|
||||
raise ValueError(
|
||||
"set_text refuses to write to the encrypted hf_token row; "
|
||||
"use set_hf_token() for secrets"
|
||||
"set_text refuses to write to an encrypted secret row; "
|
||||
"use set_hf_token()/set_secret() for secrets"
|
||||
)
|
||||
from core.db import db_conn
|
||||
|
||||
|
||||
@@ -0,0 +1,334 @@
|
||||
"""
|
||||
sherpa-onnx live-dictation ASR backend.
|
||||
|
||||
Adds the k2-fsa/sherpa-onnx ONNX runtime as a *dictation* engine alongside the
|
||||
existing Whisper/NeMo family — without touching any of them. The whole point of
|
||||
this engine is **live, faster-than-real-time dictation on CPU**:
|
||||
|
||||
• STREAMING models (OnlineRecognizer) emit partial text frame-by-frame as the
|
||||
user speaks, finalising on sherpa's built-in endpoint (silence) detection.
|
||||
• OFFLINE models (OfflineRecognizer) re-transcribe a growing buffer on a short
|
||||
cadence so the user still sees live partials, finalising on EOF/silence.
|
||||
|
||||
CPU provider only (strict cross-platform-default parity rule): identical
|
||||
behaviour on macOS arm64+x86_64, Windows x64, Linux. No CUDA dependency.
|
||||
|
||||
Model weights are the small int8 ONNX checkpoints published under
|
||||
``csukuangfj/`` on HuggingFace; they download on first use through the same HF
|
||||
cache the rest of the app uses (``snapshot_download``). Exact asset filenames
|
||||
were verified against the live HF repo trees (see ``_MODELS`` below) — the
|
||||
streaming zipformer repos use the plain ``encoder-epoch-99-avg-1.int8.onnx``
|
||||
naming, NOT a ``-chunk-16-left-64`` variant.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
logger = logging.getLogger("omnivoice.asr.sherpa")
|
||||
|
||||
# CPU only — strict cross-platform default-parity rule. Overridable for
|
||||
# power users on a verified GPU build, but the default never diverges.
|
||||
_PROVIDER = os.environ.get("OMNIVOICE_SHERPA_ASR_PROVIDER", "cpu")
|
||||
_NUM_THREADS = int(os.environ.get("OMNIVOICE_SHERPA_ASR_THREADS", "2"))
|
||||
|
||||
|
||||
def _endpoint_rules() -> tuple[float, float]:
|
||||
"""Trailing-silence endpoint rules (seconds) for streaming recognizers.
|
||||
|
||||
Wispr-Flow-speed defaults (dictation v2): rule2 commits ~0.6s after speech
|
||||
stops, rule1 flushes after 1.0s of trailing non-speech — down from the
|
||||
upstream 2.4/1.2, which made every committed sentence feel laggy. Read at
|
||||
call time so the env overrides apply without a restart.
|
||||
"""
|
||||
def _f(env: str, default: float) -> float:
|
||||
try:
|
||||
return float(os.environ.get(env, "") or default)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
return (_f("OMNIVOICE_DICTATION_ENDPOINT_R1", 1.0),
|
||||
_f("OMNIVOICE_DICTATION_ENDPOINT_R2", 0.6))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SherpaModelSpec:
|
||||
"""One downloadable sherpa-onnx dictation model.
|
||||
|
||||
``files`` maps a logical role (encoder/decoder/joiner/tokens) to the EXACT
|
||||
asset filename in the HF repo. ``kind`` selects the recognizer factory:
|
||||
``offline-transducer`` | ``offline-whisper`` | ``online-transducer`` |
|
||||
``online-paraformer``. ``tag`` is the frontend-facing "offline"/"streaming".
|
||||
"""
|
||||
id: str
|
||||
repo_id: str
|
||||
label: str
|
||||
tag: str # "offline" | "streaming"
|
||||
kind: str # recognizer factory selector
|
||||
size_gb: float
|
||||
languages: str
|
||||
files: dict[str, str]
|
||||
recommended: bool = False
|
||||
model_type: str = "" # offline transducer only (nemo_transducer)
|
||||
extra: dict = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def streaming(self) -> bool:
|
||||
return self.tag == "streaming"
|
||||
|
||||
|
||||
# ── The 7 models (HF repo ids under csukuangfj/, filenames VERIFIED against the
|
||||
# live HF /api/models/<repo>/tree/main on 2026-06-25; int8 variants pinned).
|
||||
_MODELS: dict[str, SherpaModelSpec] = {
|
||||
"sherpa-parakeet-tdt-v3": SherpaModelSpec(
|
||||
id="sherpa-parakeet-tdt-v3",
|
||||
repo_id="csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8",
|
||||
label="Parakeet TDT v3",
|
||||
tag="offline",
|
||||
kind="offline-transducer",
|
||||
size_gb=0.18,
|
||||
languages="25 European languages",
|
||||
recommended=True,
|
||||
model_type="nemo_transducer",
|
||||
files={
|
||||
"encoder": "encoder.int8.onnx",
|
||||
"decoder": "decoder.int8.onnx",
|
||||
"joiner": "joiner.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-parakeet-tdt-v2": SherpaModelSpec(
|
||||
id="sherpa-parakeet-tdt-v2",
|
||||
repo_id="csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8",
|
||||
label="Parakeet TDT v2",
|
||||
tag="offline",
|
||||
kind="offline-transducer",
|
||||
size_gb=0.17,
|
||||
languages="English",
|
||||
model_type="nemo_transducer",
|
||||
files={
|
||||
"encoder": "encoder.int8.onnx",
|
||||
"decoder": "decoder.int8.onnx",
|
||||
"joiner": "joiner.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-zipformer-bilingual-zh-en": SherpaModelSpec(
|
||||
id="sherpa-zipformer-bilingual-zh-en",
|
||||
repo_id="csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20",
|
||||
label="Zipformer Bilingual",
|
||||
tag="streaming",
|
||||
kind="online-transducer",
|
||||
size_gb=0.13,
|
||||
languages="Chinese + English",
|
||||
files={
|
||||
"encoder": "encoder-epoch-99-avg-1.int8.onnx",
|
||||
"decoder": "decoder-epoch-99-avg-1.int8.onnx",
|
||||
"joiner": "joiner-epoch-99-avg-1.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-paraformer-bilingual-zh-en": SherpaModelSpec(
|
||||
id="sherpa-paraformer-bilingual-zh-en",
|
||||
repo_id="csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en",
|
||||
label="Paraformer Bilingual",
|
||||
tag="streaming",
|
||||
kind="online-paraformer",
|
||||
size_gb=0.115,
|
||||
languages="Chinese + English",
|
||||
files={
|
||||
"encoder": "encoder.int8.onnx",
|
||||
"decoder": "decoder.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-zipformer-en-20m": SherpaModelSpec(
|
||||
id="sherpa-zipformer-en-20m",
|
||||
repo_id="csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17",
|
||||
label="Zipformer Streaming EN",
|
||||
tag="streaming",
|
||||
kind="online-transducer",
|
||||
size_gb=0.128,
|
||||
languages="English",
|
||||
files={
|
||||
"encoder": "encoder-epoch-99-avg-1.int8.onnx",
|
||||
"decoder": "decoder-epoch-99-avg-1.int8.onnx",
|
||||
"joiner": "joiner-epoch-99-avg-1.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-zipformer-zh-14m": SherpaModelSpec(
|
||||
id="sherpa-zipformer-zh-14m",
|
||||
repo_id="csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23",
|
||||
label="Zipformer Streaming ZH",
|
||||
tag="streaming",
|
||||
kind="online-transducer",
|
||||
size_gb=0.074,
|
||||
languages="Chinese",
|
||||
files={
|
||||
"encoder": "encoder-epoch-99-avg-1.int8.onnx",
|
||||
"decoder": "decoder-epoch-99-avg-1.int8.onnx",
|
||||
"joiner": "joiner-epoch-99-avg-1.int8.onnx",
|
||||
"tokens": "tokens.txt",
|
||||
},
|
||||
),
|
||||
"sherpa-whisper-tiny": SherpaModelSpec(
|
||||
id="sherpa-whisper-tiny",
|
||||
repo_id="csukuangfj/sherpa-onnx-whisper-tiny",
|
||||
label="Whisper Tiny",
|
||||
tag="offline",
|
||||
kind="offline-whisper",
|
||||
size_gb=0.116,
|
||||
languages="90+ languages (auto-detect)",
|
||||
files={
|
||||
"encoder": "tiny-encoder.int8.onnx",
|
||||
"decoder": "tiny-decoder.int8.onnx",
|
||||
"tokens": "tiny-tokens.txt",
|
||||
},
|
||||
),
|
||||
}
|
||||
|
||||
DEFAULT_MODEL_ID = "sherpa-parakeet-tdt-v3"
|
||||
|
||||
# repo_id → model id, so the model-store list (keyed by repo_id) can be
|
||||
# enriched with the dictation metadata, and so capture can map either key.
|
||||
_REPO_TO_ID: dict[str, str] = {m.repo_id: mid for mid, m in _MODELS.items()}
|
||||
|
||||
|
||||
def list_specs() -> list[SherpaModelSpec]:
|
||||
return list(_MODELS.values())
|
||||
|
||||
|
||||
def get_spec(model_id: str) -> SherpaModelSpec | None:
|
||||
"""Look up a spec by its dictation id OR its HF repo_id."""
|
||||
if model_id in _MODELS:
|
||||
return _MODELS[model_id]
|
||||
if model_id in _REPO_TO_ID:
|
||||
return _MODELS[_REPO_TO_ID[model_id]]
|
||||
return None
|
||||
|
||||
|
||||
def is_sherpa_model(model_id: str | None) -> bool:
|
||||
return bool(model_id) and get_spec(model_id) is not None
|
||||
|
||||
|
||||
def sherpa_available() -> tuple[bool, str]:
|
||||
try:
|
||||
import sherpa_onnx # noqa: F401
|
||||
return True, "ready"
|
||||
except ImportError as e:
|
||||
return False, f"sherpa-onnx not installed: {e}. Install with: uv add sherpa-onnx"
|
||||
|
||||
|
||||
def _resolve_model_dir(spec: SherpaModelSpec, *, download: bool = True) -> str:
|
||||
"""Return the local directory containing this model's ONNX assets.
|
||||
|
||||
Tries the HF cache offline first (``local_files_only=True``); on a miss,
|
||||
downloads on first use (like every other engine) unless ``download=False``.
|
||||
Restricts the fetch to the exact int8 assets we pin via ``allow_patterns``
|
||||
so we never pull the bundled fp32 weights or test wavs.
|
||||
"""
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
wanted = list(spec.files.values())
|
||||
try:
|
||||
return snapshot_download(
|
||||
repo_id=spec.repo_id,
|
||||
local_files_only=True,
|
||||
allow_patterns=wanted,
|
||||
)
|
||||
except Exception:
|
||||
if not download:
|
||||
raise
|
||||
logger.info("sherpa dictation: downloading %s on first use", spec.repo_id)
|
||||
return snapshot_download(repo_id=spec.repo_id, allow_patterns=wanted)
|
||||
|
||||
|
||||
def is_installed(spec: SherpaModelSpec) -> bool:
|
||||
"""True if every pinned asset is already present in the HF cache."""
|
||||
try:
|
||||
d = _resolve_model_dir(spec, download=False)
|
||||
except Exception:
|
||||
return False
|
||||
return all(os.path.isfile(os.path.join(d, f)) for f in spec.files.values())
|
||||
|
||||
|
||||
# ── Recognizers ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_offline_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
"""Construct an ``OfflineRecognizer`` for an offline transducer/whisper model."""
|
||||
import sherpa_onnx
|
||||
|
||||
d = _resolve_model_dir(spec, download=download)
|
||||
|
||||
def p(role: str) -> str:
|
||||
return os.path.join(d, spec.files[role])
|
||||
|
||||
if spec.kind == "offline-transducer":
|
||||
return sherpa_onnx.OfflineRecognizer.from_transducer(
|
||||
encoder=p("encoder"),
|
||||
decoder=p("decoder"),
|
||||
joiner=p("joiner"),
|
||||
tokens=p("tokens"),
|
||||
num_threads=_NUM_THREADS,
|
||||
provider=_PROVIDER,
|
||||
decoding_method="greedy_search",
|
||||
model_type=spec.model_type or "nemo_transducer",
|
||||
)
|
||||
if spec.kind == "offline-whisper":
|
||||
return sherpa_onnx.OfflineRecognizer.from_whisper(
|
||||
encoder=p("encoder"),
|
||||
decoder=p("decoder"),
|
||||
tokens=p("tokens"),
|
||||
num_threads=_NUM_THREADS,
|
||||
provider=_PROVIDER,
|
||||
language="", # auto-detect
|
||||
task="transcribe",
|
||||
)
|
||||
raise ValueError(f"{spec.id} is not an offline model (kind={spec.kind})")
|
||||
|
||||
|
||||
def build_online_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
"""Construct an ``OnlineRecognizer`` (true streaming) with endpoint detection.
|
||||
|
||||
Endpoint (silence) detection drives the live "final" boundary: sherpa
|
||||
commits a sentence after trailing silence so we can flush a ``final`` and
|
||||
reset the stream for the next utterance — all within one WS session.
|
||||
"""
|
||||
import sherpa_onnx
|
||||
|
||||
d = _resolve_model_dir(spec, download=download)
|
||||
rule1, rule2 = _endpoint_rules()
|
||||
|
||||
def p(role: str) -> str:
|
||||
return os.path.join(d, spec.files[role])
|
||||
|
||||
if spec.kind == "online-transducer":
|
||||
return sherpa_onnx.OnlineRecognizer.from_transducer(
|
||||
tokens=p("tokens"),
|
||||
encoder=p("encoder"),
|
||||
decoder=p("decoder"),
|
||||
joiner=p("joiner"),
|
||||
num_threads=_NUM_THREADS,
|
||||
provider=_PROVIDER,
|
||||
decoding_method="greedy_search",
|
||||
enable_endpoint_detection=True,
|
||||
rule1_min_trailing_silence=rule1,
|
||||
rule2_min_trailing_silence=rule2,
|
||||
rule3_min_utterance_length=20,
|
||||
)
|
||||
if spec.kind == "online-paraformer":
|
||||
return sherpa_onnx.OnlineRecognizer.from_paraformer(
|
||||
tokens=p("tokens"),
|
||||
encoder=p("encoder"),
|
||||
decoder=p("decoder"),
|
||||
num_threads=_NUM_THREADS,
|
||||
provider=_PROVIDER,
|
||||
decoding_method="greedy_search",
|
||||
enable_endpoint_detection=True,
|
||||
rule1_min_trailing_silence=rule1,
|
||||
rule2_min_trailing_silence=rule2,
|
||||
rule3_min_utterance_length=20,
|
||||
)
|
||||
raise ValueError(f"{spec.id} is not a streaming model (kind={spec.kind})")
|
||||
@@ -42,11 +42,25 @@ IDEAL_REF_DURATION_S = 8.0 # target window — long enough for prosody, short e
|
||||
# is the empirical floor below which our zero-shot clone gets unstable.
|
||||
MIN_SEGMENT_REF_DURATION_S = 3.0
|
||||
|
||||
# Clone-purity guards (speaker-hint fix): a per-speaker reference cut from
|
||||
# mislabeled or boundary-adjacent audio mixes two people's voices and the
|
||||
# resulting clone sounds "made up".
|
||||
# * A slice below MIN_SLICE_DURATION_S is too short to be a reliable
|
||||
# single-speaker sample (and diarization boundary jitter dominates it).
|
||||
# * A slice whose edges come within ADJACENT_TURN_GUARD_S of a *different*
|
||||
# speaker's turn risks bleeding that speaker's audio across the imprecise
|
||||
# boundary — deprioritized (scoring preference, not a hard filter, so
|
||||
# extraction still succeeds on dense dialogue).
|
||||
MIN_SLICE_DURATION_S = 1.5
|
||||
ADJACENT_TURN_GUARD_S = 0.3
|
||||
|
||||
|
||||
def extract_speaker_clones(
|
||||
vocals_path: str,
|
||||
segments: list[dict],
|
||||
out_dir: str,
|
||||
*,
|
||||
labels_source: str | None = None,
|
||||
) -> dict[str, dict]:
|
||||
"""Build a per-speaker reference sample from `vocals_path` + `segments`.
|
||||
|
||||
@@ -63,7 +77,20 @@ def extract_speaker_clones(
|
||||
|
||||
Speakers whose segments total < MIN_REF_DURATION_S are skipped — we'd
|
||||
rather fall back to the default TTS voice than ship a bad clone.
|
||||
|
||||
``labels_source`` records where the ``speaker_id`` labels came from
|
||||
(``"pyannote"`` | ``"turns"`` | ``"heuristic"``; ``None`` = unknown,
|
||||
treated as trusted for backward compatibility). ``"heuristic"`` labels
|
||||
are silence-gap *estimates*, not voice identity — a reference cut from
|
||||
them routinely concatenates two people's audio, so extraction is skipped
|
||||
entirely (the caller warns the user and falls back to the default voice).
|
||||
"""
|
||||
if labels_source == "heuristic":
|
||||
logger.info(
|
||||
"speaker_clone: skipping auto-clone extraction — speaker labels "
|
||||
"are gap-based heuristic estimates, not voice identity"
|
||||
)
|
||||
return {}
|
||||
if not vocals_path or not os.path.exists(vocals_path):
|
||||
logger.info("speaker_clone: no vocals track at %s; skipping", vocals_path)
|
||||
return {}
|
||||
@@ -88,7 +115,12 @@ def extract_speaker_clones(
|
||||
out: dict[str, dict] = {}
|
||||
|
||||
for speaker_id, items in by_speaker.items():
|
||||
chosen = _pick_reference_slices(items)
|
||||
chosen = _pick_reference_slices(
|
||||
items,
|
||||
speaker_id=speaker_id,
|
||||
all_segments=segments,
|
||||
labels_source=labels_source,
|
||||
)
|
||||
if not chosen:
|
||||
logger.info(
|
||||
"speaker_clone: %s has <%ss of usable audio; will fall back to default voice",
|
||||
@@ -191,34 +223,144 @@ def extract_segment_refs(
|
||||
return out
|
||||
|
||||
|
||||
def refine_ref_text(ref_audio_path: str, asr_backend, fallback_text: str) -> str:
|
||||
"""Re-transcribe a written reference clip and return that transcript.
|
||||
|
||||
`extract_speaker_clones`/`extract_segment_refs` pair each audio slice with
|
||||
the ASR segment's OWN text field, on the assumption that the segment's
|
||||
timestamps and its transcribed text agree. They routinely don't — Whisper
|
||||
(and friends) frequently drift on segment boundaries: a trailing word
|
||||
audible in `[start, end]` but missing from `text`, or vice versa. When the
|
||||
(ref_audio, ref_text) pair disagrees, zero-shot TTS prompt-priming breaks
|
||||
down and the clone can speak the mismatched reference text itself instead
|
||||
of the target-language text it was given to synthesize (issue #1004).
|
||||
|
||||
Re-transcribing the *actual written clip* guarantees the pair matches by
|
||||
construction — the model doesn't care whether the original ASR text was
|
||||
right, only that ref_text is what's really in ref_audio. `asr_backend` is
|
||||
the caller's already-loaded active backend (duck-typed:
|
||||
`.transcribe(path, word_timestamps=...) -> dict` with a `chunks` list of
|
||||
`{"text": ...}`); the model is already warm, so this costs one more short
|
||||
transcribe call, not a fresh load. Falls back to `fallback_text` — never
|
||||
raises — so a re-transcribe failure is a strict no-op, never a regression
|
||||
from the original (matching) behavior.
|
||||
"""
|
||||
if asr_backend is None:
|
||||
return fallback_text
|
||||
try:
|
||||
result = asr_backend.transcribe(ref_audio_path, word_timestamps=False)
|
||||
text = " ".join(
|
||||
(c.get("text") or "").strip() for c in (result.get("chunks") or [])
|
||||
).strip()
|
||||
return text or fallback_text
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"speaker_clone: re-transcribe of %s failed, keeping original ref_text: %s",
|
||||
ref_audio_path, e,
|
||||
)
|
||||
return fallback_text
|
||||
|
||||
|
||||
def refine_ref_texts(clones: dict[str, dict], asr_backend) -> dict[str, dict]:
|
||||
"""Apply `refine_ref_text` to every entry's `ref_text` in place.
|
||||
|
||||
Batches the whole dict (per-speaker `clones` from `extract_speaker_clones`
|
||||
or per-segment `seg_clones` from `extract_segment_refs`) into the single
|
||||
executor round-trip the caller submits to the GPU pool, rather than one
|
||||
dispatch per reference. Mutates and returns `clones` for a convenient
|
||||
call-and-reassign at the call site.
|
||||
"""
|
||||
for entry in clones.values():
|
||||
entry["ref_text"] = refine_ref_text(
|
||||
entry["ref_audio"], asr_backend, entry.get("ref_text", "")
|
||||
)
|
||||
return clones
|
||||
|
||||
|
||||
# ── Internals ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _pick_reference_slices(items: list[tuple[int, dict]]) -> list[tuple[int, dict]]:
|
||||
def _adjacent_to_other_speaker(
|
||||
seg: dict, speaker_id: str, all_segments: list[dict] | None
|
||||
) -> bool:
|
||||
"""True when `seg`'s edges come within ADJACENT_TURN_GUARD_S of (or
|
||||
overlap) a segment attributed to a *different* speaker — a boundary where
|
||||
imprecise diarization timestamps risk bleeding the other voice into the
|
||||
reference slice."""
|
||||
if not all_segments:
|
||||
return False
|
||||
s0 = float(seg.get("start", 0.0))
|
||||
s1 = float(seg.get("end", 0.0))
|
||||
for other in all_segments:
|
||||
if other is seg:
|
||||
continue
|
||||
if (other.get("speaker_id") or "Speaker 1") == speaker_id:
|
||||
continue
|
||||
o0 = float(other.get("start", 0.0))
|
||||
o1 = float(other.get("end", 0.0))
|
||||
# Signed gap between the two spans; negative = overlap.
|
||||
if max(o0 - s1, s0 - o1) < ADJACENT_TURN_GUARD_S:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _pick_reference_slices(
|
||||
items: list[tuple[int, dict]],
|
||||
*,
|
||||
speaker_id: str | None = None,
|
||||
all_segments: list[dict] | None = None,
|
||||
labels_source: str | None = None,
|
||||
) -> list[tuple[int, dict]]:
|
||||
"""Select the subset of a speaker's segments to use as reference audio.
|
||||
|
||||
Strategy: take the single longest segment; if it's short, accumulate the
|
||||
next longest ones in original order until we clear IDEAL_REF_DURATION_S.
|
||||
Cap at MAX_REF_DURATION_S. Return [] if we can't reach MIN_REF_DURATION_S.
|
||||
Strategy: rank candidates clean-first (not temporally adjacent to a
|
||||
different speaker's turn — see ``_adjacent_to_other_speaker``), longest
|
||||
first within each tier, and accumulate until IDEAL_REF_DURATION_S is
|
||||
cleared. Adjacency is a scoring preference, NOT a hard filter — on dense
|
||||
dialogue where every slice borders another speaker, extraction still
|
||||
succeeds using the adjacent ones. Two hard guards protect clone purity:
|
||||
|
||||
* slices shorter than MIN_SLICE_DURATION_S are rejected outright
|
||||
(boundary jitter dominates them, so they're the likeliest to carry a
|
||||
second speaker's audio);
|
||||
* ``labels_source="heuristic"`` returns [] — gap-based labels are not
|
||||
voice identity, so no slice of them is safe to clone from.
|
||||
|
||||
Cap at MAX_REF_DURATION_S. Return [] if we can't reach
|
||||
MIN_REF_DURATION_S. When ``all_segments``/``speaker_id`` are not
|
||||
provided (legacy callers), adjacency scoring degrades to duration-only —
|
||||
the pre-guard behavior.
|
||||
"""
|
||||
if not items:
|
||||
return []
|
||||
if labels_source == "heuristic":
|
||||
return []
|
||||
if speaker_id is None:
|
||||
speaker_id = items[0][1].get("speaker_id") or "Speaker 1"
|
||||
|
||||
# Longest-first candidates. Keep original indices so we can preserve order.
|
||||
by_dur = sorted(
|
||||
def _dur(pair) -> float:
|
||||
return max(0.0, float(pair[1].get("end", 0.0)) - float(pair[1].get("start", 0.0)))
|
||||
|
||||
# Rank: clean (non-adjacent) before adjacent, longest first within each
|
||||
# tier. Keep original indices so we can restore transcript order below.
|
||||
ranked = sorted(
|
||||
items,
|
||||
key=lambda pair: (pair[1].get("end", 0.0) - pair[1].get("start", 0.0)),
|
||||
reverse=True,
|
||||
key=lambda pair: (
|
||||
_adjacent_to_other_speaker(pair[1], speaker_id, all_segments),
|
||||
-_dur(pair),
|
||||
),
|
||||
)
|
||||
|
||||
picked: list[tuple[int, dict]] = []
|
||||
total = 0.0
|
||||
for idx, seg in by_dur:
|
||||
dur = max(0.0, float(seg.get("end", 0.0)) - float(seg.get("start", 0.0)))
|
||||
if dur <= 0.0:
|
||||
for idx, seg in ranked:
|
||||
dur = _dur((idx, seg))
|
||||
if dur < MIN_SLICE_DURATION_S:
|
||||
continue
|
||||
if total + dur > MAX_REF_DURATION_S and picked:
|
||||
break
|
||||
# Ranking is no longer duration-monotonic, so a later (shorter or
|
||||
# adjacent) slice may still fit — skip, don't stop.
|
||||
continue
|
||||
picked.append((idx, seg))
|
||||
total += dur
|
||||
if total >= IDEAL_REF_DURATION_S:
|
||||
|
||||
+151
-10
@@ -18,9 +18,17 @@ import logging
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from services.llm_backend import get_active_llm_backend, OffBackend
|
||||
# Shared LLM-output divergence guard (length window + target-script +
|
||||
# critique-echo). Lives in translator; translator never imports this module,
|
||||
# so there is no import cycle.
|
||||
from services.translator import refine_output_ok
|
||||
|
||||
logger = logging.getLogger("omnivoice.speech_rate")
|
||||
|
||||
# LLM Skills registry id — Settings → LLM Skills can disable the slot-fit
|
||||
# LLM pass or route it to a specific provider. Disabled == the no-llm path.
|
||||
_SKILL_ID = "slot_fitting"
|
||||
|
||||
# Per-language read-speed estimates (chars/sec at natural pace, counting
|
||||
# Python `len()` codepoints — not phonemes or graphemes). These are
|
||||
# rough; real speakers vary wildly. Numbers below come from a mix of
|
||||
@@ -87,9 +95,15 @@ _EXPAND_PROMPT = """\
|
||||
You are a dubbing writer. The user will give you a translated line + the exact
|
||||
time slot it must fit. The current line is TOO SHORT — add natural filler or
|
||||
gently flesh out the thought while keeping the meaning the same. Aim for a
|
||||
reading duration that matches the slot.
|
||||
reading duration that matches the slot. Never invent new information, names,
|
||||
or dialogue that is not already in the line; do not more than double the line.
|
||||
Reply with ONLY the new line. No quotes, no commentary."""
|
||||
|
||||
# Below this predicted rate ratio a line can never honestly fill its slot —
|
||||
# any LLM "expansion" that far would be fabricated dialogue. Skip the expand
|
||||
# pass entirely and keep the short line (slot-aware TTS absorbs the silence).
|
||||
_MIN_EXPANDABLE_RATIO = 0.15
|
||||
|
||||
|
||||
def adjust_for_slot(
|
||||
text: str,
|
||||
@@ -97,16 +111,47 @@ def adjust_for_slot(
|
||||
slot_seconds: float,
|
||||
target_lang: str,
|
||||
source_text: Optional[str] = None,
|
||||
strict: bool = False,
|
||||
) -> dict:
|
||||
"""Return `{text, rate_ratio, attempts, error?}`.
|
||||
|
||||
Falls back to the input text if the LLM is off or the loop gives up.
|
||||
"""
|
||||
initial_ratio = rate_ratio(text, slot_seconds, target_lang)
|
||||
if TOL_LOW <= initial_ratio <= TOL_HIGH:
|
||||
return {"text": text, "rate_ratio": initial_ratio, "attempts": 0}
|
||||
|
||||
llm = get_active_llm_backend()
|
||||
``strict`` (Autofit mode) changes exactly one thing: the accepted upper
|
||||
bound is 1.0 instead of ``TOL_HIGH`` — the line must fit *within* the
|
||||
slot, never overrun it — so the target-language reading time can't exceed
|
||||
the segment and push the video timing out. Lines under ``TOL_LOW`` still
|
||||
go through the LLM expand pass in strict mode too (same as loose mode);
|
||||
padding is bounded by the divergence guard below, and a line under
|
||||
``_MIN_EXPANDABLE_RATIO`` is never expanded at all — it could only "fill"
|
||||
the slot with fabricated dialogue, so it stays short. Best-effort: after
|
||||
``MAX_ATTEMPTS`` it returns the closest candidate seen, so a stubborn line
|
||||
degrades gracefully.
|
||||
|
||||
Every LLM reply is validated with ``translator.refine_output_ok`` against
|
||||
the ORIGINAL input ``text`` (not the previous candidate — divergence
|
||||
compounds across attempts otherwise). A reply that fails the guard is
|
||||
discarded: the attempt is burned, ``current``/``best`` stay put, and if
|
||||
nothing valid ever came back the input text is returned with
|
||||
``error="fit-diverged"`` — a hallucinating model can no longer invent the
|
||||
dub line (v0.3.9 field report).
|
||||
"""
|
||||
tol_high = 1.0 if strict else TOL_HIGH
|
||||
initial_ratio = rate_ratio(text, slot_seconds, target_lang)
|
||||
if TOL_LOW <= initial_ratio <= tol_high:
|
||||
return {"text": text, "rate_ratio": initial_ratio, "attempts": 0}
|
||||
if initial_ratio < _MIN_EXPANDABLE_RATIO:
|
||||
return {
|
||||
"text": text,
|
||||
"rate_ratio": initial_ratio,
|
||||
"attempts": 0,
|
||||
"error": "fit-skip-short",
|
||||
}
|
||||
|
||||
from services import llm_skills
|
||||
# `active=` forwards this module's (monkeypatch-able) name so the
|
||||
# no-override path is byte-identical to the pre-skills behavior.
|
||||
llm = llm_skills.skill_backend(_SKILL_ID, active=lambda: get_active_llm_backend())
|
||||
if isinstance(llm, OffBackend):
|
||||
return {
|
||||
"text": text,
|
||||
@@ -117,9 +162,10 @@ def adjust_for_slot(
|
||||
|
||||
current = text
|
||||
best = (current, initial_ratio)
|
||||
diverged = False
|
||||
for attempt in range(1, MAX_ATTEMPTS + 1):
|
||||
r = rate_ratio(current, slot_seconds, target_lang)
|
||||
if TOL_LOW <= r <= TOL_HIGH:
|
||||
if TOL_LOW <= r <= tol_high:
|
||||
return {"text": current, "rate_ratio": r, "attempts": attempt - 1}
|
||||
|
||||
system = _TRIM_PROMPT if r > 1.0 else _EXPAND_PROMPT
|
||||
@@ -134,23 +180,43 @@ def adjust_for_slot(
|
||||
user_lines.append(f"Source line (for meaning): {source_text}")
|
||||
|
||||
try:
|
||||
next_text = llm.chat(system=system, user="\n".join(user_lines))
|
||||
next_text = llm.chat(
|
||||
system=system, user="\n".join(user_lines),
|
||||
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("speech-rate attempt %d failed: %s", attempt, e)
|
||||
return {"text": best[0], "rate_ratio": best[1], "attempts": attempt - 1, "error": str(e)}
|
||||
|
||||
if next_text and next_text.strip():
|
||||
current = next_text.strip()
|
||||
candidate = next_text.strip()
|
||||
# Divergence guard — validate against the ORIGINAL text, not
|
||||
# `current`: each accepted reply becomes the next prompt's input,
|
||||
# so per-step checks would let drift compound across attempts.
|
||||
ok, reason = refine_output_ok(text, candidate, target_lang)
|
||||
if not ok:
|
||||
diverged = True
|
||||
logger.warning(
|
||||
"speech-rate attempt %d rejected (%s) — discarding candidate",
|
||||
attempt, reason,
|
||||
)
|
||||
continue # attempt burned; current/best untouched
|
||||
current = candidate
|
||||
new_r = rate_ratio(current, slot_seconds, target_lang)
|
||||
# Keep the best candidate seen so far in case we exhaust retries.
|
||||
if abs(new_r - 1.0) < abs(best[1] - 1.0):
|
||||
best = (current, new_r)
|
||||
|
||||
return {
|
||||
out = {
|
||||
"text": best[0],
|
||||
"rate_ratio": best[1],
|
||||
"attempts": MAX_ATTEMPTS,
|
||||
}
|
||||
# Every usable reply diverged and the input text survived unchanged —
|
||||
# surface it on the row (rate_error in dub_translate, like fit-budget).
|
||||
if diverged and best[0] == text:
|
||||
out["error"] = "fit-diverged"
|
||||
return out
|
||||
|
||||
|
||||
def adjust_many(pairs: Iterable[tuple[str, float, str, Optional[str]]]) -> list[dict]:
|
||||
@@ -162,3 +228,78 @@ def adjust_many(pairs: Iterable[tuple[str, float, str, Optional[str]]]) -> list[
|
||||
adjust_for_slot(t, slot_seconds=s, target_lang=tl, source_text=src)
|
||||
for (t, s, tl, src) in pairs
|
||||
]
|
||||
|
||||
|
||||
async def adjust_for_slot_many(
|
||||
items: Iterable[tuple],
|
||||
*,
|
||||
executor=None,
|
||||
concurrency: Optional[int] = None,
|
||||
deadline: Optional[float] = None,
|
||||
loop=None,
|
||||
) -> dict:
|
||||
"""Fan `adjust_for_slot` out across many segments concurrently, bounded by a
|
||||
shared wall-clock ``deadline``.
|
||||
|
||||
``items``: iterable of ``(key, text, slot_seconds, target_lang,
|
||||
source_text_or_None, strict)``. Returns ``{key: adjust_for_slot_result}``.
|
||||
|
||||
Why this exists: the Autofit fit pass used to run one `adjust_for_slot` per
|
||||
segment *sequentially* and *outside* any budget, so a 50-segment dub against
|
||||
a slow/rate-limited LLM spun ~50×(per-call timeout) unbounded. Here every
|
||||
segment runs on the executor under a bounded ``asyncio.Semaphore``, and any
|
||||
segment still running when the shared ``deadline`` passes degrades to a
|
||||
no-fit result (input text kept, predicted ``rate_ratio``, ``error`` =
|
||||
``"fit-budget"``) instead of hanging the translate. ``deadline`` is an
|
||||
absolute ``loop.time()``; ``None`` disables the bound (run to completion).
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
loop = loop or asyncio.get_running_loop()
|
||||
items = list(items)
|
||||
if not items:
|
||||
return {}
|
||||
sem = asyncio.Semaphore(concurrency or int(os.environ.get("OMNIVOICE_LLM_CONCURRENCY", "6")))
|
||||
|
||||
async def _one(key, text, slot, tgt, src, strict):
|
||||
async with sem:
|
||||
res = await loop.run_in_executor(
|
||||
executor,
|
||||
lambda: adjust_for_slot(
|
||||
text, slot_seconds=slot, target_lang=tgt,
|
||||
source_text=src, strict=strict,
|
||||
),
|
||||
)
|
||||
return key, res
|
||||
|
||||
def _degraded(text, slot, tgt) -> dict:
|
||||
return {
|
||||
"text": text,
|
||||
"rate_ratio": rate_ratio(text, slot, tgt),
|
||||
"attempts": 0,
|
||||
"error": "fit-budget",
|
||||
}
|
||||
|
||||
tasks = [asyncio.ensure_future(_one(*it)) for it in items]
|
||||
|
||||
if deadline is None:
|
||||
pairs_out = await asyncio.gather(*tasks)
|
||||
return dict(pairs_out)
|
||||
|
||||
timeout = max(0.0, deadline - loop.time())
|
||||
done, _pending = await asyncio.wait(tasks, timeout=timeout)
|
||||
out: dict = {}
|
||||
for task, it in zip(tasks, items):
|
||||
key, text, slot, tgt = it[0], it[1], it[2], it[3]
|
||||
if task in done and not task.cancelled():
|
||||
try:
|
||||
k, res = task.result()
|
||||
out[k] = res
|
||||
continue
|
||||
except Exception as e: # noqa: BLE001 — one slow seg must not sink the pass
|
||||
logger.warning("fit segment %s failed: %s", key, e)
|
||||
else:
|
||||
task.cancel() # stop awaiting; the executor thread is abandoned (#730 pattern)
|
||||
out[key] = _degraded(text, slot, tgt)
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,471 @@
|
||||
"""Storage usage report for Settings → Storage.
|
||||
|
||||
Computes, for everything the app owns on disk:
|
||||
|
||||
* per-volume totals (total / used / free, grouped by ``st_dev`` so two
|
||||
roots on the same disk are reported once),
|
||||
* per-category directory sizes — the HF model cache (with the largest
|
||||
model dirs), the app data dir (broken into voices / outputs / dub_jobs /
|
||||
batch / preview / database / logs / other subtotals), the per-engine
|
||||
venvs under ``backend/engines/*/.venv`` (+ the app venv), and any
|
||||
``omnivoice*`` entries in the OS temp dir,
|
||||
* server-side ``warnings`` (low disk, volume pressure, unreadable paths)
|
||||
so every client renders the same guidance.
|
||||
|
||||
Directory walks are **bounded**: each top-level category gets a deadline
|
||||
(default 10 s) and returns a partial total (``complete: false`` + an
|
||||
``unreadable`` warning with ``reason: "timeout"``) when it expires. Results
|
||||
are cached in-process for 5 minutes; ``refresh`` bypasses the cache. The API
|
||||
layer runs the whole build in a worker thread so the event loop never blocks.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
CACHE_TTL_SECONDS = 300.0
|
||||
CATEGORY_TIMEOUT_SECONDS = 10.0
|
||||
TOP_MODEL_COUNT = 10
|
||||
VOLUME_PRESSURE_PERCENT = 90.0
|
||||
DEFAULT_MIN_FREE_GB = 10 # callers pass setup.wizard.MIN_FREE_GB — this is the standalone fallback
|
||||
|
||||
# DATA_DIR children we know by name (core.config constants + routers that
|
||||
# write there). Anything else lands in the "other" subtotal so the numbers
|
||||
# always add up to the real on-disk footprint.
|
||||
_DATA_CHILD_DIRS = ("voices", "outputs", "dub_jobs", "batch", "preview")
|
||||
_DB_PREFIX = "omnivoice.db" # omnivoice.db + -wal / -shm / -journal
|
||||
_LOG_FILES = ("crash_log.txt", "error_journal.jsonl")
|
||||
_LOG_PREFIX = "omnivoice.log" # rolling log + rotations
|
||||
|
||||
_GB = 1024 ** 3
|
||||
|
||||
|
||||
def default_engines_dir() -> str:
|
||||
"""``backend/engines`` — where per-engine venvs live (`<id>/.venv`)."""
|
||||
return str(Path(__file__).resolve().parents[1] / "engines")
|
||||
|
||||
|
||||
def default_app_venv() -> str | None:
|
||||
"""The venv this backend runs from, when it is one (None for system python)."""
|
||||
if sys.prefix != getattr(sys, "base_prefix", sys.prefix):
|
||||
return sys.prefix
|
||||
return None
|
||||
|
||||
|
||||
def _existing_ancestor(path: str) -> str:
|
||||
"""Deepest existing ancestor of ``path`` (for disk_usage on missing dirs)."""
|
||||
p = os.path.abspath(path)
|
||||
while p and not os.path.exists(p):
|
||||
parent = os.path.dirname(p)
|
||||
if parent == p:
|
||||
break
|
||||
p = parent
|
||||
return p
|
||||
|
||||
|
||||
def _mount_point(path: str) -> str:
|
||||
"""Mount point of the volume holding ``path`` (best-effort, cheap)."""
|
||||
p = _existing_ancestor(path)
|
||||
try:
|
||||
while p and not os.path.ismount(p):
|
||||
parent = os.path.dirname(p)
|
||||
if parent == p:
|
||||
break
|
||||
p = parent
|
||||
except OSError:
|
||||
pass
|
||||
return p or os.path.abspath(os.sep)
|
||||
|
||||
|
||||
def _dir_size(path: str, deadline: float) -> tuple[int, bool, str | None]:
|
||||
"""du-style size of ``path``: ``(bytes, complete, first_unreadable_path)``.
|
||||
|
||||
Never follows symlinks (lstat + walk default), never raises. Stops early
|
||||
and reports ``complete=False`` once ``deadline`` (time.monotonic) passes.
|
||||
"""
|
||||
err_path: str | None = None
|
||||
|
||||
def _onerror(e: OSError) -> None:
|
||||
nonlocal err_path
|
||||
if err_path is None:
|
||||
err_path = getattr(e, "filename", None) or path
|
||||
|
||||
try:
|
||||
if not os.path.exists(path):
|
||||
return 0, True, None
|
||||
if not os.path.isdir(path):
|
||||
return os.lstat(path).st_size, True, None
|
||||
except OSError:
|
||||
return 0, True, path
|
||||
|
||||
total = 0
|
||||
complete = True
|
||||
for root, _dirs, files in os.walk(path, onerror=_onerror):
|
||||
if time.monotonic() > deadline:
|
||||
complete = False
|
||||
break
|
||||
for name in files:
|
||||
fp = os.path.join(root, name)
|
||||
try:
|
||||
total += os.lstat(fp).st_size
|
||||
except OSError:
|
||||
if err_path is None:
|
||||
err_path = fp
|
||||
return total, complete, err_path
|
||||
|
||||
|
||||
def _sum_files(paths: list[str]) -> int:
|
||||
total = 0
|
||||
for p in paths:
|
||||
try:
|
||||
total += os.lstat(p).st_size
|
||||
except OSError:
|
||||
pass
|
||||
return total
|
||||
|
||||
|
||||
def _hf_model_dirs(cache_dir: str) -> list[str]:
|
||||
"""`models--org--name` dirs in the cache root and its `hub/` child.
|
||||
|
||||
HF_HUB_CACHE points straight at the hub dir; HF_HOME needs `/hub`
|
||||
appended — scanning both covers either env resolution.
|
||||
"""
|
||||
out: list[str] = []
|
||||
for base in (cache_dir, os.path.join(cache_dir, "hub")):
|
||||
try:
|
||||
with os.scandir(base) as it:
|
||||
out.extend(
|
||||
e.path for e in it
|
||||
if e.name.startswith("models--") and e.is_dir(follow_symlinks=False)
|
||||
)
|
||||
except OSError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def _model_display_name(dir_name: str) -> str:
|
||||
return dir_name.removeprefix("models--").replace("--", "/")
|
||||
|
||||
|
||||
def build_report(
|
||||
*,
|
||||
data_dir: str,
|
||||
hf_cache_dir: str,
|
||||
engines_dir: str | None = None,
|
||||
app_venv: str | None = None,
|
||||
temp_root: str | None = None,
|
||||
min_free_gb: float = DEFAULT_MIN_FREE_GB,
|
||||
category_timeout: float = CATEGORY_TIMEOUT_SECONDS,
|
||||
) -> dict:
|
||||
"""Build the full storage report (synchronous; call from a worker thread)."""
|
||||
engines_dir = engines_dir if engines_dir is not None else default_engines_dir()
|
||||
temp_root = temp_root if temp_root is not None else tempfile.gettempdir()
|
||||
warnings: list[dict] = []
|
||||
categories: list[dict] = []
|
||||
|
||||
def _warn_unreadable(category_id: str, path: str, reason: str) -> None:
|
||||
warnings.append({
|
||||
"kind": "unreadable",
|
||||
"severity": "warning",
|
||||
"category_id": category_id,
|
||||
"path": path,
|
||||
"reason": reason,
|
||||
})
|
||||
|
||||
def _finish(category_id: str, cat: dict, complete: bool, err_path: str | None) -> None:
|
||||
cat["complete"] = complete
|
||||
if not complete:
|
||||
_warn_unreadable(category_id, cat["path"], "timeout")
|
||||
if err_path is not None:
|
||||
_warn_unreadable(category_id, err_path, "permission")
|
||||
|
||||
# ── 1. HF model cache (+ top model dirs) ───────────────────────────────
|
||||
deadline = time.monotonic() + category_timeout
|
||||
hf_total = 0
|
||||
hf_complete = True
|
||||
hf_err: str | None = None
|
||||
models: list[dict] = []
|
||||
model_dirs = set(_hf_model_dirs(hf_cache_dir))
|
||||
seen: set[str] = set()
|
||||
for mdir in sorted(model_dirs):
|
||||
size, ok, err = _dir_size(mdir, deadline)
|
||||
hf_total += size
|
||||
hf_complete = hf_complete and ok
|
||||
hf_err = hf_err or err
|
||||
models.append({"name": _model_display_name(os.path.basename(mdir)), "bytes": size})
|
||||
seen.add(os.path.realpath(mdir))
|
||||
# Non-model remainder of the cache (datasets, xet chunks, token file, …):
|
||||
# walk the top-level entries that aren't model dirs so the category total
|
||||
# reflects the whole cache, not just models.
|
||||
try:
|
||||
with os.scandir(hf_cache_dir) as it:
|
||||
entries = list(it)
|
||||
except OSError:
|
||||
entries = []
|
||||
if os.path.exists(hf_cache_dir):
|
||||
hf_err = hf_err or hf_cache_dir
|
||||
for e in entries:
|
||||
if os.path.realpath(e.path) in seen:
|
||||
continue
|
||||
if e.name == "hub":
|
||||
# hub/ holds the model dirs (already counted) + misc; count the rest.
|
||||
try:
|
||||
with os.scandir(e.path) as hub_it:
|
||||
for h in hub_it:
|
||||
if os.path.realpath(h.path) in seen:
|
||||
continue
|
||||
size, ok, err = _dir_size(h.path, deadline)
|
||||
hf_total += size
|
||||
hf_complete = hf_complete and ok
|
||||
hf_err = hf_err or err
|
||||
except OSError:
|
||||
hf_err = hf_err or e.path
|
||||
continue
|
||||
size, ok, err = _dir_size(e.path, deadline)
|
||||
hf_total += size
|
||||
hf_complete = hf_complete and ok
|
||||
hf_err = hf_err or err
|
||||
models.sort(key=lambda m: m["bytes"], reverse=True)
|
||||
hf_cat = {
|
||||
"id": "hf_cache",
|
||||
"path": hf_cache_dir,
|
||||
"exists": os.path.isdir(hf_cache_dir),
|
||||
"bytes": hf_total,
|
||||
"items": models[:TOP_MODEL_COUNT],
|
||||
}
|
||||
_finish("hf_cache", hf_cat, hf_complete, hf_err)
|
||||
categories.append(hf_cat)
|
||||
|
||||
# ── 2. App data dir, broken into subtotals ─────────────────────────────
|
||||
deadline = time.monotonic() + category_timeout
|
||||
data_complete = True
|
||||
data_err: str | None = None
|
||||
children: list[dict] = []
|
||||
claimed: set[str] = set()
|
||||
|
||||
for name in _DATA_CHILD_DIRS:
|
||||
p = os.path.join(data_dir, name)
|
||||
size, ok, err = _dir_size(p, deadline)
|
||||
data_complete = data_complete and ok
|
||||
data_err = data_err or err
|
||||
claimed.add(name)
|
||||
children.append({"id": name, "path": p, "bytes": size, "complete": ok})
|
||||
|
||||
db_files = sorted(glob.glob(os.path.join(glob.escape(data_dir), _DB_PREFIX + "*")))
|
||||
claimed.update(os.path.basename(p) for p in db_files)
|
||||
children.append({
|
||||
"id": "database",
|
||||
"path": os.path.join(data_dir, _DB_PREFIX),
|
||||
"bytes": _sum_files(db_files),
|
||||
"complete": True,
|
||||
})
|
||||
|
||||
log_files = sorted(glob.glob(os.path.join(glob.escape(data_dir), _LOG_PREFIX + "*")))
|
||||
log_files += [os.path.join(data_dir, n) for n in _LOG_FILES]
|
||||
claimed.update(os.path.basename(p) for p in log_files)
|
||||
children.append({
|
||||
"id": "logs",
|
||||
"path": data_dir,
|
||||
"bytes": _sum_files(log_files),
|
||||
"complete": True,
|
||||
})
|
||||
|
||||
other_bytes = 0
|
||||
try:
|
||||
with os.scandir(data_dir) as it:
|
||||
for e in it:
|
||||
if e.name in claimed:
|
||||
continue
|
||||
if e.is_dir(follow_symlinks=False):
|
||||
size, ok, err = _dir_size(e.path, deadline)
|
||||
other_bytes += size
|
||||
data_complete = data_complete and ok
|
||||
data_err = data_err or err
|
||||
else:
|
||||
try:
|
||||
other_bytes += e.stat(follow_symlinks=False).st_size
|
||||
except OSError:
|
||||
data_err = data_err or e.path
|
||||
except OSError:
|
||||
if os.path.exists(data_dir):
|
||||
data_err = data_err or data_dir
|
||||
children.append({"id": "other", "path": data_dir, "bytes": other_bytes, "complete": True})
|
||||
|
||||
data_cat = {
|
||||
"id": "data",
|
||||
"path": data_dir,
|
||||
"exists": os.path.isdir(data_dir),
|
||||
"bytes": sum(c["bytes"] for c in children),
|
||||
"children": children,
|
||||
}
|
||||
_finish("data", data_cat, data_complete, data_err)
|
||||
categories.append(data_cat)
|
||||
|
||||
# ── 3. Engine venvs (+ the app venv) ───────────────────────────────────
|
||||
deadline = time.monotonic() + category_timeout
|
||||
venv_total = 0
|
||||
venv_complete = True
|
||||
venv_err: str | None = None
|
||||
venv_items: list[dict] = []
|
||||
try:
|
||||
with os.scandir(engines_dir) as it:
|
||||
engine_dirs = sorted(e.path for e in it if e.is_dir(follow_symlinks=False))
|
||||
except OSError:
|
||||
engine_dirs = []
|
||||
for edir in engine_dirs:
|
||||
venv_dir = os.path.join(edir, ".venv")
|
||||
if not os.path.isdir(venv_dir):
|
||||
continue
|
||||
size, ok, err = _dir_size(venv_dir, deadline)
|
||||
venv_total += size
|
||||
venv_complete = venv_complete and ok
|
||||
venv_err = venv_err or err
|
||||
venv_items.append({"name": os.path.basename(edir), "bytes": size})
|
||||
if app_venv:
|
||||
size, ok, err = _dir_size(app_venv, deadline)
|
||||
venv_total += size
|
||||
venv_complete = venv_complete and ok
|
||||
venv_err = venv_err or err
|
||||
venv_items.append({"name": "app", "bytes": size})
|
||||
venv_items.sort(key=lambda m: m["bytes"], reverse=True)
|
||||
venv_cat = {
|
||||
"id": "engine_venvs",
|
||||
"path": engines_dir,
|
||||
"exists": os.path.isdir(engines_dir),
|
||||
"bytes": venv_total,
|
||||
"items": venv_items,
|
||||
}
|
||||
_finish("engine_venvs", venv_cat, venv_complete, venv_err)
|
||||
categories.append(venv_cat)
|
||||
|
||||
# ── 4. Temp/working files the app owns (omnivoice* in the OS temp dir) ─
|
||||
deadline = time.monotonic() + category_timeout
|
||||
tmp_total = 0
|
||||
tmp_complete = True
|
||||
tmp_err: str | None = None
|
||||
for p in sorted(glob.glob(os.path.join(glob.escape(temp_root), "omnivoice*"))):
|
||||
size, ok, err = _dir_size(p, deadline)
|
||||
tmp_total += size
|
||||
tmp_complete = tmp_complete and ok
|
||||
tmp_err = tmp_err or err
|
||||
tmp_cat = {
|
||||
"id": "temp",
|
||||
"path": temp_root,
|
||||
"exists": os.path.isdir(temp_root),
|
||||
"bytes": tmp_total,
|
||||
"items": [],
|
||||
}
|
||||
_finish("temp", tmp_cat, tmp_complete, tmp_err)
|
||||
categories.append(tmp_cat)
|
||||
|
||||
# ── Volumes: group category roots by device, disk_usage once each ──────
|
||||
roots = {"hf_cache": hf_cache_dir, "data": data_dir, "engine_venvs": engines_dir, "temp": temp_root}
|
||||
by_dev: dict[object, dict] = {}
|
||||
for cid, root in roots.items():
|
||||
anchor = _existing_ancestor(root)
|
||||
try:
|
||||
dev: object = os.stat(anchor).st_dev
|
||||
except OSError:
|
||||
dev = anchor
|
||||
if dev not in by_dev:
|
||||
try:
|
||||
usage = shutil.disk_usage(anchor)
|
||||
except OSError:
|
||||
continue
|
||||
by_dev[dev] = {
|
||||
"path": _mount_point(anchor),
|
||||
"total_bytes": usage.total,
|
||||
"used_bytes": usage.used,
|
||||
"free_bytes": usage.free,
|
||||
"used_percent": round(usage.used / usage.total * 100.0, 1) if usage.total else 0.0,
|
||||
"roots": [],
|
||||
}
|
||||
by_dev[dev]["roots"].append(cid)
|
||||
volumes = list(by_dev.values())
|
||||
|
||||
# ── Server-side warnings ────────────────────────────────────────────────
|
||||
for v in volumes:
|
||||
free_gb = v["free_bytes"] / _GB
|
||||
base = {
|
||||
"path": v["path"],
|
||||
"free_gb": round(free_gb, 1),
|
||||
"min_free_gb": min_free_gb,
|
||||
"roots": v["roots"],
|
||||
}
|
||||
if free_gb < min_free_gb:
|
||||
warnings.append({"kind": "low_disk", "severity": "critical", **base})
|
||||
elif free_gb < 2 * min_free_gb:
|
||||
warnings.append({"kind": "low_disk", "severity": "low", **base})
|
||||
if v["used_percent"] > VOLUME_PRESSURE_PERCENT and ({"hf_cache", "data"} & set(v["roots"])):
|
||||
warnings.append({
|
||||
"kind": "volume_pressure",
|
||||
"severity": "warning",
|
||||
"path": v["path"],
|
||||
"used_percent": v["used_percent"],
|
||||
"roots": v["roots"],
|
||||
})
|
||||
|
||||
# Order: critical first, then the rest in computed order (stable sort).
|
||||
warnings.sort(key=lambda w: 0 if w["severity"] == "critical" else 1)
|
||||
|
||||
return {
|
||||
"generated_at": time.time(),
|
||||
"min_free_gb": min_free_gb,
|
||||
"volumes": volumes,
|
||||
"categories": categories,
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
|
||||
# ── In-process cache (5-minute TTL, refresh bypasses) ──────────────────────
|
||||
_cache_lock = threading.Lock()
|
||||
_cache: dict = {"key": None, "ts": 0.0, "report": None}
|
||||
|
||||
|
||||
def get_report(
|
||||
*,
|
||||
data_dir: str,
|
||||
hf_cache_dir: str,
|
||||
engines_dir: str | None = None,
|
||||
app_venv: str | None = None,
|
||||
temp_root: str | None = None,
|
||||
min_free_gb: float = DEFAULT_MIN_FREE_GB,
|
||||
category_timeout: float = CATEGORY_TIMEOUT_SECONDS,
|
||||
refresh: bool = False,
|
||||
ttl: float = CACHE_TTL_SECONDS,
|
||||
) -> dict:
|
||||
"""Cached ``build_report``. ``refresh=True`` forces a rescan."""
|
||||
key = (data_dir, hf_cache_dir, engines_dir, app_venv, temp_root, min_free_gb)
|
||||
if not refresh:
|
||||
with _cache_lock:
|
||||
fresh = (
|
||||
_cache["report"] is not None
|
||||
and _cache["key"] == key
|
||||
and (time.monotonic() - _cache["ts"]) < ttl
|
||||
)
|
||||
if fresh:
|
||||
return {**_cache["report"], "cached": True}
|
||||
report = build_report(
|
||||
data_dir=data_dir,
|
||||
hf_cache_dir=hf_cache_dir,
|
||||
engines_dir=engines_dir,
|
||||
app_venv=app_venv,
|
||||
temp_root=temp_root,
|
||||
min_free_gb=min_free_gb,
|
||||
category_timeout=category_timeout,
|
||||
)
|
||||
with _cache_lock:
|
||||
_cache.update(key=key, ts=time.monotonic(), report=report)
|
||||
return {**report, "cached": False}
|
||||
|
||||
|
||||
def clear_cache() -> None:
|
||||
"""Testing hook — drop the in-process cache."""
|
||||
with _cache_lock:
|
||||
_cache.update(key=None, ts=0.0, report=None)
|
||||
@@ -132,6 +132,10 @@ class IsolatedFasterWhisperBackend(SubprocessASRBackend):
|
||||
|
||||
id = "faster-whisper-isolated"
|
||||
display_name = "Faster-Whisper (crash-isolated subprocess)"
|
||||
# Same engine as FasterWhisperBackend, so the same device support — the
|
||||
# sidecar picks cuda/cpu itself via `_device()`. Without this the registry
|
||||
# default ("cpu",) would dishonestly report cpu_only routing on CUDA hosts.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
"""
|
||||
Deterministic polish for dictation finals (dictation v2).
|
||||
|
||||
Every ``final`` that leaves ``/ws/transcribe`` passes through
|
||||
:func:`polish_text` so pasted dictation reads like typed text:
|
||||
|
||||
* leading capital -- Latin scripts only (CJK/Cyrillic/etc. untouched),
|
||||
* terminal punctuation -- a period is appended unless the text already
|
||||
ends with sentence-terminal punctuation (incl. the CJK fullwidth forms),
|
||||
* doubled spaces collapsed, leading/trailing whitespace stripped.
|
||||
|
||||
Purely rule-based -- no model, no locale detection, no network -- so it is
|
||||
byte-for-byte reproducible and idempotent (``polish(polish(x)) == polish(x)``).
|
||||
|
||||
CJK codepoints below are ``\\u``-escaped on purpose: this is functional
|
||||
punctuation handling (allowed), and the escapes keep this file outside the
|
||||
literal-CJK scan in ``tests/test_no_hardcoded_cjk.py`` without growing its
|
||||
allowlist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Sentence-terminal punctuation that already "closes" a final -- Latin plus
|
||||
# the CJK fullwidth forms (U+3002 ideographic full stop, U+FF01 !, U+FF1F ?)
|
||||
# and ellipsis. A trailing closing quote/bracket after one of these still
|
||||
# counts as terminated ("He said \"hi.\"").
|
||||
_TERMINAL = ".!?\u2026\u3002\uff01\uff1f"
|
||||
_CLOSERS = "\"'\u201d\u2019\u00bb\u203a)]}\u300d\u300f\uff09\u3011"
|
||||
|
||||
# A dangling clause separator at the very end (ASR often stops mid-breath on
|
||||
# a comma) is swapped for a stop instead of stacking ",." punctuation.
|
||||
# Latin , ; : plus the CJK forms U+3001 U+FF0C U+FF1B U+FF1A.
|
||||
_DANGLING = ",;:\u3001\uff0c\uff1b\uff1a"
|
||||
|
||||
# CJK codepoints (kana, unified ideographs, compatibility + halfwidth forms)
|
||||
# -- used to pick the fullwidth stop U+3002 over "." for CJK sentences.
|
||||
_CJK = re.compile(
|
||||
"[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uff66-\uff9f]"
|
||||
)
|
||||
|
||||
_MULTISPACE = re.compile(r"[ \t]{2,}")
|
||||
|
||||
|
||||
def _is_latin_lower(ch: str) -> bool:
|
||||
"""Lowercase letter in a Latin block (ASCII, Latin-1, Latin Extended-A/B).
|
||||
|
||||
Capitalization is meaningless (CJK) or presumptuous (Cyrillic, Greek --
|
||||
the model's casing is trusted) outside Latin scripts.
|
||||
"""
|
||||
return ch.islower() and ord(ch) <= 0x024F
|
||||
|
||||
|
||||
def polish_text(text: str) -> str:
|
||||
"""Normalise one dictation final. Empty/whitespace-only input -> ``""``."""
|
||||
if not text:
|
||||
return ""
|
||||
out = _MULTISPACE.sub(" ", text).strip()
|
||||
if not out:
|
||||
return ""
|
||||
|
||||
# Leading capital (Latin scripts only).
|
||||
if _is_latin_lower(out[0]):
|
||||
out = out[0].upper() + out[1:]
|
||||
|
||||
# Already terminated -- possibly behind a closing quote/bracket?
|
||||
body = out.rstrip(_CLOSERS)
|
||||
if body and body[-1] in _TERMINAL:
|
||||
return out
|
||||
|
||||
# Swap a dangling comma/colon for the stop instead of stacking ",.".
|
||||
if out[-1] in _DANGLING:
|
||||
out = out[:-1].rstrip()
|
||||
if not out:
|
||||
return ""
|
||||
|
||||
# Script-matched stop: fullwidth U+3002 when the sentence ends in CJK.
|
||||
out += "\u3002" if _CJK.search(out[-1]) else "."
|
||||
return out
|
||||
@@ -92,9 +92,11 @@ REGISTRY: dict[str, dict] = {
|
||||
"category": "llm",
|
||||
"needs_key": True,
|
||||
"notes": (
|
||||
"Any OpenAI-compatible endpoint: GPT-4/5 (OpenAI), Claude (via OpenRouter), "
|
||||
"Gemini (OpenAI-compat mode), DeepSeek, Qwen, Ollama, LM Studio. "
|
||||
"Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY + TRANSLATE_MODEL."
|
||||
"Uses the LLM provider you configure in Settings → LLM Providers "
|
||||
"(route it via the 'Dub translation' skill in Settings → LLM Skills): "
|
||||
"GPT (OpenAI), Claude (via OpenRouter), Gemini, DeepSeek, Qwen, "
|
||||
"Ollama, LM Studio. Power-user env override: TRANSLATE_BASE_URL + "
|
||||
"TRANSLATE_API_KEY + TRANSLATE_MODEL."
|
||||
),
|
||||
},
|
||||
}
|
||||
@@ -120,16 +122,62 @@ def _probe(entry: dict) -> tuple[bool, str]:
|
||||
return False, f"import {mod!r} failed: {e}"
|
||||
|
||||
|
||||
def install_command(engine: "str | dict | None") -> str | None:
|
||||
"""The exact shell command that makes this engine importable, or None.
|
||||
|
||||
Single source of truth for the install string. BOTH the proactive Install
|
||||
affordance in the Engine selector (via list_engines' ``install_command``
|
||||
field) AND the translate-time 400 error (dub_translate.py) read from here,
|
||||
so the command a user is told to run can never drift between the two
|
||||
surfaces. Returns None when the engine needs no separate install — either
|
||||
it's unknown or its dependency is a core dep already pinned in
|
||||
``pyproject.toml`` (e.g. NLLB → transformers), in which case a
|
||||
``uv pip install`` line would be misleading.
|
||||
"""
|
||||
entry = engine if isinstance(engine, dict) else REGISTRY.get(engine) if engine else None
|
||||
pkg = entry.get("pip_package") if entry else None
|
||||
return f"uv pip install {pkg}" if pkg else None
|
||||
|
||||
|
||||
def _llm_configured() -> tuple[bool, "str | None"]:
|
||||
"""Whether the LLM translation engine has something to call, and via what.
|
||||
|
||||
Resolution mirrors the translate-time path in dub_translate.py: the
|
||||
"dub_translation" LLM skill (per-skill override → active provider from
|
||||
Settings → LLM Providers) first, then the TRANSLATE_* env override. Lets
|
||||
the Engine dropdown say "ready via <provider>" / "needs setup" up front
|
||||
instead of a per-segment failure after the user clicks Translate.
|
||||
"""
|
||||
try:
|
||||
from services import llm_skills
|
||||
res = llm_skills.resolve_skill("dub_translation")
|
||||
if res.ready and res.provider is not None:
|
||||
return True, res.provider.display_name
|
||||
except Exception: # noqa: BLE001 — a probe must never break list_engines()
|
||||
logger.debug("dub_translation skill probe failed", exc_info=True)
|
||||
if os.environ.get("TRANSLATE_BASE_URL") or os.environ.get("TRANSLATE_API_KEY"):
|
||||
return True, "env"
|
||||
return False, None
|
||||
|
||||
|
||||
def list_engines() -> list[dict]:
|
||||
"""Return a UI-ready list with per-engine availability stamped in."""
|
||||
out = []
|
||||
for e in REGISTRY.values():
|
||||
installed, reason = _probe(e)
|
||||
out.append({
|
||||
entry = {
|
||||
**e,
|
||||
"installed": installed,
|
||||
"availability_reason": reason,
|
||||
})
|
||||
"install_command": install_command(e),
|
||||
}
|
||||
# LLM engines additionally need a provider/key — surface configured-ness
|
||||
# so the UI can distinguish "importable" from "actually ready to call".
|
||||
if e.get("category") == "llm":
|
||||
configured, via = _llm_configured()
|
||||
entry["configured"] = configured
|
||||
entry["configured_via"] = via
|
||||
out.append(entry)
|
||||
return out
|
||||
|
||||
|
||||
@@ -177,6 +225,16 @@ async def run_pip(args: list[str], timeout: float = 600.0) -> tuple[int, str]:
|
||||
"""
|
||||
base = _installer_cmd()
|
||||
using_uv = base[:1] == ["uv"]
|
||||
# Pin `uv pip` to the interpreter the backend ACTUALLY runs under. The desktop
|
||||
# spawns `<venv>/bin/python -m uvicorn` WITHOUT exporting VIRTUAL_ENV, so bare
|
||||
# `uv pip install` finds no venv and 500s with "No virtual environment found"
|
||||
# (#529/#527) — and the `--system` branch below never fires, because the
|
||||
# running interpreter genuinely IS in a venv (uv just can't auto-discover it).
|
||||
# `--python sys.executable` targets the same interpreter _probe()/is_installed()
|
||||
# import from, and takes precedence when both flags are present, so the Docker
|
||||
# `--system` path is unaffected.
|
||||
if using_uv and args and args[0] in ("install", "uninstall") and "--python" not in args:
|
||||
args = [args[0], "--python", sys.executable, *args[1:]]
|
||||
if using_uv and not _in_virtualenv() and args and args[0] in ("install", "uninstall") and "--system" not in args:
|
||||
args = [args[0], "--system", *args[1:]]
|
||||
cmd = base + args
|
||||
|
||||
+196
-36
@@ -59,8 +59,9 @@ _ADAPT_PROMPT = """\
|
||||
You are a cinematic dubbing writer. Rewrite the literal translation using the
|
||||
editor's critique so it sounds natural, in-character, and fits the speaker's
|
||||
time slot. Keep meaning faithful but prefer native idiom over word-for-word
|
||||
accuracy. The output MUST be written in the same target language and script
|
||||
as the literal translation — never switch language or transliterate.
|
||||
accuracy. Never introduce facts, names, or dialogue that are not present in
|
||||
the source line. The output MUST be written in the same target language and
|
||||
script as the literal translation — never switch language or transliterate.
|
||||
Reply ONLY with the adapted translation — no quotes, no headers, no code
|
||||
fences, no commentary."""
|
||||
|
||||
@@ -93,28 +94,138 @@ def _looks_like_target_script(text: str, code: str, threshold: float = 0.5) -> b
|
||||
return (inside / len(letters)) >= threshold
|
||||
|
||||
|
||||
def _llm_client():
|
||||
"""Lazy-build the OpenAI-compatible client. Returns None if no key + no local base_url."""
|
||||
# ── Divergence guard (shared with speech_rate's Autofit fit pass) ────────────
|
||||
# For every Latin-script target `_looks_like_target_script` passes ANY text
|
||||
# unconditionally (no `_SCRIPT_RANGES` entry), so it was the only — and for
|
||||
# es/de/fr/… a no-op — gate on the ADAPT/fit LLM output. These checks close
|
||||
# that gap for the whole class: runaway length (hallucinated dialogue,
|
||||
# refusals, commentary) and the REFLECT critique echoed back as the "line".
|
||||
|
||||
_SHORT_REF_CHARS = 20 # below this, a length *ratio* is meaningless
|
||||
_SHORT_REF_ABS_SLACK = 120 # …use an absolute cap instead: ref + this many chars
|
||||
|
||||
|
||||
def _refine_ratio_bounds() -> tuple[float, float]:
|
||||
"""Accepted ``len(candidate)/len(reference)`` window for LLM refine output.
|
||||
Anything outside is treated as divergence and the caller degrades to its
|
||||
input text. Defaults [0.4, 2.5]; env-tunable like the cinematic budget."""
|
||||
try:
|
||||
from openai import OpenAI
|
||||
except ImportError:
|
||||
logger.warning("openai package not installed — cinematic mode unavailable.")
|
||||
return None
|
||||
base_url = os.environ.get("TRANSLATE_BASE_URL")
|
||||
api_key = (
|
||||
os.environ.get("TRANSLATE_API_KEY")
|
||||
or os.environ.get("OPENAI_API_KEY")
|
||||
or ("local" if base_url else None) # local providers often accept any key
|
||||
)
|
||||
if not api_key:
|
||||
return None
|
||||
kw = {"api_key": api_key}
|
||||
if base_url:
|
||||
kw["base_url"] = base_url
|
||||
return OpenAI(**kw)
|
||||
lo = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MIN", "0.4"))
|
||||
except ValueError:
|
||||
lo = 0.4
|
||||
try:
|
||||
hi = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MAX", "2.5"))
|
||||
except ValueError:
|
||||
hi = 2.5
|
||||
return lo, hi
|
||||
|
||||
|
||||
def _norm_overlap_text(s: str) -> str:
|
||||
return " ".join(s.lower().split())
|
||||
|
||||
|
||||
def _echoes_critique(candidate: str, critique: str) -> bool:
|
||||
"""True when the "adaptation" is really the REFLECT critique leaking through.
|
||||
|
||||
Deterministic on purpose (no fuzzy matching): exact match after
|
||||
case/whitespace normalization; containment — the full critique inside the
|
||||
candidate always counts, the candidate inside the critique only when it
|
||||
covers most of it (critiques legitimately quote short phrases from the
|
||||
line); or >0.8 token-set overlap.
|
||||
"""
|
||||
c = _norm_overlap_text(candidate)
|
||||
k = _norm_overlap_text(critique)
|
||||
if not c or not k:
|
||||
return False
|
||||
if c == k:
|
||||
return True
|
||||
if k in c: # critique embedded in the output
|
||||
return True
|
||||
if c in k and len(c) >= 0.6 * len(k): # output ≈ a big chunk of the critique
|
||||
return True
|
||||
ct, kt = set(c.split()), set(k.split())
|
||||
union = ct | kt
|
||||
return bool(union) and len(ct & kt) / len(union) > 0.8
|
||||
|
||||
|
||||
def refine_output_ok(
|
||||
reference: str,
|
||||
candidate: str,
|
||||
target_lang: str,
|
||||
*,
|
||||
critique: str | None = None,
|
||||
max_ratio: float | None = None,
|
||||
) -> tuple[bool, str | None]:
|
||||
"""Sanity-check one LLM refine output against the text it was rewriting.
|
||||
|
||||
Shared by the Cinematic ADAPT step here and by ``speech_rate``'s Autofit
|
||||
fit pass (speech_rate imports this; translator never imports speech_rate,
|
||||
so there is no cycle). Returns ``(ok, reason)`` — ``reason`` is ``None``
|
||||
when ok, otherwise a short machine-readable tag for logs/error mapping.
|
||||
|
||||
Checks, in order:
|
||||
• script — candidate must look like the target language's script
|
||||
(``_looks_like_target_script``; Latin-script targets pass, as before);
|
||||
• length — ``len(candidate)/len(reference)`` must sit inside
|
||||
[``OMNIVOICE_REFINE_RATIO_MIN``, ``OMNIVOICE_REFINE_RATIO_MAX``]
|
||||
(default 0.4–2.5; ``max_ratio`` overrides the upper bound). References
|
||||
shorter than ~20 chars use an absolute cap (reference + 120 chars)
|
||||
instead — a two-word line legitimately doubles or halves;
|
||||
• critique echo — the candidate must not be the critique itself.
|
||||
"""
|
||||
cand = (candidate or "").strip()
|
||||
ref = (reference or "").strip()
|
||||
if not cand:
|
||||
return False, "empty"
|
||||
if not _looks_like_target_script(cand, target_lang):
|
||||
return False, f"wrong-script:{target_lang}"
|
||||
lo, hi = _refine_ratio_bounds()
|
||||
if max_ratio is not None:
|
||||
hi = max_ratio
|
||||
if ref:
|
||||
if len(ref) < _SHORT_REF_CHARS:
|
||||
if len(cand) > len(ref) + _SHORT_REF_ABS_SLACK:
|
||||
return False, f"length-abs:{len(cand)}>{len(ref)}+{_SHORT_REF_ABS_SLACK}"
|
||||
else:
|
||||
ratio = len(cand) / len(ref)
|
||||
if not (lo <= ratio <= hi):
|
||||
return False, f"length-ratio:{ratio:.2f}"
|
||||
if critique and _echoes_critique(cand, critique):
|
||||
return False, "critique-echo"
|
||||
return True, None
|
||||
|
||||
|
||||
# The LLM Skills registry entry this pipeline resolves through — lets the
|
||||
# user disable Cinematic/Autofit's LLM use or route it to a specific provider
|
||||
# (Settings → LLM Skills) independently of the other LLM features.
|
||||
_SKILL_ID = "cinematic_translation"
|
||||
|
||||
|
||||
def _llm_client():
|
||||
"""Lazy-build the OpenAI-compatible client for the Cinematic skill.
|
||||
|
||||
Resolves through the LLM Skills registry: per-skill provider override →
|
||||
global active provider (Settings → LLM Providers). The registry's
|
||||
``custom`` provider still maps ``TRANSLATE_BASE_URL``/``TRANSLATE_API_KEY``,
|
||||
so legacy env setups keep working. Returns None if the skill is disabled
|
||||
or no provider is configured — the callers' Fast-fallback path.
|
||||
|
||||
The registry builds the client with ``max_retries=0`` (see
|
||||
``llm_skills.resolve_skill_client``) so a 429 + long Retry-After can't make
|
||||
one call sleep+retry past the cinematic wall-clock budget from inside a
|
||||
single request. The pass-level budget (``cinematic_refine_many``) and the
|
||||
per-call timeout stay the only bounds.
|
||||
"""
|
||||
from services import llm_skills
|
||||
handle = llm_skills.resolve_skill_client(_SKILL_ID)
|
||||
return handle.client if handle is not None else None
|
||||
|
||||
|
||||
def _llm_model() -> str:
|
||||
from services import llm_providers, llm_skills
|
||||
p = llm_skills.effective_provider(_SKILL_ID)
|
||||
if p is not None:
|
||||
return llm_providers.resolve_model(p)
|
||||
return os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
|
||||
|
||||
|
||||
@@ -125,6 +236,16 @@ def _llm_timeout() -> float:
|
||||
return 45.0
|
||||
|
||||
|
||||
def _cinematic_budget() -> float:
|
||||
"""Overall wall-clock cap for a whole cinematic/autofit refine pass (seconds).
|
||||
Unfinished segments degrade to their literal (Fast) translation once hit, so
|
||||
a slow provider can't hang the translate. Default 180s; <=0 disables."""
|
||||
try:
|
||||
return float(os.environ.get("OMNIVOICE_CINEMATIC_BUDGET_S", "180"))
|
||||
except ValueError:
|
||||
return 180.0
|
||||
|
||||
|
||||
def _glossary_text(glossary: Iterable[dict] | None) -> str:
|
||||
"""Format the project glossary as a preamble for the LLM prompts.
|
||||
|
||||
@@ -154,6 +275,7 @@ def _chat(client, *, system: str, user: str) -> str:
|
||||
res = client.chat.completions.create(
|
||||
model=_llm_model(),
|
||||
timeout=_llm_timeout(),
|
||||
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
@@ -255,21 +377,28 @@ def cinematic_refine_sync(
|
||||
}
|
||||
|
||||
final = (adapted or "").strip() or literal_text
|
||||
# Refuse adaptations that drifted off the target script (e.g. local LLM
|
||||
# rewrote a Devanagari line in Latin/German). Caller still gets the
|
||||
# critique so the UI can show what happened, but the live text falls
|
||||
# back to the literal translation rather than corrupting the dub.
|
||||
if final is not literal_text and not _looks_like_target_script(final, target_lang):
|
||||
logger.warning(
|
||||
"cinematic adapt produced wrong-script output for %s — falling back to literal",
|
||||
target_lang,
|
||||
)
|
||||
return {
|
||||
"text": literal_text,
|
||||
"literal": literal_text,
|
||||
"critique": critique,
|
||||
"error": f"adapt-wrong-script:{target_lang}",
|
||||
}
|
||||
# Refuse adaptations that diverged from the line they were rewriting:
|
||||
# wrong script (e.g. a local LLM rewrote a Devanagari line in
|
||||
# Latin/German), runaway length (hallucinated dialogue, refusals,
|
||||
# commentary — the script check alone passes ANY text for Latin-script
|
||||
# targets), or the critique echoed back as the "adaptation". Caller still
|
||||
# gets the critique so the UI can show what happened, but the live text
|
||||
# falls back to the literal translation rather than corrupting the dub.
|
||||
if final is not literal_text:
|
||||
ok, reason = refine_output_ok(literal_text, final, target_lang, critique=critique)
|
||||
if not ok:
|
||||
logger.warning(
|
||||
"cinematic adapt diverged for %s (%s) — falling back to literal",
|
||||
target_lang, reason,
|
||||
)
|
||||
wrong_script = (reason or "").startswith("wrong-script")
|
||||
return {
|
||||
"text": literal_text,
|
||||
"literal": literal_text,
|
||||
"critique": critique,
|
||||
"error": (f"adapt-wrong-script:{target_lang}" if wrong_script
|
||||
else "adapt-diverged"),
|
||||
}
|
||||
return {
|
||||
"text": final,
|
||||
"literal": literal_text,
|
||||
@@ -320,4 +449,35 @@ async def cinematic_refine_many(
|
||||
)
|
||||
return {"id": seg_id, **res}
|
||||
|
||||
return await asyncio.gather(*(_one(sid, src, lit) for sid, src, lit in pairs))
|
||||
# Overall wall-clock budget for the whole pass. Per-call timeout + bounded
|
||||
# concurrency already cap it, but a slow/rate-limited provider on a large dub
|
||||
# can still stall the "Translating…" spinner for minutes. Bound it: segments
|
||||
# that finish in time keep their cinematic refine; any still-running segment
|
||||
# degrades to its literal (Fast) translation so the translate ALWAYS returns
|
||||
# within the budget instead of hanging. 0/negative disables the bound.
|
||||
budget = _cinematic_budget()
|
||||
tasks = [asyncio.ensure_future(_one(sid, src, lit)) for sid, src, lit in pairs]
|
||||
if budget <= 0:
|
||||
return await asyncio.gather(*tasks)
|
||||
|
||||
done, pending = await asyncio.wait(tasks, timeout=budget)
|
||||
if pending:
|
||||
logger.warning(
|
||||
"Cinematic pass hit its %.0fs budget with %d/%d segment(s) unfinished "
|
||||
"— falling back to the literal translation for those (slow LLM "
|
||||
"provider?). Raise OMNIVOICE_CINEMATIC_BUDGET_S or pick a faster "
|
||||
"provider.", budget, len(pending), len(tasks),
|
||||
)
|
||||
out: list[dict] = []
|
||||
for task, (sid, _src, lit) in zip(tasks, pairs):
|
||||
if task in done and not task.cancelled():
|
||||
try:
|
||||
out.append(task.result())
|
||||
continue
|
||||
except Exception as e: # noqa: BLE001 — never let one seg sink the pass
|
||||
logger.warning("cinematic segment %s failed: %s", sid, e)
|
||||
else:
|
||||
task.cancel() # stop awaiting; the executor thread is abandoned (#730 pattern)
|
||||
out.append({"id": sid, "text": lit, "literal": lit, "critique": "",
|
||||
"error": "cinematic-budget"})
|
||||
return out
|
||||
|
||||
+410
-12
@@ -55,6 +55,59 @@ def _mask_hf_tokens(value):
|
||||
return _HF_TOKEN_MASK_RE.sub(_HF_TOKEN_MASK, value)
|
||||
|
||||
|
||||
# ── HF Hub closed-client recovery (#880) ────────────────────────────────────
|
||||
#
|
||||
# huggingface_hub ≥1.x shares ONE global httpx client across every download.
|
||||
# If anything closes it mid-lifecycle, every later hub call — e.g. an engine's
|
||||
# first-use model download inside the generate path — dies with httpx's
|
||||
# "Cannot send a request, as the client has been closed". The client is
|
||||
# recoverable: ``close_session()`` drops it and the next hub call builds a
|
||||
# fresh one, so the correct handling is a single targeted retry, not a
|
||||
# user-facing failure.
|
||||
|
||||
|
||||
def _is_closed_client_error(e) -> bool:
|
||||
"""True iff ``e`` (or anything in its __cause__/__context__ chain) is
|
||||
httpx's closed-client lifecycle error. Cycle-safe."""
|
||||
seen, stack = set(), [e]
|
||||
while stack:
|
||||
exc = stack.pop()
|
||||
if exc is None or id(exc) in seen:
|
||||
continue
|
||||
seen.add(id(exc))
|
||||
low = str(exc).lower()
|
||||
if "client has been closed" in low or "cannot send a request" in low:
|
||||
return True
|
||||
stack.append(exc.__cause__)
|
||||
stack.append(exc.__context__)
|
||||
return False
|
||||
|
||||
|
||||
def _retry_once_with_fresh_hf_client(loader, what: str):
|
||||
"""Run ``loader()`` — a model constructor that may download from the HF
|
||||
Hub on first use. On the specific closed-client failure above, reset the
|
||||
hub's shared client and retry exactly ONCE. Any other failure (and a
|
||||
repeat closed-client failure) propagates untouched, where the generation
|
||||
error classifier labels it as a network problem (#880)."""
|
||||
try:
|
||||
return loader()
|
||||
except Exception as e:
|
||||
if not _is_closed_client_error(e):
|
||||
raise
|
||||
logger.warning(
|
||||
"%s: HF Hub httpx client was closed mid-download (%s); "
|
||||
"retrying once with a fresh client.", what, e,
|
||||
)
|
||||
try:
|
||||
from huggingface_hub.utils import close_session
|
||||
close_session()
|
||||
except Exception: # pragma: no cover — hub too old / API renamed
|
||||
logger.warning(
|
||||
"%s: couldn't reset the HF Hub client; retrying anyway.", what,
|
||||
)
|
||||
return loader()
|
||||
|
||||
|
||||
# ── Protocol ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -89,13 +142,40 @@ class TTSBackend(ABC):
|
||||
#: (e.g. "young female, warm tone, British accent") without reference audio.
|
||||
supports_voice_design: bool = False
|
||||
|
||||
def ensure_ready(self) -> None:
|
||||
"""Load model weights now (blocking), so callers can separate the
|
||||
LOAD budget from the GENERATE budget (#1033/#1037 class).
|
||||
|
||||
Every adapter lazily loads inside ``generate()`` via a private
|
||||
``_ensure_loaded()`` — which meant a cold first call spent its whole
|
||||
``OMNIVOICE_GENERATE_TIMEOUT_S`` window (default 300s) downloading /
|
||||
loading weights and got killed with a misleading "too heavy for the
|
||||
available compute" error (measured in the wild on a fresh install:
|
||||
multi-GB checkpoint download, 0% GPU util, #1014). Routes call this
|
||||
first under the model-load budget (``OMNIVOICE_MODEL_LOAD_TIMEOUT``,
|
||||
default 1200s), then start the generate clock on an already-warm
|
||||
engine. Default implementation dispatches to the adapter's own
|
||||
``_ensure_loaded`` when present; engines without lazy state no-op.
|
||||
Must be called on the GPU pool (it's blocking), same as generate.
|
||||
"""
|
||||
loader = getattr(self, "_ensure_loaded", None)
|
||||
if callable(loader):
|
||||
loader()
|
||||
|
||||
#: Whether this engine already emits mastered, studio-grade audio and should
|
||||
#: therefore skip the shared apply_mastering() chain (Compressor + Reverb,
|
||||
#: therefore skip the shared apply_mastering() chain (highpass + Compressor,
|
||||
#: tuned for OmniVoice's 24 kHz output). Studio engines like VoxCPM2 (native
|
||||
#: 48 kHz) set this True so their clean output isn't pumped/reverbed. Loudness
|
||||
#: 48 kHz) set this True so their clean output isn't pumped. Loudness
|
||||
#: normalisation is applied regardless — it's a benign peak scale.
|
||||
applies_own_mastering: bool = False
|
||||
|
||||
#: Whether this engine can clone an arbitrary voice from reference audio
|
||||
#: (`ref_audio=`), as opposed to only offering a fixed set of preset
|
||||
#: voices. Default True — most engines clone. Dub/batch gate on this
|
||||
#: (issue #312 class) before committing to a job that needs it, instead
|
||||
#: of silently falling back to OmniVoice or mis-cloning per segment.
|
||||
supports_cloning: bool = True
|
||||
|
||||
#: GPU/accelerator targets the engine can run on. Surfaced via the
|
||||
#: Engine Compatibility Matrix (Plan 02-04 / ENGINE-06) so users can
|
||||
#: tell at a glance which engines will use their hardware. Defaults to
|
||||
@@ -550,6 +630,7 @@ class KittenTTSBackend(TTSBackend):
|
||||
display_name = "KittenTTS (English, 8 preset voices, CPU realtime)"
|
||||
# KittenTTS ships as an ONNX CPU graph; no CUDA/MPS path today.
|
||||
gpu_compat = ("cpu",)
|
||||
supports_cloning = False # fixed preset voices only; ref_audio is ignored
|
||||
|
||||
PRESET_VOICES = [
|
||||
"expr-voice-2-m", "expr-voice-2-f",
|
||||
@@ -587,7 +668,13 @@ class KittenTTSBackend(TTSBackend):
|
||||
"OMNIVOICE_KITTENTTS_MODEL", "KittenML/kitten-tts-mini-0.8"
|
||||
)
|
||||
logger.info("Loading KittenTTS from %s", checkpoint)
|
||||
self._model = KittenTTS(checkpoint)
|
||||
# #880: the first-use load downloads ~80 MB from the HF Hub inside the
|
||||
# generate path; if the hub's shared httpx client was closed
|
||||
# mid-lifecycle, retry once with a fresh client instead of failing
|
||||
# the whole generation.
|
||||
self._model = _retry_once_with_fresh_hf_client(
|
||||
lambda: KittenTTS(checkpoint), what="KittenTTS"
|
||||
)
|
||||
|
||||
def generate(self, text: str, **kw) -> torch.Tensor:
|
||||
import numpy as np
|
||||
@@ -624,6 +711,54 @@ class KittenTTSBackend(TTSBackend):
|
||||
# ── MLX-Audio (mac-ARM engine multiplexer) ──────────────────────────────────
|
||||
|
||||
|
||||
# #977: Kokoro's own ALIASES table (mlx_audio.tts.models.kokoro.pipeline) only
|
||||
# recognizes ISO-ish tokens ("en", "es", "fr-fr", "pt-br", …) — it has no idea
|
||||
# what a full language name is. OmniVoice's `language` kwarg is normally a
|
||||
# full display name from frontend/src/languages.json (e.g. "Dutch",
|
||||
# "Spanish"), forwarded verbatim by the frontend and by
|
||||
# `OmniVoiceBackend.generate()`. Translate the subset Kokoro actually
|
||||
# supports to the ISO token its own ALIASES expects; a caller that already
|
||||
# passes an ISO code (or one of Kokoro's own single-letter codes) is
|
||||
# resolved unchanged by `resolve_kokoro_lang_code()` below.
|
||||
_KOKORO_ISO_BY_FULL_NAME = {
|
||||
"english": "en",
|
||||
"spanish": "es",
|
||||
"french": "fr",
|
||||
"hindi": "hi",
|
||||
"italian": "it",
|
||||
"portuguese": "pt",
|
||||
"japanese": "ja",
|
||||
"chinese": "zh",
|
||||
}
|
||||
|
||||
|
||||
def resolve_kokoro_lang_code(language: str) -> str:
|
||||
"""Map a full language name / ISO code to Kokoro's single-letter
|
||||
`lang_code`, against the AUTHORITATIVE table read from the installed
|
||||
mlx-audio package (never a hardcoded guess — the vendored table is the
|
||||
only source of truth and can change across mlx-audio versions).
|
||||
|
||||
Raises ``ValueError`` for anything Kokoro doesn't support, naming what
|
||||
it *does* support — instead of forwarding a bogus code into Kokoro's
|
||||
`assert lang_code in LANG_CODES`, which crashes with an unreadable
|
||||
``(lang_code, LANG_CODES)`` tuple/dict repr (#977).
|
||||
"""
|
||||
from mlx_audio.tts.models.kokoro.pipeline import ALIASES, LANG_CODES
|
||||
|
||||
key = language.strip().lower()
|
||||
iso = _KOKORO_ISO_BY_FULL_NAME.get(key, key)
|
||||
code = ALIASES.get(iso, iso)
|
||||
if code not in LANG_CODES:
|
||||
supported = ", ".join(sorted(name.title() for name in _KOKORO_ISO_BY_FULL_NAME))
|
||||
raise ValueError(
|
||||
f"mlx-audio's Kokoro model (mlx-community/Kokoro-82M-bf16) doesn't "
|
||||
f"support language={language!r}. Kokoro supports: {supported}. "
|
||||
f"Pick one of those, leave language as 'Auto', or switch to a "
|
||||
f"multilingual engine (e.g. OmniVoice) for other languages."
|
||||
)
|
||||
return code
|
||||
|
||||
|
||||
class MLXAudioBackend(TTSBackend):
|
||||
"""Blaizzy/mlx-audio — Apple-Silicon-only wrapper over 14+ TTS engines
|
||||
(Kokoro, CSM, Dia, Qwen3-TTS, Chatterbox, MeloTTS, OuteTTS, Spark,
|
||||
@@ -662,7 +797,16 @@ class MLXAudioBackend(TTSBackend):
|
||||
def __init__(self):
|
||||
self._model = None
|
||||
self._sr = 24000 # most mlx-audio engines emit 24 kHz mono
|
||||
key = os.environ.get("OMNIVOICE_MLX_AUDIO_MODEL", self.DEFAULT_MODEL_KEY)
|
||||
# Env var > persisted UI choice (#981 — Settings → Engines curated-
|
||||
# model picker) > default. Mirrors active_backend_id()'s resolution
|
||||
# order exactly so power-users can still pin a model without the UI
|
||||
# silently undoing it.
|
||||
from core import prefs
|
||||
key = prefs.resolve(
|
||||
"mlx_audio_model_id",
|
||||
env="OMNIVOICE_MLX_AUDIO_MODEL",
|
||||
default=self.DEFAULT_MODEL_KEY,
|
||||
)
|
||||
# Accept either a curated key ("kokoro") or a full HF repo id
|
||||
# ("mlx-community/Kokoro-82M-bf16") — flexibility for power users.
|
||||
self._model_id = self.CURATED_MODELS.get(key, key)
|
||||
@@ -700,6 +844,18 @@ class MLXAudioBackend(TTSBackend):
|
||||
# silently ignores languages it doesn't know.
|
||||
return ["multi"]
|
||||
|
||||
@property
|
||||
def supports_cloning(self) -> bool:
|
||||
"""Model-dependent — this adapter multiplexes 7+ curated models and
|
||||
only some take a reference-audio speaker prompt. `generate()` passes
|
||||
`ref_audio` through when present (~kwargs below) but silently retries
|
||||
without it on a TypeError, so an engine picked for cloning that's
|
||||
actually running Kokoro/Qwen3-TTS/etc. would clone nothing. Of the
|
||||
curated set, only CSM (`mlx-community/csm-1b-8bit`) is confirmed to
|
||||
accept a reference prompt — default False for every other model,
|
||||
curated or user-supplied, until positively confirmed."""
|
||||
return self._model_id == self.CURATED_MODELS.get("csm")
|
||||
|
||||
def _ensure_loaded(self):
|
||||
if self._model is not None:
|
||||
return
|
||||
@@ -713,6 +869,7 @@ class MLXAudioBackend(TTSBackend):
|
||||
|
||||
voice = kw.get("voice")
|
||||
ref_audio = kw.get("ref_audio")
|
||||
ref_text = kw.get("ref_text")
|
||||
language = kw.get("language")
|
||||
speed = float(kw.get("speed", 1.0))
|
||||
|
||||
@@ -723,7 +880,31 @@ class MLXAudioBackend(TTSBackend):
|
||||
kwargs = {"text": text, "speed": speed}
|
||||
if voice: kwargs["voice"] = voice
|
||||
if ref_audio: kwargs["ref_audio"] = ref_audio
|
||||
if language: kwargs["lang_code"] = language[:2].lower()
|
||||
# CSM (sesame.py) only builds its cloning context when BOTH ref_audio
|
||||
# AND ref_text are present — with ref_text missing, its context list
|
||||
# stays empty and indexing into it raises an opaque
|
||||
# "IndexError: list index out of range" deep inside mlx-audio,
|
||||
# instead of ever attempting the clone. Community-diagnosed (#1012).
|
||||
if ref_audio and ref_text: kwargs["ref_text"] = ref_text
|
||||
if language and language != "Auto":
|
||||
if self._model_id == self.CURATED_MODELS.get("kokoro"):
|
||||
# Kokoro's vendored pipeline hard-asserts `lang_code` against
|
||||
# its own single-letter table — a bogus code crashes with an
|
||||
# unreadable AssertionError instead of failing cleanly
|
||||
# (#977). Resolve against the authoritative installed table
|
||||
# instead of guessing via `language[:2]`.
|
||||
kwargs["lang_code"] = resolve_kokoro_lang_code(language)
|
||||
else:
|
||||
# `lang_code`-as-2-letter-truncation is Kokoro's own
|
||||
# convention, not mlx-audio's in general — other curated
|
||||
# models either ignore unrecognized kwargs (CSM/Dia/OuteTTS
|
||||
# accept **kwargs and drop it) or expect something else
|
||||
# entirely (Qwen3-TTS's own docstring: "lang_code: Language
|
||||
# code (auto, chinese, english, etc.)" — a full name, not a
|
||||
# 2-letter code). Kokoro's strict validation doesn't apply to
|
||||
# them, so don't reject a language that's valid for whatever
|
||||
# model is actually active.
|
||||
kwargs["lang_code"] = language[:2].lower()
|
||||
|
||||
pieces = []
|
||||
try:
|
||||
@@ -1052,6 +1233,7 @@ class SherpaOnnxBackend(TTSBackend):
|
||||
# Sherpa-ONNX uses the onnxruntime providers — CPU is the universal
|
||||
# baseline; CUDA provider is available on Linux/Windows installs.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
supports_cloning = False # VITS speaker-id only; no ref_audio support
|
||||
|
||||
def __init__(self):
|
||||
self._tts = None
|
||||
@@ -1061,13 +1243,34 @@ class SherpaOnnxBackend(TTSBackend):
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
try:
|
||||
import sherpa_onnx # noqa: F401
|
||||
return True, "ready"
|
||||
except ImportError as e:
|
||||
return False, (
|
||||
f"sherpa-onnx not installed: {e}. "
|
||||
"Install with: pip install sherpa-onnx. "
|
||||
"Download models from https://github.com/k2-fsa/sherpa-onnx/releases"
|
||||
)
|
||||
# #919: sherpa-onnx ships no bundled default model — it can only
|
||||
# synthesize once OMNIVOICE_SHERPA_MODEL points at a downloaded model
|
||||
# directory. Gate on it here (like the other path-configured opt-in
|
||||
# engines: Confucius4/dots/MOSS) so the picker marks it unavailable-
|
||||
# with-a-reason instead of letting a user select it, generate, and hit
|
||||
# a config error that used to be mislabeled as out-of-memory.
|
||||
model_dir = os.environ.get("OMNIVOICE_SHERPA_MODEL", "").strip()
|
||||
if not model_dir:
|
||||
return False, (
|
||||
"OMNIVOICE_SHERPA_MODEL not set. Point it to a sherpa-onnx TTS "
|
||||
"model directory (containing model.onnx + tokens.txt), then "
|
||||
"restart OmniVoice. Download models from "
|
||||
"https://github.com/k2-fsa/sherpa-onnx/releases"
|
||||
)
|
||||
if not os.path.isfile(os.path.join(model_dir, "model.onnx")):
|
||||
return False, (
|
||||
f"No model.onnx in OMNIVOICE_SHERPA_MODEL ({model_dir}). Point "
|
||||
"it at a sherpa-onnx TTS model directory containing model.onnx "
|
||||
"+ tokens.txt. Download models from "
|
||||
"https://github.com/k2-fsa/sherpa-onnx/releases"
|
||||
)
|
||||
return True, "ready"
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
@@ -1151,6 +1354,19 @@ _LAZY_REGISTRY: dict[str, tuple[str, str]] = {
|
||||
# this module for TTSBackend). The class is resolved on first
|
||||
# attribute access via the LazyRegistry below.
|
||||
"supertonic3": ("engines.supertonic3", "Supertonic3Backend"),
|
||||
# Issue #498: MOSS-TTS-v1.5 (8B) and dots.tts (2B) — both opt-in,
|
||||
# subprocess-isolated with their own venv because each pins a
|
||||
# transformers version that conflicts with the parent's >=5.3
|
||||
# (MOSS == 5.0.0, dots.tts == 4.57.0). Same dedicated-venv pattern as
|
||||
# IndexTTS2. Lazy for the same import-cycle reason as the entries above.
|
||||
"moss-tts-v15": ("engines.moss_tts_v15", "MossTTSV15Backend"),
|
||||
"dots-tts": ("engines.dots_tts", "DotsTTSBackend"),
|
||||
# Issue #590: Confucius4-TTS (netease-youdao) — LLM-based, 14-language
|
||||
# cross-lingual zero-shot cloning, Apache-2.0. Opt-in + subprocess-isolated
|
||||
# (own Python 3.10 venv) like the entries above. Validated end-to-end
|
||||
# 2026-07-02 (CPU, Apple Silicon; 22.05 kHz output). Gated behind
|
||||
# OMNIVOICE_CONFUCIUS4_TTS_DIR so it's inert until enabled.
|
||||
"confucius4-tts": ("engines.confucius4", "Confucius4Backend"),
|
||||
}
|
||||
|
||||
|
||||
@@ -1187,7 +1403,13 @@ class _LazyRegistry(dict):
|
||||
# effect on every list_backends() call — we keep iteration light
|
||||
# and let the caller's __getitem__ trigger the import.
|
||||
seen: set[str] = set()
|
||||
for k in dict.__iter__(self):
|
||||
# Snapshot the live keys before yielding. A concurrent thread's lazy
|
||||
# __getitem__ inserts into self (self[key] = cls), and list_backends()
|
||||
# runs in a FastAPI threadpool — so holding a *live* dict iterator open
|
||||
# across the per-engine is_available() probes would raise
|
||||
# "dictionary changed size during iteration". list() consumes the
|
||||
# iterator atomically under the GIL, closing that window.
|
||||
for k in list(dict.__iter__(self)):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in _LAZY_REGISTRY:
|
||||
@@ -1244,6 +1466,41 @@ _INSTALL_HINTS: dict[str, str] = {
|
||||
"sherpa-onnx": "pip install sherpa-onnx (universal ONNX runtime, WASM-ready)",
|
||||
"omnivoice-gguf":"Bundled — runs the C++ omnivoice-tts binary in bin/. Quants download lazily from Serveurperso/OmniVoice-GGUF on first generate.",
|
||||
"supertonic3": "uv sync --extra supertonic (CPU-only ONNX, 31 langs, ~400 MB model on first use; OpenRAIL-M model license)",
|
||||
"moss-tts-v15": "git clone OpenMOSS/MOSS-TTS + set OMNIVOICE_MOSS_TTS_V15_DIR (own venv, transformers==5.0; 8B, ~16 GB weights; CUDA/CPU, no MPS; Apache-2.0)",
|
||||
"dots-tts": "git clone rednote-hilab/dots.tts + set OMNIVOICE_DOTS_TTS_DIR (own venv, transformers==4.57; 2B, ~9 GB weights; CUDA/CPU, Linux/macOS only — no Windows; Apache-2.0)",
|
||||
"confucius4-tts":"git clone netease-youdao/Confucius4-TTS + set OMNIVOICE_CONFUCIUS4_TTS_DIR (own Python 3.10 venv; 14-lang cross-lingual zero-shot clone; ~5 GB weights auto-download; CUDA/CPU, no MPS; Apache-2.0)",
|
||||
}
|
||||
|
||||
|
||||
# Copy-paste-ready setup line for opt-in engines gated behind a filesystem-path
|
||||
# env var (issue #498 / #590). The install_hint tells users a var exists; this
|
||||
# is the *exact* `export VAR=...` line to run, so they don't have to reconstruct
|
||||
# it from the docs. Surfaced verbatim in the Compat Matrix's "Why unavailable?"
|
||||
# disclosure with a Copy button. Single-sourced here so it can't drift from the
|
||||
# var each engine's is_available() actually reads. bash/zsh form (the dominant
|
||||
# clone-and-run workflow for these engines; dots.tts is *nix-only anyway).
|
||||
_SETUP_SNIPPETS: dict[str, str] = {
|
||||
"indextts2": "export OMNIVOICE_INDEXTTS_DIR=/path/to/index-tts",
|
||||
"moss-tts-v15": "export OMNIVOICE_MOSS_TTS_V15_DIR=/path/to/MOSS-TTS",
|
||||
"dots-tts": "export OMNIVOICE_DOTS_TTS_DIR=/path/to/dots.tts",
|
||||
"confucius4-tts": "export OMNIVOICE_CONFUCIUS4_TTS_DIR=/path/to/Confucius4-TTS",
|
||||
# #919: sherpa-onnx gates on a downloaded model dir (model.onnx + tokens.txt).
|
||||
"sherpa-onnx": "export OMNIVOICE_SHERPA_MODEL=/path/to/sherpa-onnx-model",
|
||||
}
|
||||
|
||||
|
||||
# Short, readable labels for mlx-audio's curated models (#981) — surfaced in
|
||||
# the Settings → Engines model picker so users see more than a bare key.
|
||||
# Single-sourced here rather than on MLXAudioBackend.CURATED_MODELS itself so
|
||||
# the class dict stays a plain key → repo-id map (what __init__ needs).
|
||||
_MLX_AUDIO_MODEL_LABELS: dict[str, str] = {
|
||||
"kokoro": "Kokoro (default, fast)",
|
||||
"csm": "CSM (voice cloning)",
|
||||
"qwen3-tts": "Qwen3-TTS (voice design)",
|
||||
"dia": "Dia",
|
||||
"chatterbox": "Chatterbox",
|
||||
"melotts": "MeloTTS (lightweight)",
|
||||
"outetts": "OuteTTS",
|
||||
}
|
||||
|
||||
|
||||
@@ -1258,6 +1515,7 @@ def list_backends() -> list[dict]:
|
||||
"available": bool,
|
||||
"reason": Optional[str], # message when not available
|
||||
"install_hint": Optional[str],
|
||||
"setup_snippet": Optional[str], # exact `export VAR=...` for path-gated opt-in engines
|
||||
"last_error": Optional[str], # cached most-recent failure
|
||||
"isolation_mode": "in-process" | "subprocess",
|
||||
"gpu_compat": list[str], # subset of {cuda, rocm, mps, xpu, cpu}
|
||||
@@ -1321,12 +1579,30 @@ def list_backends() -> list[dict]:
|
||||
"available": ok,
|
||||
"reason": None if ok else _mask_hf_tokens(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
# Exact `export VAR=...` line for path-gated opt-in engines, or None.
|
||||
"setup_snippet": _SETUP_SNIPPETS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": isolation,
|
||||
"gpu_compat": list(gpu_compat),
|
||||
# effective_device / routing_status / routing_reason (scrubbed):
|
||||
**routing_fields(gpu_compat, caps),
|
||||
})
|
||||
# #981: mlx-audio multiplexes 7+ curated models behind one backend id
|
||||
# — surface the roster + the currently-active pick so Settings can
|
||||
# render a model picker instead of always defaulting to Kokoro.
|
||||
# mlx-audio ONLY; every other backend loads a single fixed model.
|
||||
if bid == "mlx-audio":
|
||||
from core import prefs
|
||||
active_model = prefs.resolve(
|
||||
"mlx_audio_model_id",
|
||||
env="OMNIVOICE_MLX_AUDIO_MODEL",
|
||||
default=cls.DEFAULT_MODEL_KEY,
|
||||
)
|
||||
out[-1]["curated_models"] = [
|
||||
{"key": key, "label": _MLX_AUDIO_MODEL_LABELS.get(key, key), "repo_id": repo_id}
|
||||
for key, repo_id in cls.CURATED_MODELS.items()
|
||||
]
|
||||
out[-1]["active_model_id"] = active_model
|
||||
return out
|
||||
|
||||
|
||||
@@ -1336,6 +1612,29 @@ def get_backend_class(backend_id: str) -> type[TTSBackend]:
|
||||
return _REGISTRY[backend_id]
|
||||
|
||||
|
||||
def cloning_capable_engine_ids() -> list[str]:
|
||||
"""Engine ids that support reference-audio voice cloning — used to build
|
||||
an actionable error when the active engine can't (dub/batch gating).
|
||||
|
||||
Iterates the same registry ``list_backends()`` uses, via ``.items()`` so
|
||||
lazy entries resolve through ``_LazyRegistry``'s snapshot-safe iteration
|
||||
(see ``_LazyRegistry.__iter__``) exactly like every other registry scan
|
||||
in this module.
|
||||
|
||||
A class-level ``getattr`` on a *property* returns the descriptor object
|
||||
itself (always truthy) rather than its computed value — so a
|
||||
model-dependent adapter like ``MLXAudioBackend`` (only some of its 7+
|
||||
curated models can clone) would always show up here regardless of which
|
||||
model is actually configured. Excluded rather than falsely recommended:
|
||||
``isinstance(..., bool)`` is False for a descriptor, True for a plain
|
||||
class attribute.
|
||||
"""
|
||||
return [
|
||||
bid for bid, cls in _REGISTRY.items()
|
||||
if isinstance((v := getattr(cls, "supports_cloning", True)), bool) and v
|
||||
]
|
||||
|
||||
|
||||
def active_routing() -> dict | None:
|
||||
"""Routing verdict for the currently-active TTS engine, or ``None`` if it
|
||||
can't be determined (no engine / probe failure).
|
||||
@@ -1404,15 +1703,21 @@ def active_backend_id() -> str:
|
||||
# call the outgoing engine's unload() before switching.
|
||||
_active_instance: "TTSBackend | None" = None
|
||||
_active_instance_id: "str | None" = None
|
||||
# mlx-audio multiplexes 7+ curated models behind one backend id — a model-only
|
||||
# switch (same "mlx-audio" id, different curated model) must also invalidate
|
||||
# the cache, or picking a different model in Settings has no effect until the
|
||||
# app restarts (#981). Only meaningful when _active_instance_id == "mlx-audio".
|
||||
_active_mlx_model_key: "str | None" = None
|
||||
|
||||
|
||||
def reset_active_backend() -> None:
|
||||
"""Unload + clear the cached active backend. For app shutdown and tests.
|
||||
Idempotent and best-effort — a raising unload() never propagates."""
|
||||
global _active_instance, _active_instance_id
|
||||
global _active_instance, _active_instance_id, _active_mlx_model_key
|
||||
inst = _active_instance
|
||||
_active_instance = None
|
||||
_active_instance_id = None
|
||||
_active_mlx_model_key = None
|
||||
if inst is not None:
|
||||
try:
|
||||
inst.unload()
|
||||
@@ -1430,13 +1735,32 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
|
||||
``model=`` (caller already holds a loaded model), we return a fresh view
|
||||
over the shared singleton rather than caching it — but a switch *away from*
|
||||
a different engine still triggers that engine's unload().
|
||||
|
||||
For mlx-audio specifically, the backend id alone doesn't capture *which*
|
||||
curated model is loaded (#981) — so we also track the resolved model key
|
||||
and treat a model-only change as a switch, reusing the exact same
|
||||
unload-and-reconstruct path as an id switch.
|
||||
"""
|
||||
global _active_instance, _active_instance_id
|
||||
global _active_instance, _active_instance_id, _active_mlx_model_key
|
||||
bid = active_backend_id()
|
||||
|
||||
# Switching engines: release the outgoing one first. Best-effort so a bad
|
||||
# unload() can never block the switch.
|
||||
if _active_instance is not None and _active_instance_id != bid:
|
||||
mlx_model_key = None
|
||||
if bid == "mlx-audio":
|
||||
from core import prefs
|
||||
mlx_model_key = prefs.resolve(
|
||||
"mlx_audio_model_id",
|
||||
env="OMNIVOICE_MLX_AUDIO_MODEL",
|
||||
default=MLXAudioBackend.DEFAULT_MODEL_KEY,
|
||||
)
|
||||
|
||||
# Switching engines (or, for mlx-audio, switching curated models): release
|
||||
# the outgoing one first. Best-effort so a bad unload() can never block
|
||||
# the switch.
|
||||
switching = _active_instance is not None and (
|
||||
_active_instance_id != bid
|
||||
or (bid == "mlx-audio" and mlx_model_key != _active_mlx_model_key)
|
||||
)
|
||||
if switching:
|
||||
try:
|
||||
_active_instance.unload()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
@@ -1444,6 +1768,7 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
|
||||
type(_active_instance).__name__, exc)
|
||||
_active_instance = None
|
||||
_active_instance_id = None
|
||||
_active_mlx_model_key = None
|
||||
|
||||
cls = get_backend_class(bid)
|
||||
if cls is OmniVoiceBackend and model is not None:
|
||||
@@ -1455,9 +1780,82 @@ def get_active_tts_backend(*, model=None) -> TTSBackend:
|
||||
if _active_instance is None or _active_instance_id != bid:
|
||||
_active_instance = OmniVoiceBackend(model=model) if cls is OmniVoiceBackend else cls()
|
||||
_active_instance_id = bid
|
||||
_active_mlx_model_key = mlx_model_key
|
||||
return _active_instance
|
||||
|
||||
|
||||
# ── Shared generation-time engine resolution (issue #312 class) ───────────
|
||||
#
|
||||
# dub_generate.py and batch.py used to call services.model_manager.get_model()
|
||||
# directly, hardcoding OmniVoice regardless of the engine selected in
|
||||
# Settings → Engines — a SILENT fallback: pick VoxCPM2, dub anyway with
|
||||
# OmniVoice, no error. This is the single resolution path both routers now
|
||||
# call instead, mirroring generation.py's /generate resolution (engine id →
|
||||
# is_available() → routing gate) plus a voice-cloning capability gate that
|
||||
# /generate doesn't need (OmniVoice's native path always clones).
|
||||
|
||||
|
||||
async def resolve_generation_backend(
|
||||
*, require_cloning: bool = False, cloning_purpose: str = "dubbing",
|
||||
) -> TTSBackend:
|
||||
"""Resolve + validate the active TTS engine for a generation call.
|
||||
|
||||
Returns the live backend instance (:func:`get_active_tts_backend`) —
|
||||
cached, and properly unload()ed on an engine switch. Raises ``ValueError``
|
||||
with an actionable message (never silently falls back to OmniVoice) when:
|
||||
|
||||
* the configured engine id is unknown (bad env var / stale pref),
|
||||
* the engine reports itself unavailable (``is_available()``),
|
||||
* the engine needs an accelerator this host lacks and has no CPU path
|
||||
(``routing_status == "unavailable"``),
|
||||
* ``require_cloning`` is True and the resolved backend can't clone
|
||||
from reference audio (``supports_cloning`` False) — checked on the
|
||||
live *instance*, not the class, so a model-dependent adapter like
|
||||
MLX-Audio (Kokoro vs. CSM) is judged by what's actually loaded.
|
||||
"""
|
||||
engine_id = active_backend_id()
|
||||
try:
|
||||
backend_cls = get_backend_class(engine_id)
|
||||
except ValueError as e:
|
||||
raise ValueError(
|
||||
f"Active TTS engine '{engine_id}' is not a recognized backend ({e}). "
|
||||
"Check Settings → Engines or the OMNIVOICE_TTS_BACKEND env var."
|
||||
) from e
|
||||
|
||||
try:
|
||||
ok, msg = backend_cls.is_available()
|
||||
except Exception as exc: # noqa: BLE001 — surface as an actionable ValueError
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise ValueError(f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}")
|
||||
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing
|
||||
routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if routing["routing_status"] == "unavailable":
|
||||
raise ValueError(routing["routing_reason"])
|
||||
|
||||
_model = None
|
||||
if backend_cls is OmniVoiceBackend:
|
||||
# OmniVoice needs its model pre-loaded before construction: called
|
||||
# from an async context, OmniVoiceBackend._ensure_loaded() refuses to
|
||||
# bootstrap its own event loop (see its docstring) — same reason
|
||||
# generation.py's /generate special-cases this backend.
|
||||
from services.model_manager import get_model
|
||||
_model = await get_model()
|
||||
backend = get_active_tts_backend(model=_model)
|
||||
|
||||
if require_cloning and not getattr(backend, "supports_cloning", True):
|
||||
raise ValueError(
|
||||
f"The active TTS engine '{engine_id}' doesn't support voice cloning, "
|
||||
f"so {cloning_purpose} can't preserve speaker voices. Switch to one "
|
||||
f"of: {', '.join(cloning_capable_engine_ids())} in Settings → "
|
||||
"Engines, or use OmniVoice for this job."
|
||||
)
|
||||
|
||||
return backend
|
||||
|
||||
|
||||
# ── PEP 562 lazy attribute re-export ───────────────────────────────────────
|
||||
#
|
||||
# Allows ``from services.tts_backend import IndexTTS2Backend`` to keep
|
||||
|
||||
@@ -25,7 +25,17 @@ sys.modules["core.config"] = _config
|
||||
|
||||
whisperx = pytest.importorskip("whisperx")
|
||||
|
||||
from services.asr_backend import WhisperXBackend # noqa: E402
|
||||
from services.asr_backend import ( # noqa: E402
|
||||
WhisperXBackend,
|
||||
_is_compute_type_error,
|
||||
)
|
||||
|
||||
# The exact ValueError CTranslate2 raises at model construction on a GPU
|
||||
# without efficient fp16 (older Maxwell/Pascal, GTX 16xx) or a cuDNN mismatch.
|
||||
_FP16_ERR = (
|
||||
"Requested float16 compute type, but the target device or backend do not "
|
||||
"support efficient float16 computation"
|
||||
)
|
||||
|
||||
|
||||
def test_cuda_oom_falls_back_to_cpu(monkeypatch):
|
||||
@@ -52,8 +62,13 @@ def test_cuda_oom_falls_back_to_cpu(monkeypatch):
|
||||
|
||||
|
||||
def test_non_oom_runtime_error_still_raises(monkeypatch):
|
||||
msg = "some other failure"
|
||||
# A generic non-OOM, non-compute-type RuntimeError must still propagate —
|
||||
# the new compute_type fallback must NOT swallow it.
|
||||
assert _is_compute_type_error(msg) is False
|
||||
|
||||
def fake_load_model(name, device, compute_type, **kw):
|
||||
raise RuntimeError("some other failure") # not an OOM → must propagate
|
||||
raise RuntimeError(msg) # not an OOM, not compute-type → must propagate
|
||||
|
||||
monkeypatch.setattr(whisperx, "load_model", fake_load_model)
|
||||
|
||||
@@ -62,3 +77,63 @@ def test_non_oom_runtime_error_still_raises(monkeypatch):
|
||||
be._allow_vad_pickle_globals = lambda: None
|
||||
with pytest.raises(RuntimeError, match="some other failure"):
|
||||
be._ensure_asr()
|
||||
|
||||
|
||||
def test_float16_unsupported_falls_back_to_int8(monkeypatch):
|
||||
"""#551: a GPU without efficient fp16 raises a ValueError at load for both
|
||||
float16 AND int8_float16; the backend must degrade to int8 on the SAME
|
||||
device (cuda) without raising — not fall to CPU and not crash."""
|
||||
calls = []
|
||||
|
||||
def fake_load_model(name, device, compute_type, **kw):
|
||||
calls.append((device, compute_type))
|
||||
if device == "cuda" and compute_type in ("float16", "int8_float16"):
|
||||
raise ValueError(_FP16_ERR)
|
||||
return object() # cuda int8 succeeds
|
||||
|
||||
monkeypatch.setattr(whisperx, "load_model", fake_load_model)
|
||||
|
||||
be = WhisperXBackend()
|
||||
be._device, be._compute_type = "cuda", "float16"
|
||||
be._allow_vad_pickle_globals = lambda: None
|
||||
|
||||
be._ensure_asr()
|
||||
|
||||
assert be._asr is not None # recovered, no raise
|
||||
assert be._device == "cuda" and be._compute_type == "int8" # same device, int8
|
||||
assert calls == [("cuda", "float16"), ("cuda", "int8_float16"), ("cuda", "int8")]
|
||||
|
||||
|
||||
def test_faster_whisper_float16_unsupported_falls_back_to_int8(monkeypatch):
|
||||
"""Mirror for FasterWhisperBackend: float16 + int8_float16 raise the fp16
|
||||
ValueError, int8 succeeds → loads on (cuda, int8) without raising."""
|
||||
import services.asr_backend as asr_backend
|
||||
from services.asr_backend import FasterWhisperBackend
|
||||
|
||||
calls = []
|
||||
|
||||
class FakeWhisperModel:
|
||||
def __init__(self, name, device, compute_type, **kw):
|
||||
calls.append((device, compute_type))
|
||||
if device == "cuda" and compute_type in ("float16", "int8_float16"):
|
||||
raise ValueError(_FP16_ERR)
|
||||
# cuda int8 succeeds
|
||||
|
||||
fake_fw = types.ModuleType("faster_whisper")
|
||||
fake_fw.WhisperModel = FakeWhisperModel
|
||||
monkeypatch.setitem(sys.modules, "faster_whisper", fake_fw)
|
||||
|
||||
# Force the CUDA starting point regardless of the CI host's hardware by
|
||||
# making torch.cuda.is_available() return True inside _ensure_model.
|
||||
fake_torch = types.ModuleType("torch")
|
||||
fake_torch.cuda = types.SimpleNamespace(
|
||||
is_available=lambda: True, empty_cache=lambda: None
|
||||
)
|
||||
monkeypatch.setitem(sys.modules, "torch", fake_torch)
|
||||
|
||||
be = FasterWhisperBackend()
|
||||
be._ensure_model()
|
||||
|
||||
assert be._model is not None # recovered, no raise
|
||||
assert be._device == "cuda" and be._compute_type == "int8"
|
||||
assert calls == [("cuda", "float16"), ("cuda", "int8_float16"), ("cuda", "int8")]
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
"""speechbrain LazyModule cross-platform guard (#630/#611/#647).
|
||||
|
||||
speechbrain 1.x suppresses optional-integration imports (k2_fsa, numba, …) that
|
||||
are triggered merely by introspection from the stdlib `inspect` module. Its
|
||||
guard checked `filename.endswith("/inspect.py")` — a hardcoded POSIX separator —
|
||||
so on Windows (backslash paths) the guard MISSED and a stray access to the
|
||||
`speechbrain.k2_integration` redirect actually imported the (absent) k2 package,
|
||||
raising `ImportError: Lazy import of LazyModule(...k2_fsa...) failed` that aborted
|
||||
WhisperX transcription with zero segments.
|
||||
|
||||
`_harden_speechbrain_lazy_imports()` re-implements `ensure_module` with an
|
||||
`os.path.basename` check so the guard fires on every platform. These tests fake
|
||||
the importer frame (both Windows- and POSIX-style `inspect.py` paths, plus a
|
||||
real-caller path) so they pin the behaviour regardless of the host OS.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
importutils = pytest.importorskip(
|
||||
"speechbrain.utils.importutils",
|
||||
reason="speechbrain not installed in this environment",
|
||||
)
|
||||
from services.asr_backend import _harden_speechbrain_lazy_imports # noqa: E402
|
||||
|
||||
|
||||
class _FakeFrameInfo:
|
||||
def __init__(self, filename):
|
||||
self.filename = filename
|
||||
|
||||
|
||||
def _bogus_lazy_module():
|
||||
# A LazyModule whose target can never import — so we can observe whether the
|
||||
# inspect.py guard fired (AttributeError) or the import was attempted (ImportError).
|
||||
return importutils.LazyModule(
|
||||
"omnivoice_nonexistent_zzz",
|
||||
"omnivoice_nonexistent_zzz_target",
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"inspect_path",
|
||||
[
|
||||
r"C:\Python311\Lib\inspect.py", # Windows — the case the old guard missed
|
||||
"/usr/lib/python3.11/inspect.py", # POSIX — already worked, must keep working
|
||||
],
|
||||
)
|
||||
def test_guard_fires_for_inspect_frame_on_any_separator(monkeypatch, inspect_path):
|
||||
_harden_speechbrain_lazy_imports()
|
||||
lm = _bogus_lazy_module()
|
||||
monkeypatch.setattr(
|
||||
importutils.inspect, "getframeinfo",
|
||||
lambda *_a, **_k: _FakeFrameInfo(inspect_path),
|
||||
)
|
||||
# Guard must treat an inspect.py-triggered access as "attribute absent"
|
||||
# (AttributeError) rather than attempting the doomed import (ImportError).
|
||||
with pytest.raises(AttributeError):
|
||||
lm.ensure_module(0)
|
||||
|
||||
|
||||
def test_real_caller_still_surfaces_import_error(monkeypatch):
|
||||
"""A genuine access from real user code (not inspect.py) with the target
|
||||
missing must still raise ImportError — we only suppress inspect-triggered
|
||||
spurious imports, never legitimate failures."""
|
||||
_harden_speechbrain_lazy_imports()
|
||||
lm = _bogus_lazy_module()
|
||||
monkeypatch.setattr(
|
||||
importutils.inspect, "getframeinfo",
|
||||
lambda *_a, **_k: _FakeFrameInfo(r"C:\Users\me\app\real_caller.py"),
|
||||
)
|
||||
with pytest.raises(ImportError):
|
||||
lm.ensure_module(0)
|
||||
|
||||
|
||||
def test_patch_is_idempotent():
|
||||
_harden_speechbrain_lazy_imports()
|
||||
first = importutils.LazyModule.ensure_module
|
||||
_harden_speechbrain_lazy_imports()
|
||||
assert importutils.LazyModule.ensure_module is first
|
||||
assert getattr(importutils.LazyModule, "_omnivoice_xplat_guard", False) is True
|
||||
@@ -0,0 +1,238 @@
|
||||
"""Whole-file ASR transcribe must be wall-clock bounded (TamKieu / Vietnam report).
|
||||
|
||||
The chunked dub pipeline already bounds each chunk, but the whole-file paths
|
||||
(dub QC re-transcribe, dictation, OpenAI-compat) ran unbounded — a slow/stuck
|
||||
transcribe (e.g. large-v3 on a VRAM-starved GPU) hung the request *and* held a
|
||||
GPU-pool worker, surfacing in the UI as the misleading "can't reach the local
|
||||
backend". `run_transcribe_guarded` bounds them and raises `ASRTimeoutError` with
|
||||
actionable guidance. These tests pin the timeout path, the pass-through path, and
|
||||
that the error message tells the user what to do.
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
from services import asr_backend # noqa: E402
|
||||
from services.asr_backend import ( # noqa: E402
|
||||
ASRTimeoutError,
|
||||
ASR_TRANSCRIBE_TIMEOUT_S,
|
||||
reset_pool_after_wedge,
|
||||
run_transcribe_guarded,
|
||||
)
|
||||
from concurrent.futures import ThreadPoolExecutor # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_timeout_streak(monkeypatch):
|
||||
"""The consecutive-timeout streak (#730 residual B) is process-global
|
||||
session state; zero it per test so ordering can't leak recommendations,
|
||||
and pin the active engine so a dev box's prefs can't flip the hint."""
|
||||
monkeypatch.setattr(asr_backend, "_timeout_streak", 0)
|
||||
monkeypatch.setattr(asr_backend, "active_backend_id", lambda: "whisperx")
|
||||
|
||||
|
||||
def test_default_timeout_is_env_overridable(monkeypatch):
|
||||
# The constant is read at import; just assert it's a sane positive default.
|
||||
assert ASR_TRANSCRIBE_TIMEOUT_S > 0
|
||||
|
||||
|
||||
def test_slow_transcribe_raises_actionable_timeout():
|
||||
pool = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
def _hang():
|
||||
time.sleep(5) # would block far past our tiny timeout
|
||||
return "never"
|
||||
|
||||
async def _go():
|
||||
with pytest.raises(ASRTimeoutError) as ei:
|
||||
await run_transcribe_guarded(pool, _hang, what="QC", timeout=0.2)
|
||||
msg = str(ei.value)
|
||||
# Message must reassure (backend alive) + give concrete remedies.
|
||||
assert "backend is running" in msg
|
||||
assert "Settings → Models" in msg
|
||||
assert "CPU" in msg
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_fast_transcribe_passes_through():
|
||||
pool = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
def _quick():
|
||||
return {"segments": [{"text": "hi"}]}, "whisperx"
|
||||
|
||||
async def _go():
|
||||
out = await run_transcribe_guarded(pool, _quick, what="Dictation", timeout=5.0)
|
||||
assert out == ({"segments": [{"text": "hi"}]}, "whisperx")
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=True)
|
||||
|
||||
|
||||
def test_timeout_error_is_a_timeouterror_subclass():
|
||||
# Routers that catch broad TimeoutError (openai_compat) must also catch ours.
|
||||
assert issubclass(ASRTimeoutError, TimeoutError)
|
||||
|
||||
|
||||
def test_timeout_resets_a_resilient_pool_to_restore_capacity():
|
||||
# #730: a wedged transcribe holds its GPU-pool worker forever; with a 1-2
|
||||
# worker pool that starves TTS generate and surfaces as "can't reach
|
||||
# backend". On timeout, run_transcribe_guarded must reset() a pool that
|
||||
# supports it (the real _ResilientGpuPool) so the next submit gets a fresh
|
||||
# worker — capacity restored without an app restart.
|
||||
class _FakePool(ThreadPoolExecutor):
|
||||
def __init__(self):
|
||||
super().__init__(max_workers=1)
|
||||
self.reset_calls = 0
|
||||
|
||||
def reset(self):
|
||||
self.reset_calls += 1
|
||||
|
||||
pool = _FakePool()
|
||||
|
||||
def _hang():
|
||||
time.sleep(5)
|
||||
return "never"
|
||||
|
||||
async def _go():
|
||||
with pytest.raises(ASRTimeoutError):
|
||||
await run_transcribe_guarded(pool, _hang, what="Dub", timeout=0.2)
|
||||
|
||||
asyncio.run(_go())
|
||||
assert pool.reset_calls == 1
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_timeout_without_reset_capable_pool_does_not_crash():
|
||||
# A plain ThreadPoolExecutor (no reset) must still bound + raise cleanly —
|
||||
# the reset() is best-effort, never required.
|
||||
pool = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
def _hang():
|
||||
time.sleep(5)
|
||||
return "never"
|
||||
|
||||
async def _go():
|
||||
with pytest.raises(ASRTimeoutError):
|
||||
await run_transcribe_guarded(pool, _hang, what="QC", timeout=0.2)
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
# ── Residual B on #730: consecutive timeouts recommend the isolated engine ──
|
||||
|
||||
|
||||
def _hang_forever():
|
||||
time.sleep(5)
|
||||
return "never"
|
||||
|
||||
|
||||
async def _timeout_once(pool, timeout=0.1) -> str:
|
||||
with pytest.raises(ASRTimeoutError) as ei:
|
||||
await run_transcribe_guarded(pool, _hang_forever, what="Dub", timeout=timeout)
|
||||
return str(ei.value)
|
||||
|
||||
|
||||
def test_second_consecutive_timeout_recommends_isolated_engine():
|
||||
"""When guarded timeouts hit twice in a row in one session, pool resets
|
||||
clearly aren't recovering the hang — the error the user sees must name the
|
||||
crash-isolated escape-hatch engine (and make clear we never auto-switch)."""
|
||||
pool = ThreadPoolExecutor(max_workers=2)
|
||||
|
||||
async def _go():
|
||||
first = await _timeout_once(pool)
|
||||
assert "faster-whisper-isolated" not in first # one timeout ≠ a pattern
|
||||
second = await _timeout_once(pool)
|
||||
assert "faster-whisper-isolated" in second
|
||||
assert "Settings → Engines" in second
|
||||
assert "never switches engines automatically" in second
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_successful_transcribe_resets_the_timeout_streak():
|
||||
"""'Consecutive' must mean consecutive: a transcribe that completes between
|
||||
two timeouts proves the pool recovered, so the recommendation must not fire."""
|
||||
pool = ThreadPoolExecutor(max_workers=3)
|
||||
|
||||
async def _go():
|
||||
await _timeout_once(pool)
|
||||
out = await run_transcribe_guarded(pool, lambda: "ok", what="Dub", timeout=5.0)
|
||||
assert out == "ok"
|
||||
second = await _timeout_once(pool)
|
||||
assert "faster-whisper-isolated" not in second
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_no_recommendation_when_already_on_isolated_engine(monkeypatch):
|
||||
"""Recommending the isolated engine to a user already running it is noise —
|
||||
the base message's smaller-model/CPU guidance is all that's left."""
|
||||
monkeypatch.setattr(
|
||||
asr_backend, "active_backend_id", lambda: "faster-whisper-isolated"
|
||||
)
|
||||
pool = ThreadPoolExecutor(max_workers=2)
|
||||
|
||||
async def _go():
|
||||
await _timeout_once(pool)
|
||||
second = await _timeout_once(pool)
|
||||
assert "faster-whisper-isolated) in Settings" not in second
|
||||
assert "never switches engines automatically" not in second
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_timeout_env_name_is_parameterized():
|
||||
"""The chunked dub path passes its own knob; the message must name IT, not
|
||||
the whole-file env var (actionable errors point at the right dial)."""
|
||||
pool = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
async def _go():
|
||||
with pytest.raises(ASRTimeoutError) as ei:
|
||||
await run_transcribe_guarded(
|
||||
pool, _hang_forever, what="Dub chunk 1/3", timeout=0.1,
|
||||
timeout_env="OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S",
|
||||
)
|
||||
msg = str(ei.value)
|
||||
assert "OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S" in msg
|
||||
assert "OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S" not in msg
|
||||
|
||||
asyncio.run(_go())
|
||||
pool.shutdown(wait=False)
|
||||
|
||||
|
||||
def test_reset_pool_after_wedge_is_shared_and_best_effort():
|
||||
"""One reset mechanism for every transcribe path (#730 residual A): it
|
||||
resets a reset-capable pool, no-ops a plain executor, and never raises."""
|
||||
|
||||
class _Pool:
|
||||
resets = 0
|
||||
|
||||
def reset(self):
|
||||
self.resets += 1
|
||||
|
||||
p = _Pool()
|
||||
assert reset_pool_after_wedge(p, what="Dub chunk 1/2") is True
|
||||
assert p.resets == 1
|
||||
|
||||
plain = ThreadPoolExecutor(max_workers=1)
|
||||
try:
|
||||
assert reset_pool_after_wedge(plain) is False
|
||||
finally:
|
||||
plain.shutdown(wait=False)
|
||||
|
||||
class _Broken:
|
||||
def reset(self):
|
||||
raise RuntimeError("reset blew up")
|
||||
|
||||
assert reset_pool_after_wedge(_Broken()) is False # must not raise
|
||||
Regular → Executable
@@ -0,0 +1,352 @@
|
||||
# Migration — Per-Component `.css` → Tailwind v4 Utilities
|
||||
|
||||
**Status:** Plan (not yet executed) · **Drafted:** 2026-06-30 · **Type:** Incremental styling migration, no intended visual change
|
||||
**Owner stance:** leans toward a full migration but values not breaking the UI · **This plan's recommendation:** *bounded* migration (utilities for layout/spacing/typography everywhere; keep CSS for the hard stuff). See §8.
|
||||
|
||||
## Why
|
||||
|
||||
`frontend/src` carries **74 `.css` files / 16,615 lines** of global, BEM-ish CSS
|
||||
(`dub-col`, `models-row__role`, `readiness-checklist__title`, …). Tailwind v4 is
|
||||
**already wired** — `src/index.css` imports `tailwindcss/theme.css` +
|
||||
`tailwindcss/utilities.css`, has an `@theme` block, and `vite.config.js` runs
|
||||
`@tailwindcss/vite`. So the runtime cost of utilities is already paid; we are just
|
||||
not using them. Editing a layout today means hunting a class across a 989-line
|
||||
file and a JSX `className`. Utilities put the layout where it's read — in the JSX —
|
||||
and shrink the per-component CSS to only what utilities can't express.
|
||||
|
||||
This is **not** a redesign. Every step must render pixel-identical. The honest
|
||||
blocker is that **there are zero visual-regression tests** — the prior page
|
||||
refactors (see `docs/maintenance-pages-modularization.md`) verified "no change"
|
||||
by diffing `className` strings, and *that trick is useless here because the whole
|
||||
point is that class names change*. Closing that gap is the first real task (§4),
|
||||
not an afterthought.
|
||||
|
||||
## Current state (measured 2026-06-30)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|------:|
|
||||
| `.css` files | 74 |
|
||||
| Total CSS lines | 16,615 |
|
||||
| `var(--…)` token references across CSS | ~3,200 |
|
||||
| Files using `display:flex` | 64 |
|
||||
| Files using `display:grid` / `grid-template` | 27 |
|
||||
| Files using `transition:` | 46 |
|
||||
| Files using `box-shadow` | 43 |
|
||||
| Files using `@media` | 25 |
|
||||
| Files using `linear/radial-gradient` | 22 |
|
||||
| Files using `@keyframes` (73 blocks total) | 30 |
|
||||
| Files using `animation:` | 34 |
|
||||
| Files using `backdrop-filter`/glass blur | 11 |
|
||||
| Files using `::before`/`::after` | 11 |
|
||||
| Files using `:has()` | 3 |
|
||||
| Files using `!important` | 14 |
|
||||
|
||||
Biggest files (conversion ROI ranked by layout density, not raw size):
|
||||
`index.css` 2532 · `FirstRunSetup.css` 1020 · `DubTab.css` 989 ·
|
||||
`VoiceGallery.css` 541 · `StoriesEditor.css` 525 · `LogsFooter.css` 507 ·
|
||||
`Settings.css` 469 · `CloneDesignTab.css` 458 · `settings/primitives/primitives.css` 368.
|
||||
|
||||
The token system (do **not** redesign it):
|
||||
|
||||
- `src/ui/tokens.css` (157 lines, ~82 custom props): the declared "single source of
|
||||
truth" — colors, a 4px spacing scale (`--space-0..9`), radius, fonts, type scale,
|
||||
weights, shadows, motion, z-index, focus ring, glass blur. Imported via `src/ui/index.js`.
|
||||
- `src/ui/themes.css` (188 lines): per-theme overrides of the semantic color tokens,
|
||||
keyed on `[data-theme="midnight|nord|solarized|…"]` on `<html>`. Default (no attribute)
|
||||
= Gruvbox Dark.
|
||||
- `src/index.css` `@theme { … }`: maps a subset of tokens into Tailwind's theme
|
||||
namespace (`--color-*`, `--radius-*`, `--font-*`) so utilities like `bg-bg`,
|
||||
`text-fg`, `rounded-lg`, `font-mono` exist. **It hardcodes hex literals that
|
||||
duplicate `tokens.css`** — the known drift bug (see §2).
|
||||
|
||||
Load order today: `index.css` (`@theme` → `theme` layer, lowest priority) is
|
||||
imported in `App.jsx`; `tokens.css` + `themes.css` are **unlayered** `:root` /
|
||||
`[data-theme]` rules imported via `ui/index.js`. Because unlayered CSS outranks
|
||||
`@layer theme`, **`tokens.css` already wins for the default values and theming
|
||||
already works** — the `@theme` hex literals are effectively a *losing duplicate*
|
||||
that exists only so Tailwind knows the utility names. That is precisely why they
|
||||
drift silently: nothing at runtime reads them, so a stale value never shows up.
|
||||
|
||||
## Strategy (the shape of the whole thing)
|
||||
|
||||
1. **Incremental, component-by-component — never big-bang.** One component (or one
|
||||
small cluster) per PR. Each PR is independently shippable and CI-green. A
|
||||
half-migrated component is fine; a half-migrated *codebase* is the steady state
|
||||
for months and that's acceptable.
|
||||
2. **Utilities-first for the mechanical 80%:** flexbox, grid, gap, padding/margin,
|
||||
width/height, `text-*`/`font-*`, `rounded-*`, `border`, simple `bg-*`/`text-*`
|
||||
color, `hidden`, `truncate`, basic `hover:`/`focus:` color states. These map 1:1
|
||||
to utilities and are where the line-count win lives.
|
||||
3. **Keep `.css` for the hard 20%:** glassmorphism (layered gradients +
|
||||
`backdrop-filter`), `::before`/`::after`, `@keyframes`, `:has()` and other complex
|
||||
combinators, `[data-theme]`-specific rules, and anything with `!important`
|
||||
fighting specificity. Utilities don't express these cleanly and forcing them
|
||||
(arbitrary-value soup, `[&::before]:…`) trades readable CSS for unreadable JSX.
|
||||
4. **One source of truth via the token bridge (§2):** utilities reference the same
|
||||
CSS vars the remaining `.css` reads, so a value lives in exactly one place and
|
||||
`data-theme` switching keeps working for both.
|
||||
5. **No file is "done" until it's deleted or demonstrably minimal.** Success is
|
||||
measured in CSS LOC removed and `.css` files deleted, not files "touched."
|
||||
|
||||
## 2. Token-bridge prerequisite (P0 — gates everything)
|
||||
|
||||
The migration is only safe if a utility and the leftover CSS in the same component
|
||||
resolve a token to the *same* value, including after a theme switch. Today the
|
||||
`@theme` literals duplicate `tokens.css`; once components start mixing `bg-bg`
|
||||
(utility) with `background: var(--color-bg)` (CSS), any drift becomes a visible,
|
||||
theme-dependent bug. Fix the source-of-truth **before** converting anything.
|
||||
|
||||
**Recommended fix — Solution A (lowest churn, no rename):** Make `@theme` the
|
||||
single declared home for the **already-overlapping** groups only — colors, radius,
|
||||
fonts — and **delete those default declarations from `tokens.css`** (leave a
|
||||
one-line pointer comment). Everything else in `tokens.css` (spacing, type scale,
|
||||
weights, shadows, motion, z-index, focus ring, glass blur) stays put.
|
||||
|
||||
Why this is correct and safe:
|
||||
|
||||
- Tailwind needs the keys present in `@theme` to generate the utility names
|
||||
(`--color-fg` → `text-fg`/`bg-fg`; `--radius-lg` → `rounded-lg`; `--font-mono` →
|
||||
`font-mono`). Keeping the keys there is non-negotiable.
|
||||
- `@theme` emits `:root { --color-fg: … }` into the low-priority `theme` layer.
|
||||
`themes.css` `[data-theme]` rules are unlayered and still outrank it, so
|
||||
**theme switching is unchanged** — verify with a quick manual cycle through all
|
||||
themes after the edit.
|
||||
- Removing the duplicate `:root` color/radius/font lines from `tokens.css` leaves
|
||||
exactly one literal per value. All ~3,200 existing `var(--…)` references keep
|
||||
resolving (the var still exists on `:root`, now sourced from `@theme`).
|
||||
|
||||
**Guard against recurrence (required, per the "fix the class" rule):** add
|
||||
`frontend/src/__tests__/theme-token-parity.test.js` (vitest, no browser) that
|
||||
parses `index.css` `@theme` + `tokens.css` + `themes.css` and asserts:
|
||||
(a) no token key is declared with a literal in **both** `@theme` and `tokens.css`
|
||||
(catches re-introduced duplication), and (b) every `@theme` color key is overridden
|
||||
by every `[data-theme]` block in `themes.css` (catches a theme that forgot a color).
|
||||
This test is the thing that makes the de-dup *stay* de-duped.
|
||||
|
||||
**Rejected alternative — Solution B (purist):** rename source tokens to a private
|
||||
namespace (`--ov-color-fg`) and bridge with `@theme inline { --color-fg:
|
||||
var(--ov-color-fg) }`. This honors "`tokens.css` is the source" literally and is
|
||||
the textbook Tailwind pattern, **but** it forces renaming all ~3,200 `var(--color-*)`
|
||||
references across 74 files in one shot — a massive, high-risk diff that violates
|
||||
"low-risk, incremental." Not worth it. (`@theme inline` referencing the *same* name
|
||||
is circular and is not an option.)
|
||||
|
||||
**Optionally, later:** add `--spacing` to `@theme` so `p-*`/`gap-*`/`m-*` map onto
|
||||
the existing 4px scale (`--space-1 = 2px` … `--space-9 = 44px`). Tailwind's default
|
||||
spacing is a 0.25rem multiplier; OmniVoice's scale is custom, so without this,
|
||||
`gap-3` ≠ `var(--space-3)`. Two choices, decide in P0:
|
||||
- **Map to the scale:** set `--spacing: 2px` won't reproduce the non-linear steps;
|
||||
instead define explicit `--spacing-1..9` in `@theme` mirroring `--space-1..9`,
|
||||
and use `gap-2`/`p-5` etc. Cleanest for readers, but utility numbers won't match
|
||||
Tailwind defaults — document it.
|
||||
- **Use arbitrary values bridged to the var:** `gap-[var(--space-3)]`,
|
||||
`p-[var(--space-5)]`. Zero ambiguity, slightly noisier JSX, guarantees identical
|
||||
pixels. **Recommended for P1–P2** (safest for "no visual change"); revisit named
|
||||
spacing once confidence is high.
|
||||
|
||||
## 3. What converts cleanly vs. what stays CSS
|
||||
|
||||
**Converts cleanly → utilities** (concrete, from real files):
|
||||
|
||||
- `ReadinessChecklist.css` `.readiness-checklist { display:flex; flex-direction:column;
|
||||
gap:var(--space-3); padding:var(--space-5); border:1px solid var(--color-border);
|
||||
border-radius:var(--radius-lg); font-size:var(--text-sm); }`
|
||||
→ `className="flex flex-col gap-[var(--space-3)] p-[var(--space-5)] border
|
||||
border-border rounded-lg text-sm"` (or mapped `text-sm` if the type scale is
|
||||
bridged). The `backdrop-filter` line on the same selector **stays in CSS** (see below).
|
||||
- `.readiness-checklist__title { font-weight:var(--weight-semibold);
|
||||
color:var(--color-fg); display:flex; align-items:center; gap:var(--space-3); }`
|
||||
→ `font-semibold text-fg flex items-center gap-[var(--space-3)]`.
|
||||
- Generic layout rows/cols (`dub-col`, `models-row`) — flex/grid/gap/padding → utilities.
|
||||
|
||||
**Stays in `.css`** (criteria + real examples):
|
||||
|
||||
- **Glassmorphism / layered backgrounds.** `Panel.css` `.ui-panel--glass` stacks two
|
||||
`radial-gradient`s + a `linear-gradient` + `backdrop-filter: var(--glass-blur-md)`.
|
||||
Leave entirely in CSS. (11 files use glass blur.)
|
||||
- **Pseudo-elements.** `Panel.css` `.ui-panel--glass::before` (top hairline gradient);
|
||||
`DubTab.css` `.dub-stepper__step::before` (connector line). 11 files. Stay.
|
||||
- **Keyframes + animations.** 73 `@keyframes` blocks across 30 files
|
||||
(`@keyframes mesh/spin/pulse/shimmer` in `index.css`; `dub-pulse`,
|
||||
`dub-stepper-spin`, `dub-skel-shimmer` in `DubTab.css`). Keep the `@keyframes` and
|
||||
the `animation:` shorthand in CSS; a `className="animate-…"` only helps if you
|
||||
register the animation in `@theme`, which isn't worth it for one-off effects.
|
||||
- **`:has()` and complex combinators** (3 files), **`[data-theme]`-specific rules**
|
||||
(all of `themes.css` + scattered overrides), **`!important` blocks** (14 files,
|
||||
e.g. `DubTab.css` `.dub-footer-panel::before { display:none !important; }`).
|
||||
- **Media queries** (25 files): convertible to `sm:`/`md:`/`lg:` **only** if the
|
||||
breakpoints match Tailwind's; OmniVoice's are custom, so leave responsive blocks in
|
||||
CSS unless a component's breakpoints are first added to `@theme`. Low priority.
|
||||
|
||||
Rule of thumb for a reviewer: *if a declaration reads a single token and sets one
|
||||
box/text/flex property, it's a utility; if it composes multiple values, targets a
|
||||
pseudo-element/state combinator, or animates, it stays.*
|
||||
|
||||
## 4. Risk mitigation — the no-visual-test gap (the gating risk)
|
||||
|
||||
This is the make-or-break item. Be honest: **without a visual baseline, "no change"
|
||||
is unverifiable**, and `className`-diffing (what the page refactors relied on) cannot
|
||||
work when class names are the thing changing. Two layers, do both:
|
||||
|
||||
**(a) Establish a screenshot baseline before touching components (part of P0).**
|
||||
Add Playwright component/page screenshots for the surfaces being migrated. The repo
|
||||
already references Playwright tooling in its docs stack; wire a minimal
|
||||
`tests/visual/` that boots the Vite app (or Storybook-less direct route renders) and
|
||||
captures per-component PNGs at a fixed viewport for **the default theme + one dark +
|
||||
one light theme** (catches token-bridge regressions specifically). Commit baselines.
|
||||
Each migration PR runs `playwright test --update-snapshots=none` and **fails on any
|
||||
pixel diff above a tiny threshold**. This converts "did it change?" from a human
|
||||
guess into a CI gate. Capture baselines *first*, on `main`, so they reflect
|
||||
pre-migration truth.
|
||||
|
||||
- Scope realistically: snapshotting all 74 surfaces up front is its own project.
|
||||
Snapshot **per phase, just-in-time** — before P1 leaf work, baseline the leaf
|
||||
components; before P3, baseline the big pages. Baselines for a component land in
|
||||
the same PR that prepares to migrate it (separate from the conversion PR so the
|
||||
baseline diff is reviewable on its own).
|
||||
|
||||
**(b) A per-component manual checklist** (belt-and-suspenders, and the fallback for
|
||||
surfaces that are hard to screenshot deterministically — anything with animation,
|
||||
canvas/waveform, or live backend data):
|
||||
1. Default theme: side-by-side before/after at the same viewport.
|
||||
2. Cycle every `[data-theme]` — confirm colors still swap (token-bridge check).
|
||||
3. Hover/focus/active/disabled states on interactive elements.
|
||||
4. The component's `@keyframes`/animation still runs.
|
||||
5. `prefers-reduced-motion` path unaffected (e.g. `#root` launch animation).
|
||||
6. No console warnings; `bun run build` + `bun run lint` clean.
|
||||
|
||||
If neither (a) nor (b) is in place for a surface, **do not migrate it** — defer it to
|
||||
the "leave as CSS" bucket rather than fly blind.
|
||||
|
||||
## 5. Phasing
|
||||
|
||||
Each phase = one or more independently shippable, CI-green PRs. Ordered
|
||||
leaf-inward so blast radius grows only as confidence does.
|
||||
|
||||
### P0 — Token bridge + tooling + visual baseline (no component conversions)
|
||||
- De-dup `@theme` ↔ `tokens.css` (§2 Solution A) + the parity test.
|
||||
- Decide + document the spacing approach (arbitrary-value bridge recommended).
|
||||
- Add `prettier-plugin-tailwindcss` (or confirm oxlint/oxfmt class-sort) and wire
|
||||
class sorting (§6).
|
||||
- Update `CONTRIBUTING.md` (§6 — currently says *"Vanilla CSS … no Tailwind"*, which
|
||||
now contradicts reality and **must** change in this same PR per the docs-sync rule).
|
||||
- Stand up `tests/visual/` Playwright harness (no per-component baselines yet — just
|
||||
the runner + theme matrix).
|
||||
- **Effort:** ~1–2 days. **Success:** parity test green; theme switch verified across
|
||||
all themes; CI gains a class-sort check; zero pixels changed (this PR ships no
|
||||
component edits).
|
||||
|
||||
### P1 — Leaf / presentational components (lowest risk)
|
||||
Targets: small `ui/` primitives and stateless components where CSS is mostly
|
||||
flex/grid/spacing/type — e.g. `Badge`, `UpdateStatusChip`, `NetworkToggle`,
|
||||
`ReadinessChecklist`, `ReadinessChecklist`, `DemoPresetGrid`, `KeyboardCheatsheet`,
|
||||
`MultiLangPicker`. Skip glass-heavy ones for now.
|
||||
- Per component: baseline screenshot PR → conversion PR. Convert layout/spacing/type
|
||||
to utilities; keep any glass/`::before`/animation lines in a now-tiny `.css`; delete
|
||||
the `.css` entirely if nothing remains and remove its import.
|
||||
- **Effort:** ~3–5 days across ~10–15 components. **Success:** ~10 `.css` files deleted
|
||||
or reduced >70%; visual diffs clean; a repeatable per-component recipe proven.
|
||||
|
||||
### P2 — Panels & mid-size components
|
||||
Targets: `settings/*Panel.css`, `Sidebar`, `NotificationPanel`, `CastingView`,
|
||||
`ExportModal`, `EngineCompatibilityMatrix`, `donate/Postcard`, etc. More state,
|
||||
some glass — convert the layout skeleton, leave glass/pseudo/animation.
|
||||
- **Effort:** ~1–1.5 weeks. **Success:** settings panels are thin utility JSX + a
|
||||
shared `primitives.css` for the glass/control look; CSS LOC down materially.
|
||||
|
||||
### P3 — Big pages
|
||||
Targets in ROI order: `DubTab` (989), `VoiceGallery` (541), `StoriesEditor` (525),
|
||||
`LogsFooter` (507), `Settings` (469), `CloneDesignTab` (458), `FirstRunSetup` (1020).
|
||||
These pair naturally with the already-planned page modularization
|
||||
(`docs/maintenance-pages-modularization.md`) — **sequence the modularization first**,
|
||||
then migrate the smaller extracted components (P3 becomes "P1 again" on the pieces).
|
||||
Convert layout/spacing; the pipeline steppers, overlays, gradients, and keyframes
|
||||
stay as CSS.
|
||||
- **Effort:** ~2–3 weeks. **Success:** each page's `.css` drops to the
|
||||
glass/animation/pseudo residue; biggest single LOC reductions land here.
|
||||
|
||||
### P4 — Retire `index.css` globals last
|
||||
`index.css` (2532 lines) is foundation: `@theme`, `@keyframes`, `::selection`, root
|
||||
rendering, base resets, and shared global classes. Convert only the **global utility
|
||||
classes** that components reuse into real utilities or component-scoped CSS; **keep**
|
||||
the `@theme`, keyframes, resets, and `::selection`. Do this last because everything
|
||||
depends on it.
|
||||
- **Effort:** ~1 week. **Success:** `index.css` shrinks to foundation only; no
|
||||
orphaned global classes.
|
||||
|
||||
## 6. Tooling
|
||||
|
||||
- **Class sorting / formatting.** The repo lints with **oxlint** (`bun run lint`,
|
||||
gate) and an advisory ESLint for hooks. For Tailwind class ordering, add
|
||||
**`prettier-plugin-tailwindcss`** (canonical, understands `@theme`) wired to run on
|
||||
`*.jsx`, *or* adopt oxfmt's Tailwind class-sorting if the team prefers a single
|
||||
formatter. Either way the goal is deterministic class order so diffs stay readable
|
||||
and merge-clean.
|
||||
- **Regression prevention.** Add an oxlint/convention guard so new components don't
|
||||
reintroduce sprawling CSS: a soft rule (warn-only first, per "keep main green") that
|
||||
flags new `.css` files over a small line budget for components that should be
|
||||
utility-first, and the §2 parity test as a hard gate on token drift.
|
||||
- **CONTRIBUTING update (required).** `CONTRIBUTING.md` currently states *"CSS:
|
||||
Vanilla CSS in component-level files — no Tailwind."* That is now false. Replace it
|
||||
with the utilities-first standard: *layout/spacing/typography/simple color via
|
||||
Tailwind utilities; component `.css` only for glass, pseudo-elements, keyframes,
|
||||
`:has()`, `[data-theme]` rules, and `!important` overrides; tokens live in
|
||||
`tokens.css`/`@theme`, never hardcoded.* Per the docs-sync hard rule this lands in
|
||||
the **same PR** as P0.
|
||||
- **No new build infra** — `@tailwindcss/vite` already does everything; no PostCSS
|
||||
config, no Tailwind config file (v4 is CSS-first via `@theme`).
|
||||
|
||||
## 7. Non-goals / when to stop
|
||||
|
||||
- **No 100% conversion target.** ~20% of the CSS (the 11 glass files, 30 keyframe
|
||||
files, 11 pseudo-element files, 3 `:has()` files, 14 `!important` files, custom-
|
||||
breakpoint media queries) is **genuinely better as CSS** and should stay. Forcing it
|
||||
into arbitrary-value utilities makes JSX unreadable for zero benefit.
|
||||
- **No token-system redesign.** `tokens.css`/`themes.css` and the `data-theme` model
|
||||
stay as-is (only the §2 de-dup).
|
||||
- **No visual redesign.** Pixel-identical is the contract; restyling is a separate task.
|
||||
- **No `.jsx` → `.tsx`**, no engine/backend/Tauri/Python surface, no version bump,
|
||||
no dependency change beyond the dev-only formatter plugin + Playwright (frontend-only).
|
||||
- **Stop conditions for an individual file:** if after pulling out layout/spacing the
|
||||
remaining CSS is all glass/animation/pseudo, it's *done* — don't chase the last 10%.
|
||||
- **Hands off** `BootstrapSplash.css`, `WaveformPlayer.css`/`SegmentTrack.css`
|
||||
(canvas-adjacent), and other animation/`::before`-dominated files unless a clear
|
||||
layout win exists.
|
||||
|
||||
## 8. Effort + recommendation
|
||||
|
||||
**Total rough effort:** ~5–7 focused weeks for P0–P4 at the *bounded* scope below,
|
||||
spread across many small PRs (it parallelizes and pauses cleanly — it never has to be
|
||||
one big push).
|
||||
|
||||
**Recommendation — bounded migration, not 100%.** The owner leans full-migration and
|
||||
prizes not breaking things; those two goals partly conflict, and the honest call is:
|
||||
|
||||
- **Do** convert layout/spacing/typography/simple color **everywhere** — that's the
|
||||
real maintainability win, it's where ~80% of the 16.6k lines live, and it's the
|
||||
low-risk part.
|
||||
- **Keep ~15–25% as CSS** (glass, keyframes, pseudo-elements, `:has()`,
|
||||
`[data-theme]`, `!important`, custom-breakpoint media). Converting these buys
|
||||
unreadable JSX and *raises* visual-regression risk on exactly the components where
|
||||
diffs are hardest to verify.
|
||||
- **Gate on the visual baseline (§4).** This is the single most important decision: if
|
||||
the Playwright screenshot harness doesn't ship in P0, do **not** start P1 — without
|
||||
it the "won't break the UI" requirement is unmet by construction. The token-bridge
|
||||
de-dup (§2) is the other hard prerequisite; both are cheap and both are P0.
|
||||
|
||||
A realistic end state: ~60 `.css` files deleted or reduced >70%, perhaps ~10–12k of
|
||||
the 16.6k CSS lines removed, the rest a deliberate, documented residue of effects
|
||||
utilities can't express. That delivers nearly all the maintainability benefit of a
|
||||
"full" migration at a fraction of the regression risk.
|
||||
|
||||
## Constraints honored
|
||||
|
||||
- **Keep main green** — every phase is an independently CI-green PR; lint/format and
|
||||
parity-test guards are warn-first where they'd otherwise churn.
|
||||
- **Docs-sync** — the `CONTRIBUTING.md` rewrite lands in the same PR as P0.
|
||||
- **No versioning/Docker/Tauri/Python impact** — frontend-only; dev-dependency-only
|
||||
tooling additions; no `package.json` *version* bump (a devDependency add still
|
||||
requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile`
|
||||
per the Docker-green rule).
|
||||
- **Local-first / cross-platform parity** — pure styling; no behavior, no platform
|
||||
divergence.
|
||||
@@ -14,6 +14,15 @@ download UI couldn't show real bytes/speed. Until a proper Xet progress hook
|
||||
lands, the app forces the **classic LFS path**, which streams through the
|
||||
standard progress reporter and gives accurate downloaded/remaining/speed.
|
||||
|
||||
To keep that path **fast** despite Xet being off, the app runs a built-in
|
||||
**multi-connection (segmented) downloader on by default** — it fetches each file
|
||||
over parallel byte-ranges (IDM/uGet style), so the legacy-LFS path is no longer
|
||||
single-stream. It reports real live speed/ETA and **falls back to the normal
|
||||
download on any error**, so it can never compromise a correct install. Adding a
|
||||
free Hugging Face token (first-run setup, or Settings → Credentials) makes this
|
||||
faster still — authenticated downloads get higher rate limits and fewer stalls.
|
||||
To force the old single-stream path, set `OMNIVOICE_SEGMENTED_DOWNLOAD=0`.
|
||||
|
||||
State is reported at **Settings → About** / `GET /system/info`:
|
||||
|
||||
- `fast_download.xet_installed` — `hf_xet` present (true)
|
||||
@@ -54,12 +63,13 @@ When a download starts you'll see, in order:
|
||||
|
||||
## Advanced / opt-in tuning
|
||||
|
||||
All of these default **off** and apply to every platform identically. Set them
|
||||
as environment variables (or via **Settings → API keys / environment**).
|
||||
These apply to every platform identically. Set them as environment variables (or
|
||||
via **Settings → API keys / environment**). The segmented accelerator is **on by
|
||||
default** (set its var to `0` to disable); the rest default **off**.
|
||||
|
||||
| Setting | Env var | Effect |
|
||||
|---|---|---|
|
||||
| Segmented accelerator | `OMNIVOICE_SEGMENTED_DOWNLOAD=1` | Multi-connection downloader (parallel byte-ranges) for the legacy-LFS path — restores parallel speed **and** shows live byte speed/ETA. Falls back to the normal download on any error; files land in the standard cache. Best paired with Xet disabled (the default). |
|
||||
| Segmented accelerator | `OMNIVOICE_SEGMENTED_DOWNLOAD=0` | **On by default** (see above). Set to `0` to force the old single-stream legacy-LFS download instead of the parallel byte-range one. |
|
||||
| Max parallel files | `OMNIVOICE_DOWNLOAD_MAX_WORKERS` (default 8) | Files fetched at once. Xet already parallelises *within* a file, so raising this rarely helps and uses more memory. |
|
||||
| High-performance mode | `HF_XET_HIGH_PERFORMANCE=1` | Maximum throughput. Needs lots of RAM and bandwidth — can **hurt** low-RAM machines. Leave off unless you have headroom. |
|
||||
| Spinning-disk (HDD) | `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` | Sequential writes; avoids parallel-write thrash on HDDs. Leave off on SSD/NVMe. |
|
||||
@@ -72,8 +82,12 @@ If `huggingface.co` is slow or blocked, point the client at a mirror:
|
||||
HF_ENDPOINT=https://hf-mirror.com
|
||||
```
|
||||
|
||||
Set it as an environment variable (or in **Settings → environment**) before
|
||||
downloading. Caveats:
|
||||
Set it in **Settings → Models → Hugging Face mirror** (quick-pick presets
|
||||
included), or as an environment variable before launching. On first run, the
|
||||
setup wizard offers the same mirror quick-pick right on the system-check
|
||||
screen when the endpoint is unreachable — the network check is a warning, not
|
||||
a blocker, so an offline or firewalled machine can still finish setup once
|
||||
models are available (mirror, or manual download below). Caveats:
|
||||
|
||||
- A mirror serves the **classic** download path, **not Xet** — you lose
|
||||
chunk-dedup and Xet's parallel fetch, but you gain reachability. On the
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# Translation engines (Dub tab)
|
||||
|
||||
OmniVoice dubs in two steps: **transcribe → translate → speak**. The *translate*
|
||||
step is pluggable — pick the engine in the Dub tab's **Engine** dropdown. Two
|
||||
engines are **built in** and always available offline; the rest need a small
|
||||
optional Python package.
|
||||
|
||||
| Engine | Category | Needs a package? | Key needed? |
|
||||
|--------|----------|------------------|-------------|
|
||||
| **Argos** (Local, Fast) | offline | `argostranslate` (bundled) | no |
|
||||
| **NLLB-200** (Local, Heavy) | offline | none (uses core `transformers`) | no |
|
||||
| Google Translate (Free) | online | `deep_translator` | no |
|
||||
| DeepL | online | `deep_translator` | yes (`DEEPL_API_KEY`) |
|
||||
| Microsoft Translator | online | `deep_translator` | yes (`MICROSOFT_API_KEY`) |
|
||||
| MyMemory | online | `deep_translator` | no |
|
||||
| LLM (OpenAI-compatible) | llm | `openai` | usually yes |
|
||||
|
||||
If you pick an engine whose package isn't importable yet, the Engine label shows
|
||||
a **highlighted Install affordance**, and — if you try to translate anyway — the
|
||||
backend returns a single, actionable error telling you exactly what to install
|
||||
(the install command is single-sourced, so the button and the error never
|
||||
disagree).
|
||||
|
||||
## Installing optional translation engines (from-source vs packaged build)
|
||||
|
||||
How you add an engine depends on **how you installed OmniVoice**.
|
||||
|
||||
### From-source / dev install (one-click)
|
||||
|
||||
If you cloned the repo and run OmniVoice from source (`uv sync` + the dev
|
||||
launcher) or via Docker, the app can install engines for you:
|
||||
|
||||
1. In the Dub tab, open the translation settings and pick the engine you want
|
||||
(e.g. **Google Translate**) from the **Engine** dropdown.
|
||||
2. A highlighted **Install** button appears next to the *Engine* label. Click it.
|
||||
3. OmniVoice runs the install into the **same** Python environment the backend
|
||||
is using (`uv pip install <package> --python <backend-interpreter>`), then
|
||||
re-probes. When it reports *"restart the backend to load it"*, restart so the
|
||||
freshly-installed module is importable.
|
||||
|
||||
You can also install by hand into the backend venv:
|
||||
|
||||
```
|
||||
uv pip install deep_translator # Google / DeepL / Microsoft / MyMemory
|
||||
uv pip install argostranslate # Argos (already bundled; rarely needed)
|
||||
uv pip install openai # LLM (OpenAI-compatible) provider
|
||||
```
|
||||
|
||||
Then restart the backend.
|
||||
|
||||
### Packaged / installer build (read-only — use the popover)
|
||||
|
||||
The signed desktop installers (`.dmg`, `.msi`, AppImage, `.deb`) ship a
|
||||
**read-only, code-signed Python environment**. Installing extra packages into it
|
||||
would break the signature, so **in-app install is intentionally disabled** on
|
||||
these builds. Selecting an uninstalled engine there shows a highlighted button
|
||||
that opens a small popover with everything you need:
|
||||
|
||||
- **The exact command** to run (with a copy-to-clipboard button) if you *do*
|
||||
have a from-source checkout somewhere and want the online engines there.
|
||||
- **Switch to Argos (bundled, offline)** — one click. Argos and NLLB are always
|
||||
importable in every build, so this is the guaranteed escape hatch: you can
|
||||
keep dubbing immediately, fully offline, no install required.
|
||||
- A link back to this page.
|
||||
|
||||
**Recommendation for packaged builds:** just use **Argos** (fast, offline) or
|
||||
**NLLB-200** (heavier, higher quality, offline). They need nothing installed and
|
||||
never leave your machine. Reach for the online engines only from a from-source
|
||||
install where you can add their package.
|
||||
|
||||
## Translation quality: Fast, Autofit, Cinematic
|
||||
|
||||
The **Quality** control in the Dub tab (and Settings → Translation) picks how the
|
||||
translation is produced:
|
||||
|
||||
- **Fast** — a direct one-shot translation from the selected engine (Argos, NLLB,
|
||||
Google, …). No LLM, no timing awareness.
|
||||
- **Cinematic** — an LLM refines the literal translation (reflect → adapt) for
|
||||
natural, in-context phrasing.
|
||||
- **Autofit** — Cinematic **plus** a strict fit-to-time pass: the LLM rewrites
|
||||
each line so its target-language reading time fits **within** the segment's
|
||||
slot (never overruns it). This keeps the video timing intact and avoids the
|
||||
stressed audio time-stretch you get when a translation is too long for its
|
||||
slot. Fit is per-language pronunciation-speed aware.
|
||||
|
||||
Cinematic and Autofit **require an LLM** (below). If none is configured, they
|
||||
fall back to Fast with a notice.
|
||||
|
||||
## LLM Providers (for Cinematic / Autofit)
|
||||
|
||||
**Settings → System → LLM Providers** is the one place to set up the LLM. Pick a
|
||||
provider, paste its API key, choose a model, **Test** it, and "use for
|
||||
translation." Supported: OpenAI, OpenRouter, Groq, Cerebras, Google AI (Gemini),
|
||||
Mistral, Cohere, NVIDIA, GitHub Models, Cloudflare, Hugging Face, SambaNova,
|
||||
SiliconFlow, **local Ollama / LM Studio** (offline, no key), and a **Custom**
|
||||
OpenAI-compatible endpoint.
|
||||
|
||||
Keys entered here are stored **encrypted** on your machine and never returned to
|
||||
the UI. For a fully offline setup, pick **Ollama** (`ollama pull llama3.1`) or
|
||||
**LM Studio** — nothing leaves the machine. Power users can still override any
|
||||
provider via environment variables (e.g. `GROQ_API_KEY`, or the legacy
|
||||
`TRANSLATE_BASE_URL` / `TRANSLATE_API_KEY` / `TRANSLATE_MODEL`, which map to the
|
||||
**Custom** provider).
|
||||
|
||||
### Pinning the active provider with `LLM_DEFAULT_PROVIDER`
|
||||
|
||||
By default the LLM used for Cinematic/Autofit is the one you mark "use for
|
||||
translation" in **Settings → LLM Providers**. To force a specific provider
|
||||
regardless of that stored selection — handy for headless/CI/Docker runs or a
|
||||
shared machine — set the `LLM_DEFAULT_PROVIDER` environment variable to a
|
||||
provider id before launching the backend:
|
||||
|
||||
```
|
||||
LLM_DEFAULT_PROVIDER=groq # or openai, openrouter, cerebras, ollama, custom, …
|
||||
```
|
||||
|
||||
Resolution order for the active provider is: `LLM_DEFAULT_PROVIDER` (env) →
|
||||
your saved selection → the first provider that has a key → none. The id must be
|
||||
one OmniVoice knows (the ids shown in **Settings → LLM Providers**); an unknown
|
||||
value is ignored and resolution falls through to your saved selection. While
|
||||
this env var is set it wins over the in-app picker, so if the UI selection
|
||||
appears to have "no effect," check whether `LLM_DEFAULT_PROVIDER` is exported.
|
||||
|
||||
## LLM Skills (per-feature routing)
|
||||
|
||||
**Settings → System → LLM Skills** lists every LLM-powered feature — Cinematic &
|
||||
Autofit translation, speech-rate slot fitting, glossary auto-extract, direction
|
||||
parsing, and dictation cleanup — and lets you toggle each one or route it to a
|
||||
specific provider instead of the global active one. That way sensitive work
|
||||
(e.g. dictation cleanup) can stay on a local Ollama/LM Studio model while
|
||||
heavier jobs use a remote provider. A disabled skill degrades exactly like
|
||||
having no LLM configured: Cinematic/Autofit falls back to Fast, dictation
|
||||
cleanup passes the raw transcript through, direction parsing uses the keyword
|
||||
heuristic. Everything defaults to enabled + "use active provider", so existing
|
||||
setups behave unchanged.
|
||||
|
||||
## API keys (online MT engines)
|
||||
|
||||
The non-LLM online engines need a key, set as an environment variable before
|
||||
launching the backend (or in **Settings → Credentials**):
|
||||
|
||||
- **DeepL:** `DEEPL_API_KEY` (optionally `DEEPL_BASE_URL` for a self-hosted /
|
||||
pro endpoint).
|
||||
- **Microsoft Translator:** `MICROSOFT_API_KEY` (optionally `MICROSOFT_BASE_URL`).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **"The 'google' translation engine needs the optional deep_translator Python
|
||||
package…"** — the package isn't installed. On a from-source install, click the
|
||||
Install button (or run the command above) and restart. On a packaged build,
|
||||
switch to Argos/NLLB via the popover.
|
||||
- **Install button does nothing / says "disabled in packaged builds"** — you're
|
||||
on a signed installer build (expected). Use Argos/NLLB, or add the package in a
|
||||
from-source checkout.
|
||||
- **Installed it but still "needs install"** — restart the backend so Python
|
||||
picks up the newly-installed module.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Confucius4-TTS (opt-in engine)
|
||||
|
||||
> **Status: validated end-to-end (2026-07-02).** The integration (engine
|
||||
> registration, dedicated-venv bootstrap, sidecar wire protocol, opt-in gating)
|
||||
> is done, the sidecar's pure logic is unit-tested
|
||||
> (`tests/test_confucius4_sidecar.py`), and a live synthesis run on Apple
|
||||
> Silicon (CPU) produced audible cloned speech — confirming the model API and
|
||||
> the true output sample rate of **22 050 Hz**. CUDA is the recommended
|
||||
> hardware; CPU works but is slow (~17× realtime — roughly 100 s for 6 s of
|
||||
> audio). MPS also runs but is *slower* than CPU (~64× realtime), so the
|
||||
> sidecar deliberately never selects it. The engine is gated behind
|
||||
> `OMNIVOICE_CONFUCIUS4_TTS_DIR`, so it's completely inert until you opt in —
|
||||
> it can't affect the default install on any platform.
|
||||
|
||||
[Confucius4-TTS](https://github.com/netease-youdao/Confucius4-TTS) (netease-youdao)
|
||||
is an LLM-based multilingual / cross-lingual zero-shot voice-cloning TTS.
|
||||
|
||||
- **14 languages**: Chinese, English, Japanese, Korean, German, French, Spanish,
|
||||
Indonesian, Italian, Thai, Portuguese, Russian, Malay, Vietnamese.
|
||||
- **Unconstrained cloning** — no reference transcript required.
|
||||
- **Cross-lingual voice transfer** — keep one voice across languages.
|
||||
- **License:** Apache-2.0. **Hardware:** NVIDIA GPU (CUDA 12.6) recommended;
|
||||
CPU validated on Apple Silicon but ~17× realtime. Output: 22 050 Hz mono.
|
||||
|
||||
Like IndexTTS-2 / MOSS-TTS-v1.5 / dots.tts, it runs in its **own subprocess venv**
|
||||
so its dependency stack never touches the default OmniVoice interpreter.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/netease-youdao/Confucius4-TTS.git
|
||||
cd Confucius4-TTS
|
||||
uv venv --python 3.10
|
||||
uv pip install -r requirements.txt
|
||||
```
|
||||
|
||||
> Upstream ships **no `pyproject.toml`/`setup.py`**, so there is nothing to
|
||||
> `pip install -e` — don't try; it fails. The OmniVoice sidecar puts the clone
|
||||
> on `sys.path` itself (the same thing upstream's `example.py` does).
|
||||
|
||||
**Model weights — all fetched automatically from HuggingFace on first
|
||||
synthesis (~5 GB total, cached in `$HF_HUB_CACHE`):**
|
||||
|
||||
- `netease-youdao/Confucius4-TTS` — `t2s_model.safetensors` + `s2a_model.pt`
|
||||
(the tokenizer + `wav2vec2bert_stats.pt` already ship in the clone's
|
||||
`checkpoints/`).
|
||||
- `facebook/w2v-bert-2.0` — semantic feature extractor (~2.3 GB).
|
||||
- `funasr/campplus` — speaker-style encoder (small).
|
||||
- `nvidia/bigvgan_v2_22khz_80band_256x` — vocoder (BigVGAN and CAMPPlus
|
||||
*code* is vendored in the clone's `external/`; no Amphion install needed).
|
||||
|
||||
Set your `HF_TOKEN` (Settings → Credentials) if you hit rate limits.
|
||||
|
||||
Then point OmniVoice at the clone and restart:
|
||||
|
||||
- **macOS/Linux:** `export OMNIVOICE_CONFUCIUS4_TTS_DIR=/path/to/Confucius4-TTS`
|
||||
- **Windows (PowerShell):** `[Environment]::SetEnvironmentVariable("OMNIVOICE_CONFUCIUS4_TTS_DIR","C:\path\to\Confucius4-TTS","User")`
|
||||
|
||||
Select **Confucius4-TTS** in Settings → Engines. The first synthesize triggers
|
||||
the weight downloads above, then generates.
|
||||
|
||||
### Optional overrides
|
||||
|
||||
- `OMNIVOICE_CONFUCIUS4_CONFIG` — path to `inference_config.yaml` if it isn't at
|
||||
`<clone>/config/inference_config.yaml`.
|
||||
|
||||
## Validation record (2026-07-02, Apple Silicon M-series, CPU)
|
||||
|
||||
The sidecar (`backend/engines/confucius4/main.py`) uses:
|
||||
|
||||
```python
|
||||
from confuciustts.cli.inference import ConfuciusTTS
|
||||
model = ConfuciusTTS(config_path=..., device="cuda") # or "cpu"
|
||||
audio = model.generate(text=..., lang="en", prompt_wav="ref.wav") # → tensor
|
||||
sr = model.sample_rate # 22050
|
||||
```
|
||||
|
||||
- ✅ **Live end-to-end run**: English zero-shot clone from a 9.5 s reference —
|
||||
6.06 s of audible speech (peak 0.85) in 102 s on CPU. `model.sample_rate`
|
||||
returned **22 050**, matching `target_sample_rate` in
|
||||
`config/inference_config.yaml`; `CONFUCIUS_SAMPLE_RATE` /
|
||||
`_DEFAULT_SAMPLE_RATE` are pinned to it (regression-tested).
|
||||
- ✅ **Not pip-installable upstream** — discovered live; the bootstrap now skips
|
||||
the editable install unless upstream ships packaging, and both the import
|
||||
probe and the sidecar resolve `confuciustts` via the clone on `sys.path`.
|
||||
- ✅ **MPS probed and rejected**: runs, but ~4× slower than CPU (Metal op
|
||||
fallbacks) — the sidecar selects CUDA when available, else CPU, never MPS.
|
||||
- ✅ **Sidecar logic unit-tested** (`tests/test_confucius4_sidecar.py`):
|
||||
language normalization, tensor→PCM (mono/stereo/clip), config-path
|
||||
resolution, clone sys.path injection, wire framing, synthesize dispatch.
|
||||
@@ -1,10 +1,11 @@
|
||||
# Engine venvs & disk usage
|
||||
|
||||
Most engines run in-process in OmniVoice's main environment. A few
|
||||
(**IndexTTS2**, and any engine whose dependencies conflict with the parent's
|
||||
`torch`/`transformers` pins) run in a **dedicated sidecar venv** so their pins
|
||||
can't break the rest of the app. Those sidecars are where disk adds up — this
|
||||
page explains why, and how the on-disk cost is kept down.
|
||||
(**IndexTTS2**, **MOSS-TTS-v1.5**, **dots.tts**, and any engine whose
|
||||
dependencies conflict with the parent's `torch`/`transformers` pins) run in a
|
||||
**dedicated sidecar venv** so their pins can't break the rest of the app. Those
|
||||
sidecars are where disk adds up — this page explains why, and how the on-disk
|
||||
cost is kept down.
|
||||
|
||||
## Why a sidecar needs its own venv
|
||||
|
||||
@@ -48,6 +49,13 @@ the parent whenever the engine allows it.** When it doesn't (IndexTTS2's
|
||||
`transformers<5` forces an older torch line), the second copy is the
|
||||
unavoidable price of isolation — not a bug.
|
||||
|
||||
The opt-in #498 engines illustrate both sides: **dots.tts** pins
|
||||
`torch==2.8.0` — the **same** build the parent constrains to — so it shares
|
||||
almost all of torch with the main venv and only its `transformers==4.57` +
|
||||
model deps are new. **MOSS-TTS-v1.5** pins `torch==2.9.1+cu128`, a **different**
|
||||
build, so it pays a full extra multi-GB torch copy on CUDA hosts (the price of
|
||||
running an 8B model whose stack pins `transformers==5.0`).
|
||||
|
||||
> On Linux, the `nvidia-*` CUDA packages are separate wheels, so even across
|
||||
> *different* torch versions any `nvidia-*` whose pinned version happens to
|
||||
> match is still shared. On Windows the CUDA DLLs live inside the one torch
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user