fix(engines): Confucius4-TTS validated E2E — clone sys.path import, 22.05 kHz, real install docs (#590) (#872)
Co-authored-by: mergetest <test@local> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
mergetest
Claude Fable 5
parent
ae516cae63
commit
4eed552153
@@ -10,6 +10,17 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Confucius4-TTS is now validated end-to-end — and actually loads.** The
|
||||
opt-in engine's first live run (Apple Silicon, CPU) caught three
|
||||
scaffold-era faults: the sidecar could never import `confuciustts` (upstream
|
||||
ships no packaging, so the documented `pip install -e` fails — the sidecar
|
||||
and bootstrap probe now put the clone on `sys.path`, like upstream's own
|
||||
example), the assumed 24 kHz sample rate was wrong (confirmed **22 050 Hz**,
|
||||
now regression-tested), and the docs demanded an Amphion/MaskGCT install
|
||||
that doesn't exist (all weights auto-download from HuggingFace). CPU is
|
||||
~17× realtime, so CUDA stays the recommended path; `gpu_compat` now
|
||||
advertises `("cuda", "cpu")`. (#590)
|
||||
|
||||
- **Parakeet TDT transcription now works without an NVIDIA GPU.** The
|
||||
`nemo-parakeet` ASR engine (parakeet-tdt-0.6b-v3, 25 languages, word
|
||||
timestamps) was hard-gated behind CUDA — but a live measurement on an Apple
|
||||
|
||||
@@ -292,11 +292,11 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
|
||||
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
|
||||
>
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10/CUDA venv from a clone, NVIDIA-only; see [Confucius4-TTS](docs/engines/confucius4-tts.md). ⚠️ scaffold — needs on-hardware validation before production use.
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
|
||||
|
||||
### ASR Engines
|
||||
|
||||
|
||||
@@ -10,12 +10,13 @@ parent. It is **opt-in** — selected in the engine picker and enabled only when
|
||||
the user points ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` at a clone — so it can never
|
||||
become a broken default on any platform (the strict default-parity rule).
|
||||
|
||||
Status (#590): the synthesis API (``confuciustts.cli.inference.ConfuciusTTS`` →
|
||||
``.generate(text, lang, prompt_wav)`` → tensor, ``model.sample_rate``) is
|
||||
validated against the upstream repo, and the sidecar's pure logic is unit-tested
|
||||
(``tests/test_confucius4_sidecar.py``). A one-time end-to-end run on a CUDA 12.6
|
||||
GPU is still needed to confirm the live model call and the true output sample
|
||||
rate. Gated off by default, so this affects no one until they opt in.
|
||||
Status (#590): **validated end-to-end** (2026-07-02, Apple Silicon, CPU) — the
|
||||
synthesis API (``confuciustts.cli.inference.ConfuciusTTS`` →
|
||||
``.generate(text, lang, prompt_wav)`` → tensor, ``model.sample_rate``) produced
|
||||
audible speech at 22 050 Hz; the sidecar's pure logic is unit-tested
|
||||
(``tests/test_confucius4_sidecar.py``). CPU inference is slow (~17× realtime),
|
||||
so CUDA is the recommended path. Gated off by default, so this affects no one
|
||||
until they opt in.
|
||||
|
||||
Three entry points: ``Confucius4Backend`` (this module), ``main.py`` (the sidecar,
|
||||
runs under the Confucius4 venv — never imported by the parent), and
|
||||
@@ -35,7 +36,7 @@ logger = logging.getLogger("omnivoice.confucius4")
|
||||
|
||||
|
||||
class Confucius4Backend(SubprocessBackend):
|
||||
"""Confucius4-TTS (netease-youdao) — LLM-based, 14 langs, zero-shot clone, CUDA.
|
||||
"""Confucius4-TTS (netease-youdao) — LLM-based, 14 langs, zero-shot clone.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv. First synthesize cold-loads the checkpoint; subsequent calls
|
||||
@@ -45,22 +46,25 @@ class Confucius4Backend(SubprocessBackend):
|
||||
|
||||
git clone https://github.com/netease-youdao/Confucius4-TTS.git
|
||||
cd Confucius4-TTS
|
||||
uv venv --python 3.10 && uv pip install -r requirements.txt && uv pip install -e .
|
||||
uv venv --python 3.10 && uv pip install -r requirements.txt
|
||||
|
||||
(Upstream ships no pyproject.toml/setup.py, so there is nothing to
|
||||
``pip install -e`` — the sidecar sys.path-inserts the clone instead.)
|
||||
Then set ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` to the clone root and restart.
|
||||
License: Apache-2.0. Requires an NVIDIA GPU (CUDA 12.6); no MPS / CPU path.
|
||||
License: Apache-2.0. CUDA recommended; CPU validated but ~17× realtime.
|
||||
"""
|
||||
|
||||
id = "confucius4-tts"
|
||||
display_name = (
|
||||
"Confucius4-TTS (LLM, 14 langs, cross-lingual zero-shot clone, CUDA, Apache-2.0)"
|
||||
"Confucius4-TTS (LLM, 14 langs, cross-lingual zero-shot clone, CUDA/CPU, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # timbre comes from a reference clip
|
||||
# Upstream vocoder sample rate is not documented; re-read from the sidecar's
|
||||
# ready/audio frames. 24 kHz is the conservative default until confirmed.
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# NVIDIA/CUDA only — no MPS or CPU branch documented upstream.
|
||||
gpu_compat = ("cuda",)
|
||||
# Upstream vocoder rate (config target_sample_rate) — confirmed 22 050 Hz by
|
||||
# a live run (2026-07-02); still re-read from the sidecar's ready/audio frames.
|
||||
_DEFAULT_SAMPLE_RATE = 22050
|
||||
# CUDA fast path + CPU fallback, both exercised (CPU end-to-end validated).
|
||||
# No MPS claim — upstream has no Metal path.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
@@ -75,15 +79,15 @@ class Confucius4Backend(SubprocessBackend):
|
||||
return False, (
|
||||
"Confucius4-TTS venv not found. Set OMNIVOICE_CONFUCIUS4_TTS_DIR "
|
||||
"to your Confucius4-TTS clone (the directory containing "
|
||||
"requirements.txt) and restart OmniVoice. NVIDIA GPU (CUDA) "
|
||||
"required. See docs/engines/confucius4-tts.md."
|
||||
"requirements.txt) and restart OmniVoice. CUDA GPU recommended "
|
||||
"(CPU works but is slow). See docs/engines/confucius4-tts.md."
|
||||
)
|
||||
if not CONFUCIUS4_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"Confucius4-TTS sidecar script missing at "
|
||||
f"{CONFUCIUS4_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA)"
|
||||
return True, "ok"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
|
||||
@@ -11,13 +11,13 @@ Probe order (existing power-user installs win — zero migration):
|
||||
1. ``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv/`` — the user's clone-level venv.
|
||||
2. ``backend/engines/confucius4/.venv/`` — this package's own venv.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -r <clone>/requirements.txt``
|
||||
+ ``uv pip install -e <clone>`` (the upstream stack).
|
||||
(+ ``uv pip install -e <clone>`` only if upstream ever ships packaging).
|
||||
|
||||
⚠️ Scaffold (#590): the install/import details mirror the documented upstream
|
||||
layout but have NOT been validated on hardware — confirm the package name
|
||||
(``confuciustts``) and requirements path against your clone before relying on
|
||||
the auto-bootstrap. The engine is opt-in (env-dir gated) and never touched
|
||||
unless ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` is set, so this can't affect the default
|
||||
Validated end-to-end 2026-07-02 (Apple Silicon, CPU): upstream ships **no
|
||||
pyproject.toml/setup.py**, so ``confuciustts`` is importable only with the
|
||||
clone root on ``sys.path`` — the import probe and the sidecar both handle
|
||||
that. The engine is opt-in (env-dir gated) and never touched unless
|
||||
``OMNIVOICE_CONFUCIUS4_TTS_DIR`` is set, so this can't affect the default
|
||||
install on any platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -113,11 +113,20 @@ def _probe_paths() -> list[Path]:
|
||||
return out
|
||||
|
||||
|
||||
def _import_probe_code() -> str:
|
||||
"""Probe snippet mirroring the sidecar's import semantics: upstream is not
|
||||
pip-installable, so ``confuciustts`` resolves via the clone on sys.path."""
|
||||
clone = os.environ.get(_CLONE_DIR_ENV, "")
|
||||
if clone:
|
||||
return f"import sys; sys.path.insert(0, {clone!r}); import {_IMPORT_PROBE}"
|
||||
return f"import {_IMPORT_PROBE}"
|
||||
|
||||
|
||||
def _venv_can_import(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify ``import confuciustts`` works."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", f"import {_IMPORT_PROBE}"],
|
||||
[str(python_path), "-c", _import_probe_code()],
|
||||
capture_output=True, timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
@@ -173,10 +182,14 @@ def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"-r", str(requirements)],
|
||||
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
subprocess.run(
|
||||
[uv, "pip", "install", "--python", str(python_path), "-e", str(clone_dir)],
|
||||
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
# Editable install only if upstream ever ships packaging metadata —
|
||||
# as of 2026-07 there is none, and `uv pip install -e` on a bare clone
|
||||
# fails outright. Import resolution is handled via sys.path instead.
|
||||
if (clone_dir / "pyproject.toml").is_file() or (clone_dir / "setup.py").is_file():
|
||||
subprocess.run(
|
||||
[uv, "pip", "install", "--python", str(python_path), "-e", str(clone_dir)],
|
||||
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install failed during Confucius4 bootstrap "
|
||||
|
||||
@@ -18,10 +18,10 @@ Op flow: ready → ping/pong → synthesize (→ progress, → audio) → shutdo
|
||||
Status (#590): the model API below
|
||||
(``confuciustts.cli.inference.ConfuciusTTS(config_path=…, device=…)`` and
|
||||
``model.generate(text=, lang=, prompt_wav=)`` → audio tensor, ``model.sample_rate``)
|
||||
is validated against the upstream repo; this sidecar's pure logic is unit-tested
|
||||
in ``tests/test_confucius4_sidecar.py``. Still pending a one-time CUDA-12.6 GPU
|
||||
run to confirm the live model call + true sample rate. Opt-in, so it affects no
|
||||
one until enabled.
|
||||
is **validated end-to-end** (2026-07-02, Apple Silicon, CPU): live generate()
|
||||
produced audible speech at 22 050 Hz. This sidecar's pure logic is unit-tested
|
||||
in ``tests/test_confucius4_sidecar.py``. Opt-in, so it affects no one until
|
||||
enabled.
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code. NO logging of os.environ.
|
||||
"""
|
||||
@@ -36,9 +36,11 @@ import traceback
|
||||
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: Conservative default until the upstream vocoder rate is confirmed; the real
|
||||
#: value is re-read from each generate() result if the model exposes it.
|
||||
CONFUCIUS_SAMPLE_RATE = 24000
|
||||
#: Upstream BigVGAN vocoder rate — ``target_sample_rate: 22050`` in
|
||||
#: ``config/inference_config.yaml``, confirmed by a live end-to-end run
|
||||
#: (2026-07-02). The real value is still re-read from ``model.sample_rate``
|
||||
#: on each generate() so a future upstream change can't corrupt audio.
|
||||
CONFUCIUS_SAMPLE_RATE = 22050
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
@@ -87,14 +89,29 @@ def _config_path() -> str:
|
||||
return os.path.join(clone, "config", "inference_config.yaml")
|
||||
|
||||
|
||||
def _ensure_clone_on_sys_path() -> None:
|
||||
"""Make ``import confuciustts`` resolve from the user's clone.
|
||||
|
||||
Upstream Confucius4-TTS is **not pip-installable** (no pyproject.toml /
|
||||
setup.py as of 2026-07); its own ``example.py`` sys.path-inserts the repo
|
||||
root instead. Mirror that here so the sidecar works from a plain
|
||||
``uv pip install -r requirements.txt`` venv. Inserted at position 0 so the
|
||||
clone the user pointed at always wins over any stale installed copy.
|
||||
"""
|
||||
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
|
||||
if clone and clone not in sys.path:
|
||||
sys.path.insert(0, clone)
|
||||
|
||||
|
||||
def _load_model(stdout):
|
||||
"""Cold-construct the Confucius4 model (CUDA, else CPU as a last resort)."""
|
||||
"""Cold-construct the Confucius4 model (CUDA, else CPU — both validated)."""
|
||||
global _model
|
||||
if _model is not None:
|
||||
return _model
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
_ensure_clone_on_sys_path()
|
||||
import torch
|
||||
from confuciustts.cli.inference import ConfuciusTTS # type: ignore[import-not-found]
|
||||
|
||||
|
||||
@@ -1160,9 +1160,9 @@ _LAZY_REGISTRY: dict[str, tuple[str, str]] = {
|
||||
"dots-tts": ("engines.dots_tts", "DotsTTSBackend"),
|
||||
# Issue #590: Confucius4-TTS (netease-youdao) — LLM-based, 14-language
|
||||
# cross-lingual zero-shot cloning, Apache-2.0. Opt-in + subprocess-isolated
|
||||
# (own Python 3.10 + CUDA venv) like the entries above. ⚠️ scaffold: the
|
||||
# sidecar's synthesis API is README-derived and needs hardware validation;
|
||||
# it's gated behind OMNIVOICE_CONFUCIUS4_TTS_DIR so it's inert until enabled.
|
||||
# (own Python 3.10 venv) like the entries above. Validated end-to-end
|
||||
# 2026-07-02 (CPU, Apple Silicon; 22.05 kHz output). Gated behind
|
||||
# OMNIVOICE_CONFUCIUS4_TTS_DIR so it's inert until enabled.
|
||||
"confucius4-tts": ("engines.confucius4", "Confucius4Backend"),
|
||||
}
|
||||
|
||||
@@ -1259,7 +1259,7 @@ _INSTALL_HINTS: dict[str, str] = {
|
||||
"supertonic3": "uv sync --extra supertonic (CPU-only ONNX, 31 langs, ~400 MB model on first use; OpenRAIL-M model license)",
|
||||
"moss-tts-v15": "git clone OpenMOSS/MOSS-TTS + set OMNIVOICE_MOSS_TTS_V15_DIR (own venv, transformers==5.0; 8B, ~16 GB weights; CUDA/CPU, no MPS; Apache-2.0)",
|
||||
"dots-tts": "git clone rednote-hilab/dots.tts + set OMNIVOICE_DOTS_TTS_DIR (own venv, transformers==4.57; 2B, ~9 GB weights; CUDA/CPU, Linux/macOS only — no Windows; Apache-2.0)",
|
||||
"confucius4-tts":"git clone netease-youdao/Confucius4-TTS + set OMNIVOICE_CONFUCIUS4_TTS_DIR (own Python 3.10/CUDA venv; 14-lang cross-lingual zero-shot clone; NVIDIA GPU required; Apache-2.0)",
|
||||
"confucius4-tts":"git clone netease-youdao/Confucius4-TTS + set OMNIVOICE_CONFUCIUS4_TTS_DIR (own Python 3.10 venv; 14-lang cross-lingual zero-shot clone; ~5 GB weights auto-download; CUDA/CPU, no MPS; Apache-2.0)",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -1,18 +1,16 @@
|
||||
# Confucius4-TTS (opt-in engine)
|
||||
|
||||
> **Status: implementation complete — GPU inference run pending.** The
|
||||
> integration (engine registration, dedicated-venv bootstrap, sidecar wire
|
||||
> protocol, opt-in gating) is done, and the sidecar's synthesis API
|
||||
> (`confuciustts.cli.inference.ConfuciusTTS(config_path, device)` →
|
||||
> `generate(text, lang, prompt_wav)` → tensor, `model.sample_rate`) is
|
||||
> **validated against the upstream repo** and the sidecar's pure logic
|
||||
> (language normalization, tensor→PCM, config resolution, wire framing,
|
||||
> synthesize dispatch with the model mocked) is **unit-tested**
|
||||
> (`tests/test_confucius4_sidecar.py`). What remains is a one-time **end-to-end
|
||||
> run on a CUDA 12.6 GPU** to confirm the live model call and the true output
|
||||
> sample rate — no maintainer GPU box is available yet. The engine is gated
|
||||
> behind `OMNIVOICE_CONFUCIUS4_TTS_DIR`, so it's completely inert until you opt
|
||||
> in — it can't affect the default install on any platform.
|
||||
> **Status: validated end-to-end (2026-07-02).** The integration (engine
|
||||
> registration, dedicated-venv bootstrap, sidecar wire protocol, opt-in gating)
|
||||
> is done, the sidecar's pure logic is unit-tested
|
||||
> (`tests/test_confucius4_sidecar.py`), and a live synthesis run on Apple
|
||||
> Silicon (CPU) produced audible cloned speech — confirming the model API and
|
||||
> the true output sample rate of **22 050 Hz**. CUDA is the recommended
|
||||
> hardware; CPU works but is slow (~17× realtime — roughly 100 s for 6 s of
|
||||
> audio). MPS also runs but is *slower* than CPU (~64× realtime), so the
|
||||
> sidecar deliberately never selects it. The engine is gated behind
|
||||
> `OMNIVOICE_CONFUCIUS4_TTS_DIR`, so it's completely inert until you opt in —
|
||||
> it can't affect the default install on any platform.
|
||||
|
||||
[Confucius4-TTS](https://github.com/netease-youdao/Confucius4-TTS) (netease-youdao)
|
||||
is an LLM-based multilingual / cross-lingual zero-shot voice-cloning TTS.
|
||||
@@ -21,8 +19,8 @@ is an LLM-based multilingual / cross-lingual zero-shot voice-cloning TTS.
|
||||
Indonesian, Italian, Thai, Portuguese, Russian, Malay, Vietnamese.
|
||||
- **Unconstrained cloning** — no reference transcript required.
|
||||
- **Cross-lingual voice transfer** — keep one voice across languages.
|
||||
- **License:** Apache-2.0. **Hardware:** NVIDIA GPU, CUDA 12.6, Python 3.10.
|
||||
No CPU/MPS path documented; not advertised on Apple Silicon.
|
||||
- **License:** Apache-2.0. **Hardware:** NVIDIA GPU (CUDA 12.6) recommended;
|
||||
CPU validated on Apple Silicon but ~17× realtime. Output: 22 050 Hz mono.
|
||||
|
||||
Like IndexTTS-2 / MOSS-TTS-v1.5 / dots.tts, it runs in its **own subprocess venv**
|
||||
so its dependency stack never touches the default OmniVoice interpreter.
|
||||
@@ -34,53 +32,59 @@ git clone https://github.com/netease-youdao/Confucius4-TTS.git
|
||||
cd Confucius4-TTS
|
||||
uv venv --python 3.10
|
||||
uv pip install -r requirements.txt
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
**External dependencies (per upstream) — needed before first synthesis:**
|
||||
> Upstream ships **no `pyproject.toml`/`setup.py`**, so there is nothing to
|
||||
> `pip install -e` — don't try; it fails. The OmniVoice sidecar puts the clone
|
||||
> on `sys.path` itself (the same thing upstream's `example.py` does).
|
||||
|
||||
- **MaskGCT codec** from the [Amphion](https://github.com/open-mmlab/Amphion)
|
||||
repo — Confucius4's semantic-to-acoustic stage uses it. Follow upstream's
|
||||
README for the exact vendor/install step.
|
||||
- **`facebook/w2v-bert-2.0`** (Wav2Vec2-BERT) — pulled from HuggingFace on first
|
||||
use; make sure your `HF_TOKEN` (Settings → Credentials) is set if rate-limited.
|
||||
- **Checkpoint** `netease-youdao/Confucius4-TTS` (~2–4 GB: `t2s_model.safetensors`,
|
||||
`s2a_model.pt`, `wav2vec2bert_stats.pt`, tokenizer files) into `checkpoints/`.
|
||||
**Model weights — all fetched automatically from HuggingFace on first
|
||||
synthesis (~5 GB total, cached in `$HF_HUB_CACHE`):**
|
||||
|
||||
These are large and CUDA-only; budget disk + a first-run download.
|
||||
- `netease-youdao/Confucius4-TTS` — `t2s_model.safetensors` + `s2a_model.pt`
|
||||
(the tokenizer + `wav2vec2bert_stats.pt` already ship in the clone's
|
||||
`checkpoints/`).
|
||||
- `facebook/w2v-bert-2.0` — semantic feature extractor (~2.3 GB).
|
||||
- `funasr/campplus` — speaker-style encoder (small).
|
||||
- `nvidia/bigvgan_v2_22khz_80band_256x` — vocoder (BigVGAN and CAMPPlus
|
||||
*code* is vendored in the clone's `external/`; no Amphion install needed).
|
||||
|
||||
Set your `HF_TOKEN` (Settings → Credentials) if you hit rate limits.
|
||||
|
||||
Then point OmniVoice at the clone and restart:
|
||||
|
||||
- **macOS/Linux:** `export OMNIVOICE_CONFUCIUS4_TTS_DIR=/path/to/Confucius4-TTS`
|
||||
- **Windows (PowerShell):** `[Environment]::SetEnvironmentVariable("OMNIVOICE_CONFUCIUS4_TTS_DIR","C:\path\to\Confucius4-TTS","User")`
|
||||
|
||||
Select **Confucius4-TTS** in Settings → Engines. First synthesize downloads the
|
||||
checkpoint from `netease-youdao/Confucius4-TTS` (HuggingFace).
|
||||
Select **Confucius4-TTS** in Settings → Engines. The first synthesize triggers
|
||||
the weight downloads above, then generates.
|
||||
|
||||
### Optional overrides
|
||||
|
||||
- `OMNIVOICE_CONFUCIUS4_CONFIG` — path to `inference_config.yaml` if it isn't at
|
||||
`<clone>/config/inference_config.yaml`.
|
||||
|
||||
## Validation status (for the maintainer)
|
||||
## Validation record (2026-07-02, Apple Silicon M-series, CPU)
|
||||
|
||||
The sidecar (`backend/engines/confucius4/main.py`) uses:
|
||||
|
||||
```python
|
||||
from confuciustts.cli.inference import ConfuciusTTS
|
||||
model = ConfuciusTTS(config_path=..., device="cuda")
|
||||
model = ConfuciusTTS(config_path=..., device="cuda") # or "cpu"
|
||||
audio = model.generate(text=..., lang="en", prompt_wav="ref.wav") # → tensor
|
||||
sr = model.sample_rate
|
||||
sr = model.sample_rate # 22050
|
||||
```
|
||||
|
||||
- ✅ **Validated against upstream** — import path/package (`confuciustts`), the
|
||||
`ConfuciusTTS(config_path, device)` constructor, and `generate(text, lang,
|
||||
prompt_wav)` → tensor all match the repo.
|
||||
- ✅ **Sidecar logic unit-tested** (`tests/test_confucius4_sidecar.py`, 22 cases):
|
||||
language normalization, tensor→PCM (mono/stereo/clip), config-path resolution,
|
||||
wire framing, and synthesize dispatch with the model mocked.
|
||||
- ⏳ **Pending a CUDA 12.6 GPU run:** confirm the live `generate()` call end-to-end
|
||||
and the **true output sample rate** — if it isn't 24 kHz, update
|
||||
`CONFUCIUS_SAMPLE_RATE` in `main.py` / `_DEFAULT_SAMPLE_RATE` in `__init__.py`.
|
||||
To validate: set `OMNIVOICE_CONFUCIUS4_TTS_DIR`, select the engine, synthesize
|
||||
one clip on an NVIDIA box, and confirm audible output.
|
||||
- ✅ **Live end-to-end run**: English zero-shot clone from a 9.5 s reference —
|
||||
6.06 s of audible speech (peak 0.85) in 102 s on CPU. `model.sample_rate`
|
||||
returned **22 050**, matching `target_sample_rate` in
|
||||
`config/inference_config.yaml`; `CONFUCIUS_SAMPLE_RATE` /
|
||||
`_DEFAULT_SAMPLE_RATE` are pinned to it (regression-tested).
|
||||
- ✅ **Not pip-installable upstream** — discovered live; the bootstrap now skips
|
||||
the editable install unless upstream ships packaging, and both the import
|
||||
probe and the sidecar resolve `confuciustts` via the clone on `sys.path`.
|
||||
- ✅ **MPS probed and rejected**: runs, but ~4× slower than CPU (Metal op
|
||||
fallbacks) — the sidecar selects CUDA when available, else CPU, never MPS.
|
||||
- ✅ **Sidecar logic unit-tested** (`tests/test_confucius4_sidecar.py`):
|
||||
language normalization, tensor→PCM (mono/stereo/clip), config-path
|
||||
resolution, clone sys.path injection, wire framing, synthesize dispatch.
|
||||
|
||||
@@ -19,7 +19,7 @@ def test_registered_in_lazy_registry():
|
||||
def test_backend_class_metadata():
|
||||
from engines.confucius4 import Confucius4Backend
|
||||
assert Confucius4Backend.id == "confucius4-tts"
|
||||
assert Confucius4Backend.gpu_compat == ("cuda",) # CUDA-only, no MPS claim
|
||||
assert Confucius4Backend.gpu_compat == ("cuda", "cpu") # CPU validated E2E; no MPS claim
|
||||
assert Confucius4Backend.supports_voice_design is False
|
||||
|
||||
|
||||
|
||||
@@ -2,12 +2,11 @@
|
||||
|
||||
The upstream synthesis API (``confuciustts.cli.inference.ConfuciusTTS`` →
|
||||
``generate(text, lang, prompt_wav)`` → tensor, ``model.sample_rate``) is
|
||||
validated against the netease-youdao/Confucius4-TTS repo. Actual audio
|
||||
generation needs a CUDA 12.6 GPU + the ~2-4 GB checkpoint (Amphion/MaskGCT
|
||||
codec + w2v-bert-2.0), so it can't run in CI — but the sidecar's *pure* logic
|
||||
(language normalization, tensor→PCM, config-path resolution, wire framing) and
|
||||
the bootstrap probe are fully testable here, with the model mocked. These pin
|
||||
that logic so a real GPU run only has to confirm the model call itself.
|
||||
**validated end-to-end** (2026-07-02, Apple Silicon, CPU): audible speech at
|
||||
22 050 Hz. Full generation needs ~5 GB of weights, so it can't run in CI — but
|
||||
the sidecar's *pure* logic (language normalization, tensor→PCM, config-path
|
||||
resolution, sys.path clone injection, wire framing) and the bootstrap probe are
|
||||
fully testable here, with the model mocked.
|
||||
|
||||
The sidecar is stdlib-only at import time (the model/torch imports are lazy),
|
||||
so we import it directly without spawning the engine venv.
|
||||
@@ -85,6 +84,42 @@ def test_pcm_accepts_numpy(sc):
|
||||
assert (sr, n) == (16000, 3)
|
||||
|
||||
|
||||
# ── Sample rate (confirmed 22 050 Hz by the 2026-07-02 live run) ──────────
|
||||
|
||||
def test_sample_rate_constant_is_confirmed_upstream_rate(sc):
|
||||
# Upstream config target_sample_rate — regression-pins the live-run value
|
||||
# so the pre-validation 24 000 guess can't come back.
|
||||
assert sc.CONFUCIUS_SAMPLE_RATE == 22050
|
||||
|
||||
|
||||
def test_sample_rate_lockstep_with_backend_default(sc):
|
||||
import os as _os, sys as _sys
|
||||
_sys.path.insert(0, _os.path.join(
|
||||
_os.path.dirname(_os.path.dirname(_os.path.abspath(__file__))), "backend"))
|
||||
from engines.confucius4 import Confucius4Backend
|
||||
assert Confucius4Backend._DEFAULT_SAMPLE_RATE == sc.CONFUCIUS_SAMPLE_RATE
|
||||
|
||||
|
||||
# ── Clone sys.path injection (upstream is not pip-installable) ────────────
|
||||
|
||||
def test_clone_dir_inserted_at_sys_path_front(sc, monkeypatch):
|
||||
import sys as _sys
|
||||
monkeypatch.setenv("OMNIVOICE_CONFUCIUS4_TTS_DIR", "/clone")
|
||||
monkeypatch.setattr(_sys, "path", ["existing"])
|
||||
sc._ensure_clone_on_sys_path()
|
||||
assert _sys.path[0] == "/clone"
|
||||
sc._ensure_clone_on_sys_path() # idempotent — no dup
|
||||
assert _sys.path.count("/clone") == 1
|
||||
|
||||
|
||||
def test_no_sys_path_change_without_clone_dir(sc, monkeypatch):
|
||||
import sys as _sys
|
||||
monkeypatch.delenv("OMNIVOICE_CONFUCIUS4_TTS_DIR", raising=False)
|
||||
monkeypatch.setattr(_sys, "path", ["existing"])
|
||||
sc._ensure_clone_on_sys_path()
|
||||
assert _sys.path == ["existing"]
|
||||
|
||||
|
||||
# ── Config path resolution ────────────────────────────────────────────────
|
||||
|
||||
def test_config_path_explicit_override(sc, monkeypatch):
|
||||
|
||||
Reference in New Issue
Block a user