Palash DebnathandClaude Fable 5
b79ba9bd3b
* docs(readme): lead with download + first clone; seed benchmarks page Quickstart (installers, install guides, a three-step first-clone walkthrough) moves above What's-new/Features in both READMEs — visitors get the action before the pitch. New docs/benchmarks.md anchors measured per-engine/device numbers on the bench_pipeline.py harness, community-contributed, no estimates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(changelog): entry for the README conversion restructure (#1555) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): emit RTF + CUDA peak VRAM; guard NaN RAM; define the benchmarks schema Bot harvest on #1555: the tts stage now prints RTF per warm measurement and CUDA peak VRAM (None elsewhere — no made-up zeros), the stage floor refuses unmeasurable RAM instead of sailing past a NaN comparison (FLOOR_GB=0 overrides), docs/benchmarks.md columns map 1:1 to what the harness prints, and the download badges say they open the release page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): link palash.dev from the maker section Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): name the resolved engine, track VRAM from resolution, comment the guards Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): the quick-switch gif is the hero image The hero shows motion now; the Launchpad screenshot moves into the 0.5.0 What's-new slot so nothing appears twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): peak VRAM is reserved memory; adapter engines name their model Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): subprocess-isolated engines report VRAM n/a, not a parent-side zero Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): out-of-process detection is declarative; sherpa rows name their model 'runs_out_of_process' is now a TTSBackend attribute set by SubprocessBackend AND omnivoice-gguf (which inherits TTSBackend directly but spawns a binary per generate — the isinstance check missed it). Duck-typed for the same module-purge reason as _is_subprocess_isolated. Sherpa-onnx identity comes from _model_dir's basename when _model_id is absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): backends self-report model identity via TTSBackend.model_identity() Greptile enumerated the adapter engines one at a time (mlx _model_id, sherpa _model_dir, cosyvoice env-only) — the attribute sniffing rots per engine. The hook fixes the class: each multi-model backend reports its own identity, the profiler just asks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2.4 KiB
Benchmarks
Measured numbers per engine and device — how long a generation actually takes on real hardware. Every number here is produced by the in-repo harness, on named hardware, at a named version; nothing is estimated.
How numbers are measured
# stop the app first — a running backend holds a model and skews numbers
uv run python scripts/bench_pipeline.py # everything
uv run python scripts/bench_pipeline.py tts # just the TTS stage
scripts/bench_pipeline.py profiles each pipeline stage one at a time,
memory-safely: it refuses to start a stage without enough free RAM and
unloads models between stages. See performance.md for
what each stage spends its time on.
The tts stage emits the two values this table collects:
- RTF (real-time factor) — seconds of compute per second of generated audio, printed next to each warm measurement. RTF < 1 means faster than real time. Use the short line (warm) RTF for the table.
- Peak VRAM — printed on CUDA only. MPS is unified memory and CPU has
no VRAM; subprocess-isolated engines allocate outside the harness's view
(it prints
n/afor them). Leave the column blank in all those cases.
Results
No verified rows yet — this table fills from maintainer runs and community submissions.
| Engine | Device | RTF (warm) | Peak VRAM (GB) | App version | Source |
|---|---|---|---|---|---|
| none yet — contribute yours below |
Column meanings: Engine — the TTS engine the harness resolved (printed
at stage start). Device — one string naming what ran the model, e.g.
RTX 3060 12 GB, Apple M2 Pro, Ryzen 7 5800X (CPU). RTF (warm) —
the short-line warm RTF from the harness. Peak VRAM — the harness's
CUDA peak, blank on MPS/CPU. App version — from Settings → About.
Source — a link to the PR that added the row.
Contributing a row
- Run the harness on an otherwise-idle machine (app stopped) and copy its summary table.
- Open a PR adding one row using the column meanings above, and paste the raw harness output into the PR description — that PR link becomes the row's Source.
- One row per engine+device pair; a newer app version replaces the old row.
Numbers from different machines aren't directly comparable — that's fine. The point is honest expectations ("this engine on this class of GPU ≈ this fast"), not a leaderboard.