Palash DebnathandClaude Fable 5
b79ba9bd3b
* docs(readme): lead with download + first clone; seed benchmarks page Quickstart (installers, install guides, a three-step first-clone walkthrough) moves above What's-new/Features in both READMEs — visitors get the action before the pitch. New docs/benchmarks.md anchors measured per-engine/device numbers on the bench_pipeline.py harness, community-contributed, no estimates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(changelog): entry for the README conversion restructure (#1555) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): emit RTF + CUDA peak VRAM; guard NaN RAM; define the benchmarks schema Bot harvest on #1555: the tts stage now prints RTF per warm measurement and CUDA peak VRAM (None elsewhere — no made-up zeros), the stage floor refuses unmeasurable RAM instead of sailing past a NaN comparison (FLOOR_GB=0 overrides), docs/benchmarks.md columns map 1:1 to what the harness prints, and the download badges say they open the release page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): link palash.dev from the maker section Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): name the resolved engine, track VRAM from resolution, comment the guards Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme): the quick-switch gif is the hero image The hero shows motion now; the Launchpad screenshot moves into the 0.5.0 What's-new slot so nothing appears twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): peak VRAM is reserved memory; adapter engines name their model Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): subprocess-isolated engines report VRAM n/a, not a parent-side zero Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): out-of-process detection is declarative; sherpa rows name their model 'runs_out_of_process' is now a TTSBackend attribute set by SubprocessBackend AND omnivoice-gguf (which inherits TTSBackend directly but spawns a binary per generate — the isinstance check missed it). Duck-typed for the same module-purge reason as _is_subprocess_isolated. Sherpa-onnx identity comes from _model_dir's basename when _model_id is absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(bench): backends self-report model identity via TTSBackend.model_identity() Greptile enumerated the adapter engines one at a time (mlx _model_id, sherpa _model_dir, cosyvoice env-only) — the attribute sniffing rots per engine. The hook fixes the class: each multi-model backend reports its own identity, the profiler just asks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
57 lines
2.4 KiB
Markdown
57 lines
2.4 KiB
Markdown
# Benchmarks
|
|
|
|
Measured numbers per engine and device — how long a generation actually
|
|
takes on real hardware. Every number here is produced by the in-repo
|
|
harness, on named hardware, at a named version; nothing is estimated.
|
|
|
|
## How numbers are measured
|
|
|
|
```bash
|
|
# stop the app first — a running backend holds a model and skews numbers
|
|
uv run python scripts/bench_pipeline.py # everything
|
|
uv run python scripts/bench_pipeline.py tts # just the TTS stage
|
|
```
|
|
|
|
`scripts/bench_pipeline.py` profiles each pipeline stage one at a time,
|
|
memory-safely: it refuses to start a stage without enough free RAM and
|
|
unloads models between stages. See [performance.md](performance.md) for
|
|
what each stage spends its time on.
|
|
|
|
The `tts` stage emits the two values this table collects:
|
|
|
|
- **RTF** (real-time factor) — seconds of compute per second of generated
|
|
audio, printed next to each warm measurement. RTF < 1 means faster than
|
|
real time. Use the **short line (warm)** RTF for the table.
|
|
- **Peak VRAM** — printed on CUDA only. MPS is unified memory and CPU has
|
|
no VRAM; subprocess-isolated engines allocate outside the harness's view
|
|
(it prints `n/a` for them). Leave the column blank in all those cases.
|
|
|
|
## Results
|
|
|
|
No verified rows yet — this table fills from maintainer runs and community
|
|
submissions.
|
|
|
|
| Engine | Device | RTF (warm) | Peak VRAM (GB) | App version | Source |
|
|
|---|---|---|---|---|---|
|
|
| _none yet — contribute yours below_ | | | | | |
|
|
|
|
Column meanings: **Engine** — the TTS engine the harness resolved (printed
|
|
at stage start). **Device** — one string naming what ran the model, e.g.
|
|
`RTX 3060 12 GB`, `Apple M2 Pro`, `Ryzen 7 5800X (CPU)`. **RTF (warm)** —
|
|
the short-line warm RTF from the harness. **Peak VRAM** — the harness's
|
|
CUDA peak, blank on MPS/CPU. **App version** — from `Settings → About`.
|
|
**Source** — a link to the PR that added the row.
|
|
|
|
## Contributing a row
|
|
|
|
1. Run the harness on an otherwise-idle machine (app stopped) and copy its
|
|
summary table.
|
|
2. Open a PR adding one row using the column meanings above, and paste the
|
|
raw harness output into the PR description — that PR link becomes the
|
|
row's **Source**.
|
|
3. One row per engine+device pair; a newer app version replaces the old row.
|
|
|
|
Numbers from different machines aren't directly comparable — that's fine.
|
|
The point is honest expectations ("this engine on this class of GPU ≈ this
|
|
fast"), not a leaderboard.
|