Refresh current v0.5.0/0.5 tag examples, document API-key and share-PIN behavior, and require encrypted private-overlay access for remote deployments. Keeps the Docker install guide and changelog synchronized.
Fix server-mode admin authentication recovery without trapping PIN-only deployments, and prevent stale 403 responses from clearing or superseding newly issued sessions. Includes backend/frontend regression coverage, docs, and changelog credit for @paoloantinori.
* feat(omnivoice): port upstream VoiceClonePrompt persistence + FlashInfer opt-in
Upstream k2-fsa teardown ports, verified with generated voice samples:
- VoiceClonePrompt.save()/.load() (upstream format v1, weights_only-safe)
on the vendored model, and a disk layer under the in-memory prompt LRU
(DATA_DIR/prompt_cache, keyed by ref path+mtime+ref_text+preprocess,
32 newest kept, OMNIVOICE_PROMPT_DISK_CACHE=0 opts out). First generation
of a session with a known voice skips the reference re-encode and any
auto-transcription pass — verified across two real processes (encodes=1
then encodes=0, same voice).
- omnivoice_flashinfer.py ported (packed CFG attention, fused kernels,
optional CUDA graphs), schedule adapted to our num_step+1 divergence.
Opt-in via OMNIVOICE_FLASHINFER=1|graph, CUDA-only, replaces
torch.compile for the session; missing package / apply failure / runtime
failure all degrade with a named reason (same #278 contract as compile:
classify → unapply → retry once, session latch). Measured 2.20x at
batch=1 on an RTX 4090 with byte-identical text and clean ASR round-trip.
- Docs: OmniVoice guide gains instruct+reference combination semantics
(consistent instruct stabilizes cloning, reference wins conflicts),
inline pronunciation control (pinyin / CMU), prompt persistence, and
corrects the 'no voice design' claim; performance.md documents both new
env knobs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: point changelog entries at the real PR number (#1565)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pr): harden FlashInfer lifecycle + prompt-cache writes per review
Bot harvest round 1 (#1565): unapply on apply-failure (half-patched model
could crash the next render); pin eager-mode FlashInfer inference to one
thread too — the attention plan and packed position ids are per-generation
module state, so interleaved _gpu_pool workers would corrupt each other;
restore the CAPTURED pre-apply attention impl (could be flash_attention_2)
instead of assuming sdpa; unique tmp name per prompt-cache write; correct
the _forward_logits layout docstring; resolve VoiceClonePrompt at test
runtime; docs — Known limits keeps only the limitation, performance.md
states the VRAM cost and scopes the fallback claim to classified kernel
failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pr): round-2 review — publish only a fully restored model, redact latch reason, tighten CPU-persistence test
Greptile: the runtime fallback now unapplies BEFORE swapping generate, so
a concurrent render keeps queuing behind the thread-affinity wrapper while
teardown mutates modules. CodeRabbit: FlashInfer failure reasons pass
through core.failure.sanitize before latching/logging (wheel paths embed
the user's home); the save-portability test now creates the tokens on CUDA
when available and asserts the persisted payload itself is CPU-resident.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pr): fail-closed latch reason when the sanitizer itself breaks
CodeQL empty-except + CodeRabbit round 3: if core.failure.sanitize raises,
the raw reason (home paths, wheel paths) was latched anyway. Now only the
exception class survives with a fixed redaction note; two regression tests
(normal redaction + sanitizer failure).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(settings): compute-device override — auto | CUDA | ROCm | XPU | MPS | CPU
Auto-detect stays the default; the override kills the 'auto-detect picked
wrong' issue class. Applied at the single choke point (_probe()'s family
selection) so routing, get_best_device(), and every badge inherit it.
Resolution: OMNIVOICE_DEVICE env > Settings pick (prefs.json) > auto (#981
pattern). An override can steer, never invent hardware: a family the host
lacks is noted and ignored; cpu is always honorable. Applies at next
backend start (host caps are immutable per process — same restart contract
as the rest of the Performance tab, RestartBadge shown).
GET/PUT /api/settings/compute-device (admin-gated) reports resolved vs
applied so the panel shows restart-required truthfully and disables itself
under an env pin instead of pretending.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): entry for the compute-device override (#1557)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(device-override): harvest — the override reaches CT2 ASR, full i18n, honest edge states
- _ctranslate2_cuda_ok() and the ASR sidecar now gate on the probe's family,
so a cpu pin (or ROCm host) can never hand CTranslate2 a CUDA device —
the override reaches every CT2 loader through one shared gate
- override_ignored exposed by the API and shown by the panel (env pin naming
a device this machine lacks: auto is in effect, restart won't change it)
- all 8 panel strings + 5 device-family labels translated into all 21
locales; failed saves keep their error visible through the re-sync
- test isolation: cleanup drops OMNIVOICE_DEVICE before re-probing so no
overridden caps leak into later tests; panel tests wait for loaded state
- xpu/intel search keywords; oxfmt formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(device-override): round 2 — fail-safe probe fallbacks, complete i18n, combined pin state
- a broken capability probe now means CPU everywhere (CT2 gate + ASR
sidecar) — never a torch-derived guess that would bypass a cpu pin or
re-open #1529 on ROCm; regression test added
- env-pinned AND not-detected shows both facts in one subtitle
- device_load_failed/perf_save_failed translated into all 21 locales;
CJK/th/vi/ar strings no longer say literal 'Auto'
- test_ctranslate2_never_gets_cuda_on_a_rocm_build pins the probe family
(it was order-dependent on the lru_cache before)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): pin the probe family in the faster-whisper OOM-fallback test
Same class as the rocm-build test: it mocked torch but not the probe the
new override gate consults first, so on a cpu-family CI host the CUDA
fallback chain under test was unreachable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs(engines): a guide for every engine + index; fix two engine-metadata bugs
21 new pages under docs/engines/ (10 TTS, 10 ASR, index README) — every
registered engine now has one: what it's for, platform support, model env
vars, quirks with issue refs. Linked from both READMEs' engine sections.
Code fixes found while verifying facts against the registries:
- KittenTTS docstring claimed default voice 'Jasper'; the code default is
expr-voice-2-f
- the isolated-ASR sidecar read only ASR_MODEL_FW while the download
preflight read ASR_MODEL_FASTER — set one and the other quietly used a
different model; both now resolve ASR_MODEL_FW-override → ASR_MODEL_FASTER
- moonshine's install hint named 'useful-moonshine', a package the backend
never imports; now moonshine-onnx / moonshine-voice
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): entries for the engine guides + sidecar model fix (#1556)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(engines): second-harvest fixes — TLS guidance, matrix/code alignment, CN counts
- README matrix aligned to gpu_compat (the code is the source of truth):
CosyVoice macOS is CPU not MPS, IndexTTS and GGUF gain their real
CUDA/CPU/MPS cells
- gpt-sovits guide: prefer https/tunnel for non-loopback servers, plaintext
warning; first-use download guidance on both OmniVoice pages
- preflight empty-env fallback matches the sidecar (ASR_MODEL_FASTER='' no
longer resolves a different repo)
- nano installs via uv pip; kitten log level wording; index links install
guides incl. the Gatekeeper step; README_CN engine counts 16/11
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(readme-cn): the all-engines-local claim now excludes the remote client
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs(readme): lead with download + first clone; seed benchmarks page
Quickstart (installers, install guides, a three-step first-clone walkthrough)
moves above What's-new/Features in both READMEs — visitors get the action
before the pitch. New docs/benchmarks.md anchors measured per-engine/device
numbers on the bench_pipeline.py harness, community-contributed, no estimates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): entry for the README conversion restructure (#1555)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): emit RTF + CUDA peak VRAM; guard NaN RAM; define the benchmarks schema
Bot harvest on #1555: the tts stage now prints RTF per warm measurement and
CUDA peak VRAM (None elsewhere — no made-up zeros), the stage floor refuses
unmeasurable RAM instead of sailing past a NaN comparison (FLOOR_GB=0
overrides), docs/benchmarks.md columns map 1:1 to what the harness prints,
and the download badges say they open the release page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(readme): link palash.dev from the maker section
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): name the resolved engine, track VRAM from resolution, comment the guards
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(readme): the quick-switch gif is the hero image
The hero shows motion now; the Launchpad screenshot moves into the 0.5.0
What's-new slot so nothing appears twice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): peak VRAM is reserved memory; adapter engines name their model
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): subprocess-isolated engines report VRAM n/a, not a parent-side zero
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): out-of-process detection is declarative; sherpa rows name their model
'runs_out_of_process' is now a TTSBackend attribute set by SubprocessBackend
AND omnivoice-gguf (which inherits TTSBackend directly but spawns a binary
per generate — the isinstance check missed it). Duck-typed for the same
module-purge reason as _is_subprocess_isolated. Sherpa-onnx identity comes
from _model_dir's basename when _model_id is absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(bench): backends self-report model identity via TTSBackend.model_identity()
Greptile enumerated the adapter engines one at a time (mlx _model_id,
sherpa _model_dir, cosyvoice env-only) — the attribute sniffing rots per
engine. The hook fixes the class: each multi-model backend reports its
own identity, the profiler just asks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(startup): bind the socket in ~1s and narrate startup step by step
The structural fix for the "can't reach the local backend" class (~1 in 5
of every issue ever filed): uvicorn served nothing until torch import
(10-20s cold), the 30-router fan-out, an import-time DB migration, the
cuDNN preload, and alembic all finished — every slow or fragile step
rendered as an unexplained dead backend.
main.py now keeps module scope fast and defers the heavy work:
- _phase_a_build (executor thread): prefs/env restore + #963 migration,
yt-dlp overlay, cuDNN preload, torchaudio, model_manager, router
imports — order preserved, literal imports so PyInstaller still traces.
- _phase_a_finalize (event loop, no awaits → atomic wrt requests):
include_router, mounts, MCP, SPA, openapi bust.
- _phase_b: the old lifespan startup body; handles on app.state so
shutdown survives a startup that never finished.
- Eager mode (pytest / OMNIVOICE_EAGER_INIT=1) runs everything at import
— byte-equivalent behavior for the ~100 lifespan-less TestClient sites
and for embedders (dump_api_routes, probe boot runner opt in).
While starting: /health answers 503 with the current step, new
/startup/progress serves the full ledger (always 200), and
StartupGateMiddleware 503s everything else with the [starting] marker
(same skip-the-Report-button convention as [shutting_down]). A deferred
failure keeps import-crash semantics: traceback to stderr → shell crash
forensics, run sentinel stays uncleared, exit 1 names the failed step.
Shell: startup_progress() probe (marker-header-gated so a foreign
responder can't narrate the splash) feeds per-step log lines into the
launch poll and the supervisor's reconnect wait. --health-check absorbs
the deferred init (60→180s); --diagnose runs Phase A up front so it
still sees restored prefs. Docker HEALTHCHECK semantics unchanged
(curl -f fails on 503 exactly as it did on connection-refused).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(startup): join the Phase A thread on shutdown; async fail-path sleep
Bot-review harvest on #1550: cancelling the deferred-startup task cannot
stop the executor thread inside Phase A's blocking imports — shutdown now
waits (bounded, only when a build started and hasn't finished) on a
thread-completion event so interpreter teardown can't race a mid-import
(#1000 class). The failure path's last-poll beat is now awaited, not
time.sleep — a blocking sleep froze the very loop that beat exists to let
serve. Also: dump_api_routes forces eager (assignment, not setdefault),
and the integration test's child gets DEVNULL instead of an undrained
pipe that could wedge a cold boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(startup): close the Phase A submission race; CodeQL nits
Review finds on #1550: shutdown could sample _phase_a_started unset
while the executor callable was queued-but-not-running, skipping the
thread join. started is now set BEFORE submission, the submission is
shielded so a cancel can't strand a queued callable that would never set
_phase_a_finished, and the wrapper sets finished on every exit including
the already-built early return. Contract pinned by
test_phase_a_thread_join_contract. Plus explanatory comments on the new
bare excepts and a consistent return in the gate's websocket branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Defer and coalesce omnivoice.app and omni_ui persistence behind a 250 ms
quiet window with a 1,000 ms hard maximum, preserving storage schemas,
synchronous pending reads, legacy formats, Factory Reset semantics,
widget read-only ownership, and lifecycle (pagehide/visibilitychange)
durability. Adds scheduler, restore, reset, StrictMode, concurrent-render,
role-ownership, and migration regression tests plus an opt-in
production-bundle responsiveness harness.
Lands #1541 by @bultodepapas (maintainer landing branch; the out-of-scope
attribution-policy commit was dropped).
Co-authored-by: Gius <bultodepapas@gmail.com>
CodeRabbit's last-cycle Minor on #1540, applied as the immediate
follow-up the docs-sync rule prescribes: three OmniVoice mentions in
docs/remote-workers.md now carry the product's name.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: bump to 0.5.0
Owner-requested minor bump. package.json is the source of truth; the
three mirrors (Cargo.toml, pyproject.toml, _FALLBACK_VERSION), the three
lockfiles and the branding pin move in lockstep, and the accumulated
Unreleased section becomes the curated 0.5.0 release notes — quiet
Highlights first, one-liner subsections after, duplicates folded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: fresh README for the VoiceStudio era; 0.5.0 notes wear the release
README: 551 lines from 656 — a What's-new-in-0.5.0 section with real
captures, the engine tables corrected to the actual 16 TTS registrations,
a stale Settings path and a broken Colab link fixed, roadmap/FAQ/credits
trimmed to what earns its place.
Release notes: the quick-switch GIF and catalogue/gallery screenshots,
captured from the running app during the pre-bump test pass, embedded
after the Highlights; #1542's gallery work and the ffmpeg CI fallback
recorded in their subsections.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: the feature inventory's Remote Model Downloads mention survives the README trim
check-docs-drift requires every docs/features.yaml name verbatim in the
README; the overhaul folded the phrase away. It now lives in the remote
workers feature line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: one blockquote, and an architecture claim that survives the opt-ins
CodeRabbit on #1540: MD028 blank line inside adjacent blockquotes, and
'every layer is on your machine' contradicted the opt-in remote paths
documented two sections away — it now states local-by-default with the
opt-ins named.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(gallery): save gallery voices as profiles, with validated audio references
Work-in-progress lifted from the concurrent gallery session at the
owner's request (its uncommitted working tree, preserved verbatim from
base 92b1ee5d; safety snapshot remains at rescue/gallery-wip):
- gallery voices can be saved as local profiles: audio is copied into
the profile store with content-addressed filenames, existing profiles
are detected and refreshed only when the source clip changed
- backend/core/audio_validation.py: symlink-rejecting, root-contained
resolution for persisted profile WAV references, with tests
- archetype/community routers and the Voice Gallery UI updated for the
save-as-profile handoff (spec: docs/specs/longform/26-gallery-use-handoff.md)
- locale updates for the new gallery strings across all 21 files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: drop a stray local screenshot script that rode in with the tree copy
* fix(community): explain the tolerated Content-Length parse failure; drop an unused import
CodeQL on #1542: the empty except now says why it is safe (the streamed
byte counter enforces the same cap regardless), and the test file loses
an unused Path import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(gallery): review findings — copy outside the write lock, no stale completions
CodeRabbit on #1542, all findings addressed:
- the profile-audio copy stages to a .part temp BEFORE BEGIN IMMEDIATE
and publishes via atomic os.replace inside it — other backend writers
no longer block for the duration of an audio copy; a mid-copy failure
leaves no temp droppings and no profile row (both pinned by tests)
- VoiceGallery async ops carry per-operation generation tokens: a
preview or save-as-profile that resolves after unmount (or after a
newer operation) can no longer play audio, redirect into a workspace,
or touch state — three fail-before regression tests
- VoiceGalleryActions imports the page at test runtime; the e2e locator
uses a stable data-testid instead of a translated string; symlink
tests skip cleanly where the OS can't create symlinks; the changelog
line carries its PR ref
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* ci: static ffmpeg fallback when the chocolatey feed is down
Third feed outage to break a PR run (2026-07-20, 2026-07-28, today —
three attempts, three 'installed 0/1'). Chocolatey is a distribution
channel, not the dependency: after the retry loop exhausts, fetch the
static gyan.dev build from its GitHub release mirror and put it on
PATH — same binary, no feed in the path. URL verified live (HTTP 200).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(security): replace persistent admin keys with sessions
Exchange the remote administrator key once for bounded, revocable credentials. Canonicalize backend principals, enforce cookie CSRF and exact origins, and use path-bound one-use WebSocket tickets.
Migrate the bundled UI away from durable master-key storage and credential-bearing URLs. Add unit, integration, static-hygiene, and production-browser regressions plus synchronized operator documentation.
* docs: link session hardening to PR 1528
* fix(security): key session indexes with process pepper
Use HMAC-SHA-256 instead of an unkeyed digest for in-memory session and WebSocket-ticket indexes. This preserves constant-size lookup identifiers, makes copied records unusable without the process pepper, and resolves CodeQL's weak sensitive-data hash finding.
* fix(auth): align empty bearer migration precedence
Centralize the Authorization-channel presence decision with canonical principal parsing. Bearer followed only by spaces now remains an empty channel during legacy cookie migration, while unsupported or invalid explicit credentials stay authoritative and fail closed.
* fix(security): harden admin session review boundaries
* fix(security): derive key generations with HKDF
* fix(auth): anchor the admin-session store so module reloads cannot fork it
test_master_exchange_does_not_bypass_pin_on_normal_routes failed in full-suite
runs: test_mcp_bindings' client fixture purges the services.* tree from
sys.modules and reloads main, so api.routers.auth re-imported a fresh
services.admin_sessions (new AdminSessionStore) while core.auth kept its
import-time reference to the old one — the exchange issued the cookie into
one store and the middleware resolved it against another, turning the
expected "PIN required" into "API key required".
Root cause is the class of bug, not the one test: a process-global auth
store defined as a bare module-level singleton forks under importlib.reload
or purge-and-reimport. Fix at the source: admin_session_store now resolves
through a synthetic sys.modules anchor (_omnivoice_admin_session_store_anchor)
that reloads never re-execute and package-prefix purges never match, so every
copy of the module shares the one per-process store. No consumer or behavior
changes.
Regression test reproduces both fork vectors (in-place reload and
sys.modules purge + fresh import) and asserts previously issued sessions
still resolve and the store identity is preserved; it fails before this fix
and passes after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(auth): honor X-Forwarded-Proto for CSRF origin and Secure cookies behind TLS proxies
Behind Tailscale Serve (docs/remote-gpu.md) or any TLS-terminating proxy,
the browser talks https while the backend hop stays http, so exact-origin
CSRF compared an https Origin against an http expectation and rejected
every legitimate request, and the session cookie shipped without Secure.
uvicorn's ProxyHeadersMiddleware only rewrites the scope for loopback
peers, which misses Docker and any non-loopback proxy topology.
New core.csrf.effective_scheme derives the client-facing scheme: resolved
scope first (uvicorn's trusted-proxy rewrite wins), then an upgrade-only
read of X-Forwarded-Proto's first value — https/wss promotes http to
https, everything else is ignored, and a genuine TLS hop can never be
downgraded. Used by both the destination-origin comparison and
auth._secure_cookie so the WS-ticket/logout CSRF paths and the cookie
Secure flag agree. Spoofing gains nothing: the host:port half of the
origin tuple is untouched, browsers cannot attach the header cross-site
without a preflight this API never grants, and forging it on plain http
only adds Secure (the browser then drops the cookie — self-harm only).
Regression tests: proxied https origin accepted (origin check, Secure
flag, logout), comma-separated chains, scope-fallback path, spoofed
header still rejects cross-origin, cannot downgrade real https, junk
values ignored.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(auth): consume the stored admin key only after a successful exchange
A remote-backend user upgrading with their backend unreachable lost the
only stored copy of OMNIVOICE_API_KEY: every migration path deleted the
durable ov_api_key BEFORE the session exchange settled, stranding them
until they recovered the key from the server box. Close the whole class:
- client.ts bootstrap: read the legacy key, exchange first, and remove
the durable copy only after the exchange succeeds; on failure the key
stays so the next launch retries the migration (auth gate still rises).
- authSession.ts exchangeApiKey: move removeLegacyMaster from before the
fetch to the cookie/bearer success paths — the key never coexists with
a live session, but a rejected or hung exchange no longer consumes it.
- remoteBackendProbe.ts configuredRemoteBackend: stop wiping the key on
every app mount.
- RemoteBackendPanel: a connection test or an aborted save no longer
wipes the pending key; only disabling the remote backend discards it.
- prefKeys.js: ov_api_key moves from PREF_KEYS to PRESERVED_KEYS —
factory reset preserves the pending connection credential exactly like
ov_backend_url; the successful migration is what deletes it.
Tighten the credential-hygiene static guard to match: it accepted
sessionStorage.setItem('ov_api_key', …) — the exact class it exists to
close. The guard now flags .setItem(<master key>) on any storage
receiver, quote style, or injected-store alias, with a self-test pinning
what it catches and what stays legal.
Fail-before/pass-after regression tests: backend unreachable retains the
key and the next bootstrap retries it; a successful exchange removes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(auth): make session validation occupancy-independent
* test(auth): catch optional master-key storage calls
* feat(docs): add PR control document for bultodepapas in VoiceStudio
* docs: keep the PR tracking board in the fork; credit the changelog line
The pr-control document is excellent process discipline, but it is the
contributor's own operational board (their inventory, their update
commands) — it lives naturally in their fork, and docs/agents/ here is
context every repo agent loads. Removed with appreciation; the changelog
line gains its contributor credit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: debpalash <4178343+debpalash@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refactor(launchpad): quieter, borderless design refresh
The launchpad carried decoration from an earlier direction: icon chips,
corner-hung count badges, a permanently visible filled arrow, uppercase
mono card titles, and a dotted stipple divider — plus a frame that had
been invisible since the app-wide border tokens were zeroed.
Rework it around what the borderless direction actually implies:
- Feature tiles get a whisper-faint surface instead of a dead frame, and
read as three bands (bare glyph + count / title + arrow / description).
`--card-hue` is spent sparingly — the glyph at rest, the surface, count
and arrow only once raised. Titles move to sans sentence case; counts
are plain tabular numerals. Lift softened 4px -> 2px, coloured glow ->
neutral shadow, plus an explicit focus ring and a staggered entrance.
- Hero drops the boxed "646" pill and the filled A/B-Compare button for
quiet type, with a hairline standing in for the separation.
- Section labels trade the dotted stipple for a single fading hairline;
rows are transparent until hover and reveal "Open" on hover/focus (it
stays in the DOM, so AT and keyboard always reach it).
- Hero, tiles, recent files, callout and project lists now share one
1180px column — previously only the top half was capped, so lists ran
edge-to-edge on a wide display while the deck stayed centred.
Two bugs found and fixed while doing it:
- Buttons that had `border border-solid border-transparent` removed fell
back to the UA default border and rendered a visible 1px outline. They
now carry `border-0` explicitly.
- `.lp-animate` used `animation-fill-mode: both`, so after the entrance
it kept owning `transform` — and animation-origin declarations outrank
normal ones, which silently killed the card hover lift. Now `backwards`,
which still holds the from-state through the stagger delay.
Also drops CSS the page has not rendered since #904: the cursor-spotlight
layer, the breath ring, and the per-card waveform strip.
Verified with headless renders at 1600/1280/940 and the empty state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(dictation): decode Wayland portal signals and show the capture pill
The GlobalShortcuts portal declares Activated/Deactivated as
(o session, s shortcut_id, t timestamp, a{sv} options). We decoded the
timestamp as u32, so zbus rejected every signal with
Signature mismatch: got `(osta{sv})`, expected `(osua{sv})`
and the press was dropped as an invalid signal. Registration succeeded
and the desktop even reported the bound chord back, so the hotkey looked
wired up while doing nothing at all — on every Wayland compositor, for
the whole life of the feature (#1490). Decode the 64-bit timestamp, and
keep the 32-bit spelling as a fallback so a non-conforming portal
degrades to working rather than to silence.
With presses arriving, the second half of the failure showed: nothing
had shown the widget window since it became a hidden recorder host, so a
capture ran with no pill on screen — and a mic or Accessibility failure
rendered into a window nobody could see. Add show_dictation_pill, which
bottom-centres the capsule on the monitor under the pointer and shows it
without taking focus (Windows keeps SW_SHOWNOACTIVATE so paste still
lands in the user's document), and call it from the widget for every
state but idle. Wayland denies clients their own placement, so the
compositor picks the spot there; the pill still appears.
dispatch_dictation_capture now logs whether a press was emitted or
queued — a press that reaches Rust and produces nothing was otherwise
indistinguishable from one the compositor never delivered.
Tests: portal signals decode at both timestamp widths (the 64-bit case
fails before this change with the exact production error); pill
placement centres, respects a second monitor's origin, and clamps rather
than going off-screen; the widget shows for a state needing the user,
stays hidden while idle, and never shows for a press that arrives while
dictation is disabled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: sync in-progress workspace changes
Uncommitted work already in the tree, checkpointed so the branch matches
the local machine:
- Remote GPU workers: join-from-the-app flow, one-time secrets, QR join
codes, a Compute control in the status bar, and the device-list
Workers panel (#1516)
- Model Catalogue workspace, with Settings pointing at it
- Settings sidebar search and keyboard navigation
- Demo assets for dubbing, dictation and voice design, plus the scripts
that render them
- Backend: validation-error handling, ASR request-path degradation, and
the accompanying tests
- CHANGELOG entries for the above and for the Wayland dictation fix
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(tests): follow Engines to the Model Catalogue, and green the sweep
- test_supertonic3 asserted the license gate points at "Settings" while
the engine now names Model Catalogue → Engines, which is where the
accept button actually lives. The assertion follows the move; what it
pins is unchanged — the hint must name a place the user can reach it.
- Carries the CJK allowlist entries for the rendered dub bundle (#1517)
and the regenerated route snapshot for /workers/agent (#1516), both of
which this branch inherits from the workspace sync.
- docs/install/linux.md: the dictation capsule is bottom-anchored
everywhere except Wayland, where the protocol gives applications no
say in their placement. Documented rather than left as a surprise
(CodeRabbit).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: stop a flaky dependency fetch from failing green runs
en-core-web-sm resolves to a direct GitHub release URL, and github.com
intermittently answers `http2 error: refused stream before processing
any application logic`. uv's own three retries all land within the same
few seconds and fail together, so the whole job dies on a dependency
that has nothing to do with the change under test — it cost #1518 and
#1517 an otherwise-green run tonight.
Two changes: back off between whole `uv sync` attempts, which is what
actually clears it, and pass --no-sync to the pytest steps. `uv run`
re-resolves the environment before running, so every test step was a
fresh chance to hit the same fetch even though the install step had
already synced — that is exactly how #1518 failed, in the isolated
backend/tests step, with all 5467 tests already passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: one retry seam for every uv sync, not just the job that failed last
en-core-web-sm resolves to a direct GitHub *release* URL rather than a
package index, and github.com intermittently answers `http2 error:
refused stream before processing any application logic`. uv's own
retries all land inside the same ~10 seconds and fail together, so a job
dies on a dependency unrelated to the change under test. Tonight that
cost four otherwise-green runs across #1515, #1517 and #1518 — and the
first fix only covered the Tests job, so the next failure simply moved
to Smoke (Linux), which syncs separately.
The fetch is per-job, so the fix has to be per-job: scripts/uv-sync-retry.sh
backs off between whole attempts (15s, 45s, 90s) and every workflow that
syncs now goes through it — ci.yml (tests + the platform matrix),
release.yml, security.yml, evals.yml. It still fails loudly after four
attempts, so a genuinely broken lockfile is not disguised as a flake.
The Tests job also lacked the UV_HTTP_TIMEOUT / UV_HTTP_RETRIES the smoke
matrix has always set, which is part of why it was the one that kept
dying; it has them now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(ci): pin the Intel-Mac contract by intent, not by command spelling
test_ci_verifies_intel_mac_as_the_documented_remote_only_host asserted
the literal line `run: uv sync --extra pockettts`, so routing every sync
through scripts/uv-sync-retry.sh read as a broken Intel-Mac contract. The
contract it exists to protect is that the pockettts extra installs ONLY
on backend_supported legs — which the regex now pins, while leaving how
the sync is invoked free to change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: keep every uv run out of the resolver, and bound the retry budget
CodeRabbit, #1517:
- `uv run` re-resolves before running, so the smoke suite, the
worker-artifact tests, the release test run and the eval run were each
a fresh chance to hit the flaky direct-URL fetch outside the retry
loop. All of them pass --no-sync now; the environment is already
synced by the step that owns the retries. security.yml's
`uv run --with pip-audit` is deliberately left alone — it layers an
ephemeral package rather than running the project's own tests.
- The retry count multiplied uv's own budget (UV_HTTP_RETRIES=5 with a
120 s timeout on the smoke matrix). Three attempts and 60 s of total
backoff outlast the refusals actually observed while staying well
inside the jobs' timeout-minutes.
- The Intel-Mac contract test pinned the smoke command literally too, so
--no-sync tripped it exactly like the sync line did. Same fix: assert
the contract (smoke runs only on backend_supported legs), not its
spelling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(workers): join from the app, share by QR, and a status-bar Compute control
Remote workers shipped with a hole in the middle: the control plane could
mint join codes, and on the other machine there was nothing to paste them
into. Becoming a worker meant launching with OMNIVOICE_WORKER_MODE and
OMNIVOICE_WORKER_TOKEN in the environment and relaunching — on the machine
that is usually the least convenient one to configure by hand.
Backend
- GET /workers/agent, POST /workers/agent/join, POST /workers/agent/enabled.
Join redeems a code and starts the agent live; no restart.
- Worker mode now persists in settings as well as the environment (env still
wins, and the panel is told so it can disable a switch it cannot honour),
and it is written only after a join that actually worked — a failed
enrolment must not have the app retrying on every launch.
- The endpoint carried by the redeemed code is remembered. Without that a
machine that joined from the UI came back up enrolled but with nowhere to
dial, and the only fix was OMNIVOICE_WORKER_ENDPOINT.
UI
- "Lend this machine's GPU": paste the code, Join. Once joined it offers a
switch rather than another code, because the pinned certificate survives.
- <OneTimeSecret/> renders join codes and connection strings as a QR next to
the text, with a live expiry countdown, and is used by both halves. QR
generation is best-effort: a string past the format's capacity still shows
the code and Copy, because losing the QR is a degraded share and losing the
only copy of a one-time secret is data loss.
- Status-bar Compute control: pick local or a machine, flip the feature, mint
a join code — without opening Settings. Absent entirely until the user has
opted in or enrolled something.
- Remote workers now reads as a device list: status dot, address, latency,
live task meter, resident models, last seen; housekeeping actions revealed
on hover; a three-step empty state.
- Approve is on the row. A worker could connect, sit there labelled "Not
approved" and never be usable, with no way out of it in the UI.
Fixes found on the way
- Status dots and menu surfaces in the GPU picker were painted from fixed
Tailwind palette classes (bg-emerald-400, text-amber-400, hover:bg-white/5),
so on Midnight or Catppuccin they showed Gruvbox colours next to the
theme's own. Both controls now paint from themed --color-* tokens, shared
in computeTarget.jsx along with the JSON wrapper all three copies duplicated.
- Button funnels every child into one <span>, so an icon passed as a child
renders glued to its label — the flex gap only applies to the `leading`
slot. Six buttons across these panels were affected.
- InboundNodePanel passed `variant="warning"` to Badge, which takes `tone`;
the "on your network" warning rendered as an ordinary neutral pill.
Docs updated in the same change (docs/remote-workers.md): the join flow, the
QR, the status-bar control, and the new environment variable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(changelog): stamp the remote-workers entries with their PR ref
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(workers): a join is not done until the control plane accepts it
Review findings on #1516:
- Greptile P1: `start()` only SCHEDULES the dial-out loop, so a control
plane that rejected this worker — expired token, wrong address, a
server that never answers — looked identical to a successful join. The
route persisted worker mode, reported success, and the machine retried
forever on every launch. The agent now signals first registration, and
join waits for it before persisting anything.
- CodeRabbit: a failed REJOIN left the machine unable to reconnect to the
control plane it was already serving, because pinning the new
certificate overwrites the old one on disk. Snapshot the pinned
certificate, endpoint and setting up front, and restore them (and the
running agent) when the join fails.
- CodeRabbit: join and the enable toggle awaited stop()/start() with no
exclusion, so two concurrent requests could interleave their pairs and
have `start()` return early — reporting success for a control plane it
never dialled. Both now hold one lifecycle lock.
- CodeRabbit: with OMNIVOICE_WORKER_MODE set, the toggle still started or
stopped the agent and wrote a setting the rest of the app ignores,
contradicting the env_pinned status it reports. It now answers 409 and
says which variable is in charge.
- CodeRabbit: the QR code kept encoding the previous secret until the new
one finished encoding, so the code on screen could disagree with the
text beside it.
CI: regenerated tests/fixtures/api_routes.txt for the three
/workers/agent routes.
Tests: a join the control plane never accepts is a 409 that persists
nothing and leaves no agent dialling; a failed rejoin restores the
previous certificate, endpoint and setting; an env-pinned machine
refuses the toggle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(workers): the environment pin governs joining too, not just the toggle
CodeRabbit, #1516:
- join_control_plane skipped the env_pinned guard set_agent_enabled
enforces, and joining is precisely what ENABLES worker mode: under
OMNIVOICE_WORKER_MODE it wrote a setting nothing consults, and with the
variable pinned off it handed back a machine that reported a successful
join and lent nothing. One shared guard now covers both routes.
- Two of the three rollback assertions could not fail before the fix
(nothing wrote those settings on the failure path). The test now pins
the behaviour only the rollback produces: the previous enrollment is
dialling again, rather than left stopped until someone notices.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared voice model's unload emptied the allocator caches and *then*
dropped the reference. That frees nothing: the weights are still reachable
when gc.collect() runs, empty_cache() only returns blocks the allocator
already considered free, and the reference drops a moment later into a cache
nothing will flush again. The unload logs success, the engine leaves the
registry, and nvidia-smi does not move.
Six modules open-coded the same two lines. Exactly one had them inverted --
OmniVoiceBackend.unload, which is the path the engine-registry idle sweep
reaches, which is the sweep a headless worker node runs. So every unload a
user could trigger from the UI worked, and the one that runs unattended on a
machine lending its GPU held 3.6 GB indefinitely. Found on hardware: the
sweep fired on schedule, logged "Released 1 idle engine(s)", and VRAM stayed
flat at 3656 MiB for the next two minutes.
Replace all six with model_manager.unload_shared_model(), which clears the
reference, drops the clone-prompt side cache, then frees -- in that order,
in one place. Two callers gain the side-cache drop they were missing
(/system/flush-memory and the shutdown path), which is the same defect one
step down: an unload that kept the encoded reference tensors belonging to the
model it had just released.
A source guard asserts nothing outside model_manager assigns the shared
reference, so the next caller cannot reintroduce the ordering. It caught the
sixth site while being written.
Also give the AudioSeal watermark models the bargain every other model in the
app already makes: they loaded on the first embed and stayed resident for the
life of the process. CPU-resident, so this is system RAM rather than VRAM,
and the machines that notice are the ones running batches.
The error text on a failing unload changes with the ordering. "Could not be
unloaded, retry after the current generation finishes" was accurate when the
cache flush ran first and aborted before the release; now the release has
already happened and only the flush can fail, so it says that instead of
sending the user to repeat work that is done.
Watching a ten-minute rule take effect means waiting ten minutes, so it tends
not to get watched. Both numbers are now env-tunable:
OMNIVOICE_ENGINE_IDLE_UNLOAD_SECONDS and OMNIVOICE_IDLE_SWEEP_SECONDS.
They are documented as a pair, because shortening only the threshold still
means waiting a full sweep interval to see it fire — which reads as a broken
sweep and sends you looking for a bug that is not there.
Unparseable values and anything below the floor are ignored with a warning
rather than honoured. A zero threshold would hand back a model the instant it
went idle and reload it for the very next request, which is worse than the
behaviour being tuned.
Two more found on hardware.
Disconnect ended the session and the panel redialled two seconds later, so the
log read disconnected and connected in the same breath and the button appeared
to do nothing. A kicked key now sits out for a minute — long enough that the
disconnect is real and the person notices, short enough that it is plainly not
a revocation, which stays a separate and permanent action. The docs now say
which of the two buttons does which.
Re-pasting a connection string for an already-connected machine saved the new
string and then short-circuited on the existing session, so a wrong key
reported success, kept running on the old connection, and only failed after a
restart — by which point nothing pointed back at the paste that caused it. The
live session is now torn down before the new one is dialled.
Wires the two transport halves into something a user can actually turn on.
Two independent switches, deliberately not one. "Accept connections" makes this
machine a node others dial; "saved connections" are the nodes this panel dials
out to. A workstation with a GPU that also drives jobs on a second box does
both, so neither implies the other.
Binding stays on 127.0.0.1 until someone explicitly widens it, and widening is
its own field rather than a flag riding along with the enable toggle. With no
encryption that boundary is the difference between a credential on one machine
and a credential on a network, so it is never crossed as a side effect. The
API reports `exposed` so the UI can say which side of it the user is on.
Saved nodes are redialled only after the control plane is up, since the
connector hands frames to its servicer. Failing to listen records the reason
rather than leaving the feature looking enabled while it quietly accepts
nothing.
Docs say plainly that this mode is unencrypted, that the connection string is a
password crossing the network in the clear, and that dial-out remains the
better choice when one machine is enough. The Security section no longer
implies its TLS guarantees cover both modes.
Remote workers connect outbound: the node dials the control plane, spends an
enrollment token, pins a certificate. That stays the default and is unchanged.
It is also structurally 1:1 — a worker process holds one endpoint, one pinned
certificate and one worker id — so a second person wanting the same GPU box has
to get shell access to it, repoint the start script at their own address and
restart, which disconnects whoever was using it. Sharing a GPU requires root on
it and evicts the incumbent, and no amount of UI work fixes that, because the
constraint is the shape of the connection.
This adds the other arrangement: the node listens, and any panel holding a key
connects to it, concurrently, with no shell access to the machine.
* NodeService mirrors WorkerService. Transport roles invert; message roles do
not — the node still sends WorkerMessage and the panel still sends
ServerMessage, so every state machine on both sides is untouched. Register
folds into the stream as the first exchange and reuses the existing
request/response messages rather than growing parallel ones.
* Keys are per panel, not per node. Revoking one person leaves everyone else
connected; a shared key would be revoked by nobody and leave no record of
who used it. Stored hashed, compared in constant time against every key so
the reply time is not an oracle, and the plaintext exists exactly once.
* Failed authentication is throttled per source address, so one stale
bookmark cannot lock out a different panel.
* A connection log records every attach, refusal and disconnect, and any
session can be kicked. That is what replaces per-job approval, which would
make a shared GPU unusable and train people to click yes.
* Artifacts invert too: the panel pushes inputs before assigning, and fetches
results after. The node stages both under one contained directory and
trusts no id or filename off the wire.
Runs in plaintext by deliberate decision, recorded with its accepted risk in
docs/adr/inbound-node-mode.md, and scoped there to LAN and self-hosted use —
never a fleet transport, which goal_v2 B2/B5.2 still require to dial out.
Off by default, and bound to 127.0.0.1 until someone explicitly widens it.
Add OrcaRouter to the Settings → LLM Providers registry (OpenAI-compatible
gateway, base_url https://api.orcarouter.ai/v1, default openai/gpt-5.5).
Env surface follows the existing provider pattern: ORCAROUTER_API_KEY /
ORCAROUTER_BASE_URL / ORCAROUTER_MODEL.
- registry: Provider entry after OpenRouter
- llm_backend: include OrcaRouter in the not-configured hint
- settings search: 'orcarouter' keyword on the LLM Providers category
- docs: list OrcaRouter in the supported-provider docs (docs-sync)
- test: registry test covers the new id
Co-Authored-By: Claude <noreply@anthropic.com>
Three conflicts, all additive on both sides — resolved by keeping both
rather than choosing, since either side's entries were real shipped work:
* CHANGELOG.md — remote-GPU entries against branding, IndexTTS 2.5 and
the recording-input work
* setup/download.py — the per-target progress reset against main's
active-install tracking; both belong in the same finally block
* docs/features.yaml — the remote-worker and model docs against
docs/branding.md
Backend 5349 passed, frontend 1871 passed. `bun install --frozen-lockfile`
reports no changes, so the Docker build sees the same tree CI does.
The remote-GPU line, verified on hardware rather than asserted.
**Dubbing renders on the worker.** dub_generate.py dispatches the coarse
`dub_segments` operation through the gateway, following the audiobook
pattern: per-unit local fallback after consecutive remote failures, one
aggregated notice rather than one per segment. A 40-minute dub that loses
its worker at segment 200 degrades instead of producing 200 error rows.
**An out-of-date worker is now refused by name.** This was the worst
defect in the plan and it was silent: an un-upgraded worker registered
cleanly, then ignored `inputs` and rendered a clone with NO reference
audio — returned as success. A plausible wrong result with nothing
anywhere to surface it. Workers now declare features, and one missing
them is turned away with the features named and `no task was run`.
Verified live: a worker one commit behind was correctly refused.
**"Offline" and "cannot run this" are different facts.** Asking a live
worker for an engine it lacks answered "is offline or cannot be reached.
Wake the selected worker" — while that worker reported ready, one free
slot and 3.6 ms latency. The user was sent to wake a machine that was
already awake. The scheduler now distinguishes absent from present-but-
incapable, and names the engine rather than the operation, because the
engine is the thing a user can install.
**An engine with no catalog entry is no longer hidden.** A `repo_ids`
non-emptiness check had been implemented as a runtime filter, so a worker
silently refused to advertise any engine lacking a models.yaml entry —
which is four registered engines, including CosyVoice. Users with those
already installed would have lost remote support with only a log line.
Empty `repo_ids` now means "not downloadable here", never "not runnable".
**And a script so this stops being done by hand.**
scripts/verify-remote-worker.sh runs the per-phase acceptance checks
against a live worker, non-destructively. Its preconditions are the
mistakes that cost the most time: exactly one listener on the control
port (two instances silently shared it), and never detecting the worker
with a pgrep pattern that matches the ssh shell running it.
Its first real run found the dubbing picker claiming remote placement.
That turned out to be the CHECK being stale, not the picker — the port
had landed since it was written. It now asserts self-consistency instead:
the picker may claim remote only for an operation the control plane
actually advertises as remotely producible, which cannot rot the next
time an op is ported.
Backend 5291 passed, frontend 1812 passed. Acceptance script: no
automated failures across Phases 4-8 on an RTX 4090. Four checks remain
MANUAL by design — true airplane mode, concurrent downloads, killing a
worker mid-audiobook, and the model-list UI — and are reported as
unverified rather than passed.
Five workstreams that finish the remote-GPU line, plus the test hole that
let a broken signature reach a commit.
**Downloads go through the normal path** (Phase 5). Rather than a second
remote-only route, the existing Models install flow became target-aware,
so a model landing on a worker uses the same code, the same progress
events and the same UI as a local one. Progress rows key on
(target, repo_id) — the aggregator keyed on bare repo_id, so the same
model downloading here and on a worker at once collapsed into one row
that told the user nothing true about either.
**Audiobooks render chapter by chapter on the worker** (Phase 8), with
per-chapter local fallback and ONE aggregated notice. The failure that
shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200
must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED.
Dictation is deliberately NOT ported — it runs ASR per utterance inside a
live WebSocket loop, and paying queue admission plus a round trip there
would spend the one thing that route is for.
**Dubbing stays local, and says so** (Phase 7). The coarse worker
operation is not finished, so the picker still reports dubbing as local
rather than showing a green remote chip over work this machine is doing.
What could not wait is the in-loop OOM retry: it sniffed the error string
and flushed the *local* CUDA cache, which under remote execution is the
wrong machine's GPU entirely. That is fixed now, before the path that
would have exercised it exists.
**Two instances can no longer share the control plane.** A second
VoiceStudio silently bound the same worker port and coexisted, so remote
workers landed on whichever process won the race — a session that
registers with one instance and appears dead to the other. This produced
hours of misdiagnosis during hardware testing and would hit any user with
the app open twice. The second instance now keeps running locally and
explains the conflict instead of quietly competing.
**And the hole that allowed all this to be missable.** gpu_gateway called
Scheduler.submit(pinned_worker_id=...) one commit before that parameter
existed. Every remote generation raised TypeError; 5236 tests passed
anyway, because nothing exercised the gateway against the real scheduler.
tests/test_gpu_gateway_scheduler_contract.py now runs that path for real
and binds every gateway→dependency call signature. Verified by renaming
the parameter away and watching both tests fail with the original error.
Gallery previews also fall back to a local render when a downloaded clip
cannot be decoded, rather than yielding silence.
Backend 5274 passed, frontend 1812 passed.
Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and
its artifact transport have been proven on a real GPU.
Three phases that only make sense together: a job that names a worker,
a worker that reports honestly what it can actually run, and the small
defects that made both lie.
**Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places
that choose a worker — `eligible_workers` and `select_worker` build
independent lists, so applying it to one silently leaked work onto
whichever machine was least busy. The pin persists across a restart via
an additive column, deliberately not alembic (justified in the code, per
the precedent already in db.py): quitting mid-render used to drop it
without a word. `max_attempts=1` was rejected as the mechanism — it makes
the FIRST failure terminal, including the penalty-free ones a stale
advisory view produces routinely.
Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero
callers, so cancelling released the slot while the GPU thread kept
running, and a late result could resurrect the task as COMPLETED —
`commit_result` assigned that state directly, bypassing the transition
table where CANCELLED is terminal by construction.
**Honest capabilities** (Phase 4). A worker now probes whether weights
are actually present, and a job stops BEFORE dispatch with a typed 409
naming the model and the machine, instead of failing mid-task. The probe
fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed
clone outside the HF layout, so only a positive "absent" refuses.
Refusing an engine that works today would break the compatibility
promise. `pool.supports` deliberately still ignores `downloaded` — had it
not, the scheduler would drop the worker and answer with a terminal
NO_CAPABLE_WORKER, which tells the user to check their install when the
truth is one download away. The frontend no longer offers "Report this
bug" for that state; it offers the download.
Catalog tags resolve against the TARGET's OS/arch/backend, not this
machine's. From a Mac control plane, a CUDA worker's model list was
showing the mlx-community repos it cannot run and hiding the ones it
needs.
**And the quiet ones** (Phase 0 leftovers): a model's human label rides
its own proto field so renaming it cannot orphan breaker history; an
empty model_id no longer forks the capacity slot key into two slots for
one model; the idle sweep cannot evict an engine out from under a live
LOCAL render.
Verified on real hardware, which is the only verification that has ever
caught anything here: 2025 characters, default settings, routed to an
RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s
of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through
the artifact path rather than the control stream.
Backend 5259 passed, frontend 1808 passed.
Every enrolled worker sat at connected=False against a healthy control
plane, and the control-plane log showed no Register call arriving at all.
The worker's own log said only "connecting", then nothing.
The cause was on our side of the handshake. The client sends an HTTP/2
ping every 25 s to keep its long-lived Control RPC alive through NAT —
an interval the control plane itself configures. But the server kept
gRPC's default enforcement policy, which permits two idle pings and then
answers ENHANCE_YOUR_CALM:
GOAWAY received; Error code: 11; Debug Text: too_many_pings
So the control plane hung up on every worker for obeying the keepalive
the control plane asked for. Idle workers were hit hardest, because a
session with no traffic is exactly the case the ping exists to protect.
Fixed by accepting the interval this protocol configures: a 20 s minimum
still rate-limits an abusive peer, while removing the idle-ping count
ceiling stops a healthy session dying of its own liveness mechanism.
This is a whole-fleet fix, not a per-enrollment one.
Worth recording what this was NOT, because it looked exactly like it:
TLS pin-on-first-use was the obvious suspect, since a control plane that
regenerated its certificate on restart would strand every enrolled
worker with no useful error. Disproved — the live certificate
fingerprint and the pinned copy on the remote worker match exactly, and
the certificate survives restarts. Enrollment was never involved.
Verified live against a remote worker: the session now establishes where
previously nothing reached the server. It is not yet stable — it drops
after ~17 s and advertises zero engines — but that is a separate defect
being tracked on its own, and this fix is a prerequisite for reaching it.
debpalash