Selecting a remote worker repainted a badge and nothing else. The cause was
not subtle: `scheduler.submit` had no production caller, and `routing.decide()`
was read only by the status endpoint that paints the header. Remote execution
was a complete, tested pipeline with no producer at its head.
This adds the producer and fixes the defects that made the pipeline unable to
carry a real job:
- Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated,
**development-only** until the gateway lands) and `Scheduler.wait`, backed by
per-task futures rather than the unregisterable `on_change` listener list.
- Every task over two minutes died. No worker ever sent `TaskProgress`, so the
120s progress lease expired mid-render — including during the cold model
load, which happens after `TaskStarted`. Workers now report progress and
emit a keepalive, bounded by the phase's absolute budget so it renews the
lease without deleting the only enforced bound in the system.
- The executor rebuilt its engine per task (`return cls()`), so every job paid
a cold load. Engines now share one instance cache with the router, resolved
by the assignment's engine — never `get_active_tts_backend()`, which returns
the worker machine's own Settings preference and would silently run the
wrong engine.
- One lease expiry took a worker offline permanently: parked slots were never
reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled
against the worker's own load report — at a ceiling of one the only task such
a worker can report is the wedged one, so "busy" would drop the park and the
next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190).
- A worker that dropped and reconnected mid-render had every liveness frame
discarded: task frames were fenced on the live session epoch, which bumps on
every reconnect, while the worker echoes the ref stamped at dispatch. The
control plane then expired a task whose GPU was still rendering, and swallowed
the failure report when it went wrong. Fenced per attempt instead.
- A result from one worker could commit another's task, after which the owner's
real delivery arrived as a duplicate and its audio was discarded. "Unknown
attempt" and "another worker's attempt" are no longer the same answer.
- An oversized result was a poison pill, re-sent identically on every reconnect
and permanently disconnecting the worker. It is now a terminal
`RESULT_TOO_LARGE`, which is also classified — it was falling through to
TRANSIENT and retrying a re-render that could never fit.
- `_store_inline` joined the artifact directory with worker-supplied ids, and
`os.path.join` discards its prefix on an absolute component. Paths are now
minted control-plane-side and resolved through `core.path_security`.
- Remote synthesis bypassed `mark_synthetic`, and the guard that exists to
catch exactly that walked only `backend/api` and `backend/services` — so it
stayed green while a fourth unmarked producer shipped. Marking moved to the
worker's tensor stage; the guard now walks `backend/worker` too.
Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the
gallery no longer needs a GPU or a downloaded model. The manifest is verified
against the updater's release key already baked into the binary; a fresh
install hears voices without downloading 2.4GB first, and everything falls back
to local rendering when the gallery is unreachable.
Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090
returned 105.94s of 24kHz audio in 23.9s, committed and served from the
artifact store.
Not yet done, and deliberately not claimed: the keepalive fix cannot be
exercised end-to-end on fast hardware, because any job long enough to reach the
120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has
to land first. Pinning to the worker the user chose is also still absent, so
"Remote" reaches a remote GPU but not necessarily the one on the badge.
Remote workers was nested under Sharing, which reads backwards: everything
in Sharing is about letting something else reach THIS machine (a remote
backend, an MCP client, a share PIN), while remote workers sends work OUT
to machines you own. It is now its own System entry.
Docs-sync: every "Settings → Sharing → Remote workers" reference is
updated — the guide, the changelog, the two API error messages that tell a
user where to generate a token, and the agent's not-enrolled error.
Also ignores remote/ (local goal docs, review briefs, council reports) and
repoints the code comments that cited remote/goal_v2.md at the shipped
docs/remote-workers.md, so no committed file references a path that is not
in the repo.
Send individual jobs to GPUs on your other machines while everything else
stays local. Opt-in, off by default: with the toggle off there is no
listening socket, no certificate and no background loop.
Design follows remote/goal_v2.md, the council-revised goal doc. The
decisions that shaped the code, and why:
* A disconnect is an unknown outcome, not a failure. The original design
reassigned on disconnect while also describing the case where the worker
had already finished — following both guarantees duplicate execution. An
attempt now holds a grace window; a worker returning inside it commits
its result and no second attempt is ever made.
* At-least-once execution, exactly-once result commit. The result is
persisted BEFORE it is acknowledged, so a crash between the two cannot
silently lose a finished render.
* Deadlines are phased (accept -> model load -> execute -> deliver) and
liveness is a progress lease. The old fixed 30s execution budget was two
orders of magnitude below what this product actually does; silence is
the failure signal, not slowness.
* Capacity is derived from free VRAM, never configured: a static value
corrupts output under torch.compile thread affinity (#315) and aborts
the process on small cards (#567).
* A circuit breaker replaces the reliability-score/quarantine machinery,
which had no recovery path (no probation workload exists in a TTS
product) and penalised consumer networks for existing.
* Identity is a keypair the worker generates and never sends. A
server-assigned id is a name, not an authenticator, so revocation of one
would be theatre. Enrollment tokens are single-use and carry the control
plane's certificate fingerprint for pin-on-first-use.
Adds the domain core, scheduler, durable task store, gRPC transport,
worker agent, management API, Settings panel, and docs. Protobuf reserves
the tenant/trace/usage fields a hosted control plane would need, since
adding them later means upgrading a whole fleet.
Includes tests for the failure paths that matter: duplicate delivery,
stale-session fencing, reconnect reconciliation, grace expiry, breaker
attribution, and a real end-to-end TLS round trip.
Close unauthenticated remote mutation paths and keep filesystem destinations behind one-shot Tauri capabilities. CodeRabbit and Greptile findings were fixed on-branch; all review threads are resolved. Full CI, Security, Rust, and cross-platform smoke checks are green.
The About page reported 'Compute device: cpu / GPU active: no / VRAM 0.00 GB' on a machine with a working GPU. Every line was true and none was usable — it is also exactly what a machine with no GPU at all reports, so the report could not distinguish a driver that isn't loaded from a container that cannot open the device from a ROCm older than the card.
The probe already knew all of it; torch.cuda.is_available() returning False simply produced no note. Each cause now reads differently: a missing device node names the --device flags, a permissions failure names --group-add and how to find the host's real render/video GIDs (copied numbers are the most common way this ends up on CPU in Docker), a card newer than the shipped ROCm points at rocminfo, an HSA_OVERRIDE_GFX_VERSION that is doing more harm than good is named first because it is both likelier and cheaper to test, and an unreachable NVIDIA driver gets its own advice.
The probe never diagnoses from a measurement it did not complete: when torch.cuda.is_available() itself raises, the exception is reported and no device findings are asserted beside it. Metadata access that raises is contained too — this runs on the path whose whole job is to explain a failure, so it cannot become one.
Every subprocess engine confirms a candidate interpreter by spawning it and importing the engine package. For IndexTTS that is 'import indextts.infer_v2', which pulls in torch and transformers — seconds with a warm page cache, tens of seconds on a first run, a spinning disk, a network share, or Windows with real-time AV scanning every DLL.
The bound was 10s (15s for three peers), and elapsing it was treated as a negative: the candidate was discarded exactly as if the import had raised. A working OMNIVOICE_INDEXTTS_DIR install was reported as 'IndexTTS-2 is not installed', or fell through into the lazy bootstrap and reinstalled over a working clone. Only successful resolution was memoised, so every retry re-ran the probe and failed identically — which is why all three reported repro paths look like one bug.
A timeout is the absence of evidence, not evidence of breakage. The probe is now tri-state: yes (imported), no (ran and failed), unproven (did not finish). An unproven candidate is kept as a fallback and used only after every candidate has had its chance, so a wedged user clone cannot shadow a healthy bootstrapped venv. If an unproven venv really is broken it now fails at the sidecar handshake with a real error rather than a confident lie about the install.
Fixed as a class: backend/engines/_venv_probe.py replaces the drifted copy in each of the four bootstraps, and the bound is tunable per engine, defaulting to 60s. Zero and negative values are ignored — an unbounded probe would let one wedged candidate hang engine resolution forever.
Reported with a precise root cause by @OracleNightmare. (#1414)
Two bugs from the v0.4.2 rename sweep:
- The lazy model import was rewritten to `from omnivoice.models.omnivoice import VoiceStudio`, a class the library does not export. ImportError is not ModuleNotFoundError, so the #564 source fallback never caught it and /generate 500'd on every default-engine request. The class keeps its library name — it is a checkpoint-referenced identifier, not branding.
- alembic resolved a bare relative script_location against the process cwd, and the desktop shell launches the backend from frontend/src-tauri, so a pending migration killed startup. script_location and prepend_sys_path are now anchored with %(here)s, with path_separator = os so Windows drive letters and paths with spaces survive. alembic floor raised to >=1.16.
Regression tests verified fail-before/pass-after for both.
Portable mode put everything in OmniVoiceStudio-Data beside the app — the only storage row on the setup screen you could look at but not change, while installed mode had a picker for all three of its directories.
The pin had a reason: portable_base() is computed from the executable's location and never read from config, which is what makes a portable install self-discovering. A user-chosen folder breaks that, because the only record of the location would live inside the folder being located.
So it is recorded somewhere findable, in order: a portable.path marker beside the app; then portableDir in the per-user config for app folders that are read-only; then the historical default, so existing portable installs resolve byte-identically.
The marker stores a RELATIVE path whenever the folder sits inside the app's own directory — the USB-stick case portable mode exists for, where app and folder move as a unit and the mount path is free to change. Anywhere else only an absolute path can be stored, and the setup screen says the install is tied to it rather than promising portability it cannot keep.
Two traps worth naming, both caught in review: the per-user fallback would recurse forever through load_config (config_path to portable_config_file to portable_base), so it reads the platform config file directly; and clear_portable_dir was reading through that same chain, silently failing to clear the machine record so the old folder kept winning after the user chose the default.
This also retires the greyed-out Portable option after a default Program Files install (#766) — you point it at a writable disk and get the machine-bound variant.
Rust 109 passed, frontend 1705 passed. All 21 locales in lockstep; docs/install/windows.md rewritten.
The dictation hotkey could leave a blank dark square stuck on the desktop with no way to dismiss it. Three defects compounded: the tray listener's effect depended on [state], so it detached across an await on every state change and a press landing in that gap was lost; an idle pill renders null, so the window Rust had already shown was empty; and the opaque chrome background made that empty window a hard-edged square. Nothing could hide it — dismiss() is only reachable from the X button, Esc, or a post-session timer, none of which exist for a session that never started.
Fixed at the invariant rather than the call sites: the listener subscribes once for the component's lifetime, the widget window's chrome background is transparent, and an idle-but-visible window reconciles itself to hidden. The reconcile is polled (a dropped press changes no React state, so there is nothing to key an effect off) and aborts if its effect is torn down mid-check, so it can never hide a dictation that has just started.
Also in scope:
- The rename sweep had repointed three data-dir literals at a brand-named directory that does not exist, so smoke-test.sh verified a directory the backend never writes and desktop-prod.sh silently stopped clearing backend state on Windows. Both invisible on macOS, where they are usually run. A guard test now pins the assignments specifically.
- The dictation model picker's download sizes were wrong for all seven models, in both directions — Parakeet TDT v3 (the recommended default) understated 180 MB against an actual 670 MB, while the low-RAM fallbacks were overstated threefold, discouraging exactly the choice that would have helped. Measured from the published repos and pinned by a test.
- The 0.6B Parakeet models now decode on more threads, capped by host cores and still overridable.
- uninstall.ps1 gained a UTF-8 BOM (Windows PowerShell 5.1 mis-decodes its non-ASCII output without one), and sponsor.yml lost its last OmniVoice references.
Scaffolds the per-repo agent configuration the engineering skills assume: where issues live (GitHub Issues via the gh CLI), the five canonical triage labels, and the single-context domain-doc layout. Adds docs/agents/{issue-tracker,triage-labels,domain}.md and an '## Agent skills' block in both CLAUDE.md and AGENTS.md so the two stay in sync.
Documentation only — no runtime code is touched.
Review fixes: CONTEXT.md is now described as 'read it when it exists' rather than stating its absence as permanent, and both directory-tree fences carry a language identifier (markdownlint MD040).
CTranslate2 (the engine under WhisperX and faster-whisper) requires cuDNN 8 while torch ships cuDNN 9. When the side-loaded cuDNN 8 is absent it does not raise — it __fastfail()s, killing the whole backend with 0xC0000409 and no traceback, so the shell restarts it and the next attempt dies the same way.
The defect was that the backend computed the answer and discarded it: the preload passed silently on a missing directory and on every OSError, then called into a library that treats the same condition as fatal. The answer is now kept, and the CTranslate2 engines report themselves unavailable so auto-detect falls through to pytorch-whisper.
Two more instances of the same class went with it: the crash-isolated ASR sidecar is a child process that never preloaded at all (failing every transcribe, quietly), and the preload only searched <project root>/.venv, missing any other interpreter.
Conservative by design — a false positive costs WhisperX's forced alignment, so ROCm is excluded via torch.version.hip before cuda.is_available(), which is True on HIP builds.
14 regression tests; backend suite 4374 passed.
Renames what users see. The app, the installers, the window title, the
docs and all 21 locales now say VoiceStudio, with "(previously
OmniVoice-Studio)" noted near the title of each doc surface so people
recognise it.
Deliberately NOT renamed, because renaming any of them silently breaks
an existing install — there is no legacy-path fallback anywhere in this
codebase:
- bundle identifier com.debpalash.omnivoice-studio (MSI UpgradeCode,
macOS TCC grants, managed venv, WebView localStorage, the
single-instance lock)
- data directories OmniVoice / .omnivoice and omnivoice.db
- the ~150 OMNIVOICE_* environment variables
- the X-OmniVoice-* HTTP headers (a wire protocol)
- the published Docker image paths
- the OmniVoice ENGINE, which is a model name and not this product
tests/test_identity_paths_survive_the_rename.py pins every one of those
so a future well-meaning sweep cannot orphan a user's library.
Linux .deb users install a new package name and should apt remove
omnivoice-studio; that note is in the changelog.
The repository was renamed. 724 references across 59 files now point at the new URL — README badges, docs, install guides, the updater's releases API call, CONTRIBUTING, the Colab link and the probe harness. GitHub redirects the old URLs, so nothing was broken in the meantime.
Deliberately NOT renamed, because each breaks something on a user's machine: the Tauri bundle identifier (the path to every existing user's data), /usr/lib/omnivoice-studio and the compose container names, and the published Docker image paths.
The image path needed a code change to STAY still: docker.yml derived it from github.repository, so the next build would have published to ghcr.io/debpalash/voicestudio while Docker Hub, a hardcoded literal, stayed put — everyone pulling the documented GHCR path would have kept receiving the last pre-rename image forever. It is now pinned, with a test that fails if it ever derives from the repo name again.
Also makes the probe's repo-name assertion shape-based: it hardcoded the old name and failed on every PR after the rename while the code it tests worked perfectly.
A meticulous report from an LXC/CPU-only source install surfaced three real defects: the build script deleted the libggml shared libraries a dynamically-linked build needs (first spawn died with exit 127), the hardcoded 120s per-spawn kill switch reaped legitimate CPU-only renders, and OMNIVOICE_ALLOWED_ORIGINS — the only fix for cross-origin browser access — was documented nowhere.
All platform branches of scripts/build-omnivoice-tts.sh now copy the shared libs next to the binary, the CI artifact glob uploads them, and the backend puts bin/ on the loader path for every spawn of the engine binary. The timeout defaults to 600s (above the pool guard's well-diagnosed 300s deadline), is tunable via OMNIVOICE_GGUF_GENERATE_TIMEOUT_S with non-finite values rejected, and the timeout error names the knob. CORS documented in api-auth.md with a pointer from remote-gpu.md. Regression tests pin the spawn-env rule, the per-branch copy rule, the artifact glob, and the timeout behavior.
The TRANSFORMERS_IMPORT hint and the ASR pipeline error told users to reinstall torch + torchaudio + transformers. torchvision — the package whose ABI mismatch actually produces this exact lazy-import wording (#1357's torchvision::nms, wrapped into "Could not import module 'AutoFeatureExtractor'") — was the one package the advice omitted. Following it to the letter left the broken package untouched (#1376).
Both surfaces now name the mismatch as a cause and prescribe the pinned reinstall with literal versions (desktop installs ship no deploy/, so the constraint-file form fails there) targeting the venv explicitly. A lockstep test asserts the exact command on every advice surface against deploy/torch-constraints.txt, so a pin bump stays red until the advice matches. Docs gain the same-wording-different-cause section (1a-bis).
Palash Debnath