d91beef0fd314250d8d9b94de86dfea019a8bd96
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Palash Debnath
|
c870f794ed |
fix(engines): give every sidecar a private fd for its frames (#1428)
Protect all nine sidecar frame channels from library stdout noise and pin the complete sidecar manifest in regression coverage.\n\nCloses #1428. Thanks @1335-Group for the diagnosis and tested fix. |
||
|
Palash DebnathandClaude Opus 4.8
|
e862f0faf0 |
feat(asr): crash-isolated faster-whisper subprocess backend (Wave 4.2) (#393)
* feat(asr): crash-isolated faster-whisper subprocess backend (Wave 4.2) Native ASR engines (faster-whisper / CTranslate2) can segfault on GPU teardown — a process-level crash that kills the whole backend. Running the engine in a child process turns that into a failed job: the sidecar dies, the parent raises a decorated error (engine id + device), and the next request respawns a fresh sidecar. - services/subprocess_asr.py: SubprocessASRBackend reuses SubprocessBackend's wire protocol + lifecycle — including respawn-on-dead-process (_spawn relaunches when the child isn't alive) and GPU-slot acquire/release — adding a 'transcribe' op (the TTS 'generate' surface is stubbed). IsolatedFasterWhisperBackend wraps faster-whisper using the PARENT venv (already a dep — only the process boundary is new); opt-in via OMNIVOICE_ASR_BACKEND=faster-whisper-isolated. - engines/_asr_sidecar/main.py: the faster-whisper runner (stdlib wire protocol; torch/CT2 import lazily so the ready handshake fits the timeout). - engines/_echo/main.py: a 'transcribe' echo op so the round-trip + crash recovery are testable without a real engine. - asr_backend._REGISTRY is now a lazy dict (mirrors the TTS registry) so the isolated backend lists/resolves without importing the subprocess stack unless selected. Tests (echo sidecar, stdlib-only): round-trip, single long-lived sidecar across calls, crash-mid-transcribe → decorated error + backend healthy + next call respawns, registry exposure, generate-not-supported. Spec 7 / parity program Wave 4.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(asr): deterministic crash test + drift marker for lazy ASR registry (Wave 4.2 CI) CI surfaced two issues: - The echo crash test relied on the crash-AFTER-reply hook, whose reply may still reach the parent (timing-dependent) — and a leaked OMNIVOICE_ECHO_CRASH from a sibling subprocess test poisoned the non-crash tests. Fix: a deterministic OMNIVOICE_ECHO_CRASH_NO_REPLY hook that exits BEFORE replying (guaranteed dead pipe → decorated error), and the asr fixture clears both crash envs so the round-trip/two-call tests can't inherit a leak. - check-docs-drift's _ASR_MARKER didn't match the new lazy registry line (_LazyASRRegistry({); updated the marker + the self-test fixture. Verified the no-reply crash hook by driving the sidecar directly (reply=None, exit 1); drift self-test + real-repo check green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(asr): allowlist the 'segments' op so transcribe replies aren't dropped (Wave 4.2 CI) The parent's PARENT_INBOUND_OPS frozenset gated inbound sidecar frames but never included 'segments' — the ASR transcribe reply op. _recv() dropped the frame as disallowed, tail-recursed, hit EOF, and returned None, so every transcribe surfaced as a bogus 'sidecar crashed mid-transcription'. TTS ('audio') was allowlisted; ASR ('segments') was missed. Add it (and list 'transcribe' in the informational SIDECAR_INBOUND_OPS), update the exact-shape allowlist test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Palash DebnathandClaude Opus 4.7
|
0fc5ea6cf3 |
Phase 2 Plan 02-01: SubprocessBackend primitive (Wave 1 of Phase 2) (#97)
* Phase 2 Plan 02-01: SubprocessBackend primitive + echo sidecar + ENGINE-05 wrap
Lands the durable SubprocessBackend primitive — the architectural keystone
that Plans 02-03 (IndexTTS migration), Phase 3 (Supertonic-3), and
Phase 4 (GGUF / Singing) plug into.
Files added:
- backend/services/subprocess_backend.py — base class owning spawn,
shutdown, _send/_recv (length-prefixed JSON), GPU-slot acquire-release,
atexit teardown, stderr drain, op allowlist (T-02-04), and 64 MB
frame cap (T-02-01). No multiprocessing — subprocess.Popen
exclusively so subclasses can target a *different* venv's interpreter
(Locked Decision D4 / Pitfall 1).
- backend/engines/_echo/main.py — permanent CI regression sidecar.
Stdlib-only, runs under the parent's sys.executable. Implements
ready/ping-pong/synthesize/shutdown plus test-only probe_env and
emit_unknown ops for env-forwarding and op-allowlist tests. DO NOT
DELETE — the round-trip test depends on this file.
- tests/backend/services/test_subprocess_backend.py — 13 tests:
round-trip, health_check, no-zombie, shutdown idempotency, env
forwarding (HF_TOKEN/HF_HOME/HF_ENDPOINT/HF_HUB_CACHE), oversize
frame, short read, op-allowlist drop, op-allowlist constant shape,
sidecar-crash recovery, no-multiprocessing grep gate, MAX_FRAME_BYTES.
- tests/backend/services/test_tts_backend_registry.py — 6 tests for
list_backends() resilience + shape + isolation_mode + last_error
caching + existing-engines preservation + install_hint passthrough.
Files modified:
- backend/services/tts_backend.py:
* Adds module-level _LAST_ERRORS dict for ENGINE-06.
* Rewrites list_backends() to wrap each is_available() in try/except
so one broken engine cannot blank the picker (ENGINE-05).
* Adds last_error + isolation_mode keys to each response entry
(ENGINE-06 UI in Plan 02-04 consumes via the same /engines route).
* Uses a duck-typed _is_subprocess_isolated marker rather than
issubclass(cls, SubprocessBackend) because test fixtures (token
resolver suite) purge sys.modules["services"] between tests and the
re-imported SubprocessBackend would be a different class object.
Threat-model mitigations (Plan 02-01 frontmatter):
T-02-01 DoS via length-prefix → MAX_FRAME_BYTES = 64 * 1024 * 1024
T-02-02 GPU slot leak on sidecar death → try/finally in generate
T-02-03 token bytes in stderr → drained via parent logger
(HFTokenRedactor from Phase 1 already on root)
T-02-04 unknown ops from compromised sidecar → PARENT_INBOUND_OPS
allowlist, unknown frames logged and dropped
T-02-05 Tauri group-kill scope → start_new_session=True on Unix /
CREATE_NEW_PROCESS_GROUP on Windows
Verification:
- 337 passed, 6 skipped, 12 xfailed, 1 xpassed (full suite,
`uv run pytest tests/ --ignore=tests/manual`)
- All 19 new tests pass on macOS Apple Silicon
- Smoke tests still pass: `uv run pytest tests/smoke/ -q` → 4 passed
- SoniTranslate untouched (D1 locked decision)
- Zero new Python dependencies
Closes part of ENGINE-01 + ENGINE-05.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(02-01): plan summary — public API, invariants, deviations
Documents the SubprocessBackend public API so Plan 02-03 (IndexTTS) and
Phase 3 (Supertonic-3) authors don't need to re-read the source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|