fix(memory): idle-release the dictation ASR — the real cause of the 16 GB OOM deaths (#1104)

Four "Can't reach the local OmniVoice backend" reports (#1076/#1092/#1093/#1101)
all died at the same moment: during a generate, on a 16 GB machine. Measuring it
(phys_footprint, not RSS — RSS badly undercounts MPS unified memory) showed the
generate was never the problem: it costs ~116 MB. The problem is the BASELINE —
the backend sits at ~6.2 GB *idle*: TTS 3.8 GB plus ~2 GB of warm capture ASR.

The TTS model has always been idle-unloaded (model_manager.idle_worker). The
capture/dictation ASR singleton never was — one dictation warmed it and it
stayed resident for the life of the process. So the app dutifully freed 3.8 GB
of TTS while silently holding 2 GB of ASR forever, and on a 16 GB Mac that
baseline plus the app plus macOS is enough for the OS to kill the backend
mid-generate. That death surfaces as the "can't reach the backend" error — the
class #1102 made honest and this fixes at the source.

- asr_backend.release_idle_capture_backend(idle_s): unloads the warm capture
  singleton once it's been unused that long; no-op under a live lease, when
  nothing's loaded, or when recently used; never raises (idle_worker calls it
  on a loop).
- capture_lease(): pins the singleton for a live dictation session's whole life
  (the stream holds the backend without re-resolving it, so the reaper must not
  unload the model mid-sentence); wrapped around both sherpa handlers in
  capture_ws. Releasing restarts the idle clock.
- Both capture getters stamp _touch_capture() so any handout resets the clock.
- idle_worker runs the reaper each tick with the same idle timeout the TTS model
  uses, then free_vram().

Cost: a ~1.4 s model re-warm on the next dictation after a full idle timeout —
the same bargain the TTS model already makes. 8 new unit tests (releases when
idle / never while leased / never when recently used / lease released on raise /
nested refcount / failing unload still drops the ref / no-op when empty).
Backend suite 2905 passed; verified live against the running backend.

Fixes the crash class behind #1076 #1092 #1093 #1101

Co-authored-by: mergetest <nizam4103@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -6,6 +6,12 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
 Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
 The bundled TTS model package (`pyproject.toml`) is versioned independently.
 
+## [Unreleased]
+
+### Fixed
+
+- **The backend no longer sits on ~2 GB of idle dictation model — the real reason it was being killed on 16 GB Macs.** Four reports of *"Can't reach the local OmniVoice backend"* (#1076, #1092, #1093, #1101) all died at the same moment: during a generate, on a 16 GB machine. Measuring it showed the generate was never the problem — it costs about 116 MB. The problem was the **baseline**: the backend sat at **~6.2 GB even while idle**. The TTS model has always been unloaded after an idle timeout, but the speech-recognition model used for dictation never was — so once you dictated a single time, ~2 GB stayed resident for as long as the app ran. On a 16 GB Mac, that plus the app, macOS, and your other programs is enough for the system to run out of memory and kill the backend, which surfaced as the "can't reach the backend" error. Dictation's model now gets the same idle release the TTS model already had, handing that memory back. The only cost is a ~1.4-second re-warm on your next dictation after a long pause, and a live dictation session is pinned so nothing is ever unloaded mid-sentence.
+
 ## [0.3.20] — 2026-07-12
 
 The follow-through release. v0.3.19 promised that "Can't reach the local OmniVoice backend" would stop firing while the backend was merely restarting — and then a user hit it anyway, on 0.3.19, because the fix had a race in it. That's closed properly here. Uninstalling also stopped being a thing only maintainers could do: it's now a button in the app, where the person who asked for it can actually reach it.

backend/api/routers/capture_ws.py

@@ -158,9 +158,16 @@ async def ws_transcribe(websocket: WebSocket):
     # legacy Whisper/WebM path, byte-for-byte unchanged.
     spec = _select_sherpa_spec(websocket)
     if spec is not None:
-        from services.asr_backend import SherpaDictationBackend
+        from services.asr_backend import SherpaDictationBackend, capture_lease
         ok, _reason = SherpaDictationBackend.is_available()
         if ok:
+            # A live session holds the shared capture backend for its whole
+            # lifetime without ever re-resolving it, so the idle reaper
+            # (#1101 class) must not unload the model out from under it — even
+            # if the user leaves the mic open, silent, past the idle timeout.
+            # The lease pins it for exactly this window and restarts the idle
+            # clock on the way out.
+            with capture_lease():
                 if spec.streaming:
                     await _run_sherpa_streaming(websocket, spec)
                 else:

backend/services/asr_backend.py

@@ -27,7 +27,9 @@ import asyncio
 import logging
 import os
 import re
+import contextlib
 import threading
+import time
 from abc import ABC, abstractmethod
 from collections import OrderedDict
 from typing import Optional
@@ -2238,6 +2240,74 @@ _capture_backend_key: str | None = None
 # check-then-build must be atomic to avoid two threads each building a model.
 _capture_backend_lock = threading.Lock()
 
+# ── Idle release of the warm capture/dictation ASR (#1101 class) ────────────
+#
+# The TTS model has always been idle-unloaded (model_manager.idle_worker), but
+# the capture ASR singleton above was not: once you dictated even once, its
+# model stayed resident for the life of the process. Measured on a 16 GB M2:
+# the backend sits at ~6.2 GB idle — TTS 3.8 GB plus ~2 GB of warm ASR — while
+# an actual generate costs only ~116 MB on top. That baseline, not any spike, is
+# what pushes a 16 GB machine into memory pressure until the OS kills the
+# backend mid-generate — the death behind #1076/#1092/#1093/#1101. Freeing
+# 3.8 GB of TTS while silently holding 2 GB of ASR forever was the asymmetry.
+#
+# Reclaiming it costs a model re-warm on the next dictation (~1.4 s for
+# mlx-whisper turbo) and only after a full idle timeout — the same bargain the
+# TTS model already makes.
+_capture_last_used: float = 0.0
+# Live dictation streams hold the singleton for the WHOLE session while calling
+# nothing that would refresh `_capture_last_used`, so a long session could have
+# its model unloaded mid-sentence. A lease pins it for exactly that window.
+_capture_leases: int = 0
+
+
+def _touch_capture() -> None:
+    """Mark the capture backend as used now (resets its idle clock)."""
+    global _capture_last_used
+    _capture_last_used = time.monotonic()
+
+
+@contextlib.contextmanager
+def capture_lease():
+    """Pin the warm capture backend for the duration of a live session, so the
+    idle reaper can never unload the model out from under an open dictation
+    stream. Releasing the lease restarts the idle clock."""
+    global _capture_leases
+    with _capture_backend_lock:
+        _capture_leases += 1
+    try:
+        yield
+    finally:
+        with _capture_backend_lock:
+            _capture_leases = max(0, _capture_leases - 1)
+        _touch_capture()
+
+
+def release_idle_capture_backend(idle_s: float, *, now: float | None = None) -> bool:
+    """Unload the warm capture/dictation ASR once it has gone unused for
+    ``idle_s`` seconds. Returns True when a model was actually released.
+
+    No-ops while a live session holds a lease, when nothing is loaded, or when
+    the model was used recently. Never raises — a failed unload must not take
+    the idle worker down with it."""
+    global _capture_backend, _capture_backend_key
+    now = time.monotonic() if now is None else now
+    with _capture_backend_lock:
+        if _capture_backend is None or _capture_leases > 0:
+            return False
+        if now - _capture_last_used < idle_s:
+            return False
+        backend, _capture_backend, _capture_backend_key = _capture_backend, None, None
+    try:
+        backend.unload()
+    except Exception:  # noqa: BLE001 — a stuck unload must not kill idle_worker
+        logger.warning("capture ASR unload failed", exc_info=True)
+    logger.info(
+        "Idle timeout reached. Unloading capture ASR (%s) to free memory.",
+        type(backend).__name__,
+    )
+    return True
+
 
 def get_sherpa_dictation_backend(model_id: str) -> "SherpaDictationBackend":
     """Return a shared, warm-cached :class:`SherpaDictationBackend` for
@@ -2252,6 +2322,7 @@ def get_sherpa_dictation_backend(model_id: str) -> "SherpaDictationBackend":
     :func:`get_capture_asr_backend`. Thread-safe: the recognizer is shared;
     each session creates its own decode stream (see capture_ws)."""
     global _capture_backend, _capture_backend_key
+    _touch_capture()  # any handout resets the idle clock
     with _capture_backend_lock:
         if (isinstance(_capture_backend, SherpaDictationBackend)
                 and _capture_backend_key == model_id):
@@ -2299,6 +2370,7 @@ def get_capture_asr_backend() -> ASRBackend:
     """
     global _capture_backend, _capture_backend_key
 
+    _touch_capture()  # any handout resets the idle clock (#1101 class)
     # Atomic resolve+build so the preload thread and a WS session (which may
     # call get_sherpa_dictation_backend concurrently) can't both build a model.
     with _capture_backend_lock:

backend/services/model_manager.py

@@ -1273,11 +1273,27 @@ async def idle_worker():
     torch = _lazy_torch()
     while True:
         await asyncio.sleep(30)
+        idle_timeout = _resolve_idle_timeout()
         async with _model_lock:
-            if model is not None and time.time() - _last_used > _resolve_idle_timeout():
+            if model is not None and time.time() - _last_used > idle_timeout:
                 logger.info("Idle timeout reached. Unloading OmniVoice model to free VRAM.")
                 model = None
                 free_vram()
+        # The capture/dictation ASR was never idle-released — so once a user
+        # dictated, its model stayed resident for the life of the process while
+        # the TTS model dutifully freed its 3.8 GB. On a 16 GB Mac that left the
+        # backend sitting at ~6.2 GB idle, which is what tipped it into the
+        # memory pressure that gets it killed mid-generate (#1076/#1092/#1093/
+        # #1101). Give it the same bargain the TTS model already makes. Held
+        # off while a live dictation stream has a lease, so nothing is unloaded
+        # mid-sentence.
+        try:
+            from services.asr_backend import release_idle_capture_backend
+
+            if release_idle_capture_backend(idle_timeout):
+                free_vram()
+        except Exception:  # noqa: BLE001 — the reaper must never kill idle_worker
+            logger.warning("idle capture-ASR release failed", exc_info=True)
 
 def free_vram():
     """Release cached GPU memory on any accelerator (CUDA, MPS, XPU)."""

tests/test_capture_asr_idle_release.py

@@ -0,0 +1,144 @@
+"""The warm capture/dictation ASR must be idle-released, like the TTS model.
+
+Root cause behind the "Can't reach the local OmniVoice backend" deaths on 16 GB
+Macs (#1076/#1092/#1093/#1101): the TTS model has always been unloaded after an
+idle timeout (``model_manager.idle_worker``), but the capture-ASR singleton was
+not — once a user dictated even once, its model stayed resident for the life of
+the process.
+
+Measured on a 16 GB M2: the backend sat at ~6.2 GB **idle** (TTS 3.8 GB + ~2 GB
+of warm ASR) while an actual generate cost only ~116 MB on top. That baseline —
+not any spike during generation — is what pushes the machine into memory
+pressure until the OS kills the backend mid-generate. Freeing 3.8 GB of TTS
+while silently holding 2 GB of ASR forever was the asymmetry.
+
+Fail-before: ``release_idle_capture_backend`` did not exist and ``idle_worker``
+never touched the ASR singleton.
+"""
+from __future__ import annotations
+
+import os
+
+os.environ.setdefault("OMNIVOICE_MODEL", "test")
+os.environ.setdefault("OMNIVOICE_DISABLE_FILE_LOG", "1")
+
+import pytest
+
+from services import asr_backend as ab
+
+
+class _FakeBackend:
+    """Stands in for a warm mlx-whisper / sherpa recognizer."""
+
+    def __init__(self):
+        self.unloaded = False
+
+    def unload(self):
+        self.unloaded = True
+
+
+@pytest.fixture(autouse=True)
+def _clean_singleton(monkeypatch):
+    """Isolate the module-level capture singleton for every test."""
+    monkeypatch.setattr(ab, "_capture_backend", None, raising=False)
+    monkeypatch.setattr(ab, "_capture_backend_key", None, raising=False)
+    monkeypatch.setattr(ab, "_capture_leases", 0, raising=False)
+    monkeypatch.setattr(ab, "_capture_last_used", 0.0, raising=False)
+    yield
+
+
+def _install(backend, *, last_used=0.0):
+    ab._capture_backend = backend
+    ab._capture_backend_key = "fake"
+    ab._capture_last_used = last_used
+
+
+def test_releases_the_model_once_it_has_gone_idle():
+    fake = _FakeBackend()
+    _install(fake, last_used=0.0)
+
+    # 900 s (the default idle timeout) later, with nothing holding it.
+    assert ab.release_idle_capture_backend(900.0, now=1000.0) is True
+    assert fake.unloaded is True
+    # The singleton is dropped, so the next dictation rebuilds a fresh one.
+    assert ab._capture_backend is None
+    assert ab._capture_backend_key is None
+
+
+def test_keeps_the_model_while_it_is_still_in_use():
+    fake = _FakeBackend()
+    _install(fake, last_used=990.0)  # used 10 s ago
+
+    assert ab.release_idle_capture_backend(900.0, now=1000.0) is False
+    assert fake.unloaded is False
+    assert ab._capture_backend is fake
+
+
+def test_never_unloads_underneath_a_live_dictation_session():
+    """A live stream holds the backend for its whole life without re-resolving
+    it — so an open-but-silent session must NOT have its model pulled away."""
+    fake = _FakeBackend()
+    _install(fake, last_used=0.0)  # long idle: would otherwise be reaped
+
+    with ab.capture_lease():
+        assert ab.release_idle_capture_backend(900.0, now=1_000_000.0) is False
+        assert fake.unloaded is False
+        assert ab._capture_backend is fake
+
+    # Leaving the session restarts the idle clock (it is NOT instantly reapable).
+    assert ab.release_idle_capture_backend(900.0) is False
+    assert fake.unloaded is False
+
+
+def test_lease_is_released_even_if_the_session_raises():
+    fake = _FakeBackend()
+    _install(fake, last_used=0.0)
+
+    with pytest.raises(RuntimeError):
+        with ab.capture_lease():
+            raise RuntimeError("client disconnected mid-stream")
+
+    assert ab._capture_leases == 0  # not leaked → the reaper isn't wedged forever
+
+
+def test_nested_leases_refcount_correctly():
+    fake = _FakeBackend()
+    _install(fake, last_used=0.0)
+
+    with ab.capture_lease():
+        with ab.capture_lease():
+            assert ab.release_idle_capture_backend(900.0, now=1_000_000.0) is False
+        # Inner released, outer still holds it.
+        assert ab._capture_leases == 1
+        assert ab.release_idle_capture_backend(900.0, now=1_000_000.0) is False
+    assert ab._capture_leases == 0
+
+
+def test_no_op_when_nothing_is_loaded():
+    assert ab.release_idle_capture_backend(900.0, now=1_000_000.0) is False
+
+
+def test_a_failing_unload_still_drops_the_reference():
+    """A stuck unload must not wedge the reaper or keep the model pinned —
+    idle_worker calls this on a loop and must never die."""
+
+    class _Boom(_FakeBackend):
+        def unload(self):
+            raise RuntimeError("metal context already torn down")
+
+    _install(_Boom(), last_used=0.0)
+    assert ab.release_idle_capture_backend(900.0, now=1000.0) is True
+    assert ab._capture_backend is None
+
+
+def test_getting_the_backend_resets_the_idle_clock(monkeypatch):
+    """Any handout counts as use — otherwise a freshly-warmed model built at
+    T=0 would be reaped on the very next idle tick."""
+    fake = _FakeBackend()
+    _install(fake, last_used=0.0)
+    monkeypatch.setattr(ab, "dictation_model_id", lambda: None)
+    monkeypatch.setattr(ab, "_pick_capture_whisper_backend", lambda: fake, raising=False)
+
+    before = ab._capture_last_used
+    ab._touch_capture()
+    assert ab._capture_last_used > before