fix(engines): snapshot lazy registry keys so /engines can't 500 under concurrency (#940)

* fix(engines): snapshot lazy registry keys so /engines can't 500 under concurrency

`list_backends()` runs in a FastAPI threadpool and iterates the lazy TTS/ASR
registries via `items()` → `__iter__`, which held a *live* `dict.__iter__(self)`
open across each engine's slow `is_available()` probe. Meanwhile the lazy
`__getitem__` resolves a deferred entry by mutating the dict (`self[key] = cls`).
A second concurrent `/engines` request (or any ASR op) materializing the lazy
`faster-whisper-isolated` entry therefore changed the dict size mid-iteration:

    RuntimeError: dictionary changed size during iteration
      asr_backend.py:1729 list_backends → _REGISTRY.items()
      asr_backend.py:1665 __iter__ → for k in dict.__iter__(self)

Both `_LazyRegistry` (TTS) and `_LazyASRRegistry` (ASR) now snapshot their live
keys up front with `list(dict.__iter__(self))` — consumed atomically under the
GIL — so a concurrent lazy insert can no longer trip the iteration. The slow
per-engine probes then run over the snapshot, not the live iterator.

Deterministic fail-before/pass-after regression for both registries:
tests/backend/services/test_lazy_registry_concurrency.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): add the /engines concurrency fix under [Unreleased] (#940)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

CHANGELOG.md

@@ -6,6 +6,12 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
 Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
 The bundled TTS model package (`pyproject.toml`) is versioned independently.
 
+## [Unreleased]
+
+### Fixed
+
+- **Settings → Engines can no longer 500 under concurrent loads.** The lazy TTS/ASR engine registries held a *live* dictionary iterator open across each engine's `is_available()` probe while `list_backends()` ran in a FastAPI threadpool — so a second concurrent `/engines` request materializing a lazy engine entry (`self[key] = cls`) mutated the dict mid-iteration and crashed the request with `RuntimeError: dictionary changed size during iteration`. Both registries now snapshot their keys before iterating (atomic under the GIL), immune to a concurrent insert; regression-tested for TTS and ASR. (#940)
+
 ## [0.3.9] — 2026-07-04
 
 The dictation release — and a deep reliability pass driven by live-testing the entire app. **Dictation is rebuilt end-to-end**: instant feedback with a live waveform, words that commit about half a second after you stop speaking, clean punctuation, and text insertion that never lies about success. **LLM providers get one-click connection testing** with real diagnostics and model discovery, in all 21 languages. The app now **always opens maximized**, bottom buttons **can't hide under the footer** at small window sizes, and a wave of "out of memory / can't reach the backend / stuck at preparing" reports were traced to their real causes and fixed — including the silent VRAM crash on 8 GB cards, dead-IPC startup hangs after a Windows BSOD, and misleading error labels. Intel-Mac support status is now stated honestly, Confucius4-TTS is validated end-to-end, and Parakeet — roughly 20× faster than the default transcriber on CPU — is unlocked for every machine.

backend/services/asr_backend.py

@@ -1662,7 +1662,13 @@ class _LazyASRRegistry(dict):
 
     def __iter__(self):
         seen = set()
-        for k in dict.__iter__(self):
+        # Snapshot the live keys before yielding — see _LazyRegistry.__iter__ in
+        # tts_backend.py. A concurrent lazy __getitem__ inserts into self, and
+        # list_backends() runs in a FastAPI threadpool, so a *live* dict iterator
+        # held open across the per-engine is_available() probes would raise
+        # "dictionary changed size during iteration". list() consumes it
+        # atomically under the GIL, closing the window.
+        for k in list(dict.__iter__(self)):
             seen.add(k)
             yield k
         for k in self._LAZY:

backend/services/tts_backend.py

@@ -1280,7 +1280,13 @@ class _LazyRegistry(dict):
         # effect on every list_backends() call — we keep iteration light
         # and let the caller's __getitem__ trigger the import.
         seen: set[str] = set()
-        for k in dict.__iter__(self):
+        # Snapshot the live keys before yielding. A concurrent thread's lazy
+        # __getitem__ inserts into self (self[key] = cls), and list_backends()
+        # runs in a FastAPI threadpool — so holding a *live* dict iterator open
+        # across the per-engine is_available() probes would raise
+        # "dictionary changed size during iteration". list() consumes the
+        # iterator atomically under the GIL, closing that window.
+        for k in list(dict.__iter__(self)):
             seen.add(k)
             yield k
         for k in _LAZY_REGISTRY:

tests/backend/services/test_lazy_registry_concurrency.py

@@ -0,0 +1,38 @@
+"""Regression: the lazy TTS/ASR registries must not raise "dictionary changed
+size during iteration" when a lazy ``__getitem__`` inserts a resolved key while
+another caller iterates — the ``/engines`` 500 seen in production logs.
+
+FastAPI runs ``list_backends()`` in a threadpool, so two concurrent ``/engines``
+requests race: one iterates ``_REGISTRY.items()`` (which held a *live* dict
+iterator open across the slow per-engine ``is_available()`` probes) while the
+other materializes the lazy entry via ``__getitem__`` (``self[key] = cls``).
+The insert then tripped the open iterator. ``__iter__`` now snapshots the live
+keys up front (``list(dict.__iter__(self))``, atomic under the GIL), so a
+concurrent insert can no longer trip the iteration.
+
+The tests drive the exact crash site (``__iter__``) deterministically: begin
+iterating, insert mid-iteration, then drain. Pre-fix this raises on the drain;
+post-fix it completes.
+"""
+
+
+def _assert_iter_survives_concurrent_insert(reg):
+    it = iter(reg)                     # the generator items()/list_backends() drives
+    first = next(it)                   # first yield → the real-key snapshot is taken here
+    reg["zzz-concurrent-insert"] = object()  # a concurrent lazy insert, mid-iteration
+    drained = [first, *it]             # must NOT raise "dictionary changed size during iteration"
+    assert first in drained
+
+
+def test_tts_lazy_registry_iter_survives_concurrent_insert():
+    from services.tts_backend import _LazyRegistry
+
+    reg = _LazyRegistry({"omnivoice": object(), "b": object(), "c": object()})
+    _assert_iter_survives_concurrent_insert(reg)
+
+
+def test_asr_lazy_registry_iter_survives_concurrent_insert():
+    from services.asr_backend import _LazyASRRegistry
+
+    reg = _LazyASRRegistry({"whisperx": object(), "faster-whisper": object()})
+    _assert_iter_survives_concurrent_insert(reg)