After the ordering fix, unloading the model on a 4090 still left the GPU at
1238 MiB with torch reporting 8.5 MB allocated and 803 MB reserved -- and no
number of Flush Memory presses moved it. A segment dump said why: ONE 803 MB
segment, 794.7 MB of it inactive-but-split, pinned by a single live block of
8,519,680 bytes.
That is cuBLAS's default workspace. It is taken from the caching allocator on
first use, so it lands inside whatever segment the model load had just grown,
and it is held for the life of the cuBLAS handle. empty_cache() can only
return segments that are entirely free, so one 8.5 MB block kept three
quarters of a gigabyte from ever reaching the driver again. On a machine
lending its GPU that is the difference between an idle node costing 470 MiB
and costing 1.2 GB.
free_vram() now clears the workspaces before emptying the cache, on the
unload paths only -- the next cuBLAS call re-takes one, which is cheap but
not something to pay per generate. The binding is private
(torch._C._cuda_clearCublasWorkspaces), so it is optional by construction: a
build without it keeps today's behaviour rather than failing an unload.
Found by adding reserved-vs-allocated to /system/flush-memory in 642513d2.
Allocated alone reads near zero after an unload, which is exactly why this
hid for so long -- every diagnostic we had agreed the memory was free.
The shared voice model's unload emptied the allocator caches and *then*
dropped the reference. That frees nothing: the weights are still reachable
when gc.collect() runs, empty_cache() only returns blocks the allocator
already considered free, and the reference drops a moment later into a cache
nothing will flush again. The unload logs success, the engine leaves the
registry, and nvidia-smi does not move.
Six modules open-coded the same two lines. Exactly one had them inverted --
OmniVoiceBackend.unload, which is the path the engine-registry idle sweep
reaches, which is the sweep a headless worker node runs. So every unload a
user could trigger from the UI worked, and the one that runs unattended on a
machine lending its GPU held 3.6 GB indefinitely. Found on hardware: the
sweep fired on schedule, logged "Released 1 idle engine(s)", and VRAM stayed
flat at 3656 MiB for the next two minutes.
Replace all six with model_manager.unload_shared_model(), which clears the
reference, drops the clone-prompt side cache, then frees -- in that order,
in one place. Two callers gain the side-cache drop they were missing
(/system/flush-memory and the shutdown path), which is the same defect one
step down: an unload that kept the encoded reference tensors belonging to the
model it had just released.
A source guard asserts nothing outside model_manager assigns the shared
reference, so the next caller cannot reintroduce the ordering. It caught the
sixth site while being written.
Also give the AudioSeal watermark models the bargain every other model in the
app already makes: they loaded on the first embed and stayed resident for the
life of the process. CPU-resident, so this is system RAM rather than VRAM,
and the machines that notice are the ones running batches.
The error text on a failing unload changes with the ordering. "Could not be
unloaded, retry after the current generation finishes" was accurate when the
cache flush ran first and aborted before the release; now the release has
already happened and only the flush can fail, so it says that instead of
sending the user to repeat work that is done.
* fix(ui): loaded-models panel says when a resident model is not the active engine (#TBD)
A switched-away TTS model stays resident in VRAM until Unload/Flush or the
idle evictor, so users who picked another engine (e.g. voxcpm2) saw
"OmniVoice TTS - cuda:0 - 1937 MB" in the LOADED MODELS flyout and concluded
synthesis was still routing to OmniVoice. It wasn't - the panel just gave no
hint that resident != active.
/model/loaded entries for TTS-family models (in-process OmniVoice +
subprocess sidecars) now carry engine_id + is_active_engine, computed against
active_backend_id(); attribution failure degrades to the old shape
(is_active_engine: null) and non-TTS entries (ASR, diarization) are left
unannotated. The flyout renders a muted "not active - safe to unload" tag
(i18n: header.model_not_active, en + zh-CN) on inactive entries; Unload/Flush
behavior is unchanged. Regression tests cover both attribution states, the
ASR non-label, and the degradation path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): loaded-models active-engine hint entry (#985)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mm2): string-target monkeypatch for active_backend_id — immune to sys.modules reimports
The two attribution tests patched the collection-time module alias; other
suites pop+reimport services.* modules mid-run, so in full-suite order the
patch landed on a stale module object while _active_tts_id late-imported the
fresh one (CI-only failure). String targets resolve at patch time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A first-run user whose model download was interrupted after the config/
tokenizer files landed but before the weight shard got stranded on the
Models & Engines page: GET /models computed "installed" purely from cache
size on disk, so a size-positive-but-weight-less cache reported installed=true,
the wizard hid the re-download button, and the model manager (Settings → Models)
that could repair it was unreachable behind the wizard gate.
Make install-state weight-aware. The boolean weight-floor scan now lives in
models.py (the lowest module in the setup import graph) as snapshot_has_weights()
+ cache_is_complete(); list_models() and recommendations() downgrade a truncated
cache to installed=false (+ an explicit incomplete=true on /models), so the
existing "install" action re-appears and the user can re-download in-wizard.
Fixes the whole class, not just /models: download.py's install-time validator
now delegates to the same shared scan (one source of the floors, can't drift),
matching the load-time repair in model_manager.py (#581/#606).
config_only repos (pyannote/speaker-diarization-3.1 — a pipeline whose real
weights live in referenced sub-repos and whose own cache is legitimately tiny)
carry a new config_only:true hint in models.yaml and are exempt, so they're not
false-flagged as incomplete.
Tests: tests/test_mm2_lifecycle.py — snapshot_has_weights truncated-vs-complete,
cache_is_complete on a truncated weight repo + config-only exemption, and
list_models downgrading a size-positive truncated cache to installed=false /
incomplete=true. Full backend suite green (1832 passed).
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
One coherent lifecycle surface over the in-process model, diarization, and
subprocess sidecars; fixes the engine-switch VRAM leak; tightens download
robustness. Backend-only, response shapes preserved, no new deps.
Tier 1 — correctness:
- MM2-01: get_active_tts_backend() caches one instance per backend id and
unload()s the outgoing engine on switch (fixes the VRAM leak behind #278);
adds reset_active_backend().
- MM2-02: OmniVoiceBackend.unload() releases the shared model_manager singleton
+ free_vram(); SubprocessBackend.unload() -> unload_sidecar(self.id),
inherited by all sidecar engines. Idempotent + preload-safe.
- MM2-03: /model/loaded ASR row reports the real device + a note explaining the
disabled unload button.
Tier 2 — single surface:
- MM2-04: new services/model_lifecycle.py owns list_loaded/unload/unload_all/
free_vram; system.py routers are thin delegations (shapes unchanged).
- MM2-05: idle timeouts (in-process + sidecar) resolve via prefs.resolve
(env wins, no restart); removed the duplicated _IDLE_TIMEOUT_SECONDS.
Tier 3 — robustness/observability:
- MM2-06: _install_cooldowns swept (1h TTL) + cleared on success — bounded.
- MM2-07: per-extension weight floors (onnx 64KB, tensors 5MB) OR the original
>=5MB catch — small ONNX no longer false-flagged, #352 still caught.
- MM2-08: indextts GPU sidecar self-reports vram_mb in pong; parent surfaces it
in list_live_sidecars (0 = CPU/unmeasured).
- MM2-09: is_cached scan_cache_dir->disk fallback logs WARNING w/ exc type
(#117/#118), was invisible at DEBUG.
Tests: tests/test_mm2_lifecycle.py (15). Full suite: 1379 passed.
Plan/summary: .planning/quick/260613-mm2-clean-model-management-v2/.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
debpalash