After the ordering fix, unloading the model on a 4090 still left the GPU at
1238 MiB with torch reporting 8.5 MB allocated and 803 MB reserved -- and no
number of Flush Memory presses moved it. A segment dump said why: ONE 803 MB
segment, 794.7 MB of it inactive-but-split, pinned by a single live block of
8,519,680 bytes.
That is cuBLAS's default workspace. It is taken from the caching allocator on
first use, so it lands inside whatever segment the model load had just grown,
and it is held for the life of the cuBLAS handle. empty_cache() can only
return segments that are entirely free, so one 8.5 MB block kept three
quarters of a gigabyte from ever reaching the driver again. On a machine
lending its GPU that is the difference between an idle node costing 470 MiB
and costing 1.2 GB.
free_vram() now clears the workspaces before emptying the cache, on the
unload paths only -- the next cuBLAS call re-takes one, which is cheap but
not something to pay per generate. The binding is private
(torch._C._cuda_clearCublasWorkspaces), so it is optional by construction: a
build without it keeps today's behaviour rather than failing an unload.
Found by adding reserved-vs-allocated to /system/flush-memory in 642513d2.
Allocated alone reads near zero after an unload, which is exactly why this
hid for so long -- every diagnostic we had agreed the memory was free.
memory_allocated counts live tensors only, so after an unload it reads
near zero while nvidia-smi still shows gigabytes. That gap is the whole
substance of every "flush says it worked, the GPU says it didn't" report,
and the endpoint was reporting only the half that looks good.
memory_reserved is what the caching allocator holds from the driver; the
remainder between that and the driver's own figure is the CUDA context and
kernel workspaces, which nothing in-process can hand back.
The shared voice model's unload emptied the allocator caches and *then*
dropped the reference. That frees nothing: the weights are still reachable
when gc.collect() runs, empty_cache() only returns blocks the allocator
already considered free, and the reference drops a moment later into a cache
nothing will flush again. The unload logs success, the engine leaves the
registry, and nvidia-smi does not move.
Six modules open-coded the same two lines. Exactly one had them inverted --
OmniVoiceBackend.unload, which is the path the engine-registry idle sweep
reaches, which is the sweep a headless worker node runs. So every unload a
user could trigger from the UI worked, and the one that runs unattended on a
machine lending its GPU held 3.6 GB indefinitely. Found on hardware: the
sweep fired on schedule, logged "Released 1 idle engine(s)", and VRAM stayed
flat at 3656 MiB for the next two minutes.
Replace all six with model_manager.unload_shared_model(), which clears the
reference, drops the clone-prompt side cache, then frees -- in that order,
in one place. Two callers gain the side-cache drop they were missing
(/system/flush-memory and the shutdown path), which is the same defect one
step down: an unload that kept the encoded reference tensors belonging to the
model it had just released.
A source guard asserts nothing outside model_manager assigns the shared
reference, so the next caller cannot reintroduce the ordering. It caught the
sixth site while being written.
Also give the AudioSeal watermark models the bargain every other model in the
app already makes: they loaded on the first embed and stayed resident for the
life of the process. CPU-resident, so this is system RAM rather than VRAM,
and the machines that notice are the ones running batches.
The error text on a failing unload changes with the ordering. "Could not be
unloaded, retry after the current generation finishes" was accurate when the
cache flush ran first and aborted before the release; now the release has
already happened and only the flush can fail, so it says that instead of
sending the user to repeat work that is done.
The startup preload exists so the first generate feels instant for the person
sitting in front of the app. A machine lending its GPU has nobody sitting
there, so it was several GB of VRAM held from boot against a request that may
never arrive — and the idle sweep could not reclaim it, because the sweep owns
the worker executor's engines while this is the default local model.
Measured on gpu2: a node that had run nothing still sat at 2.4 GB, and an idle
unload after a real job returned it to exactly that floor rather than below it.
Worker-mode processes now load on first request and release when idle, which is
what a node should do. A machine that is both a desktop app and a worker keeps
the warm-up — there is a real user there and the point stands.
Watching a ten-minute rule take effect means waiting ten minutes, so it tends
not to get watched. Both numbers are now env-tunable:
OMNIVOICE_ENGINE_IDLE_UNLOAD_SECONDS and OMNIVOICE_IDLE_SWEEP_SECONDS.
They are documented as a pair, because shortening only the threshold still
means waiting a full sweep interval to see it fire — which reads as a broken
sweep and sends you looking for a bug that is not there.
Unparseable values and anything below the floor are ignored with a warning
rather than honoured. A zero threshold would hand back a model the instant it
went idle and reload it for the very next request, which is worse than the
behaviour being tuned.
The ten-minute idle sweep lived inside the dial-out agent. A node that only
accepts inbound connections never starts that agent — on gpu2 it fails outright
with 'Set OMNIVOICE_WORKER_ENDPOINT' — so a machine lending its GPU to panels
that dial IN held several GB of weights forever. That is precisely the cost the
sweep exists to avoid, and it was silently missing in the mode most likely to
be a shared box.
The loop moves to module scope and both transports use it. Inbound starts it
when the listener starts and cancels it when the listener stops, and passes a
callback that re-advertises capabilities to every attached panel, so a control
plane's view of what is resident does not go stale the moment it becomes
useful. Local behaviour is unchanged: nothing sweeps unless a worker role runs.
Found by clicking Synthesize in the desktop UI — the one path nothing had
exercised.
task_store.stage_input mints inputs/<digest><ext>, a path rather than a bare
name. The node ran safe_filename over it, which rejects anything nested, so
every real clone input was refused, the dispatch failed, and the scheduler
retried about eighteen times a second while the 4090 sat idle and the user
watched a spinner.
The wire id is now hashed into a directory name rather than used as one. That
accepts any id the protocol allows while leaving placement entirely ours to
decide, which is the property the check was really buying. The declared
filename is still required to be a bare name, and a hostile one is still
refused outright — covered by its own test so the containment cannot be traded
away later to fix some future rejection.
Every earlier test used a flat id like 'ref-1' and so never met the shape
production emits.
Found on hardware. The queue was built once per connection and reused across
reconnects, so a frame a dying session left behind became the FIRST frame of
the next attach. The node requires a registration there, aborted the call, and
the two span at full speed — session epoch 2445 inside one second, the node
logging 'Locally aborted' on repeat, and the panel reporting the machine
offline while the connection list showed it connected.
Two more found on hardware.
Disconnect ended the session and the panel redialled two seconds later, so the
log read disconnected and connected in the same breath and the button appeared
to do nothing. A kicked key now sits out for a minute — long enough that the
disconnect is real and the person notices, short enough that it is plainly not
a revocation, which stays a separate and permanent action. The docs now say
which of the two buttons does which.
Re-pasting a connection string for an already-connected machine saved the new
string and then short-circuited on the existing session, so a wrong key
reported success, kept running on the old connection, and only failed after a
restart — by which point nothing pointed back at the paste that caused it. The
live session is now torn down before the new one is dialled.
Found on hardware. FetchResult seeked to request.size_bytes as though it were
a resume point, but that field is the artifact's total size — so every fetch
started at end-of-file, yielded no chunks, and failed with 'the result ended
before its final chunk' while the finished render sat on the node's disk.
ArtifactRef carries no resume field, so resumption is a protocol addition
rather than a reinterpreted one, and the fetch now always starts at zero.
Every earlier test drove publish and stage directly and never called
FetchResult with a populated ref, which is exactly why this survived them.
Found on hardware. The job ran on the GPU machine and the audio never arrived:
'gpu2 finished the job but its audio did not arrive.'
Both artifact directions were built and neither was wired. A result reported by
a dialled node is only staged on that node's disk — nothing pushes it, because
the node cannot call us — so the commit recorded an artifact path that had
never been written. Inputs had the mirror problem: nothing sent them, so a
clone would have failed on a reference file that was never delivered.
Results are now pulled when the frame naming them arrives, and inputs are
pushed before the assignment rather than alongside it, because the executor
asks for them as soon as it starts and an assignment that overtakes its own
reference audio fails on a file that is merely late.
A fetch that fails is not a silent loss: no artifact is recorded, the task
fails naming the machine, and the node keeps its copy because nothing
acknowledges a result we could not fetch.
Found on hardware. The Attach handler started the read pump and the outbound
loop but never the heartbeat loop that the outbound path starts inside
_connect_once. So a node registered, went silent, was declared dead about
ninety seconds later, reconnected, and flapped forever — and in between, work
aimed at it fell back to the local machine with 'gpu2 is offline', while the
panel had shown it ready at 3.4 ms moments earlier.
Every end-to-end test in this file finished inside three seconds, comfortably
within the grace window that hid it. The regression test therefore asserts on
the emitted heartbeat frames themselves rather than on liveness, and shortens
the advertised interval so it does that in two seconds instead of twenty.
Found on hardware. With the listener bound to 0.0.0.0 — which is what sharing
a GPU across a network requires — the issued string came out as
ovnode://...@0.0.0.0:7444. That is a legal bind and a meaningless destination,
so it would have failed on the far end with a connection error naming nothing,
and the person who pasted it had no way to tell a bad string from a firewall.
The string is now built from an advertised address rather than the bind: for a
wildcard bind, the source address the routing table would use to leave this
machine, found with a connected UDP socket that sends no packets and needs no
DNS. An explicitly typed bind is advertised verbatim, because someone who
entered a specific address meant it.
Adds the panel that makes inbound mode usable: a toggle to accept connections,
a bind field that says which side of "only this machine" you are on, per-person
connection strings with a copy button, the live list of who is connected with a
disconnect button, and a paste box for joining someone else's GPU.
Placed behind the existing Remote workers toggle rather than beside it. "Off
means off" is this feature's stated contract, and a second switch that stayed
live underneath would be exactly the surprise that promise exists to prevent.
Headless machines that only lend a GPU set OMNIVOICE_INBOUND_NODE and never see
this panel.
The unencrypted warning appears where it becomes true, not buried in a doc:
next to the bind field once it points beyond this machine, naming the address,
and again under every freshly issued connection string. The remove-access
confirm says the others stay connected, since that is the only place a user
learns keys are per person rather than one switch for everybody.
All 36 strings are translated into all 20 non-English locales in this change,
with the {{address}}, {{label}}, {{count}} and {{when}} placeholders verified
programmatically against en.json before writing — a dropped token is the exact
bug the parity test was built for, and en-only keys would have passed CI
silently while every other language read English.
Two existing WorkersPanel rename tests queried the only textbox on the page.
That was incidental, not intentional; they now name the field they mean.
Codex