The ten-minute idle sweep lived inside the dial-out agent. A node that only
accepts inbound connections never starts that agent — on gpu2 it fails outright
with 'Set OMNIVOICE_WORKER_ENDPOINT' — so a machine lending its GPU to panels
that dial IN held several GB of weights forever. That is precisely the cost the
sweep exists to avoid, and it was silently missing in the mode most likely to
be a shared box.
The loop moves to module scope and both transports use it. Inbound starts it
when the listener starts and cancels it when the listener stops, and passes a
callback that re-advertises capabilities to every attached panel, so a control
plane's view of what is resident does not go stale the moment it becomes
useful. Local behaviour is unchanged: nothing sweeps unless a worker role runs.
Found by clicking Synthesize in the desktop UI — the one path nothing had
exercised.
task_store.stage_input mints inputs/<digest><ext>, a path rather than a bare
name. The node ran safe_filename over it, which rejects anything nested, so
every real clone input was refused, the dispatch failed, and the scheduler
retried about eighteen times a second while the 4090 sat idle and the user
watched a spinner.
The wire id is now hashed into a directory name rather than used as one. That
accepts any id the protocol allows while leaving placement entirely ours to
decide, which is the property the check was really buying. The declared
filename is still required to be a bare name, and a hostile one is still
refused outright — covered by its own test so the containment cannot be traded
away later to fix some future rejection.
Every earlier test used a flat id like 'ref-1' and so never met the shape
production emits.
Found on hardware. The queue was built once per connection and reused across
reconnects, so a frame a dying session left behind became the FIRST frame of
the next attach. The node requires a registration there, aborted the call, and
the two span at full speed — session epoch 2445 inside one second, the node
logging 'Locally aborted' on repeat, and the panel reporting the machine
offline while the connection list showed it connected.
Two more found on hardware.
Disconnect ended the session and the panel redialled two seconds later, so the
log read disconnected and connected in the same breath and the button appeared
to do nothing. A kicked key now sits out for a minute — long enough that the
disconnect is real and the person notices, short enough that it is plainly not
a revocation, which stays a separate and permanent action. The docs now say
which of the two buttons does which.
Re-pasting a connection string for an already-connected machine saved the new
string and then short-circuited on the existing session, so a wrong key
reported success, kept running on the old connection, and only failed after a
restart — by which point nothing pointed back at the paste that caused it. The
live session is now torn down before the new one is dialled.
Found on hardware. FetchResult seeked to request.size_bytes as though it were
a resume point, but that field is the artifact's total size — so every fetch
started at end-of-file, yielded no chunks, and failed with 'the result ended
before its final chunk' while the finished render sat on the node's disk.
ArtifactRef carries no resume field, so resumption is a protocol addition
rather than a reinterpreted one, and the fetch now always starts at zero.
Every earlier test drove publish and stage directly and never called
FetchResult with a populated ref, which is exactly why this survived them.
Found on hardware. The Attach handler started the read pump and the outbound
loop but never the heartbeat loop that the outbound path starts inside
_connect_once. So a node registered, went silent, was declared dead about
ninety seconds later, reconnected, and flapped forever — and in between, work
aimed at it fell back to the local machine with 'gpu2 is offline', while the
panel had shown it ready at 3.4 ms moments earlier.
Every end-to-end test in this file finished inside three seconds, comfortably
within the grace window that hid it. The regression test therefore asserts on
the emitted heartbeat frames themselves rather than on liveness, and shortens
the advertised interval so it does that in two seconds instead of twenty.
Found on hardware. With the listener bound to 0.0.0.0 — which is what sharing
a GPU across a network requires — the issued string came out as
ovnode://...@0.0.0.0:7444. That is a legal bind and a meaningless destination,
so it would have failed on the far end with a connection error naming nothing,
and the person who pasted it had no way to tell a bad string from a firewall.
The string is now built from an advertised address rather than the bind: for a
wildcard bind, the source address the routing table would use to leave this
machine, found with a connected UDP socket that sends no packets and needs no
DNS. An explicitly typed bind is advertised verbatim, because someone who
entered a specific address meant it.
Wires the two transport halves into something a user can actually turn on.
Two independent switches, deliberately not one. "Accept connections" makes this
machine a node others dial; "saved connections" are the nodes this panel dials
out to. A workstation with a GPU that also drives jobs on a second box does
both, so neither implies the other.
Binding stays on 127.0.0.1 until someone explicitly widens it, and widening is
its own field rather than a flag riding along with the enable toggle. With no
encryption that boundary is the difference between a credential on one machine
and a credential on a network, so it is never crossed as a side effect. The
API reports `exposed` so the UI can say which side of it the user is on.
Saved nodes are redialled only after the control plane is up, since the
connector hands frames to its servicer. Failing to listen records the reason
rather than leaving the feature looking enabled while it quietly accepts
nothing.
Docs say plainly that this mode is unencrypted, that the connection string is a
password crossing the network in the clear, and that dial-out remains the
better choice when one machine is enough. The Security section no longer
implies its TLS guarantees cover both modes.
Completes the inbound path. The panel opens NodeService.Attach with its key in
call metadata, answers the node's register frame, and then runs the ordinary
control-plane loops against the dialled stream — the same _read_loop and
_ping_loop the outbound path uses, so assignments, cancels, results and
reconciliation all behave identically. Only who opened the socket changed.
Registration is shared rather than copied: the body of Register is now
establish_session, reached from both roads. A second copy of session issue,
capability application and in-flight reconciliation is a second thing to keep
in step forever, and the half that gets forgotten is always reconciliation.
The version and feature gates run on the inbound road too — skipping them would
let an out-of-date node register cleanly and then ignore task inputs, which is
how a clone with no reference audio once came back reported as success.
Artifacts invert with the transport: the panel pushes inputs before it assigns,
and pulls results after. Both directions verify the declared sha256 and refuse
a stream that ends without its final chunk, because a truncated file renamed
into place and called done is the failure the upload path was already hardened
against.
Two things the end-to-end tests found, neither visible from unit tests:
* Every Attach built a fresh client with an empty worker id, so the challenge
signature could never match after first enrollment — inbound could connect
once and never reconnect. The id is now kept per panel key, because each
panel keeps its own registry and the same machine is a different worker id
to each of them.
* A node that has lost the id a panel gave it could prove possession of its
key and still be refused forever, with no way back except deleting it from
both sides. It is now re-adopted on proof of key possession, narrowly: the
public key must already be the one enrolled, so this can never admit a new
key. Covered by a test that forges a valid self-signature from a different
keypair and asserts it is refused.
Remote workers connect outbound: the node dials the control plane, spends an
enrollment token, pins a certificate. That stays the default and is unchanged.
It is also structurally 1:1 — a worker process holds one endpoint, one pinned
certificate and one worker id — so a second person wanting the same GPU box has
to get shell access to it, repoint the start script at their own address and
restart, which disconnects whoever was using it. Sharing a GPU requires root on
it and evicts the incumbent, and no amount of UI work fixes that, because the
constraint is the shape of the connection.
This adds the other arrangement: the node listens, and any panel holding a key
connects to it, concurrently, with no shell access to the machine.
* NodeService mirrors WorkerService. Transport roles invert; message roles do
not — the node still sends WorkerMessage and the panel still sends
ServerMessage, so every state machine on both sides is untouched. Register
folds into the stream as the first exchange and reuses the existing
request/response messages rather than growing parallel ones.
* Keys are per panel, not per node. Revoking one person leaves everyone else
connected; a shared key would be revoked by nobody and leave no record of
who used it. Stored hashed, compared in constant time against every key so
the reply time is not an oracle, and the plaintext exists exactly once.
* Failed authentication is throttled per source address, so one stale
bookmark cannot lock out a different panel.
* A connection log records every attach, refusal and disconnect, and any
session can be kicked. That is what replaces per-job approval, which would
make a shared GPU unusable and train people to click yes.
* Artifacts invert too: the panel pushes inputs before assigning, and fetches
results after. The node stages both under one contained directory and
trusts no id or filename off the wire.
Runs in plaintext by deliberate decision, recorded with its accepted risk in
docs/adr/inbound-node-mode.md, and scoped there to LAN and self-hosted use —
never a fleet transport, which goal_v2 B2/B5.2 still require to dial out.
Off by default, and bound to 127.0.0.1 until someone explicitly widens it.
Palash Debnath