mlxrunner: keep the reused head of a cached edge safe from eviction

When a request resumes partway through a cached edge, the node holding
that edge was dropped from the active path, because the path has to end
at the live offset for close and the prefill captures to extend the trie
from its last node. Off the path, the node was an ordinary leaf with a
stale last-used time, so eviction removed it first.

The captures taken during that request start at the resume offset, but
attach rebuilds the missing node from the path's last node, so the new
node's edge begins earlier than its KV snapshot. A later request
resuming there was refused by the KV cache and re-prefilled from
scratch. If eviction first merged the node into its parent, the two
snapshots were concatenated as if adjacent, and the restore reported a
hit while the buffer held tokens from other positions.

Split the node at the live offset instead. The head stays on the path
and gets the last-used update. Only the unused tail can be evicted, and
losing it costs nothing. The split only happens when every layer can
rewind into the edge, so it never involves a recurrent layer, and the
head gets the same KV-only snapshots a close-time split already
produces. When the request follows the edge, compaction merges the
halves back.
This commit is contained in:
Jesse Gross
2026-09-10 15:20:11 -07:00
parent 45a02807e2
commit b859a94509
+11 -3
View File
@@ -283,11 +283,19 @@ pageIn:
}
}
}
// If the live offset falls inside the last node, split it so the reused
// head stays on the active path and only the unused tail can be evicted.
for i := len(c.activePath) - 1; i >= 0; i-- {
if c.activePath[i].endOffset <= minOff {
c.activePath = c.activePath[:i+1]
break
node := c.activePath[i]
if i > 0 && node.startOffset() >= minOff {
continue
}
if node.endOffset > minOff {
node = splitNode(node, minOff-node.startOffset(), c.caches, &c.pagedOutBytes)
}
c.activePath = append(c.activePath[:i], node)
break
}
// Update last-used time on only the final used node. For recurrent