mirror of
https://github.com/ollama/ollama.git
synced 2026-09-21 05:28:00 -05:00
mlxrunner: keep the reused head of a cached edge safe from eviction
When a request resumes partway through a cached edge, the node holding that edge was dropped from the active path, because the path has to end at the live offset for close and the prefill captures to extend the trie from its last node. Off the path, the node was an ordinary leaf with a stale last-used time, so eviction removed it first. The captures taken during that request start at the resume offset, but attach rebuilds the missing node from the path's last node, so the new node's edge begins earlier than its KV snapshot. A later request resuming there was refused by the KV cache and re-prefilled from scratch. If eviction first merged the node into its parent, the two snapshots were concatenated as if adjacent, and the restore reported a hit while the buffer held tokens from other positions. Split the node at the live offset instead. The head stays on the path and gets the last-used update. Only the unused tail can be evicted, and losing it costs nothing. The split only happens when every layer can rewind into the edge, so it never involves a recurrent layer, and the head gets the same KV-only snapshots a close-time split already produces. When the request follows the edge, compaction merges the halves back.
This commit is contained in:
@@ -283,11 +283,19 @@ pageIn:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// If the live offset falls inside the last node, split it so the reused
|
||||
// head stays on the active path and only the unused tail can be evicted.
|
||||
for i := len(c.activePath) - 1; i >= 0; i-- {
|
||||
if c.activePath[i].endOffset <= minOff {
|
||||
c.activePath = c.activePath[:i+1]
|
||||
break
|
||||
node := c.activePath[i]
|
||||
if i > 0 && node.startOffset() >= minOff {
|
||||
continue
|
||||
}
|
||||
if node.endOffset > minOff {
|
||||
node = splitNode(node, minOff-node.startOffset(), c.caches, &c.pagedOutBytes)
|
||||
}
|
||||
c.activePath = append(c.activePath[:i], node)
|
||||
break
|
||||
}
|
||||
|
||||
// Update last-used time on only the final used node. For recurrent
|
||||
|
||||
Reference in New Issue
Block a user