model is one package with three jobs: the contract between the runner
and the architectures, the opened checkpoint, and building nn layers
from checkpoint tensors. Its files did not say which was which. base.go
carried the folded package's name over the interfaces and the registry,
root.go held the safetensors header scan next to Root, and quant.go
mixed the nvfp4 global-scale helpers with quant parameter resolution.
base.go becomes model.go, named for what it holds. root.go keeps Root
and Open; TensorQuantInfo and the header scan join quant.go, so
everything the checkpoint says about quantization is read and resolved
in one file. The global-scale helpers move to globalscale.go with their
tests. Root.Close, a no-op with one caller, goes. No code changes
otherwise.
nn.go held every layer type in the package apart from attention,
recurrence and rope: the Linear and Embedding interfaces with their dense
and quantized types, Conv1d, RMSNorm, LayerNorm and MultiLinear, with one
test file to match. Finding a layer meant scanning the file named after
the package.
Each layer kind gets its own file: linear.go and embedding.go hold the
interface and the dense and quantized types, conv.go, norm.go and
multilinear.go take the rest, and nn_test.go splits the same way. The
Layer and MultiLinearLayer interfaces go; nothing implemented or accepted
them. No code changes otherwise.
The runner and its weight loader read a model's manifest through
x/imagegen/manifest, the last piece of the removed image generation
engine. It was a hand-rolled copy of the manifest package: its own model
name parser with the default registry and namespace spelled out, its own
blob path builder, and re-spelled media types, plus a model_index.json
reader and other helpers that nothing has called since the engine went.
The manifest package gains the three lookups the runner needs, a config
layer by path, its contents, and the tensor layers, and ReadConfigJSON is
built on the second of them. The weight loader resolves the model name
with the shared parser, which fills in the same defaults the copy did, and
locates blobs with BlobsPath. The architectures' calls to read their
config.json compile unchanged. x/imagegen is gone.
model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
Several pieces outlived the code that used them. The root tokenizer
package implemented the GGUF-side vocabularies for the Go engine and the
safetensors-to-GGUF converter; nothing has imported it since the
converter went. ml/backend.go held the Go engine's Backend, Context and
Tensor interfaces, with fs.Config existing only to be returned from them,
and a single CUDA template instance under ml/backend/ggml survived the
engine removal along with the gitattributes entries for that tree and the
CI change-filter globs for it and for the long-gone llama/llama.cpp. From
the image generation engine, an integration test group that no test
registers, its build tag, and the StepBar progress widget remained.
DeviceInfo.IsBetter has no caller at all.
All of it goes. Tidying the module file drops the regexp2 dependency and
leaves protobuf as an indirect requirement. The llama3.2 tokenizer
fixtures stay: the MLX runner's tokenizer uses them for its GGML parity
test.
The runner package used to pick an engine from the first argument of the
runner subcommand. Only the MLX engine is left, so the dispatcher has a
single arm, its README still describes the removed Go runner's flags and
endpoints, and the standalone cmd/runner binary exists only to invoke it.
We call mlxrunner.Execute directly from the hidden runner subcommand and
drop the --mlx-engine argument from the command line the MLX client
spawns. Both sides ship in the same binary, so nothing has to accept both
forms. The runner package and cmd/runner are removed.
The subcommand's help hook hands the runner a bare --help. cobra calls the
hook with no arguments for `ollama help runner`, which used to index past
the end of the slice.
Includes quantized matmul corruption fix, which impacts nvfp4 multimodal models (gemma4 vision towers). Deferring wiring up the new MLX-C thread-local stream/sync APIs for now.
* create: add server-side MLX imports and drop GGUF conversion
Support safetensors imports through the MLX create pipeline both locally and on the server, including remote upload/staging, draft layer handling, cancellation propagation, transfer limits, and shared manifest/blob writing.
Limit GGUF create to wrapping existing GGUF inputs into Ollama manifests. Remove the in-tree safetensors-to-GGUF converter, server quantization path, and converter-only dependencies so GGUF conversion and quantization stay in llama.cpp tooling.
Keep the MLX path focused on supported safetensors model creation with validation before MLX work, and expose that flow without the --experimental CLI gate.
* address comments
* add client side gguf create fast path
* address comments
* rebase adjustments
Gemma3n's MobileNetV5 projector silently produces corrupted image
embeddings on the CPU backend - no error, the model just describes the
wrong image (reproduced on llama.cpp b10760; gemma4's encoder is fine on
CPU). Without this guard the existing partial-offload, limited-VRAM, and
OOM-retry fallbacks would pick the CPU projector on exactly the small
GPUs where gemma3n lands.
Loading a model can transform tensors after reading them: qwen3.5 models
pack their linear-attention projections into one layout, and MoE models
fuse the gate and up expert stacks. The buffers those transforms consume
go back to MLX's allocator pool rather than to the system, and nothing
releases the pool until the first request finishes. On qwen3.8:27b-mlx
that is 2.15 GiB held idle on top of 16.9 GiB of weights, counted in the
runner's reported memory the whole time.
Clear the pool once the weights are evaluated. Models whose tensors load
unchanged, such as gemma4, leave nothing in the pool and are unaffected.
Models and layers checked optional weights for nil and also for a handle
that no longer refers to an array, and evaluation and weight collection
skipped such handles. No path produces one: a missing tensor is nil, and a
handle only loses its array when its scope frees it, after which using it
is a bug. The nil checks stay; the validity check is internal to the
bindings now.
The bindings freed arrays by sweeping everything not pinned, so freeing
anything required knowing what every other caller still held, and code
that never swept accumulated until memory ran out. The prefix cache's
eviction of a long stored path did exactly that: each merge copied the
KV snapshots and nothing freed the consumed copies until the request
ended, which drove a second long request past physical memory.
Every array now belongs to a scope. A function scope, entered with Scoped
or one of the ScopedEval forms, frees what was created in it when the
function returns; results leave only by being returned. A held scope is
closed by its holder and frees what was attached to it. A graph is
built in a function scope and evaluated after it, so the eval frees each
intermediate as it consumes it. Pin, Unpin, Sweep, and the array list's
mutex are gone.
On an M5 Max with qwen3.8:27b-mlx, the second 84k-token request after a
stored one peaks at 35 GB instead of 57 GB; the cold path is unchanged.
The copies themselves are untouched, so restoring an owned path can still
exceed memory.
The scheduler starts the next load as soon as Close returns. The MLX client
sent SIGINT, gave the process five seconds, then sent SIGKILL and returned
without waiting, so a runner that could not take the signal was still
exiting, with its memory still held, when the next load began. The runner
has no signal handler, so SIGINT was already a kill.
Load also started the process and recorded it without the client's mutex,
so a Close racing with a load at server shutdown could find nothing to stop
and leave the runner it missed running.
Close now kills the process and waits for it to be reaped, as the
llama-server client does. Load starts and records the process under the
mutex and refuses to start once Close has run.
On Apple silicon the scheduler's free-memory figure for the GPU is the
Metal working set minus what Ollama's own runners report. It does not see
memory held by other applications, so a second MLX model can pass the fit
check on a machine that is already short of memory, and the load pushes
the system into swap and compression.
While other models are loaded, the MLX fit check now also bounds the
available memory by the system's free memory on shared-memory GPUs, the
same rule llama-server loads already apply. A miss evicts an idle model
and retries instead of starting the load. First loads are unchanged: with
nothing else loaded, the model loads against the working-set figure alone,
as both engines do today. The check also does not cover memory that grows
after load, such as KV caches and prefix-cache snapshots.
Eviction skipped every node on the active path, so a conversation's own
turn checkpoints were never reclaimed no matter how far over budget the
trie was. On models with sliding-window or recurrent layers each turn's
checkpoint is a full copy of that state, 800 MiB per turn on
gemma4:31b-mlx, and a long chat grows without bound. The scheduler then
counts that memory as in use and evicts the model to load anything
else.
Only the frontier and branch points are protected now. Any other node,
active or not, is evicted least recently used first. On the active path
that merges a turn into the next one: the merged node keeps the newer
whole-state, and the KV snapshots there are lazy views of the live
buffer, so nothing is copied. Rewinding to an evicted turn resumes at
the newest surviving checkpoint before it.
qwen3.8:27b-mlx on an M5 Max, the same short question every turn with
24 tokens generated per reply, 8 GiB budget, 17.2 GiB of weights:
turn | before: paged out nodes reported | after: paged out nodes reported
11 | 4.61 GiB 33 21.6 GiB | 4.61 GiB 33 21.6 GiB
21 | 7.91 GiB 56 24.9 GiB | 7.92 GiB 56 24.9 GiB
31 | 8.46 GiB 60 25.5 GiB | 7.94 GiB 56 25.0 GiB
41 | 9.90 GiB 70 26.9 GiB | 7.96 GiB 56 25.0 GiB
50 | 11.19 GiB 79 28.2 GiB | 7.98 GiB 56 25.0 GiB
Fixes#17783
When a request resumes partway through a cached edge, the node holding
that edge was dropped from the active path, because the path has to end
at the live offset for close and the prefill captures to extend the trie
from its last node. Off the path, the node was an ordinary leaf with a
stale last-used time, so eviction removed it first.
The captures taken during that request start at the resume offset, but
attach rebuilds the missing node from the path's last node, so the new
node's edge begins earlier than its KV snapshot. A later request
resuming there was refused by the KV cache and re-prefilled from
scratch. If eviction first merged the node into its parent, the two
snapshots were concatenated as if adjacent, and the restore reported a
hit while the buffer held tokens from other positions.
Split the node at the live offset instead. The head stays on the path
and gets the last-used update. Only the unused tail can be evicted, and
losing it costs nothing. The split only happens when every layer can
rewind into the edge, so it never involves a recurrent layer, and the
head gets the same KV-only snapshots a close-time split already
produces. When the request follows the edge, compaction merges the
halves back.
The tokens of a non-causal media item attend to each other in both
directions, so the item has to be evaluated in one forward. Prefill
honors that when it picks chunk boundaries, but the prefix cache did
not: a snapshot could be taken partway through an item, and a request
that resumed there would evaluate the rest of the item alone and
compute different attention for it.
Snapshots scheduled inside a non-causal item now land at its end, and
a match that ends inside one resumes at its start.
KV snapshots must cover a node's edge exactly. Recurrent and
sliding-window state is only useful at a node's end, and a node may
have none: a request resuming there lands on the previous checkpoint
and begin schedules a capture at the match.
The header claimed every node carries its snapshots from creation,
which a node split out of an existing edge at close cannot. That hid a
gap: when a response is a prefix of a stored one, close lands on the
split-off head with the caches resting at its end, and pageOut skipped
the capture because the node already had a KV snapshot.
Restate the header as the rules that hold, and make pageOut capture
whatever layers a node is missing. The scheduling comment also said
eviction preserves user nodes; it only resists compaction.
Loading GGUF metadata is an expensive operation. Two caches had evolved to
mitigate this, and the two capability implementations produced inconsistent
results for some models.
This PR now extracts the metadata once per blob into a file at
<OLLAMA_MODELS>/metadata/sha256-<hex>.json. Only arrays over 4096 elements and
non-finite floats are left out.
Direct Capabilities() discovery now costs us instead of ms. /api/tags can
build directly from manifests and the extracted metadata.
progress: join the render loop in stop and do the final writes after the
goroutine exits, so Stop/StopAndClear cannot race an in-flight render on
the shared bufio.Writer.
sched: read the unload-mutable fields in runnerRef.LogValue only under a
successful refMu.TryLock and omit them when contended, since slog resolves
it on goroutines that may already hold refMu.
Eva H