Includes quantized matmul corruption fix, which impacts nvfp4 multimodal models (gemma4 vision towers). Deferring wiring up the new MLX-C thread-local stream/sync APIs for now.
* create: add server-side MLX imports and drop GGUF conversion
Support safetensors imports through the MLX create pipeline both locally and on the server, including remote upload/staging, draft layer handling, cancellation propagation, transfer limits, and shared manifest/blob writing.
Limit GGUF create to wrapping existing GGUF inputs into Ollama manifests. Remove the in-tree safetensors-to-GGUF converter, server quantization path, and converter-only dependencies so GGUF conversion and quantization stay in llama.cpp tooling.
Keep the MLX path focused on supported safetensors model creation with validation before MLX work, and expose that flow without the --experimental CLI gate.
* address comments
* add client side gguf create fast path
* address comments
* rebase adjustments
Gemma3n's MobileNetV5 projector silently produces corrupted image
embeddings on the CPU backend - no error, the model just describes the
wrong image (reproduced on llama.cpp b10760; gemma4's encoder is fine on
CPU). Without this guard the existing partial-offload, limited-VRAM, and
OOM-retry fallbacks would pick the CPU projector on exactly the small
GPUs where gemma3n lands.
Loading a model can transform tensors after reading them: qwen3.5 models
pack their linear-attention projections into one layout, and MoE models
fuse the gate and up expert stacks. The buffers those transforms consume
go back to MLX's allocator pool rather than to the system, and nothing
releases the pool until the first request finishes. On qwen3.8:27b-mlx
that is 2.15 GiB held idle on top of 16.9 GiB of weights, counted in the
runner's reported memory the whole time.
Clear the pool once the weights are evaluated. Models whose tensors load
unchanged, such as gemma4, leave nothing in the pool and are unaffected.
Models and layers checked optional weights for nil and also for a handle
that no longer refers to an array, and evaluation and weight collection
skipped such handles. No path produces one: a missing tensor is nil, and a
handle only loses its array when its scope frees it, after which using it
is a bug. The nil checks stay; the validity check is internal to the
bindings now.
The bindings freed arrays by sweeping everything not pinned, so freeing
anything required knowing what every other caller still held, and code
that never swept accumulated until memory ran out. The prefix cache's
eviction of a long stored path did exactly that: each merge copied the
KV snapshots and nothing freed the consumed copies until the request
ended, which drove a second long request past physical memory.
Every array now belongs to a scope. A function scope, entered with Scoped
or one of the ScopedEval forms, frees what was created in it when the
function returns; results leave only by being returned. A held scope is
closed by its holder and frees what was attached to it. A graph is
built in a function scope and evaluated after it, so the eval frees each
intermediate as it consumes it. Pin, Unpin, Sweep, and the array list's
mutex are gone.
On an M5 Max with qwen3.8:27b-mlx, the second 84k-token request after a
stored one peaks at 35 GB instead of 57 GB; the cold path is unchanged.
The copies themselves are untouched, so restoring an owned path can still
exceed memory.
The scheduler starts the next load as soon as Close returns. The MLX client
sent SIGINT, gave the process five seconds, then sent SIGKILL and returned
without waiting, so a runner that could not take the signal was still
exiting, with its memory still held, when the next load began. The runner
has no signal handler, so SIGINT was already a kill.
Load also started the process and recorded it without the client's mutex,
so a Close racing with a load at server shutdown could find nothing to stop
and leave the runner it missed running.
Close now kills the process and waits for it to be reaped, as the
llama-server client does. Load starts and records the process under the
mutex and refuses to start once Close has run.
On Apple silicon the scheduler's free-memory figure for the GPU is the
Metal working set minus what Ollama's own runners report. It does not see
memory held by other applications, so a second MLX model can pass the fit
check on a machine that is already short of memory, and the load pushes
the system into swap and compression.
While other models are loaded, the MLX fit check now also bounds the
available memory by the system's free memory on shared-memory GPUs, the
same rule llama-server loads already apply. A miss evicts an idle model
and retries instead of starting the load. First loads are unchanged: with
nothing else loaded, the model loads against the working-set figure alone,
as both engines do today. The check also does not cover memory that grows
after load, such as KV caches and prefix-cache snapshots.
Eviction skipped every node on the active path, so a conversation's own
turn checkpoints were never reclaimed no matter how far over budget the
trie was. On models with sliding-window or recurrent layers each turn's
checkpoint is a full copy of that state, 800 MiB per turn on
gemma4:31b-mlx, and a long chat grows without bound. The scheduler then
counts that memory as in use and evicts the model to load anything
else.
Only the frontier and branch points are protected now. Any other node,
active or not, is evicted least recently used first. On the active path
that merges a turn into the next one: the merged node keeps the newer
whole-state, and the KV snapshots there are lazy views of the live
buffer, so nothing is copied. Rewinding to an evicted turn resumes at
the newest surviving checkpoint before it.
qwen3.8:27b-mlx on an M5 Max, the same short question every turn with
24 tokens generated per reply, 8 GiB budget, 17.2 GiB of weights:
turn | before: paged out nodes reported | after: paged out nodes reported
11 | 4.61 GiB 33 21.6 GiB | 4.61 GiB 33 21.6 GiB
21 | 7.91 GiB 56 24.9 GiB | 7.92 GiB 56 24.9 GiB
31 | 8.46 GiB 60 25.5 GiB | 7.94 GiB 56 25.0 GiB
41 | 9.90 GiB 70 26.9 GiB | 7.96 GiB 56 25.0 GiB
50 | 11.19 GiB 79 28.2 GiB | 7.98 GiB 56 25.0 GiB
Fixes#17783
When a request resumes partway through a cached edge, the node holding
that edge was dropped from the active path, because the path has to end
at the live offset for close and the prefill captures to extend the trie
from its last node. Off the path, the node was an ordinary leaf with a
stale last-used time, so eviction removed it first.
The captures taken during that request start at the resume offset, but
attach rebuilds the missing node from the path's last node, so the new
node's edge begins earlier than its KV snapshot. A later request
resuming there was refused by the KV cache and re-prefilled from
scratch. If eviction first merged the node into its parent, the two
snapshots were concatenated as if adjacent, and the restore reported a
hit while the buffer held tokens from other positions.
Split the node at the live offset instead. The head stays on the path
and gets the last-used update. Only the unused tail can be evicted, and
losing it costs nothing. The split only happens when every layer can
rewind into the edge, so it never involves a recurrent layer, and the
head gets the same KV-only snapshots a close-time split already
produces. When the request follows the edge, compaction merges the
halves back.
The tokens of a non-causal media item attend to each other in both
directions, so the item has to be evaluated in one forward. Prefill
honors that when it picks chunk boundaries, but the prefix cache did
not: a snapshot could be taken partway through an item, and a request
that resumed there would evaluate the rest of the item alone and
compute different attention for it.
Snapshots scheduled inside a non-causal item now land at its end, and
a match that ends inside one resumes at its start.
KV snapshots must cover a node's edge exactly. Recurrent and
sliding-window state is only useful at a node's end, and a node may
have none: a request resuming there lands on the previous checkpoint
and begin schedules a capture at the match.
The header claimed every node carries its snapshots from creation,
which a node split out of an existing edge at close cannot. That hid a
gap: when a response is a prefix of a stored one, close lands on the
split-off head with the caches resting at its end, and pageOut skipped
the capture because the node already had a KV snapshot.
Restate the header as the rules that hold, and make pageOut capture
whatever layers a node is missing. The scheduling comment also said
eviction preserves user nodes; it only resists compaction.
Loading GGUF metadata is an expensive operation. Two caches had evolved to
mitigate this, and the two capability implementations produced inconsistent
results for some models.
This PR now extracts the metadata once per blob into a file at
<OLLAMA_MODELS>/metadata/sha256-<hex>.json. Only arrays over 4096 elements and
non-finite floats are left out.
Direct Capabilities() discovery now costs us instead of ms. /api/tags can
build directly from manifests and the extracted metadata.
progress: join the render loop in stop and do the final writes after the
goroutine exits, so Stop/StopAndClear cannot race an in-flight render on
the shared bufio.Writer.
sched: read the unload-mutable fields in runnerRef.LogValue only under a
successful refMu.TryLock and omit them when contended, since slog resolves
it on goroutines that may already hold refMu.
The runner compiled a format as a JSON Schema, the one grammar kind its
xgrammar binding exposed. A structural tag holds a schema as one node of
a larger tree and also expresses what a schema cannot: free text around
constrained spans, a thinking region that closes before constrained
content, tool calls pinned to their schemas.
The runner now compiles structural tags only; its client wraps the API's
formats into one, which compiles to the same grammar as before. The JSON
token and vocabulary caps go with it: neither bounds compile cost, which
follows the grammar's state count. The byte and nesting caps stay.
A structured-output request could not use a model's draft head: it
decoded one token at a time, at roughly half the speculative throughput
on a dense 27B MTP model.
The grammar is enforced during verification instead: each draft
position's logits are masked before rejection sampling, so an invalid
draft is never accepted and every emitted token obeys the grammar.
Drafts stay unconstrained; constraining the draft chain would stall its
pipelined forwards.
Speculative steps also now dispatch the drafts before the host builds
the verification graph, worth 4-8% end to end at a fixed draft depth on
MTP models, with or without a grammar.
* ci: rebuild MLX macOS test payloads the release can't supply
The MLX unit test dependency payload can go stale when a PR changes the
xgrammar wrapper (release dylib lacks the new symbols) or bumps MLX pins (no
matching release, so MLX tests silently skipped). Compare the matched tag's
build rules and wrapper sources against the checkout, rebuilding only
libollama_xgrammar.dylib on drift; with no matching release, build one Metal
variant at the platform default.
* address comments
Safetensors gemma4 imports served by the MLX engine now answer image
and audio chats. Images run through both vision architectures: the
transformer tower (26B, 31B, e-series) and the 12B's encoder-free
unified embedder. Audio arrives through the same intake the ollama
API already accepts for gemma4 GGUFs — WAV bytes in the images field,
OpenAI input_audio parts, and /v1/audio/transcriptions uploads — with
the e2b/e4b checkpoints running clips through their conformer audio
encoder and the 12b unified checkpoint embedding the raw waveform
directly. Clips longer than 30 seconds are split evenly into chunks
of at most 30 seconds, cut at pauses, and encoded independently.
Each modality serves only checkpoints that carry it: 26B/31B have no
audio config and reject audio input, and checkpoints with an
unrecognized vision architecture still load as text-only models and
reject image requests.
The server previously hid the vision and audio capabilities for
gemma4 safetensors because the engine served neither. Both
suppressions are removed, and existing imports start advertising the
capabilities without re-importing since import already records them.
Audio-capable models need mono PCM at their expected sample rate
before model-specific feature extraction. Like the image decoder set
in base, the supported audio containers are decided once here so
every model accepts the same formats: WAV, covering integer and float
PCM, extensible headers, and multi-channel downmix. Anything else is
rejected as unrecognized; supporting another container later means
one new decoder here, with no model or runner changes.
Input at other sample rates is resampled through a band-limiting
filter, so mismatched rates degrade gracefully instead of aliasing.
Decoded clips are capped at ten minutes: the declared rate comes from
an untrusted header, and the cap is what keeps a small file claiming
an absurdly low rate from resampling into an enormous allocation.
Models whose encoder takes clips only up to a fixed length split
longer clips with Split. Chunks are sized evenly, and each cut moves
to the quietest point within a few seconds of its even share, so a
boundary lands on a pause where the clip has one instead of severing
a word between two independently encoded chunks. The even sizing
bounds the search so no chunk can end up over the limit.
The bindings captured MLX error messages but checked almost no calls,
so a failure continued with a null output and surfaced later as zero
results, skipped evals, or an unrelated crash.
Wrap every call in mlxCheck. Paths that return an error or disable a
GPU kernel backend use mlxError instead, and the lookups where a
non-zero status means a miss read the buffer first and then treat the
status as data.
Also free the string handles behind Array.String and the log values
after the call that fills them; they were freed before it and leaked.
MLX runs on one goroutine locked to its OS thread, so the thread-local
error buffers and closure-based check helpers defended against a
calling pattern that is already invalid.
Replace them with a single buffer that the handler fills and Go reads
after every call. mlxError returns the captured message; mlxCheck
panics on it and passes the call's result through, so a checked call
is one expression. Only an int status carries a failure signal, which
lets a message next to a zero status be reported as an earlier
unchecked call.
Fix two tests that relied on errors being dropped: the laguna
mixed-precision fixture used an unsupported quantization group size,
and the compile callback test expected the callback's own panic.
Daniel Hiltgen