Checkpoints quantized with NVIDIA's Model Optimizer carry a per-tensor
scale on top of the group scales; MLX-quantized nvfp4 does not. The
expert loaders dropped it, mis-scaling every expert output for those
models. Apply it to the gather outputs on the quantized path and fold
it into the weights when dequantizing.
Gathering gate and up separately cost a third expert gather per MoE
layer. Keep gate_up packed as one tensor, joining it at load when the
checkpoint ships the halves separately, and split the gather's output
instead.
Output is byte-identical; decode is 4% faster (7.89 -> 7.58 ms/token on
M5 Max) and prefill 9% faster.
The GLM parser buffered tool calls until it observed </tool_call>, but ignored the terminal done signal. If the model omitted or partially emitted the outer closing tag, Ollama returned a successful empty response instead of a tool call or an actionable error, leaving coding agents unable to continue.
On end-of-stream, finalize only structurally complete calls for declared tools with all required arguments. Complete calls missing only the outer delimiter now proceed through the existing parser, while genuinely truncated calls return an explicit error rather than being silently dropped.
Fixes#16497
This refactors the existing integration tests into 3 priumary groups: fast,
release, and library. It also refines some of the release tests to drop some
of the older models and pick up newer models, while retaining the broad
coverage in the library group.
Add a laguna-v8 renderer/parser matching the Laguna XS 2.1 template, and fix v2 handling of embedded thinking and structured tool arguments.
Prevent FP16 overflow in Metal's quantized routed-MoE prefill path by scaling the linear branch and folding the inverse into the routing scale. Other backends and token-generation paths are unchanged.
Add comprehensive v2/v8 Jinja parity and parser tests.
Add compute capability 10.0 to the Linux CUDA v12 preset so B200-class devices can use the cuda_v12 backend with drivers that do not meet the CUDA v13 minimum.
Fixes#12583
The bare `ollama` command (and `ollama launch` with no integration) used a
bespoke `ensureServerRunning` that forked `ollama serve` directly and polled
its heartbeat forever (no timeout, no platform-aware launch). Every other
subcommand (`ollama run`, `ollama pull`, `ollama launch <integration>`, ...)
goes through `checkServerHeartbeat` -> `startApp`, so the root command behaved
differently and could hang indefinitely.
Route `runInteractiveTUI` through `checkServerHeartbeat(cmd, nil)` — the same
path `ollama launch <thing>` uses — so the root command is consistent and no
longer runs an unbounded server-spawn loop. `ensureServerRunning` and its
`backgroundServerSysProcAttr` helpers (only it referenced them) are removed,
along with the now-unused `os/exec` import.
The platform `startApp`/`waitForServer` paths are unchanged, so behavior on
macOS/Windows is identical to the other subcommands, and on Linux the root
command now errors the same way the subcommands already do when no server is
running.
Incorporate the upstream Gemma4 chat template refinements for tool-calling stability, turn closure, and multi-turn reasoning. This updates the native renderer and checked-in HF template fixtures to keep adjacent assistant/tool continuations in the same model turn, add the post-tool thought-channel cue when thinking is enabled, and match Google's default of not replaying historical thinking before a later user turn.
Also preserve null tool arguments through Gemma4 rendering/parsing and extend the Jinja2 parity coverage for these upstream behaviors.
Per-token cost of the batched head forward keeps falling until the
flush is large enough to reach the fastest kernels: NAX matmul tiles
for dense heads, and the segmented gather path for MoE heads, which
needs tokens*topK/experts >= 4. Measured across the qwen3.6 heads,
256 is the smallest cap past every threshold and within a few percent
of each head's per-token floor. The cost is bounded: up to 2.5 MiB of
pinned hiddens per request and a flush stall under one decode step.
A draft cache pairs each slot with the token that follows it, so the
deepest stored pair always names one token past what a prefix match
can verify - at generation end, the sampled-but-never-committed
final token. Restoring at the match point reuses that pair blind: a
stop token stripped from the next prompt, or any divergence at the
boundary, leaves it stale, and pairing never rewrites below the
resume position, quietly lowering draft acceptance.
Key the trie by token pairs instead: the key for offset i packs
(token i, token i+1), so matching k keys verifies k+1 tokens and
every match is a valid restore point. A pair is reused only if the
token it names matched, and prefill re-evaluates the boundary token,
rebuilding its pair with the token that actually follows. A token
gets a key only once its successor is recorded, so endings record the
final sampled token - never forwarded - and the trie stays level with
the caches. Without a look-ahead the keys are the tokens and behavior
is unchanged.
The recorded tokens' slice bounds used to reject state past them for
free; close now checks the invariant against the stored keys
directly. The test harness rests requests the way the pipeline does -
the deepest recorded token never enters the caches.
The caches, the speculation binding, and the drafter were each built
lazily inside the first request: begin constructed the caches, and open
bound the cache partition and made a fresh drafter every time. All of
it is a property of the loaded model, so build it once at load.
newPrefixCache replaces the lazy construction in begin, speculation
binds when it is created, and the drafter splits the way speculation
does: a persistent mtpDrafter constructed at load opens each request's
mtpDraftSession, whose constructor syncs the pairing cursor to the
draft caches' restored offset.
Keeping the recurrent conv state small was handled unevenly: the committed
live state was recopied on every commit — wasted work on single-token
decode, where the window is already tiny — while boundary states captured as
snapshots could still be plain slices of the forward-sized convolution
buffer. A cached slice pins that whole buffer even though the trie's eviction
accounting only counts the slice's bytes, so recurrent cache memory piled up
across requests and eviction could never reclaim it.
Compact each boundary state to its real size once, where it is produced in
the conv wrapper, so live state and snapshots own only their own bytes and
eviction sees the true cost. Single-token decode leaves the already-tiny
window as a slice.
Fixes#16698
The MTP validation forward schedules a snapshot at every drafted token, which
made CausalConv1D re-run the depthwise conv once per segment to recover each
boundary's conv tail. A conv boundary state is just the trailing convTail
input positions, so run the conv once over the whole window and slice each
boundary tail from the shared buffer, removing the per-token conv launches.
MLX now requires a macOS 26.2 deployment target for NAX kernels. Ollama's Metal v4 build still targeted 26.0, so recent MLX bumps silently built mlx_metal_v4 without NAX kernels.
Qwen3.5/Qwen3-Next architecture strings contain the substring "qwen3", so the
broad qwen3 match claimed them for the generic parser and qwen3-coder
renderer, whose template doesn't frame the thinking block — an empty
<think></think> leaked into content and think=false was ignored. Match the
family first via isQwen35Family so the parser, renderer, and
thinking-capability checks share one variant list.
