Gemma3n's MobileNetV5 projector silently produces corrupted image
embeddings on the CPU backend - no error, the model just describes the
wrong image (reproduced on llama.cpp b10760; gemma4's encoder is fine on
CPU). Without this guard the existing partial-offload, limited-VRAM, and
OOM-retry fallbacks would pick the CPU projector on exactly the small
GPUs where gemma3n lands.
Loading GGUF metadata is an expensive operation. Two caches had evolved to
mitigate this, and the two capability implementations produced inconsistent
results for some models.
This PR now extracts the metadata once per blob into a file at
<OLLAMA_MODELS>/metadata/sha256-<hex>.json. Only arrays over 4096 elements and
non-finite floats are left out.
Direct Capabilities() discovery now costs us instead of ms. /api/tags can
build directly from manifests and the extracted metadata.
progress: join the render loop in stop and do the final writes after the
goroutine exits, so Stop/StopAndClear cannot race an in-flight render on
the shared bufio.Writer.
sched: read the unload-mutable fields in runnerRef.LogValue only under a
successful refMu.TryLock and omit them when contended, since slog resolves
it on goroutines that may already hold refMu.
* ci: rebuild MLX macOS test payloads the release can't supply
The MLX unit test dependency payload can go stale when a PR changes the
xgrammar wrapper (release dylib lacks the new symbols) or bumps MLX pins (no
matching release, so MLX tests silently skipped). Compare the matched tag's
build rules and wrapper sources against the checkout, rebuilding only
libollama_xgrammar.dylib on drift; with no matching release, build one Metal
variant at the platform default.
* address comments
* Report cached prompt tokens
Add prompt_eval_cached_count to native responses and expose equivalent cached-token fields through the OpenAI- and Anthropic-compatible APIs. Keep prompt_eval_count as the logical input total while excluding cache hits from CLI and benchmark prefill rates. Surface processed and cached prompt counts in benchmark output.
Collect cache counts from llama-server and MLX, preserve coherent metrics across two-pass structured generation.
Fixes#8008
Related to #15758
* review comments
* ci: wire up MLX unit tests for PR runs
Download the latest Darwin release payload matching the current MLX and MLX-C revisions so macOS PR tests can exercise MLX without rebuilding it. If no matching release exists after a pin bump, leave MLX tests skipped until the next release.
Add whole-tree race coverage and smoke-run committed benchmarks. Verify generated UI types, and stabilize tests exposed by the broader CI coverage.
* review comments
* mlx: run tests on one pinned worker
Keep MLX tests and benchmarks on a shared pinned thread while preserving Fatal, Skip, and Cleanup semantics. Also clean stale CI payloads and ensure updater workers shut down cleanly.
* addres comments
* llama.cpp: version bump b10729
Regenerate the compat hooks patch for b10729: upstream removed the
whole-tensor load_data_for read (last consumer was llama-quantize,
which now reads slabs via load_data_range). Keep the existing hook
surface (constructor, skip loops, load_all_data, mtmd/clip) unchanged
and add maybe_load_text_tensor_range, which materializes a text load
op's output once per tensor and serves the new (offset, size) slab
reads from that cache.
* address comments
* Honor model generation defaults
Model-authored sampler defaults from GGUF metadata and HF generation_config.json were ignored, so built-in Ollama defaults could override model intent unless parameters were set in the Modelfile or request. The fix parses those defaults into model config and applies them before Modelfile/request options, preserving the expected precedence order.
* review comments
* address comments
Pi's Edit() only set baseUrl when creating a new ollama provider entry.
On subsequent launches it preserved whatever baseUrl was already in
~/.pi/agent/models.json, so switching OLLAMA_HOST to a remote server had
no effect — Pi would still connect to localhost.
Edit() now ensures baseUrl reflects the current OLLAMA_HOST. Models()
returns nil when the stored baseUrl no longer matches, so the launcher
only calls Edit() when the host has actually drifted. User-customized api
and apiKey fields are still preserved.
When a builtin parser rejects model output, the completion callback wrote the
error to an unbuffered channel and returned. The callback cannot stop
generation -- it has no error return -- so the next chunk re-entered the
callback, hit the same parse error and blocked writing to a channel the
consumer had already stopped reading after emitting its 500. The completion
never returned, the goroutine leaked and the runner request was never
released, so retrying the same prompt hung with no log output until the client
gave up.
Record the parse error, cancel the completion, and report it once the
completion has returned. Parse failures landing on the final chunk were
already terminal, which is why non-thinking requests and the direct
qwen3-coder parser path failed cleanly and only thinking mode wedged.
ChatHandler and GenerateHandler share the defect: both run the same parser in
the same shape of callback behind a consumer that stops reading at the first
error. GenerateHandler had no cancel func at all, so one is added there.
Fixes#17825
Coding clients may insert runtime system messages after the initial user turn. The shared Qwen renderer rejected these transcripts before rendering, turning a potentially usable non-standard request into an HTTP 500.
Pass non-leading system turns through the existing raw ChatML path and warn when qwen3.8 encounters one. Extend the Anthropic tool-route integration scenario to cover this message pattern and remove the obsolete rejection test.
llama-server does not currently support WebP image payloads. Detect WebP media before forwarding, and transcode it to PNG. Pass all other media through unchanged.
Replace an existing vision integration image with a lossless WebP version so we now have coverage of JPG/PNG/WebP formats.
Fixes#17753
* qwen3.8: support developer instructions
Qwen3.8 does not define a developer role, while OpenAI-compatible coding agents commonly send developer instructions before user messages. Fold the leading system/developer instruction prefix into a single system turn before Qwen3.8 validation, preserving instruction precedence without changing Qwen3.5 or other renderer behavior.
Add streaming tool-call integration coverage for the native Ollama, OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages request shapes. Each case exercises prior assistant tool calls, tool results, follow-up rendering, and parsed tool-call output. Add Qwen3.8 to the release tools sweep.
Removes an unnecessary unit test that should not have been included in the original 3.8 PR.
* review comments
Qwen3.8 keeps the Qwen3.5 model architecture and parser, but its chat template adds reasoning-effort and preserved-thinking semantics. Detect those template markers during safetensors import, select the qwen3.8 renderer, and cover thinking, tools, continuation, and malformed parser input.
Make indexed safetensors imports use the weight map's shard names instead of independently filtering files by the model-* convention. Reject unsafe shard paths, ignore unindexed tensors, and fail when an indexed weight is missing or stored in a different shard. Retain the conservative model-* scan when no index is present.
Treat Classification.Quantize as the effective tensor format and pass it to the manifest writer. This records file_type for automatic block-FP8-to-MXFP8 conversion and recognized prequantized inputs, preserves requested quantization and base-plus-draft behavior, and avoids claiming one type for mixed or unknown formats.
Normalize both supported convolution weight layouts with an explicit reshape. Add focused unit coverage for renderer selection, parser behavior, shard inventory, manifest metadata, and convolution layout; heavyweight reference-forward and release integration checks remain bring-up artifacts.
Updates Muse Glimmer Jinja reference template to the latest publisher version and mirror its explicit-system reasoning handling in the Go renderer.
Explicit system prompts now normalize "Reasoning effort" to "Reasoning strength" and skip adding a renderer-provided reasoning line when the prompt already contains one. This prevents duplicate or conflicting reasoning directives while preserving the default-system behavior.
Add reference tests for both normalization and deduplication, including Jinja-backed validation.
* launch: add Muse Code integration
Add `ollama launch muse` for Meta's Muse Code CLI.
Muse only takes a model catalog from settings.json (normally it fetches one from its provider and refuses to start otherwise), and that file's endpoint_transport is a global provider switch. So the integration writes a settings file under its own config root (~/.ollama/launch/muse-config via XDG_CONFIG_HOME), leaving a Meta-backed muse install untouched, and re-seeds it from muse's own persisted copy on later runs.
The launched model is preloaded so its catalog row carries the context length the server actually allocated, not the trained maximum; the loaded-context helpers move from cmd/agent_tui.go into cmd/launch for reuse.
Muse sends reasoning efforts outside Ollama's scale (minimal, xhigh, ultra), which were hard 400s; clamp them to the nearest tier in one helper shared by the chat and responses converters.
The registry entry stays Hidden (alias "muse-code"), like kimi and vscode.
* review comments
* skip muse test on windows (unsupported platform)
As we look to bring Linux and Windows MLX support online, instead of blocking
downloads at the registry to avoid users wasting time downloading a model they
can't run, shift the logic to the local side which knows if MLX is present or not.
Select the 3.5 parser and renderer from its checkpoint template, preserve its prompt semantics, and map medium reasoning effort to the final-user annotation expected by the reference template.
Exercise parser and renderer registration, create-time metadata inference, and exact Jinja parity so created models cannot silently fall back to the Nemotron 3 renderer.
The model occasionally emits a <|message|> boundary token in the invoke
name region, echoing the header form `to=read<|message|>`. The existing
recovery handled the tag inside a terminated name (`name="read<|message|>">`)
but not the fleet-observed shape where the tag replaces the `">` terminator
itself (`name="read<|message|><atem:parameter ...`), which failed the call
with "malformed ATEM parameter".
Replace the strip-after-cut recovery with a single name scan shared by
parseGlimmerATEM and the content fallback: the name ends at the first `">`,
boundary tokens before it are dropped, and a parameter element immediately
after a dropped token means the token replaced the terminator. Well-formed
calls are unaffected — a boundary token is never legitimate before the
terminator, and parameter values (where the literal text is preserved) only
appear after it. Murkier garbles still fail loudly, the recipient
cross-check still applies, and the recovery WARN is retained.
Add MLX support for Nemotron 3 Nano Omni, including the model implementation, Mamba2/recurrent pieces, MoE routing, and quantized NVFP4/MXFP8 expert paths.
Use a shared mapped MoE GatherQMM fast path under the generic moe_gather_qmm_mapped naming, with Metal-optimized NVFP4/MXFP8 block-mapped kernels and generic fallbacks for unsupported backends.
Serve the model's multi-token prediction head as a self-draft speculator, so speculative decoding needs no separate draft model.
Render the Nemotron prompt from the published chat template. The template the renderer was based on had drifted from the current reference; refreshing it surfaced five mismatches: stray leading newlines, the wrong turn separator and a trailing newline before the generation prompt; /think and /no_think toggles left in user turns; a trimmed system message the template leaves intact; a user block opened by a leading tool message; and Go scalar syntax for schema extras where the template applies Python str(), sending true/false/<nil> in place of True/False/None. Reference tests now render every case through the template itself.
Also harden the Nemotron parser path shared by both backends: while collecting thinking, preserve whitespace before partial </think>, <think>, and <tool_call> fakeouts, with streaming tests covering those cases.
* glimmer: implement the Muse Glimmer model
MLX model (language + vision encoder) with DFlash draft wiring, llama-server DFlash support and rope-interleave fix, renderer and parser, tokenizer fixes, and the import quantization policy.
* mlxrunner: report committed prefill chunks after the sweep and eval
The drafter's flush evaluates its report, and an eval that runs while the chunk's construction handles are still live cannot free any intermediate buffer. On media chunks that retention keeps the whole vision tower resident and grinds the Metal allocator at its limit until the request dies. Pin the report's inputs across the sweep, report after the chunk materializes, and release media items after the report so a drafter can still capture the rows its deferred flush embeds.
* ci: retry CUDA pre-release download
Also bring back version tagging the MLX library with our git hash which was
accidentally dropped when imagegen was removed. Without this, the version
claimed to be the official tagged version, but we're typically using a git hash
with different content.
rootfs.diff_ids duplicated the manifest's layer digest list into the config blob and nothing ever read it. On per-tensor safetensors models the copy grows past 100KB and create excessively large config blobs with unused redundant data. Model identity is unaffected: it is the digest of the manifest itself, which already commits to every layer hash.
The Responses API groups related tools by domain: a tool with type "namespace" carries the real function definitions in a nested tools array. The conversion dropped that array, leaving the model a single schema-less pseudo-function and making every namespaced call undeclarable.
Expand namespace declarations into their member functions with namespace-qualified names, since api.Tool carries only a flat function name.
Relates to #15921: full Responses API parity also wants the namespace preserved as a separate field on tool calls in the output, which needs new api surface and is not addressed here.
* progress: fix data races on ticker, states, spinner, and bar state
NewProgress spawned start() which wrote p.ticker while stop() read and
cleared it with no synchronization; stop() and StopAndClear() also read
p.states and p.pos outside p.mu, Spinner's start() goroutine raced
Stop() and String() on s.value/s.stopped/s.ticker, and Bar.Set raced
Bar.String on currentValue/stopped/buckets (callback goroutine vs the
render goroutine). Detected by go test -race across cmd and cmd/launch
(~20 warnings; the Bar race is latent — never flagged because tests
don't interleave it, but real in production pull/push progress).
Create tickers before spawning the render goroutines and pass the
channel in, guard Progress internals with p.mu throughout stop() (via a
renderLocked core), and give Spinner and Bar their own mutexes.
* use a more idiomatic channel based done signal
DownloadNewRelease spawned a background checkForUpdate loop that read
package-level knobs (UpdateCheckInterval et al.) and returned without
waiting for it, so under -race the next test rewrote those globals while
the orphaned goroutine was still reading them. waitDownloadIdle (from
Cancel and WaitGroup-drain the loop before DownloadNewRelease returns,
and have TestCancelOngoingDownload join its download goroutine so the
drain is observable before the test exits.
golangci-lint ran with only-new-issues, which filters findings down to the
lines a PR adds. That silently drops any issue a diff introduces at a
distance, where the report anchors to a line the diff never touched.
CI now is enabled to scan all files. This PR also fixes the last few
straggler lint glitches outside of integration, which I'll tackle
in a follow up PR.
If we have one flake, don't stop other jobs that will most likely work so when
we re-run failed jobs, only the flake and dependents need to be run. This should
help reduce the time it takes to get past a flake and finish a release build.
* model: add Laguna MLX support
Add Laguna XS 2, XS 2.1, and S 2.1 support to the MLX model and create paths.
Read the source config to apply one quantization policy across dense and routed MoE layers. Keep the tied output head and router at source precision, quantize supported attention and expert projections, selectively promote sensitive expert down projections, and emit per-tensor metadata for mixed quantization blobs.
Correct dense expert loading, BF16 source-layout handling, expert global-scale shapes and dtypes, routing-score scaling, and mixed-precision expert dispatch. Gate/up and down projections select quantized or dense execution independently so promoted BF16 down projections do not force quantized gate/up weights through the dense fallback.
Optimize the forward pass with compatible gate/up fusion, sorted standard GatherMM and GatherQMM operations for larger prefills, model-local mlx.Compile closures for elementwise MoE work, and cache-backed 512-token prefill chunks. This keeps the implementation on maintained MLX operations without custom kernels.
Add focused tests for Laguna configuration variants, quantization policy and metadata, dense and routed expert loading, mixed-precision dispatch, compiled-versus-eager parity, fused projections, routing, and prefill chunking.
* review comments and S 2.1 performance fixes
Address renderer/parser selection and mixed-precision expert quantization review feedback.
Keep Laguna weights resident on Metal to prevent repeated paging of its large, sparsely accessed expert buffers. Scope this policy to Laguna GPU execution.
Remove obsolete 512-token prefill chunking now that the runner's 2048-token path is faster.
* review comments addressed
* fix create
Configure Metal residency after the MLX runner materializes model weights.
Wire up to the smaller of active model memory and the recommended working set, leaving pageable headroom for KV caches and request allocations. If residency setup fails, warn and continue with pageable memory.
Expose recoverable MLX C API errors and verify that an oversized wired limit preserves the previous state and leaves subsequent evaluation usable.
PsHandler iterated sched.loaded without holding loadedMu, racing with
scheduler goroutines that mutate the map. It also read runnerRef fields
(model, llama, expiresAt) that unload() and the expiration path mutate
under refMu, so a concurrently unloading runner could nil model out from
under the handler.
Instead of adding locking in routes.go, give the scheduler a small
snapshot API: loadedModels() copies the runner list under loadedMu, then
captures each runner's reporting fields under its refMu, respecting the
refMu-before-loadedMu lock ordering used by the expiration path. The
zero-expiresAt estimate for still-loading models moves into the
scheduler too, since it exists because of scheduler behavior.
Also remove the dead code Scheduler.GetRunner
app/updater: TestBackgoundChecker / TestAutoUpdateDisabledSkipsDownload hit 'TempDir RemoveAll cleanup: directory not empty' on macOS because the background checker goroutine keeps writing staged files into UpdateStageDir while t.TempDir cleanup runs. The checker's context is cancelled by the time cleanup runs, and after cancellation a new download cannot reach the filesystem (DownloadNewRelease aborts at its HEAD request before any write), so it suffices to wait for any in-flight download to drain. Add a test-only waitDownloadIdle helper (polls the existing cancelDownload sentinel under its lock) and register it via t.Cleanup so TempDir cleanup runs after staged-file handles close. No production code changes.
x/transfer: TestDownloadParallelism asserted elapsed <= 1s against 50ms-per-blob delays, too tight for Windows hosted runners' ~15ms timer granularity and shared-runner jitter. Each blob costs two server sleeps (resolve GET + body GET), so model the serial baseline from the deterministic request count, raise per-blob latency to 100ms so timer quantization is a small fraction of each delay, and key the budget to 75% of the serial baseline so the check still proves parallelism while tolerating jitter.
Update llama.cpp to pick up upstream Laguna implementation and remove Ollama's local Laguna implementation. Retain a narrow Metal-only scaling workaround for routed-MoE prompt overflow.
Translate older Ollama GGUF attention-gate and SWA metadata names so existing models continue to load.
The GLM parser buffered tool calls until it observed </tool_call>, but ignored the terminal done signal. If the model omitted or partially emitted the outer closing tag, Ollama returned a successful empty response instead of a tool call or an actionable error, leaving coding agents unable to continue.
On end-of-stream, finalize only structurally complete calls for declared tools with all required arguments. Complete calls missing only the outer delimiter now proceed through the existing parser, while genuinely truncated calls return an explicit error rather than being silently dropped.
Fixes#16497
This refactors the existing integration tests into 3 priumary groups: fast,
release, and library. It also refines some of the release tests to drop some
of the older models and pick up newer models, while retaining the broad
coverage in the library group.
Add a laguna-v8 renderer/parser matching the Laguna XS 2.1 template, and fix v2 handling of embedded thinking and structured tool arguments.
Prevent FP16 overflow in Metal's quantized routed-MoE prefill path by scaling the linear branch and folding the inverse into the routing scale. Other backends and token-generation paths are unchanged.
Add comprehensive v2/v8 Jinja parity and parser tests.
Add compute capability 10.0 to the Linux CUDA v12 preset so B200-class devices can use the cuda_v12 backend with drivers that do not meet the CUDA v13 minimum.
Fixes#12583
Incorporate the upstream Gemma4 chat template refinements for tool-calling stability, turn closure, and multi-turn reasoning. This updates the native renderer and checked-in HF template fixtures to keep adjacent assistant/tool continuations in the same model turn, add the post-tool thought-channel cue when thinking is enabled, and match Google's default of not replaying historical thinking before a later user turn.
Also preserve null tool arguments through Gemma4 rendering/parsing and extend the Jinja2 parity coverage for these upstream behaviors.
MLX now requires a macOS 26.2 deployment target for NAX kernels. Ollama's Metal v4 build still targeted 26.0, so recent MLX bumps silently built mlx_metal_v4 without NAX kernels.
* llm: allow iGPU mmproj offload with fit padding
llama.cpp's fit pass sizes text-model placement before the multimodal projector is loaded. Ollama had been avoiding that risk on non-Metal iGPUs by disabling projector offload entirely, which forces CLIP onto CPU on GB10 and Strix Halo even when the projector has ample memory available.
Let integrated GPUs use the same projector-memory check as other GPUs. When projector offload is enabled, add the estimated projector memory plus the existing 1 GiB headroom to Ollama-owned LLAMA_ARG_FIT_TARGET so fit leaves space for the later projector allocation. If Ollama/device setup already supplied a fit target, add the projector pad to it. If the user set LLAMA_ARG_FIT_TARGET explicitly, leave it exactly as provided.
Fixes#16419
* review comments
Recent upstream Pascal kernel fixes let us compile native SM60/SM61 kernels again instead of relying on PTX JIT, so allow Flash Attention auto at runtime for CC 6.x devices.
Fixes#16591Fixes#16754
Use ggml_fopen for compat tensor reads so Windows paths with Unicode characters are converted through the same UTF-8-to-wide path as llama.cpp model loading.
Fixes#16493
The presets and docs had fallen out of sync with what our current ROCm versions on Linux and Windows actually support. We rely on Vulkan now to cover these older unsupported devices.
In the PS output, expose the scheduler selected size (clamped by model context size) instead of always reporting the model max context. This will help provide a hint to clients to keep the context size below this value to avoid paging and poor performance on smaller VRAM systems.
Our cuda_v12 build requires nvcc fatbin compression, which in turn requires driver 550 or newer. This change filters incompatible CUDA devices based on the runtime and driver version. This allows users to build from source with older toolkits to support older drivers.
Fixes#16449
* server: align generate with native chat templates
/api/generate rebuilt chat-like prompts through the Go template path even when the model selected its native GGUF Jinja chat template, so the same model rendered differently between generate and chat.
Route chat-like generate requests through the shared native chat preparation path, keep deprecated context and image handling working there, and keep explicit OLLAMA_GO_TEMPLATE overrides intact.
Fixes#16792
* review comments
Fall back to "{{ .Prompt }}" when lacking templates
Bump MLX to the latest selected upstream ref and update the MLX/imagegen
wrappers and tests for the new API behavior.
Fix the CUDA MLX archive so runtime NVRTC kernels work after deployment:
package CUTE/CUTLASS headers, include the CUDA runtime header closure, and
stage a coherent CUDA-toolkit-matched CCCL tree instead of MLX's fetched CCCL
for CUDA payloads. The previous archive could build successfully but crash at
runtime due to missing or incompatible JIT headers.
Existing qwen2.5vl GGUFs can contain an empty qwen25vl.vision.fullatt_block_indexes array. The compat layer translated the projector metadata but left clip.vision.n_wa_pattern unset, causing llama-server to fail loading the CLIP model.
Default the runtime compat value to the standard Qwen2.5-VL pattern when the key cannot be derived, and make the converter emit the same default for nil or empty fullatt block metadata.
Fixes#16540
* llm: size mmproj offload by projector memory
Replace the blanket 10 GiB VRAM cutoff with a projector tensor-size estimate plus backend headroom, while preserving the existing CPU-only, partial text offload, shared-memory GPU, and startup OOM retry gates.
This is a stopgap until fit accounts for mmproj memory directly.
The same limited-vram path appears in the qwen3.5 vision hang report: the logs show --no-mmproj-offload on a 7.5 GiB RTX 5050 with about 6.4 GiB free while llama-server estimates the inline mmproj at about 962 MiB.
Fixes#16496Fixes#16570
* review comments
Bump llama.cpp to b9509, which includes the upstream Gemma 4 12B multimodal projector fixes for the n_head=0 divide-by-zero crash seen on x86/CUDA/Linux/Windows.
Fixes#16479Fixes#16489Fixes#16491Fixes#16492Fixes#16495
Daniel Hiltgen