The C-level dequantize accepts a global_scale argument but rejects it
on the Metal backend, so dequantize-fallback sites hand-rolled the
same post-multiply. Take the scale in the Go wrapper and apply it on
top of the op, cast back to the output dtype. The quantized embedding
passes its scale; laguna's expert paths keep their own multiplies,
which shape per-expert scales and differ on result dtype.
mlx item<T> reinterprets without checking the dtype, so Array.Int's
8-byte read of int32 scalars took in neighboring pool bytes — masked by
Metal's zeroed allocations, corrupting token IDs on CUDA's warm pool.
Read at the element's width.
golangci-lint ran with only-new-issues, which filters findings down to the
lines a PR adds. That silently drops any issue a diff introduces at a
distance, where the report anchors to a line the diff never touched.
CI now is enabled to scan all files. This PR also fixes the last few
straggler lint glitches outside of integration, which I'll tackle
in a follow up PR.
The per-request stats are the main diagnostic for speculative
throughput, so log them at info; the controller line stays debug.
Recording chosen depths at the next beginRound dropped rounds with no
successor, so record at endRound and count resume as a depth-0 round.
Remove the x/imagegen tree (MLX image generation engine, Flux2/zimage
models, cache, C bindings) and all imagegen integration points:
- server: drop imagegen routes, scheduling, and generate handling
- api/cmd/docs: remove image generation API surface and docs
- middleware/openai: remove image endpoint support
- integration: remove imagegen test suites
- x/create: adopt the rewritten create pipeline from main; drop
imagegen create path (CreateImageGenModel, IsTensorModelDir,
model_index.json detection, Flux2KleinPipeline vision hack)
- retain x/imagegen/manifest (Ollama-store safetensors manifest
loader), still used by x/mlxrunner and x/create/client
- fix Windows MLX dl.dll install, MLX CMake version path, and the
show command after removing safetensors models
If we have one flake, don't stop other jobs that will most likely work so when
we re-run failed jobs, only the flake and dependents need to be run. This should
help reduce the time it takes to get past a flake and finish a release build.
* model: add Laguna MLX support
Add Laguna XS 2, XS 2.1, and S 2.1 support to the MLX model and create paths.
Read the source config to apply one quantization policy across dense and routed MoE layers. Keep the tied output head and router at source precision, quantize supported attention and expert projections, selectively promote sensitive expert down projections, and emit per-tensor metadata for mixed quantization blobs.
Correct dense expert loading, BF16 source-layout handling, expert global-scale shapes and dtypes, routing-score scaling, and mixed-precision expert dispatch. Gate/up and down projections select quantized or dense execution independently so promoted BF16 down projections do not force quantized gate/up weights through the dense fallback.
Optimize the forward pass with compatible gate/up fusion, sorted standard GatherMM and GatherQMM operations for larger prefills, model-local mlx.Compile closures for elementwise MoE work, and cache-backed 512-token prefill chunks. This keeps the implementation on maintained MLX operations without custom kernels.
Add focused tests for Laguna configuration variants, quantization policy and metadata, dense and routed expert loading, mixed-precision dispatch, compiled-versus-eager parity, fused projections, routing, and prefill chunking.
* review comments and S 2.1 performance fixes
Address renderer/parser selection and mixed-precision expert quantization review feedback.
Keep Laguna weights resident on Metal to prevent repeated paging of its large, sparsely accessed expert buffers. Scope this policy to Laguna GPU execution.
Remove obsolete 512-token prefill chunking now that the runner's 2048-token path is faster.
* review comments addressed
* fix create
Draft token embeddings were kept at source precision. A draft that
reuses its embedding as the output projection (the gemma4 assistant)
then reads the whole 537MB bf16 tensor on every draft step — about half
the step's cost. Draft quality only affects how many drafts are
accepted, so the output head now takes the requested type instead of the
8-bit type that protects a target's output quality.
gemma4:26b-mlx, M5 Max: MTP code decode 148 -> 157 tok/s (+26% -> +37%
over plain); prose goes from roughly zero to +2-5%; acceptance unchanged.
Configure Metal residency after the MLX runner materializes model weights.
Wire up to the smaller of active model memory and the recommended working set, leaving pageable headroom for KV caches and request allocations. If residency setup fails, warn and continue with pageable memory.
Expose recoverable MLX C API errors and verify that an oversized wired limit preserves the previous state and leaves subsequent evaluation usable.
Gathering gate and up separately cost a third expert gather per MoE
layer. Keep gate_up packed as one tensor, joining it at load when the
checkpoint ships the halves separately, and split the gather's output
instead.
Output is byte-identical; decode is 4% faster (7.89 -> 7.58 ms/token on
M5 Max) and prefill 9% faster.
PsHandler iterated sched.loaded without holding loadedMu, racing with
scheduler goroutines that mutate the map. It also read runnerRef fields
(model, llama, expiresAt) that unload() and the expiration path mutate
under refMu, so a concurrently unloading runner could nil model out from
under the handler.
Instead of adding locking in routes.go, give the scheduler a small
snapshot API: loadedModels() copies the runner list under loadedMu, then
captures each runner's reporting fields under its refMu, respecting the
refMu-before-loadedMu lock ordering used by the expiration path. The
zero-expiresAt estimate for still-loading models moves into the
scheduler too, since it exists because of scheduler behavior.
Also remove the dead code Scheduler.GetRunner
app/updater: TestBackgoundChecker / TestAutoUpdateDisabledSkipsDownload hit 'TempDir RemoveAll cleanup: directory not empty' on macOS because the background checker goroutine keeps writing staged files into UpdateStageDir while t.TempDir cleanup runs. The checker's context is cancelled by the time cleanup runs, and after cancellation a new download cannot reach the filesystem (DownloadNewRelease aborts at its HEAD request before any write), so it suffices to wait for any in-flight download to drain. Add a test-only waitDownloadIdle helper (polls the existing cancelDownload sentinel under its lock) and register it via t.Cleanup so TempDir cleanup runs after staged-file handles close. No production code changes.
x/transfer: TestDownloadParallelism asserted elapsed <= 1s against 50ms-per-blob delays, too tight for Windows hosted runners' ~15ms timer granularity and shared-runner jitter. Each blob costs two server sleeps (resolve GET + body GET), so model the serial baseline from the deterministic request count, raise per-blob latency to 100ms so timer quantization is a small fraction of each delay, and key the budget to 75% of the serial baseline so the check still proves parallelism while tolerating jitter.
The lm_head rule was asymmetric: the fp modes kept an untied head at
source precision (even under mxfp8, leaving it the only bf16 matmul in
the model), while int4 quantized it at 4 bits with no promotion. The
tied-embedding overrides (gemma4, cohere2moe) already resolve the head
to the 8-bit family type and hold quality close to bf16.
Apply the same decision to untied heads: the 8-bit type in the
requested family when it fits the shape, source precision otherwise.
int4 now promotes the head to int8, and the fp modes quantize it to
mxfp8 instead of keeping bf16.
Update llama.cpp to pick up upstream Laguna implementation and remove Ollama's local Laguna implementation. Retain a narrow Metal-only scaling workaround for routed-MoE prompt overflow.
Translate older Ollama GGUF attention-gate and SWA metadata names so existing models continue to load.
The GLM parser buffered tool calls until it observed </tool_call>, but ignored the terminal done signal. If the model omitted or partially emitted the outer closing tag, Ollama returned a successful empty response instead of a tool call or an actionable error, leaving coding agents unable to continue.
On end-of-stream, finalize only structurally complete calls for declared tools with all required arguments. Complete calls missing only the outer delimiter now proceed through the existing parser, while genuinely truncated calls return an explicit error rather than being silently dropped.
Fixes#16497
This refactors the existing integration tests into 3 priumary groups: fast,
release, and library. It also refines some of the release tests to drop some
of the older models and pick up newer models, while retaining the broad
coverage in the library group.
Add a laguna-v8 renderer/parser matching the Laguna XS 2.1 template, and fix v2 handling of embedded thinking and structured tool arguments.
Prevent FP16 overflow in Metal's quantized routed-MoE prefill path by scaling the linear branch and folding the inverse into the routing scale. Other backends and token-generation paths are unchanged.
Add comprehensive v2/v8 Jinja parity and parser tests.
Add compute capability 10.0 to the Linux CUDA v12 preset so B200-class devices can use the cuda_v12 backend with drivers that do not meet the CUDA v13 minimum.
Fixes#12583
The bare `ollama` command (and `ollama launch` with no integration) used a
bespoke `ensureServerRunning` that forked `ollama serve` directly and polled
its heartbeat forever (no timeout, no platform-aware launch). Every other
subcommand (`ollama run`, `ollama pull`, `ollama launch <integration>`, ...)
goes through `checkServerHeartbeat` -> `startApp`, so the root command behaved
differently and could hang indefinitely.
Route `runInteractiveTUI` through `checkServerHeartbeat(cmd, nil)` — the same
path `ollama launch <thing>` uses — so the root command is consistent and no
longer runs an unbounded server-spawn loop. `ensureServerRunning` and its
`backgroundServerSysProcAttr` helpers (only it referenced them) are removed,
along with the now-unused `os/exec` import.
The platform `startApp`/`waitForServer` paths are unchanged, so behavior on
macOS/Windows is identical to the other subcommands, and on Linux the root
command now errors the same way the subcommands already do when no server is
running.
Jesse Gross