Ollama publishes embeddinggemma:300m with general.architecture=gemma3
plus two extra dense projection layers stored as `dense.0.weight` /
`dense.1.weight` (the sentence-transformers post-pooling head, mapping
the 768-dim pooled embedding through 768→3072→768).
Upstream loads this model under arch=gemma-embedding, which both
disables causal attention (embeddings are bidirectional) and loads
`dense_2`/`dense_3` by name from the gemma-embedding.dense_*_feat_*
hparams. Without that arch, the gemma3 loader leaves dense.0/dense.1
unrequested and `done_getting_tensors` raises "wrong number of
tensors; expected 316, got 314".
Detection: arch=gemma3 AND has dense.0.weight tensor (only
embeddinggemma ships these — regular gemma3 chat models do not, so
the existing handle_gemma3 path remains for them).
Translation:
* switch arch_name to gemma-embedding (used by the loader to build
KV lookup prefix for subsequent reads)
* copy gemma3.* KV prefix to gemma-embedding.*
* derive gemma-embedding.dense_{2,3}_feat_{in,out} from the actual
dense tensor shapes
* rename dense.0/dense.1 → dense_2/dense_3
Verified via /api/embed: returns 768-dim vectors with correct cosine
similarity ordering (similar sentences cluster, dissimilar ones don't).
Confirmed gemma3:1b chat path is unaffected — handle_embeddinggemma's
detection only matches blobs with the dense.0 tensor present.
The previous mapping renamed Ollama's `layer_pre_norm` to upstream's
`attn_post_norm`, but in Ollama's gemma4 audio code `layer_pre_norm` is
actually the FINAL block output norm (used at the end of the conformer
block), while `ln2` (named AttnPostNorm in Ollama's Go) is the
post-attention norm.
Upstream gemma4a uses `attn_post_norm` for post-attention and `ln_2`
for the final block norm, so the correct mapping is:
ln1 → attn_pre_norm (pre-attention norm)
ln2 → attn_post_norm (post-attention norm)
layer_pre_norm → ln2 (final block output norm)
linear_pos → attn_k_rel (relative-position K projection)
Also scope the per-block renames to `a.blk.*` (not vision blocks, which
have their own ln1/ln2 that must stay as-is). Order matters: ln2 →
attn_post_norm must run before layer_pre_norm → ln2 to avoid collision.
Verified end-to-end via `ollama run gemma4:e2b` with audio input — now
produces transcription that matches the upstream-converted reference
mmproj.
Ollama's gemma4 E2B/E4B blobs ship audio tensors alongside vision.
Upstream supports audio via PROJECTOR_TYPE_GEMMA4A (Conformer encoder
+ audio multimodal embedder) — wire that up.
KV synthesis (audio side, inside the existing handle_gemma4_clip):
* has_audio bool gate detects `a.*` tensors and applies the audio
fixes only when present (vision still applies if `v.*` present;
both can run together for the monolithic blob).
* Copy `gemma4.audio.{block_count,embedding_length,
feed_forward_length,attention.head_count,
attention.layer_norm_epsilon}` → `clip.audio.*`.
* Inject defaults the migration mmproj sets explicitly:
`clip.audio.num_mel_bins = 128`, `clip.audio.projection_dim = 1536`.
* Set `clip.has_audio_encoder = true`,
`clip.audio.projector_type = "gemma4a"`.
Tensor renames — Ollama uses different leaf names for the SSCP input
projection and the encoder-out projection than upstream:
* `a.pre_encode.out.weight` (Ollama) → `a.input_projection.weight`
(upstream — REQUIRED). Same shape [1024, 1024]; semantically the
SSCP input projection that follows the conv1d front-end.
* `mm.a.fc.{weight,bias}` (Ollama) → `a.pre_encode.out.{weight,bias}`
(upstream). Same shape [1024, 1536]; the audio encoder output
projection that feeds into the multimodal embedder.
* `mm.a.input_projection.weight` already matches upstream.
Per-block leaf renames (substring — each appears once per audio block):
* `.linear_pos` → `.attn_k_rel` (relative-position K projection,
used by the conformer's Transformer-XL-style RPE attention).
* `.layer_pre_norm` → `.attn_post_norm` (post-attention norm).
Verified: clip loader recognizes both projectors (`gemma4v` for
vision, `gemma4a` for audio) with correct hparams (audio n_embd=1024,
n_layer=12, projection_dim=1536). Audio is preprocessed through the
encoder (466 prompt_tokens for ~3s mp3). End-to-end transcription
output via `ollama run` (which supplies the correct chat template);
through llama-server's chat completions the same gemma4 chat-template
artifact applies as we saw with vision.
Previous addition for glm-ocr added 4 lines to upstream-edits.patch
(1 in constructor for set_loader_path + 3 in load_all_data for the
maybe_load_text_tensor if/continue/closing-brace block). Refactor to
+1 line:
* Fold path capture into translate_metadata: signature now takes
`const char * fname` and stashes it on the per-loader registry
internally. The constructor still has just one llama_compat call
(we just changed the argument list).
* Make maybe_load_text_tensor self-contained: it derives the buffer
type from `cur->buffer` rather than the caller passing it. The
hook line in load_all_data collapses to a single
`if (llama_ollama_compat::maybe_load_text_tensor(this, cur, weight->offs)) continue;`.
* Drop the public `set_loader_path` symbol — it's now an internal
detail of translate_metadata.
Net patch growth from glm-ocr support: +1 line. Total upstream patch
is back to its original 17-line surface.
Functional: glm-ocr text generation still works ("Paris" via raw
turn-template completion), concat ops still fire (28MB per block).
glm-ocr (text side):
* Arch rename `glmocr` → `glm4` (incl. KV prefix); upstream supports
GLM-OCR via LLM_ARCH_GLM4 with n_layer=17 (16 main + 1 nextn). We
report n_layer=16 and leave nextn_predict_layers absent — the
Ollama blob doesn't ship the nextn layer's weights.
* M-RoPE: pad `rope.mrope_section` (3 elements) →
`rope.dimension_sections` (4 elements with trailing 0).
* Inject `rope.dimension_count = key_length`.
* Tokenizer pre-tokenizer rename `llama-bpe` → `chatglm-bpe`.
* Tensor renames: `attn_out`→`attn_output`, `post_attn_norm`→
`post_attention_norm`, `post_ffn_norm`→`post_ffw_norm`.
* **Per-block FFN concat**: GLM4 expects fused
`ffn_up.weight: [n_embd, n_ff*2]` (gate || up). Ollama writes
separate `ffn_gate.weight` + `ffn_up.weight` (each `[n_embd, n_ff]`).
Register a load-time concat op that stitches gate+up into the fused
upstream slot, then add a per-block skip-prefix for the orphan
`blk.X.ffn_gate.` so the n_tensors check lines up.
* Hide embedded `v.*`/`mm.*` from the text loader.
This is the first text-side compat that needs custom load-time tensor
data (the FFN concat). Until now load-op support only covered the
clip side. New plumbing:
* `set_loader_path(ml, fname)` — store the model file path on a
per-loader registry, called from the loader constructor.
* `maybe_load_text_tensor(ml, cur, off, buft)` — the text-side
counterpart to `maybe_load_tensor`; looks up the path from the
registry then delegates to the existing load-op machinery.
* Upstream patch grows two new lines: a `set_loader_path` call in
the constructor and a `maybe_load_text_tensor` hook in
`load_all_data` (before the use_mmap branch).
Verified: with --no-mmap, glm-ocr's blk.X.ffn_up.weight load fires
the concat op (28MB per block on the 1B variant) and the model emits
coherent text. Through `ollama run` the proper chat template applies.
Note: vision (clip) handler is a follow-up.
Ollama's converter writes `tokenizer.ggml.model = 'llama'` (SPM) on
gemma4 GGUFs, but the gemma4 model family actually uses BPE — upstream-
converted reference GGUFs use `'gemma4'` which selects
LLAMA_VOCAB_TYPE_BPE in src/llama-vocab.cpp.
With the wrong tokenizer type, gemma4's special tokens (e.g.
`<|thought|>`, `<|turn>`, `<|channel>`) get split into multiple SPM
subword pieces, so when the model emits them they come out as raw
text instead of being recognized as control tokens — surfaces as e.g.
the literal word "thought" appearing in the chat output instead of
being routed to the model's reasoning channel.
Ollama already supplies `tokenizer.ggml.merges` (needed for BPE) and
`tokenizer.ggml.pre = 'gemma4'`, so flipping just the model name is
enough; upstream's gemma4 init reads merges and sets pre-type
correctly. The `add_bos = False` we ship is also auto-overridden to
True by upstream's gemma4 workaround (PR 21500).
Verified: with the fix, `Hey` -> "Hello! How can I help you today?"
(coherent greeting); without the fix the model emitted raw "thought"
text into the user-visible response.
The redirect table was a stopgap for arches that llama-server (after
the runner removal) couldn't load because the Ollama-format GGUFs in
library/* differed from upstream. Every entry has now been superseded
by an in-process translator in llama/compat/, so the redirects are
dead weight: pulling library/<arch> works directly through compat
without rerouting the user to a republished blob.
Drops:
* `compatModelRedirects` table (10 entries)
* `applyCompatRedirect` / `reverseCompatRedirect` helpers
* The single `applyCompatRedirect` call in resolveName
* The `reverseCompatRedirect` call in PsHandler (the display name
is now just the model's actual ShortName)
Translates Ollama's monolithic gemma4 GGUF into the upstream gemma4v
mmproj shape so `--mmproj <same-blob>` loads alongside the text model.
KV synthesis: rewrite arch to `clip` with `clip.vision.projector_type=
gemma4v`, copy `gemma4.vision.*` → `clip.vision.*`, project
`gemma4.embedding_length` → `clip.vision.projection_dim` (= mm.input_projection
output dim). gemma4 vision uses image_mean=[0,0,0] / image_std=[1,1,1]
(the model does its own per-image normalization via v.std_bias /
v.std_scale tensors), unlike the [0.5,0.5,0.5] defaults most arches
use — inject those.
No tensor renames needed: every `v.*` and `mm.*` tensor name (incl.
v.patch_embd, v.position_embd, v.std_bias/scale, mm.input_projection,
plus the per-block ln1/ln2/attn_post_norm/ffn_post_norm/attn_q_norm/
attn_k_norm) already matches the upstream-converted reference mmproj
exactly.
F32 promote of v.patch_embd.weight (Metal IM2COL requirement, same as
gemma3 / mistral3 / deepseek-ocr).
Adds `gemma4` to the Go-side `compatClipArches` allowlist so
`ollama run gemma4 -i image.jpg ...` auto-passes the same blob as
`--mmproj`. Covers all four published sizes (E2B, E4B, 26B-A4B MoE,
31B dense).
Verified loading: 1189 tensors loaded from monolithic blob, hparams
(projector=gemma4v, n_embd=1152, n_head=16, n_ff=4304, n_layer=27,
n_merge=3, image_size=224, patch_size=16) match migration mmproj
exactly. Image encoding/decoding completes (264 vision tokens). End-
to-end testing through llama-server's chat completions surfaces a
chat-template detection issue in llama-server (Ollama's gemma4 GGUFs
ship without `tokenizer.chat_template`, so llama-server falls back to
ChatML which the model doesn't understand) — orthogonal to compat;
`ollama run` supplies the correct template via Modelfile.
Text side (handle_llama4): standard skip of `v.*` / `mm.*` from the
text loader so the n_tensors check passes. Text-side KVs and tensor
names match upstream verbatim. Tested: llama4:scout 17B-16E generates
correctly with `-c 4096` (the default 10M context OOMs even on 110GB
unified memory — that's a memory-budget issue, not a compat one).
Clip side (handle_llama4_clip): rewrites arch to `clip` with
`projector_type=llama4`, copies `llama4.vision.*` → `clip.vision.*`,
projects `llama4.embedding_length` → `clip.vision.projection_dim`,
injects pixtral-style image stats and `projector.scale_factor=2`.
Tensor renames cover three scopes:
* Vision adapter MLP → MM-MLP slots (run BEFORE the generic
`.mlp.fc{1,2}` rename so the substring stays anchored to the
adapter prefix): `v.vision_adapter.mlp.fc{1,2}` → `mm.model.mlp.{1,2}`.
* Main projector: `mm.linear_1` → `mm.model.fc`.
* Vision tower: `class_embedding`/`layernorm_pre`/`layernorm_post`/
`patch_embedding` → `class_embd`/`pre_ln`/`post_ln`/`patch_embd`,
plus `positional_embedding_vlm` → `position_embd.weight` (exact
rename — no `.weight` suffix in Ollama). Block leaves get the same
treatment as gemma3/mistral3: `attn_output`→`attn_out`, `attn_norm`/
`ffn_norm` → `ln1`/`ln2`, `mlp.fc{1,2}` → `ffn_{up,down}`.
Adds `llama4` to the Go-side `compatClipArches` allowlist.
Status: clip metadata + tensors load successfully — name mapping is
verified to match the upstream-converted reference mmproj exactly.
mtmd_encode_chunk currently fails at runtime with "failed to encode
image slice" (cause not yet diagnosed; tensor shapes/dtypes look
right, may be a graph-build mismatch in upstream's llama4 clip path).
Text-side llama4 is fully functional.
Covers two variants of the same arch:
- nemotron-3-super (120B-A12B): uses a latent-FFN variant where
experts operate in a compressed `moe_latent_size`-dim space rather
than the full embed dim. Inject `nemotron_h_moe.moe_latent_size`
from the first `ffn_latent_in.weight` ne[1], and rename Ollama's
`ffn_latent_in`/`ffn_latent_out` → upstream's `ffn_latent_down`/
`ffn_latent_up`.
- nemotron-cascade-2 (30B-A3B): no latent FFN; loads with just the
MTP skip described below.
Both variants ship MTP (Multi-Token Prediction) layers that Ollama
emits one-tensor-per-expert (`mtp.layers.X.mixer.experts.Y.{up,down}_proj.weight`
— 1040 extras on the 120B). Upstream's nemotron_h_moe loader doesn't
claim them, so without skipping they trigger
`done_getting_tensors: wrong number of tensors`.
Also tested:
- nomic-embed-text-v2-moe (`nomic-bert-moe`): loads + serves 768-dim
embeddings without any compat. No handler needed.
qwen35 (e.g. qwen3.5:9b): same SSM-hybrid + M-RoPE + MTP + monolithic-
vision converter quirks as qwen35moe — extracted shared
`apply_qwen35_text_fixes` helper and called it from both arch handlers.
gemma4 (E2B/E4B dense and 26B-A4B MoE): text-side KVs and tensor names
match upstream verbatim; only fix is to hide embedded `a.*` (audio),
`v.*` (vision), and `mm.*` (projector) tensors from the text loader so
n_tensors lines up.
deepseek-ocr (text + vision):
* Text: rename arch `deepseekocr`→`deepseek2-ocr` (incl. KV prefix),
inject `expert_feed_forward_length` from `ffn_down_exps` shape,
`expert_shared_count` from `ffn_down_shexp` shape, default
`attention.layer_norm_rms_epsilon=1e-6`, drop `s.*`/`v.*`/`mm.*`.
* Vision (clip): rewrite arch to `clip` with `projector_type=deepseekocr`,
copy `deepseekocr.vision.*` → `clip.vision.*` and `deepseekocr.sam.*` →
`clip.vision.sam.*`, inject defaults for image stats / window_size /
projection_dim / feed_forward_length. Tensor renames cover three
distinct components: SAM (prefix-only rename `s.*`→`v.sam.*` plus
leaf renames for proj/qkv/rel_pos/norm), CLIP (`self_attn.*`→
`attn_*`, `layer_norm{1,2}`→`ln{1,2}`, `mlp.fc{1,2}`→`ffn_{up,down}`,
`pre_layrnorm`→`pre_ln`), and projector (`mm.layers`→`mm.model.fc`,
`mm.image_newline`/`view_seperator`→`v.*`). Plus F32 promotes for
Metal IM2COL on `v.patch_embd.weight` / `v.sam.patch_embd.weight` /
`v.position_embd.weight`.
Bug avoided: prefix `s.` is too short for substring rename (would corrupt
`mm.layers.weight` → `mm.layerv.sam.weight`). Renamed via explicit
start-of-name iteration instead.
Adds `deepseekocr` to the Go-side `compatClipArches` allowlist.
Tested locally:
* qwen3.5:9b — text generation correct
* gemma4 e4b/26b-a4b — text generation correct (chat template required)
* deepseek-ocr text+vision — Free OCR / Parse the figure prompts
correctly describe the supplied test image
* rnj-1 (uses gemma3 arch) and nemotron-3-nano-4b (nemotron_h arch)
both load and generate without any compat handler.
The previous mistral3 vision handler loaded but produced hallucinated
descriptions of real images (e.g. described a clear photo of a person
in a suit handing money as "abstract hexagonal pattern"). Solid colors
and large color blocks worked, suggesting the vision tower was producing
*some* signal but the spatial/feature relationships were scrambled.
Root cause: Ollama's mistral3 converter only applies its LLaMA-style
RoPE repack to TEXT-side `attn_q`/`attn_k` tensors — the
`if !HasPrefix(name, "v.")` guard in convert/convert_mistral.go::Tensors
skips vision tensors entirely. Vision Q/K therefore leave the converter
in raw HF/PyTorch order. Upstream's HF→GGUF flow does permute vision
Q/K with the vision head count, because pixtral's clip graph uses
`ggml_rope_ext` in mode 0 which expects the [n_head, head_dim/2, 2, ...]
LLaMA layout.
Fix: register a load-time op that applies LlamaModel.permute equivalently
on each F16 vision attn_q.weight / attn_k.weight (24 layers × 2 = 48
tensors for ministral-3 8B). Capture file offsets *before* renames
invalidate them, same pattern as promote_tensor_to_f32 and
register_concat_load.
Verified against the upstream-format migration mmproj (which gets the
same image right) — both 8B Ollama+compat and 14B migration now produce
matching, accurate descriptions ("hand giving money to person in suit,
cartoonish star/burst above their head, blue grid background").
Translates Ollama's monolithic mistral3 GGUF into the upstream pixtral
mmproj shape so `--mmproj <same-blob>` loads alongside the text model.
KV synthesis: rewrite arch to `clip`, copy `mistral3.vision.*` →
`clip.vision.*`, set `projector_type=pixtral`, `use_silu=true`,
inject pixtral image_mean/std defaults, copy
`mistral3.spatial_merge_size` → `clip.vision.spatial_merge_size`.
Tensor renames: `v.patch_conv`→`v.patch_embd`, `v.encoder_norm`→
`v.pre_ln`, `attn_output`→`attn_out`, `attn_norm`/`ffn_norm`→
`ln1`/`ln2`, `mm.linear_{1,2}`→`mm.{1,2}`, `mm.norm`→`mm.input_norm`,
`mm.patch_merger.merging_layer`→`mm.patch_merger`.
img_break: pixtral's loader strictly requires `v.token_embd.img_break`
(the embedding row for the [IMG_BREAK] separator token). Ollama
doesn't ship it as a separate tensor; the "right" value is row 12 of
token_embd.weight, but token_embd is Q4_K and per-row dequant is
heavyweight. Reclaim the orphan `output_norm.weight` slot (already
[n_embd] F32) and zero-fill via load op — pixtral.cpp adds img_break
to row separators, so a zero embedding makes [IMG_BREAK] insertion a
no-op without breaking the rest of the vision graph.
Adds `mistral3` to the Go-side `compatClipArches` allowlist so
ollama auto-passes `--mmproj <same-blob>` for vision requests.
Tested with ministral-3 (8B, just released): text + vision both work;
solid colored test images correctly identified (red/green/blue).
Ministral-3 (and other Ollama-published mistral3 models) load with this
handler. Differences vs upstream HF format:
- RoPE YaRN parameters: Ollama writes `rope.scaling.beta_fast` /
`beta_slow` while upstream wants the `yarn_*`-prefixed names.
- Attention temperature scale: Ollama writes `rope.scaling_beta`,
upstream reads `attention.temperature_scale` (same numeric value).
- The Ollama GGUF is monolithic — embedded `v.*` / `mm.*` tensors must
be hidden from the text loader, otherwise n_tensors mismatches.
Vision (clip) translation is not yet implemented; running with
--mmproj will still fail. Ministral-3 text generation works end-to-end.
gpt-oss: rename arch "gptoss" -> "gpt-oss" (incl. KV prefix), inject
the missing `expert_feed_forward_length` from the ffn_gate_exps shape,
and rename `attn_out`/`attn_sinks`/`ffn_norm` tensors to upstream's
`attn_output`/`attn_sinks.weight`/`post_attention_norm`. Also remove
the library/gpt-oss -> dhiltgen/gpt-oss redirect now that the compat
shim handles it directly.
lfm2: rename `output_norm.weight` -> `token_embd_norm.weight` and fix
a stale `lfm2.feed_forward_length` (some Ollama blobs claim 12288 on
a model whose ffn_gate is [2048, 8192]) by reading the real value off
the ffn_gate tensor shape.
Adds two helpers to compat-util: `copy_kv` (type-preserving generic
KV copy) and `rename_kv_prefix` (bulk-copy every KV with a given
prefix to a new prefix). Old keys are left in place — harmless because
the loader queries by exact name and only the new prefix matters.
Tested locally: gpt-oss:20b and lfm2.5-thinking now load + generate
coherently against an unmodified upstream llama-server build.
Added a "Maintenance" section to llama/compat/README.md and an inline
block to llama-ollama-compat-util.h enumerating the three places the
compat layer depends on something that isn't strictly guaranteed
upstream-public API:
1. Direct ggml_tensor::{type,ne,nb} field writes (public struct,
no sanctioned post-creation mutators).
2. const_cast on gguf_get_tensor_name's return to rename in place.
3. llama_model_loader forward-decl from src/ (used only as an
opaque pointer key).
Each entry ships with its escape hatch (small upstream patch or a
local replacement). All have been stable for years; they don't need
immediate action, but surfacing them makes the maintenance story
explicit for future readers.
Main translation unit (llama-ollama-compat.cpp) is now purely per-arch
dispatch: detect_* + handle_* for each arch, plus the 4 public entry
points. Dropped from 724 lines to 430.
Everything that doesn't depend on a specific arch moves to
llama-ollama-compat-util.{h,cpp}:
- gguf KV helpers (has_key, copy_{u32,f32}_kv, inject_{u32,f32,str,bool,
f32_arr}_if_missing, truncate_{str,data}_arr)
- ggml tensor helpers (any_tensor_with_prefix, rename_tensor,
rename_tensors_containing, set_tensor_{type,shape}, reclaim_slot_as,
tensor_file_offset)
- per-loader skip-prefix registry (add_skip_prefix, should_skip_tensor_prefix)
- LoadOp registry (register_load_op, take_load_op, read_at)
- common high-level transforms (promote_tensor_to_f32, register_concat_load)
New helpers introduced while splitting:
- inject_{u32,f32,str,bool,f32_arr}_if_missing — replaces the
has_key + gguf_set_val_* idiom we were using 20+ times.
- reclaim_slot_as — extracts the "rename an orphan tensor slot as a
synthesized one" pattern used by qwen35moe's patch_embed split. Clear
name + comment explains the workaround.
CMake: target_sources now globs llama/compat/*.cpp (CONFIGURE_DEPENDS),
so new .cpp files are picked up without CMake edits.
Nothing behaviorally changed. Verified gemma3 + qwen3.5 text + vision
still work end-to-end after a clean rebuild.
Extends the compat layer with the vision side for Ollama's monolithic
qwen3.5 blobs. All changes in llama/compat/ — no new upstream patch edits.
New generic infra (reused by gemma3's existing promotion):
- LoadOp registry (g_loadops). Any dest tensor whose name is registered
gets its bytes produced by a closure instead of being read straight
from disk. maybe_load_tensor consults it.
- promote_tensor_to_f32(meta, ctx, name) now captures the source offset
at registration time and becomes a LoadOp. Gemma3 already migrated.
- register_concat_load(meta, dest, {srcs...}) captures the file offsets
of N source tensors and registers a LoadOp that concatenates them.
Assumes sources concatenate along their slowest ggml axis — which in
C order means the dest bytes are src[0] || src[1] || ... .
- set_tensor_shape / set_tensor_type helpers for in-place edits.
qwen35moe clip handler (handle_qwen35moe_clip):
- Detection reuses detect_ollama_qwen35moe; additionally requires
embedded v.* tensors so we don't fire for text-only files.
- KV synth: clip.vision.* from qwen35moe.vision.* + sensible defaults
(feed_forward_length=4304, image_size=768, layer_norm_epsilon=1e-6,
is_deepstack_layers=false[27], image_mean/std=[0.5,0.5,0.5]).
- Arch rewrite: general.architecture=clip, projector_type=qwen3vl_merger.
- QKV merge per block (27x): captures q/k/v file offsets, registers a
concat LoadOp, renames attn_q -> attn_qkv and widens its shape from
[hidden, hidden] to [hidden, 3*hidden].
- patch_embed split: source [16,16,2,3456] F16 -> two dests
[16,16,3,1152] F32, permuting (c_out*3+c_in) packed_c back into
separate c_in/c_out dims. Matches upstream convert_hf's
Qwen3VLVisionModel.modify_tensors split.
- Tensor renames (substring-matched): pos_embed -> position_embd,
merger.norm -> post_ln, merger.linear_fc1/2 -> mm.0/mm.2,
mlp.linear_fc1/2 -> ffn_up/ffn_down, norm1/2 -> ln1/ln2.
- F16 -> F32 promote for v.position_embd.weight.
Ctx-pool trick for the sibling tensor:
clip.cpp sizes its ggml_context for exactly the gguf's tensor count
(+1). ggml_new_tensor to add v.patch_embd.weight.1 overflows. Since
v.blk.0.attn_k.weight is orphaned after the QKV merge (clip only
requests the merged attn_qkv), steal that slot: rename it to
v.patch_embd.weight.1 and reshape to [16,16,3,1152] F32. Its original
file offset is ignored; the LoadOp we register overrides the read.
Go side: adds qwen35moe to the auto-mmproj arch allowlist. ollama now
passes the monolithic blob as both --model and --mmproj for qwen3.5.
Verified end-to-end: ollama run qwen3.5:35b-a3b-q4_K_M with an image
correctly describes the image ("screenshot of a chat interface...
'open the browser, open never gonna give you up on youtube'..."). Text
inference still works on the same blob.
Text-only support for Ollama's qwen3.5 (qwen35moe) blobs. Vision is more
involved (QKV merge, patch_embed reshape/split) and follows in a later
commit.
Detection markers: qwen35moe.vision.*, qwen35moe.image_token_id,
qwen35moe.ssm.v_head_reordered, qwen35moe.feed_forward_length, mtp.*
tensors, embedded v.* tensors.
Fixes applied:
- attention.head_count_kv ARRAY -> UINT32 scalar (Ollama wrote per-layer
[0,0,0,2,...] over 40 layers; upstream wants the max non-zero scalar).
- rope.dimension_sections padded from [11,11,10] to [11,11,10,0]
(4-element M-RoPE convention).
- blk.N.ssm_dt -> blk.N.ssm_dt.bias rename for all 40 layers.
- skip prefixes v.*, mm.*, mtp.* from the text loader.
Tightens detect_ollama_gemma3 so it only matches files whose
general.architecture is actually "gemma3" — without this, qwen3.5 was
triggering gemma3 clip translation on the auto-mmproj pass and crashing
on tensor-shape mismatches.
Go side: gates the auto-mmproj behavior on a small arch allowlist
(currently just gemma3). When a clip handler exists for an arch, add it
to the map; until then the model runs text-only.
Verified end-to-end via `ollama run qwen3.5:35b-a3b-q4_K_M` on the
existing library/qwen3.5 blob — answers "2 + 2 equals **4**" with the
text loader's compat fixups firing and no --mmproj flag.
Round of post-first-arch cleanup. No behavioral change — text + vision
still pass end-to-end on Ollama gemma3:latest (4B), 1B, and 270m.
Removes:
- apply_tensor_transforms() and the TransformSpec registry. Registered
the +1 RMSNorm shift at one point but empirically Ollama's blobs
already have it baked in. YAGNI until an arch actually needs a
tensor-data transform; the hook can come back data-driven.
- Corresponding llama-model.cpp patch hunk (include + post-load call).
- copy_kv<> template plumbing. Direct copy_u32_kv / copy_f32_kv are
shorter and more readable.
- set_str wrapper (one-liner around gguf_set_val_str).
- find_tensor helper (only used once; inlined as a loop).
- convert_f16_to_f32 helper (one-line inner loop, inlined).
- set_f32_if_missing wrapper (inlined at 3 call sites).
Tightens:
- Clip tensor renames are now a table (kGemma3ClipRenames) iterated by
handle_gemma3_clip. Adding a rename is one row.
- translate_clip_metadata now reuses detect_ollama_gemma3 for its
"is this an Ollama blob?" check instead of a separate ad-hoc check.
Net:
llama-ollama-compat.cpp 488 -> 343 lines (-30%)
llama-ollama-compat.h 83 -> 63 lines (-24%)
upstream-edits.patch 20 -> 16 lines of real edits, 3 -> 2 files
Two reductions:
1. Drop the gguf_rename_tensor forwarder from gguf.h/gguf.cpp.
The rename-in-place trick it does (calling ggml_set_name on an embedded
ggml_tensor) can be done from outside gguf.cpp via:
char * p = const_cast<char *>(gguf_get_tensor_name(meta, id));
strncpy(p, new_name, GGML_MAX_NAME - 1);
That pointer aims into a mutable char[GGML_MAX_NAME] inside a std::vector
element; the const on the return type is API courtesy. This is defined
behavior and has no struct-layout dependency.
2. Drop the src/CMakeLists.txt hunk that added llama-ollama-compat.cpp to
the llama target. Replace with a target_sources() call in Ollama's
llama/server/CMakeLists.txt after FetchContent_MakeAvailable. Our
compat files now stay in llama/compat/ and are never copied into the
fetched _deps/ tree.
Net patch now touches 3 files, 20 lines, all pure call-site insertions:
src/llama-model-loader.cpp +8 (include + translate + 2x should_skip)
src/llama-model.cpp +4 (include + apply_tensor_transforms)
tools/mtmd/clip.cpp +8 (include + translate_clip + maybe_load)
Verified: fresh build from scratch (rm -rf build && cmake configure)
runs PATCH_COMMAND cleanly, compiles, and ollama run gemma3 still works
end-to-end for text + vision.
The clip.cpp tensor-read loop was the fattest hook in the patch — it
duplicated the host-vs-device buffer dispatch around a call into the
compat layer. Move that dispatch into our code (maybe_load_tensor),
so the upstream patch is a single conditional call.
Net: upstream patch drops from 48 lines across 6 files to 34 lines.
Every remaining edit is either a 1-line include, a 1-line function call,
or the gguf_rename_tensor shim (which accesses gguf_context internals
and has to live in gguf.cpp).
Verified end-to-end: text + vision both still correct after rebuild.
Previous handler only fired on vision-capable gemma3 (4B/12B/27B) because
its detection looked for `gemma3.mm.tokens_per_image` or embedded v.*/mm.*
tensors. The 1B blob has neither — but its old Ollama converter emitted:
- gemma3.rope.global.freq_base (upstream uses gemma3.rope.freq_base)
- gemma3.rope.local.freq_base (upstream uses gemma3.rope.freq_base_swa)
- tokenizer.ggml.add_{padding,unknown}_token
so llama.cpp would fall back to default rope_freq_base=10000 and produce
visibly-worse output.
Also inject rope.scaling.factor=8.0 / type=linear on 4B/12B/27B — those
variants ship with that scaling in their HF config to extend the native
~16k trained context to 131072. Without this KV, llama.cpp uses factor=1.0
and the positional embeddings are subtly off everywhere.
Detection now flips on any Ollama-specific marker. All three variants
verified end-to-end via `ollama run gemma3:{latest,1b,270m}`.
FetchContent's PATCH_COMMAND runs after each update step — including on
incremental rebuilds. `git apply` fails when the patch is already applied,
which bricks the build until the dev wipes build/ entirely.
Fix by routing the apply through a small apply-patch.cmake helper that
checks `git apply --reverse --check` first. If the patch cleanly reverses,
it's already applied and we skip. Otherwise apply forward. Both branches
surface real errors (drift against upstream, missing patch file, etc.).
Verified: fresh configure+build applies the patch once; re-running the
same commands is a no-op with no errors.
Two tiny Go-side changes that let the llama/compat shim take over gemma3:
1. llm/llama_server.go: when the GGUF has embedded v.* tensors and no
projector layer is declared, pass the model file itself as --mmproj.
The in-process compat layer translates the same file into both a
text-only view (for --model) and a clip-mmproj view (for --mmproj).
2. server/model_resolver.go: drop library/gemma3 from compatModelRedirects.
The compat layer handles it directly, so no dhiltgen/ republish is
needed. Other arches stay in the redirect list until they get their
own handler in llama/compat/llama-ollama-compat.cpp.
End-to-end verified: `ollama run gemma3` answers text and image prompts
against the existing library/gemma3 blob with no re-download.
Older Ollama builds ship GGUFs that diverge slightly from upstream llama.cpp
in arch names, KV keys, tensor names, and (for vision models) file layout
(text+vision in one monolithic file). This adds a self-contained compat
layer that translates those files in memory at load time, so
~/.ollama/models/blobs/* can be served by upstream llama-server with no
re-conversion and no re-download.
Structure:
llama/compat/
llama-ollama-compat.{h,cpp} — the shim (Ollama-owned, ~500 LOC)
upstream-edits.patch — ~48 lines of call-site hooks in 6 upstream files
compat.cmake — include()-able CMake fragment
README.md — what/why/how-to-regen
Integration: llama/server/CMakeLists.txt includes compat.cmake and passes
OLLAMA_LLAMA_CPP_COMPAT_PATCH_COMMAND to FetchContent_Declare via
PATCH_COMMAND. When OLLAMA_LLAMA_CPP_SOURCE is set (dev mode), the patch is
skipped so the developer's tree stays untouched.
Currently handles gemma3 (text + vision). Pattern is data-driven — adding
other archs is a new handle_<arch>() + one dispatch line. See README for
the per-arch checklist.
Verified end-to-end: `llama-server --model BLOB --mmproj BLOB` with an
Ollama gemma3:latest blob answers both text prompts ("Paris") and vision
prompts (correct image descriptions).
Remove the vendored GGML and llama.cpp backend, CGO runner, Go model
implementations, and sample. llama-server (built from upstream llama.cpp via
FetchContent) is now the sole inference engine for GGUF-based models.
(Safetensor based models continue to run on the new MLX engine.) This allows
us to more rapidly pick up new capabilities and fixes from llama.cpp as they
come out.
On windows this now requires recent AMD driver versions to support ROCm v7 as
llama.cpp currently does not support building against v6.
If you have a long running create, and start another ollama server with the
same model dir, the GC algorithm deletes the pending blobs and breaks the
create. This adds a 1h grace period to avoid deleting in-flight creation
operations.
Following up on #15560, this change now has e2b/e4b render differently
from 26b/31b.
For backwards compatibility, we take the existing renderer name `gemma4`
and make it do dynamic resolution based on the model name/size, but the
intended use is for the models to be republished with the renderer
variant specified explicitly: `gemma4-small` or `gemma4-large`.
After the rotating buffer has wrapped (c.offset > c.maxSize) a subsequent
L>1 Update() went through a slice-to-[0, c.idx) path that discarded all
slots in [c.idx, Dim), losing the older-but-still-in-window tokens the
first Q of the new batch needs for its sliding-window attention.
Linearize the circular buffer to logical order in that wrapped case so
the existing trim + concat preserves the last (maxSize - 1) old tokens.
When the buffer has not yet wrapped (c.offset <= c.maxSize), slots
[c.idx, Dim) are grow padding or stale post-rewind data, so keep
dropping them.
Converts SiLU/GELUApprox to compiled kernels and adds SwiGLU,
matching upstream mlx/mlx_lm's activations pattern. Routes llama,
qwen3, qwen3_5 (dense + MoE), and glm4_moe_lite MLP paths through
mlx.SwiGLU so each MLP invocation runs as one fused Metal/CUDA
kernel rather than a chain of per-op launches.
Wraps MLX's mlx_compile API so Go functions can be traced into fused
kernels. Contiguous elementwise chains collapse into a single
Metal/CUDA kernel instead of launching one per op.
Exposes Compile plus arity helpers (Compile1/2/3) that mirror Python's
@mx.compile decorator shape, lazily building the closure on first call
so package-level declarations work before the MLX dylib loads.
