flake8 flagged an unused MmprojModel import, and ty flagged seven errors in
the PLE streaming path: eos_token_id can be absent, and _ple_map, _ple_path,
_ple_row_dim and _ple_rows_per_shard are all Optional at the declaration but
were dereferenced without narrowing.
The map is opened and the stride fixed before the first shard is written, and
_finish_ple_table only runs once every shard has landed, so the invariants
hold. Assert them so the checker can see it. A missing eos_token_id now raises
with the reason instead of a TypeError from int(None).
Rewrite the comments this series adds to the AGENTS.md rules: one or two lines,
no prose hard-wrapped mid-sentence, no narrative or history, and no comment that
only restates the code. Net 146 fewer comment lines, no code change.
Correct the PLE image comment: mtmd does not consume the placeholder ids. An
image is decoded as an embeddings-only batch, so ubatch->token is null and the
per-position ids never exist here. gemma3n and gemma4 hit the same case and
stand in row 0 of per_layer_token_embd; qwen4exp stands in the configured image
token id instead.
Read image_token_id straight from self.hparams in the converter. base.py merges
text_config into the root of hparams, and the key sits at the root of
config.json, so the config.json re-read was redundant.
(cherry picked from commit 205840c12169057da3e8d2f65ec4ceec3e18b980)
The PLE row indices are computed host-side from ubatch->token, and set_input
returned early when that was null. A multimodal ubatch is exactly that case:
the mtmd layer consumes the image placeholder ids and hands llama_decode
embeddings instead. The early return left the I32 index tensor uninitialised,
so ggml_get_rows indexed a 320 M row table with whatever the buffer happened to
contain, and aborted:
GGML_ASSERT(i01 >= 0 && i01 < ne01) failed
ggml_compute_forward_get_rows
mtmd_helper_decode_image_chunk -> llama_decode
Every image request crashed. Nothing caught it because the vision work had only
ever been verified by converting an mmproj, never by running one.
The reference computes the hash over input_ids, where those positions still
hold the image placeholder, so carry that id through as qwen4exp.ple.image_token_id
and hash it. The key is optional: a file converted before it existed falls back
to the PLE EOS token, which is defined and treats the image as a segment
boundary rather than crashing.
Verified end to end with llama-mtmd-cli, a Q4_K_M base and the F16 mmproj, on a
generated image with known content. The model names the red circle, the blue
square, the inverted green triangle and reads "UNSLOTH 42", each with the right
position.
The n-gram table arrives as 128 shards that were held in a dict and then
torch.cat-ed, so the peak was the shards plus the concatenation: around
300 GB of RSS on the real checkpoint, which rules out machines that could
otherwise convert this model.
Each shard is now written straight into a memory-mapped file at its final
row offset and dropped, so the resident set is one shard and the rest is
the page cache's problem. The temporary file sits beside the output and is
removed once the write finishes, including on failure.
Shards other than the last must be uniform for direct placement, which is
asserted rather than assumed, and a shard arriving before the stride is
known is held instead of misplaced.
Verified on the tiny fixture: the resulting GGUF is byte-identical to the
one the concatenating path produced (md5 2d274efac91ad1e9a6007efb0687e597).
The full-attention layers of this model do not attend to everything. An
indexer scores one mean-pooled key per block of compress_ratio tokens and
keeps a budget of the best blocks, plus the tail of tokens that do not yet
form a complete block. Below indexer_top_k + compress_ratio - 1 cached
tokens every block fits in the budget, so the result is exactly dense.
What is reused rather than rebuilt:
- the mask machinery. build_attn's DSA overload already turns a list of
token indices into a KQ mask via ggml_set_rows, so that block is lifted
out verbatim into build_attn_mask_top_k and shared with a new overload
on llm_graph_input_attn_kv. DSA's node sequence is unchanged; the new
overload exists because llama_kv_cache_dsa assumes MLA and cannot be
dropped into a hybrid model.
- the indexer key cache, which is the optional third cache added to
llama_memory_hybrid in the previous commit. It holds raw keys, because
pooling happens before the norm and the rotation.
The graph expands block scores rather than block indices: giving every
token of a block its block's score needs only a gather, where expanding
indices would need an integer multiply-add that ggml has no op for. Since
the budget is a whole number of blocks and a block's members tie exactly,
the cut still lands on a block boundary.
Everything that depends on cache layout is computed host-side in
set_input_qsa. Blocks are cuts of the position line rather than of the cell
array, so nothing assumes the cache is contiguous.
Measured on the tiny fixture against vLLM, comparing the selected token
indices directly rather than the logits:
below the budget selection identical, and 1024-token logits are
bit-identical to the pre-QSA dense path
above the budget mean jaccard 0.975
The direct index comparison is what made this correct. The reference
rectifies each head's dot product before summing over heads, which an
earlier reading of it had missed; on logits alone the resulting port looked
fine, because on a randomly initialised fixture the known-correct dense
path already disagrees with vLLM by more than the bug did. Comparing the
indices showed 0.794, and fixing the ReLU moved it to 0.975.
build_rs writes into the state tensor in place, zeroing one row and copying the
carried-over states, so calling it twice for the same layer let the second call
clobber the first write-back. The PLE layer is also a delta-net layer, so that
is exactly what happened: both convolutions gathered the same row. They now
share a single gather per layer.
The earlier claim that the conv state was carried correctly was tested on a
fixture whose conv weights are zero, where the branch contributes nothing and
chunking matches trivially. Re-running with non-zero conv weights showed the
divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one
boundary down to 90.2% at seven. With the shared gather it is bit-identical to
the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum
logprob deviation of exactly zero over 1023 positions. The delta-net-only model
stays bit-identical too, so nothing regressed there.
Also derive the delta-net conv channel count the way load_arch_tensors sizes
wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r()
only bounds the row and the convolution has to match the tensor feeding it.
test-llama-archs previously aborted on this architecture and took every later
architecture with it. qwen4exp is marked MoE-only, given the hyper-connection
keys and an ssm_d_inner consistent with its tensor derivation, and skipped for
now: the hyper-connection keys written by get_gguf_ctx are not reaching the
synthesised file, which needs a separate look. The suite completes again, 124
architectures at 0.00e+00.
Implements the decode graph for Qwen3.8-Flash-Next: the hyper-connection
residual stream, gated delta net layers, the MoE block with its gated shared
expert, and dense full attention. The QSA indexer and the PLE n-gram embedding
are not wired up yet and land in later commits.
Hyper-connections are implemented here rather than shared with deepseek4.cpp.
The two formulations agree on the [n_embd, hc, n_tokens] layout and little
else: DeepSeek-V4 mixes with a full-rank projection and Sinkhorn-normalises
it, whereas this model uses a low-rank down/silu/up sigmoid gate and collapses
by a plain mean. Only the ~10 line stream mean is genuinely common, so sharing
would mean touching DSV4's hot path and its three fused CUDA ops to reuse very
little. What is reused is the substantive part: the LLM_KV_HYPER_CONNECTION_*
keys, the n_embd_out_impl wide-residual support already in the loader, and the
layout convention.
Also allows a checkpoint to carry no PLE layers at all, which makes it
possible to bring the graph up and validate it in stages.
Validated against vLLM, the only working reference implementation. On a
scaled-down model with an init scale large enough to give non-uniform logits,
agreement with vLLM sits at the numerical noise floor: llama.cpp f32 against
its own bf16 gives 84.3% top-1 agreement over 255 positions, and this graph
against vLLM gives 85.1%. The comparison was calibrated by seeding three
deliberate bugs (silu instead of sigmoid on the delta net gate, dropping the
1/hc scale in the mix, dropping the 2x in the combine); each drops top-1 to
between 0% and 11%, an order of magnitude below the floor.
Adds LLM_ARCH_QWEN4EXP with its hparams and tensor loading. The graph
comes in the next commit; this makes the model load and report correct
metadata.
- hyper-connections set n_embd_out_impl = hc_count * n_embd, so the
residual stream is 4x wide and there is no output_norm: the final
mixer's hc_norm is the last norm in the model.
- registered as hybrid and given the same recurrent/attention memory
filters as Qwen3-Next and Qwen3.5.
- reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys as-is.
- the PLE table row count is read back from the file rather than
recomputing the vocab padding rule.
llama-model-loader gains UINT64 array support. That branch previously
threw, so no existing caller changes behaviour; it is needed because the
PLE hash multipliers do not fit in int32.
Adds the GGUF-side plumbing for HF model_type qwen4_exp:
- MODEL_ARCH.QWEN4EXP plus tensors for the low-rank hyper-connection
variant (hc_*_norm/down/up/inject) and the PLE n-gram hash embeddings.
The DeepSeek-V4 hc_*_fn/base/scale tensors are a different
parameterisation, so these are separate entries rather than reuse.
- Reuses the existing indexer, per_layer_token_embd, SSM and
compress_ratios keys unchanged.
- conversion/qwen4exp.py inherits the Qwen3.5 linear-attention V-head
reorder and interleaved mrope, concatenates the 128 PLE embedding
shards, and splits index_qk_proj into separate indexer q/k tensors.
The PLE hash multipliers reach ~2.4e13. prepare_tensors() casts every
non-float dtype to float32 before modify_tensors() runs, and GGUF array
writes infer INT32 from Python ints, so both paths are bypassed: the
constants are read from the pre-cast lazy tensors and written as
explicit UINT64 arrays.
Additive only; no existing arch changes behaviour.
This commit contains a fix for the conversion of NVIDIA Nemotron 3.5
Lightning which currently incorrectly converts when using a transformers
version later than 5.5.1.
When converting using [convert](https://github.com/ggml-org/convert) the
transformers version is 5.13.1 and this produces the following:
```console
WARNING:gguf.gguf_writer:Duplicated key name 'nemotron_h_moe.attention.head_count_kv', overwriting it with new value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] of type ARRAY
```
This does not happen with transformers 5.5.1. The reason seems to be
that the configuration is different in later versions, for example when
using 5.13.1 the configuration block looks like this:
```console
transformers 5.13.1
raw has layers_block_type: True
autoconfig has layers_block_type: True
autoconfig layers_block_type: [
'linear_attention',
'moe',
'linear_attention',
'moe',
'linear_attention',
'full_attention',
'moe',
...
]
```
And with 5.5.1 we get:
```console
transformers 5.5.1
raw has layers_block_type: True
autoconfig has layers_block_type: True
autoconfig layers_block_type: [
'mamba',
'moe',
'mamba',
'moe',
'mamba',
'attention',
'moe'
...
]
```
In our conversion script we only match for attention, not full attention
which is causing this issue.
With the changes in this commit the output with transformers 5.13.1 will
be:
```console
(venv) $ gguf-dump models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf | grep head_count_kv
INFO:gguf-dump:* Loading: models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf
29: [INT32] | 52 | nemotron_h_moe.attention.head_count_kv = [0, 0, 0, 0, 0, 2, ...]
```
Resolves: https://github.com/ggml-org/llama.cpp/issues/27718
Refs: https://github.com/ggml-org/convert/actions/runs/32949047680/job/98116096069#step:5:2391
* convert: fix get block count error for Nemotron
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* fix this in NemotronHModel.__init__ instead.
This reverts commit ca689cbc87.
---------
Signed-off-by: Rock Chen <rockchen.tw@gmail.com>
* feat(convert): Add conversion for GraniteSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(llama): Add granite_swa support
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add conversion infra for rope_pattern array
NOTE: There is other work also targeting this, so this may be
removed depending on merge order.
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(conversion): Fix SWA pattern logic and support for non-rope layers
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat(conversion): Add support for GraniteMoeSWA
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add llama_hparams::has_rope and arch constants
NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add support for per-layer rope determination
Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Fix failing flake8 for extra newlines
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver
Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Load MoE params as optional
Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Handle MoE params in conversion
branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: Remove unnecessary newline
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Remove unnecessary tensor additions to GRANITE architecture
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Correctly handle naming for ffn gate inp
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Always default hparams.rope_pattern to 1s
This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Move to has_rope for all granite model architectures
Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: No hacky rope_finetuned fallback in has_rope
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fully remove rope hparam filling in granitemoe
There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Save out rope_pattern in model-saver
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Set hparams.rope_finetuned for round trip
Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Code review cleanup
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* refactor: Keep gate/up fused for MoE path
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Skip GRANITE_SWA in model saver
https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651
Keeping is_swa_impl in the saver can break other models.
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add sliding window pattern for model in test
* style: Fix indentation
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Fix \r\n
Thanks Claude!
Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Keep shared expert fused
Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* style: More indentation fixes
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* dspark: support speculators-format checkpoints (SpecForge exports)
Speculators-format DSpark drafts (e.g. SpecForge exports for the
Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in
three ways:
- the config nests the backbone hparams under transformer_layer_config
and gives the extract layers as aux_hidden_state_layer_ids
- the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus
token, not a prediction slot. Written as dflash.bonus_anchor; such
drafts build the block and read the mask positions exactly like
DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence
sampling comes from DSpark
- the draft output vocab may be reduced (draft_vocab_size < vocab_size)
with a d2t remap table. The converter expands lm_head/markov_w2 back
to the full vocab and synthesizes an lm_head bias of -1e9 on the rows
the draft cannot produce, so the runtime needs no d2t remapping. Such
drafts ship their own (now optional) token_embd/output tensors instead
of sharing the target's
Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical
with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the speculators draft class into DSparkModel
One class now covers every DSpark variant. What used to pick the class is
a single flag, because the arch name turns out to be the only thing that
separates the two families: SpecForge also exports a flat schema that
carries no speculators_* fields yet still uses the 1+N bonus-anchor block,
so keying on those fields would silently mis-read its drafts.
Also rename i0 to i_first_pred in the draft read loop and the Markov head,
and give the head a real bonus_anchor bool instead of testing i0 > 0.
Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts
produces byte-identical GGUFs. The one behaviour change is that the
markov_head_type check now also covers the DeepSpec checkpoints, which
previously skipped it.
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Opus 5
* dspark: address review comments
- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching
the checkpoint config field; absent key still means anchor-first
- rework the reduced draft vocab to match EAGLE3: d2t is written as I64
absolute target ids and the logits are scattered at runtime, instead of
expanding lm_head/markov_w2 and synthesizing an output bias at conversion
- move the t2d skip to modify_tensors, like EAGLE3
- drop _is_specforge: the arch name only picks the sample_from_anchor
default, embed/lm_head sharing is decided by the draft vocab size
- deduplicate the tok_embd create_tensor left behind by the rebase
Verified with the RedHat gemma-4-31b speculator draft: greedy output is
byte-identical with and without the draft; acceptance 0.26 (n_max 7).
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable 5
* dspark: fold the sample_from_anchor read into the block_size block
* dspark: fix flake8 continuation indent
* clean up
* dspark: key the sample_from_anchor default off the export format
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: drop t2d in filter_tensors
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
* dspark: map model.lm_head instead of bypassing the dflash prefix
Co-authored-by: desovo7 <942845546@qq.com>
Assisted-by: Claude Fable
---------
Co-authored-by: desovo7 <942845546@qq.com>
Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
* Adding support for bailingmoe3
* Adds speculative decoding support
* Make BailingMoE3 safe gate metadata optional
* bailingmoe3: apply trained SwiGLU clamps
* common: fix Bailing V3 tool argument parsing
* llama-model-saver, instantiate float vector metadata writer
* bailingmoe3: support Q-LoRA (Ling-3.0-tiny)
Ling-3.0-flash sets q_lora_rank: None and projects Q directly, so the current
implementation loads a single ATTN_Q tensor. Ling-3.0-tiny sets q_lora_rank: 256
and routes Q through a LoRA bottleneck instead:
q_a_proj -> q_a_layernorm -> q_b_proj
Conversion therefore failed with:
ValueError: Can not map tensor 'model.layers.3.attention.q_a_layernorm.weight'
Add the missing path, mirroring the existing deepseek2 MLA implementation:
* constants.py - add ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM to BAILINGMOE3
* tensor_mapping.py - map model.layers.{bid}.attention.q_{a,b}_proj and
q_a_layernorm
* conversion - emit attention.q_lora_rank when the config has it
* bailingmoe3.cpp - read n_lora_q; create the Q-LoRA tensors and build Q
through the bottleneck when q_lora_rank > 0
Everything is gated on q_lora_rank > 0. Ling-3.0-flash's config has no
q_lora_rank, the converter only emits the key when present, hparams.n_lora_q
defaults to 0, and get_key(..., required=false) leaves the target untouched when
the key is absent - so flash keeps taking the existing direct-Q branch.
The LoRA path produces the same shape as the direct projection, so the
nope/rope split, RoPE application and wk_b absorption downstream are unchanged.
* small mtp change
* bailingmoe3: support separate MTP GGUF and Q-LoRA MTP
* gguf: remove duplicate add_kda_gate_lower_bound definition
---------
Co-authored-by: bloomer <bloomer@booper.brushtail.me>
Co-authored-by: Dyluhn <dylanranejohnston1@gmail.com>
* model: add Kimi-K3 text model
Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five
things that architecture does not have:
1. cross-layer residual attention (attn_res_block_size)
2. latent MoE (routed experts run at n_expert_latent)
3. situ activation (replaces SwiGLU everywhere)
4. MLA output gate (sigmoid gate before o_proj)
5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b)
K3's text_config reports KimiLinearForCausalLM - the older 48B architecture -
so get_model_architecture routes on the top-level name instead.
The KDA decay gate has two forms, selected by linear_attn_config's
gate_lower_bound. It is not a clamp: when set it swaps the activation entirely
(fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to
lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it
unset, so that path is unchanged.
Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is
CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels
exist.
The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is
bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale
byte, only the nibble positions within a block differ - so they are repacked
rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip.
The repack is built lazily because gguf_writer holds every added tensor until
the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so
it now shares the helper.
Verified against Moonshot's own code path (transformers + fla's Triton KDA
kernels) on a tiny model exercising every K3-specific feature. Final-position
logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the
chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source
weights with 0.0e+00 error.
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* model: fix ty errors in the Kimi-K3 converter
- `_res_parts` buffers (kind, tensor) pairs, not bare tensors
- `get_tensors` must return an Iterator, matching ModelBase
- LazyBase's `func` takes one argument, so pass the expert loaders through
`args` instead of the closure
- borrowing KimiLinearModel.set_vocab from an unrelated TextModel is
deliberate and safe, but not expressible in the signature
No behaviour change: the MXFP4 repack still dequantizes to the source weights
with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel,
corr 0.99996630).
Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update conversion/kimi_k3.py
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
* Increase LLAMA_MAX_EXPERTS from 512 to 1024
* tests : support for Kimi K3 in archs test
* chat : add Kimi K3 chat format (reasoning, content, typed tool calls)
K3's assistant output is an XTML-ish tagged format built by the template's
open_tag/close_tag macros. Two properties break generic parsing:
1. The generation prompt ends with open_tag('think'), so the completion
starts inside the think section with no opening marker in the output
(thinking_forced_open).
2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag
names ("think", "response", "message") are ordinary text tokens.
Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the
marker trio, reasoning extraction, response unwrapping, and tool-call
parsing of the tools/call/argument tag structure with argument types
taken from the tool schema. Includes the K3 chat template fixture and 9
test-chat cases derived from real generations of the full 2.8T model.
Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:
content, reasoning_content, streaming deltas, and tool_calls all correct;
finish_reason stop/tool_calls as appropriate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chat : add message_delimiters for Kimi K3
Per-role message-start markers for token-level span splitting. User and
assistant messages carry only the role attribute, so their full opener
(through <|sep|>) is used; system and tool messages continue with more
attributes (type=/tool=/index=), so those delimiters stop after the
role's closing quote. Verified against the K3 tiktoken vocabulary that
the closing quote is always a standalone token across all attribute
variants, so the token-level prefix match stays exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: apply nits from @ngxson and text fixes from @danielhanchen
* tests : added missing hyperparameters and tensors for Kimi K3 in test-llama-archs
* chore : move overly verbose header file comments to Kimi K3 source file
* tests : re-enabled KIMI_K3 in test-llama-archs for WebGPU backend
* model-saver : emit kda_gate_lower_bound for Kimi K3
Quick fix. The Kimi K3 loader reads kda_gate_lower_bound and gates a graph branch on it (it scales the KDA gate when the bound is above -INFINITY), but the model
saver never wrote the key, so a save->load roundtrip silently dropped it back to the -INFINITY default and changed the model's output. The real K3 config sets gate_lower_bound = -5.0.
I propose to emit it from the saver, and set it to -5.0 in the test-llama-archs K3 case so the roundtrip check exercises it (the roundtrip fails without the saver line).
* Refactor conditional for model architecture check
* tests : re-enabled (again) KIMI_K3 and MINIMAX_M3 in test-llama-archs for WebGPU backend
* fix code comments
* add template on conversion
* move repack_mxfp4_blocks to model base
* nits
* add_value_length
* optimize res_stack construction
* nits
---------
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Deepankar Singh <singh.deepankar39@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Caleb DeLeeuw <caleb.deleeuw@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* llama : support for MiniMax-Text-01 model
* chore : renames to match the other MiniMax models
* model : add logits mask as MiniMax-Text-01 embeddings tensor has zero-valued embeddings for tokens >= 200032 that produce zero logits disrupting the token sampling process
* llama : replace hardcoded conditions with hparams.is_recr()
* model : used build_rs() for recurrent state management
* chore : code cleanup
* model : optimized MiniMax-Text-01 by removing the state tranpose operations
* chore : removed unnecessary ggml_cont() in MiniMax-Text-01 implementation
* llama : add generic logits mask graph input
* model : permuted diag_decay dimensions to avoid doing it inside MiniMax-Text-01 graph
* chore : code cleanup
* chore : code cleanup
* model : use token positions when calculating MiniMax-Text-01 decay tensors
* convert : add support for MiniMaxM1ForCausalLM as it seems to be the same as MiniMaxText01ForCausalLM
* chat : add jinja template for MiniMax-M1
Co-authored-by: QscQ <qscqesze@gmail.com>
* chore : code cleanup
* tests : MINIMAX_01-related fixes
* chore : silence Python lint errors
* vocab : remove unnecessary vocab type
* convert : update MiniMaxText01Model conversion to use yield when modifying tensors
* convert : suppress tokens with zero-valued embeddings during MiniMax-Text-01 conversion
* llama : removed logits mask - no longer necessary as token suppression is used instead
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* model : use common functions to make MiniMax-Text-01 implementation more concise
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* convert : override non-working built-in chat template during conversion
* tests : skip arch MINIMAX_01 tests for WebGPU backend (it breaks again)
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: QscQ <qscqesze@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* fix: handle nested global_head_dim in Gemma4 config
Gemma-4 E4B models have global_head_dim inside text_config
rather than at the top level. Add fallback to support both layouts.
* fix: add fallback for global_head_dim to support per_layer_config format
* fix: read head_dim only from full_attention layers in per_layer_config and num_global_key_value_heads compatibility
* fix: added fallback for num_global_key_value_heads
* fix: read per_layer_config from root hparams
* fix: delete unused text_config
* cleanup and fixes
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* adapt the api
* text model ok
* working impl, need verify and clean up
* mtmd: build the pocket-tts transposed convolutions as GEMM + col2im
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample
was built as one convolution and one concat per channel, which floods
the graph with small nodes and makes kernel launches dominate the
decoder.
Fold both cases into the column form the seanet decoder already needs:
the general case reshapes the kernel to [IC, K * OC] and matmuls it
with the input, the depthwise case batches a matmul over the channels
so a step scales its own kernel. A single col2im_1d then scatter-adds
the columns back to the signal, with the same shape as before, so the
overlap-add tail, the streaming state and the bias are untouched.
Generation time per frame drops by 80% on CUDA and by 50% on CPU. The
output matches the previous implementation sample for sample, with a
correlation of 0.999994 and identical frame counts.
* flow_temp + frames_after_eos
* chunking
* mtmd: carry the remaining pocket-tts per-pack settings
The language packs also tune the end-of-speech padding and the padding
of short prompts, next to the temperature already carried in the
mmproj: french_24l asks for 8 tail frames instead of the guessed 3,
english_2026-01 asks for short prompts to be padded with spaces.
Write both in the mmproj as clip.gen.audio.frames_after_eos and
clip.gen.audio.pad_short_text, keyed on the pack in the conversion
script like the temperature. The loader keeps them optional, so a
mmproj without them behaves as before. Map semicolons to commas for
every pack instead, the reference only asks for it on three of them and
it costs nothing elsewhere.
Existing mmproj files must be converted again to carry the two keys.
On a long french text the port now lands within 2% of the reference:
22.96s against 23.44s, with the same peak level and the same amount of
silence.
* clip.gen.audio.model_variant
* clean up code comments
* nit: drop the dead flow_temp hparam, the pack table holds the default
* update docs
* address security problems
* less invasive base.py
* lint
* add mtmd_gen_inp_default
* add docs
* rm gen_flow_temp
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Get started with Onyx
* Add architecture
* Skip keys handled in super()
* Loading tensors
* Shorten
* Graph
* Apply suggestion from @pcuenca
* Remove norm now embedding in transformers weights
* Add eot
* Explicit output_multiplier
* Handle post_norm_eps
* No super call; unhardcode eot.
The pattern `self._set_vocab_gpt2()` seems preferred throughout the
codebase, and it allows `set_vocab()` to be called from a different part
of the Python class hierarchy: the drafter model converter that we may
need eventually.
* Register for drafting
* DFlash: inherit rope type from the linked target.
Another option would be to store it in the gguf file itself.
* mmproj conversion
Note: some fields to be renamed after the implementation works. We are
keeping compatibility with the reference Meta gguf for testing purposes.
* "clip" header declarations
* Load mmproj
* Pre-processing
* Graph
* Go back to using delimiters.
Otherwise our generations are worse.
Transformers does not use them. We need to trace inputs to verify
whether they are equivalent.
* downsample_factor -> merge_size
* Add vision graph
lol, forgot from a previous commit
* Additional renames, align with llama.cpp / transformers
* Prefer _size instead of independent _h and _w
* Fix token layout
Co-authored-by: Young Han <younghan@fb.com>
* onyx: bring the chat parser onto the onyx branch
common/chat.cpp on this branch has no Onyx handling, so a converted model
serves malformed chat: the assistant preamble leaks into content
("to=self<|message|>...") and tool calls fail with
HTTP 500 "The model produced output that does not match the expected
peg-native format"
common_chat_params_init_onyx exists on onyx-fair-patch, added there by
8bb73dd3d. It was never on this branch, so this is not a regression --
the two lines developed independently.
The code here is taken verbatim from that commit. It is the clean side of
`git merge origin/onyx-fair-patch`: chat.cpp is one of the files that
merges without conflict. The full merge is not viable -- it produces 13
conflicts, including add/add on conversion/onyx.py and src/models/onyx.cpp
where the q_norm-folding and metadata-scale approaches contradict each
other, and #4/#7 are stacked on this branch's side of that.
Verified on this branch: builds with 0 errors, converts an Onyx checkpoint,
and serving it gives "4" for "What is 2+2?" plus a correct
get_weather {"city":"Paris"} tool call, where the unported branch gives the
two failures above.
No converter or runtime changes are included, so this should not interact
with the q_norm work.
Co-authored-by: Beto de Paola <betodepaola@meta.com>
* Less params, bilinear pos-emb interpolation as a graph op instead of CPU
* Map to symbolic V_MMPROJ instead of strings
* Make a couple params explicit
* Patchify via build_inp()
* No param for rope_theta
* Small cleanup
* Restore blank line
* Unpermute, to adapt to the latest transformers checkpoint
* Apply norm after token embeddings
This follows the latest transformers approach.
* Remove duplicated function
* build_vit
* onyx: use the model rope theta on sliding-window layers
* DFlash: conversion from transformers drafter
* Revert rope_type derivation from target
NOTE: this breaks compatibility with Meta's distributed DFlash GGUFs, as
the Q/K are stored in "NEOX" (rotated half) format, like in
transformers.
* Apply suggestion from @pcuenca
* Set model type
* Remove comment that will become obsolete
* Hardcode post_norm_rms_eps instead of new param
* Derive SWA+RoPE pattern from gguf array or scalar
* Fix model type <-> number of layers
* Reorder
* Rename
* Fix typo
* DFlash: seed the draft KV cache from multimodal embedding batches
`common_speculative_impl_draft_dflash::process()` returned early on any batch carrying embeddings, so an image prefill never had its target-layer features fused through the DFlash encoder and injected into the draft's KV cache. That left a hole spanning the image's positions, and the next injection at a post-image position failed to initialize its batch:
```
decoding image batch 1/1, n_tokens_batch = 256
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1
process: llama_decode(ctx_dft) failed rc=-1 (n_tokens=17, offset=0)
srv decode: failed to process speculative batch
```
Every image request with `--spec-type draft-dflash` failed with HTTP 500. Text-only was unaffected, since those batches carry token ids and were let through.
Restore the earlier condition, which admits a batch that is either tokens or embeddings and skips only the degenerate neither/both cases. The rest of `process()` is already layout-agnostic -- it gathers features via `llama_get_embeddings_layer_inp()` and indexes `batch_in.pos[]` / `batch_in.seq_id[]`, none of which assume token ids -- so this is the whole fix.
Validated against `muse-glimmer-30B-bf16.gguf` + `mmproj-muse-glimmer-30B-bf16.gguf` + a DFlash draft head, on an image describe-the-shapes request:
- before: HTTP 500, `failed to process speculative batch`
- after: HTTP 200, draft acceptance 0.34012 (167 accepted / 491 generated), mean len 3.04
Output equivalence holds, which is the property that matters: at temperature 0 the drafted response is byte-identical to the same request served with no draft attached (1213/1213 chars), so the draft is drafting correctly through the image context rather than merely not crashing.
* Conversion: prefer rewrite to mapping
* Revert "Conversion: prefer rewrite to mapping"
This reverts commit a92d0ac584.
* fix lint
* sliding_window metadata is not optional
* disable state save/load
* Apply suggestion from @pcuenca
---------
Co-authored-by: Young Han <younghan@fb.com>
Co-authored-by: Beto de Paola <betodepaola@meta.com>
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: ruanrms <ruanslv@gmail.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* granite-switch: add llama.cpp backend (POC, CPU)
New "granite-switch" architecture: a dense, all-attention Granite-4.1
model with N embedded LoRA adapters selected per-token by control tokens.
- gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers
- conversion/granite.py: GraniteSwitchModel converter (stacks N adapters +
zero base slot into per-projection A/B tensors; emits switch metadata)
- C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp})
- src/models/granite_switch.cpp: load + per-token switched-LoRA graph via
ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token
substitution in llm_graph_input_switch::set_input
- llm_graph_input_switch in src/models/models.h
Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13)
and generate on both base and control-token paths. Sticky switch state is
single-sequence (POC); full multi-sequence machinery is a follow-up.
* granite-switch: add Mac (Metal) build + mid-sequence switch demo script
Self-contained script to build llama.cpp on Apple Silicon (Metal),
convert the composed 3b checkpoint, and run the crisp mid-sequence
adapter-switch demos verified on Vela:
- answerability: <|answerability|> mid-seq -> "unanswerable"
- query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...}
Each demo runs the same prompt twice, differing only by a control token
placed before the assistant turn, so the per-token switch is visible.
* granite-switch mac demo: add -no-cnv so each run is one-shot
The composed model ships a chat template, so llama-completion auto-enables
interactive conversation mode and halts at a `>` prompt after generating,
stalling the script. -no-cnv disables conversation mode: generate once from
the raw prompt and exit (also prints special tokens, making the switch visible).
* granite-switch: replace global sticky index with in-graph router attention
The POC computed the per-token adapter index on the CPU and carried it
across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset
only when a ubatch contained sequence position 0. That global had two
problems:
1. Concurrency: with multiple sequences in a batch it was last-writer-
wins — one sequence's adapter leaked into the others.
2. Multi-turn: an interactive `ollama run` chat continues one KV cache,
so turn 2 never saw position 0 and the index never reset — the
adapter stayed stuck on across turns.
Port the vLLM/HF backend mechanism faithfully: a single-head causal
"router" attention recovers the adapter index in-graph. Per token, only
dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain
otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single
visible control token recovers that adapter's slot (readback =
clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is
F16-safe (no F32 cache).
The router's K/V live in the model KV cache at an extra layer
R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1
so the cache allocator gives the router its own per-sequence slot, and
set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and
tensor loading are untouched and never reference layer R. The router K is
exempted from the k-shift RoPE loop (its dim-0 value is a literal
magnitude, not a rotation).
Because the selection now lives in the per-sequence KV cache, CONCURRENT
requests are isolated for free (problem 1 fixed; verified by
scratch/concurrent_switch_test.cpp). set_input becomes stateless pure
per-token maps; the global is gone.
Single-switch contract / known limitation, identical to vLLM & HF: the
gain is flat (no recency), so within one sequence there is no mechanism to
revert to base mid-sequence — once an adapter fires it stays on until that
sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF
avoid it only because each served request is a fresh sequence). A client
continuing one KV cache across turns must start a fresh sequence per turn,
or opt into a recency-biased router (a deliberate divergence, not done
here). Documented in granite_switch.cpp and asserted by
scratch/multiturn_leak_test.cpp.
Verified (CPU): both demos unchanged (answerability -> "unanswerable",
query_rewrite -> rewritten query); concurrent two-sequence isolation
passes; multi-turn carry-over matches the vLLM/HF contract.
* granite-switch: drop scratch tests and mac demo for upstream PR
Remove the local-only development artifacts that should not ship in the
upstream PR:
- granite-switch-mac-demo.sh (local Metal build + demo driver)
- scratch/concurrent_switch_test.cpp
- scratch/multiturn_leak_test.cpp
Also drop the now-dangling reference to the scratch tests from the
granite_switch.cpp header comment. Leaves only the core architecture
support (conversion, gguf constants, llama-arch/model/kv-cache, and the
granite_switch graph).
* granite-switch: trim comments to match native llama.cpp style
* granite-switch: trim conversion comments to match native style
* granite-switch: drop unused adapter_ranks metadata
* granite-switch: rename arch to graniteswitch and drop obid alias
* granite-switch: fix non-ASCII comments and document router gain assumption
* granite-switch: drop section comments from constants.py to match native style
* granite-switch: add functional tensor block comments matching Granite4 Vision style
* granite-switch: clarify n_expert_used comment
State the actual constraint: mul_mat_id needs n_expert_used == 1, and
since the GGUF carries expert_count = 0 the generic loader's
n_expert == 0 => n_expert_used == 0 assertion has already passed by the
time load_arch_hparams runs, so it is forced to 1 here.
* granite-switch: note n_layer_nextn reuse has no MTP
The router carving reuses n_layer_nextn, normally the MTP/next-token
count. Clarify in the comment that it is borrowed here purely as the
trailing-layers lever and that there is no MTP head, to spare readers
the double-take.
* granite-switch: rename source file and apply review nits
* granite-switch: don't force LoRA tensors to F16, follow --outtype instead
* granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly
* granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0
* granite-switch: derive n_slots()
* granite-switch: move llm_graph_input_switch into granite-switch.cpp
* granite-switch: cut AI-style narration comments
* granite-switch: collapse multi-line comments
* granite-switch: rename control_token_* maps to adapter_token_*
* granite-switch: cut noise comments
* granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b
* granite-switch: GGML_ASSERT token input to avoid UB on embeddings
* granite-switch: TODO for raw embedding input support
* granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix
* granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe
* granite-switch: stop forcing dense expert counts, read from config
* granite-switch: renamed control_token_gain metadata key to router_gain
* granite-switch: trim header comments to match native style
* granite-switch: collapse LoRA tensors to base name + suffix
* granite-switch: inline suffix checks in tensor op resolution
* granite-switch: drop switch-lora struct comment
* granite-switch: guard router layer index and inline n_slots
* granite-switch: group adapter metadata under {arch}.adapters.* namespace
* granite-switch: add hparams.has_rope(il) for KV-shift rope skipping
* granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO)
* granite-switch: Keys.Adapters namespace + simplify n_slots
* granite-switch: validate substitute token ids against n_vocab
* granite-switch: bound adapter count and lora rank from GGUF
* granite-switch: reject MTP context type when router_layer is set
* granite-switch: throw on bad adapter metadata instead of GGML_ASSERT
* granite-switch: use ASCII +/- in router K signal comment
* granite-switch: document n_layer_nextn repurpose and its leak points
* granite-switch: gate lora_a/lora_b op mapping on router_layer
* granite-switch: label all three preview model sizes
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.
Applies the same fix to chatglm.py.
* common/chat: update DeepSeek V4 templates
Align the DeepSeek V4 templates with the official encoders while keeping parser behavior out of this change.
- Default drop_thinking for DeepSeek V4 history so prior thinking is omitted unless preserve_reasoning is requested or tools are present.
- Add structured output response-format instructions to the V4 templates and pass the schema into template rendering.
- Add a separate Flash 0731 template for the updated high and max reasoning effort mapping.
- Cover reasoning effort, drop_thinking, structured output prompts, preserved reasoning, continuations, and empty tool arguments in template rendering tests.
Official references:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.pyhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py
Assisted-by: Codex
* Fix deepseek v4 0731 template selection
* remove unneeded lower normalization
* Fix DSML parser to consume the tool call separator
* address aldehir requests
* address aldehir comment
* llama : MTP support for DeepSeek V3.2
* model : no need to include MTP layers during DeepSeek V3.2 model type discovery
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)
Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)
Opt GLM-5.2 into the supports_mtp_export contract (post-#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Xuan Son Nguyen