upstream-archive/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-27 00:17:27 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,052 Commits 813 Branches 7,612 Tags
b11052
Commit Graph
2 Commits
Author SHA1 Message Date
Yaniss Amazouz c61b98b875 model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444) 
* hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading

G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing
(required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values
across its 40 MoE layers).

Design: rename scalar members to _impl suffix (following existing convention),
add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with
scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and
expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer).

- llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl;
  add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations.
- llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range
  il returns impl safely (shared code, no abort).
- llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive
  impl as max-of-array for validations and backward compat; zero both new arrays;
  HunyuanVL override also zeroes n_expert_used_arr.
- llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il)
  so per-layer top-k bounds the ggml_view loop correctly.
- All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl.
  Scalar arches are unaffected (broadcast fills all array slots with the single value).

(cherry picked from commit 269a81e03d)

* nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites

Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams;
derive impl as max for existing uniform GGUFs.

In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to
n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length.

In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer
top-k is used for expert routing selection.

All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection,
routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged.

(cherry picked from commit b1878a1017)

* arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl

All non-nemotron arch files continue using the scalar impl member directly.
Behaviour is identical: the impl value is the broadcast value from the GGUF scalar.

gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept
int | Sequence[int], mirroring add_feed_forward_length, so converters can write
per-layer arrays with the same existing GGUF keys.

(cherry picked from commit 8f009f54be)

* convert: support NemotronHPuzzleForCausalLM (per-block MoE config)

Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array
keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with
nextn tensors, accept the backbone.* prefix, and register the arch.
Also fix a pre-existing undeclared _experts attribute on NemotronHModel.

(cherry picked from commit d1a592f278)

* nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B)

Both have 88 layers; the per-layer expert_used_count array (heterogeneous
for Puzzle, broadcast-uniform for Super) is the discriminator.

(cherry picked from commit f824e09dc8)

* convert: accept the official Puzzle BF16 checkpoint's tensor naming

The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle-
75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings,
model.norm_f) where the original release used the NemotronH-style
backbone.*, and spells the router bias e_score_correction_bias instead of
e_score_correction.bias. Normalize both at the top of
NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every
tensor name in the official index (42683 keys, MTP head included) resolves
through the tensor map after normalization.

(cherry picked from commit 189b67fc2c)

* laguna: use n_ff_exp_impl for the uniform-MoE FFN size

Laguna landed after this branch was cut and reads hparams.n_ff_exp as a
scalar. This series turns it into a per-layer array with an n_ff_exp(il)
accessor, so the three scalar reads no longer compile. Laguna is a
uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as
deepseek2/qwen3moe/gemma4 in this series. No behaviour change.

(cherry picked from commit dbedc9e19c)

* arch: extend the n_ff_exp/n_expert_used rename to archs added upstream

kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP
block still referenced the scalar fields by their old names. n_ff_exp and
n_expert_used are accessors now, so those reads no longer compile; point the
non-per-layer archs at the _impl scalars and use the indexed form where the
call site is per-layer.

* convert: keep Puzzle opted out of the NemotronH MTP export path

#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers.
Puzzle's config carries that key, but NemotronHPuzzleModel bypasses
NemotronHModel.__init__ (its per-block config needs a different setup), so
_mtp_bid was never assigned and modify_tensors raised AttributeError on any
mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the
mtp.layers.* form the base maps.

Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare
supports_mtp_export = False so --mtp / --no-mtp fail at the CLI.

* llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors

Follow-up to review feedback: the previous revision kept the scalar
hparams fields alongside the new per-layer arrays, which duplicated
state that get_key_or_arr already handles by broadcasting a scalar
value over every layer.

Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built
exactly like the existing n_head_kv(il) and n_ff(il) accessors: they
index the array and GGML_ABORT out of range, with il defaulting to 0
so genuinely uniform call sites stay a plain n_ff_exp().

Arch loaders now read both keys through get_key_or_arr over
n_layer_all, and the n_expert_used validation checks the maximum
across layers instead of a single field.

* llama: restore per-key required flags on the expert hparam reads

The scalar-to-array conversion passed required=false at every call site,
which silently made mandatory keys optional. Each read now carries the
same required flag it had before the conversion.
 2026-09-03 08:53:08 +02:00
Gabe GoodhartandSigbjørn Skjæret 7221e24f57 model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505) 
* feat(convert): Add conversion for GraniteSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(llama): Add granite_swa support

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob, OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add conversion infra for rope_pattern array

NOTE: There is other work also targeting this, so this may be
removed depending on merge order.

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(conversion): Fix SWA pattern logic and support for non-rope layers

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(conversion): Add support for GraniteMoeSWA

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add llama_hparams::has_rope and arch constants

NOTE: This shadows the work done for Granite Speech
https://github.com/ggml-org/llama.cpp/pull/25107

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Add support for per-layer rope determination

Branch: GraniteSWAForCausalLM
AI-usage: full (Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Fix failing flake8 for extra newlines

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver

Branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Load MoE params as optional

Branch: GraniteSWAForCausalLM
AI-usage: draft (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Handle MoE params in conversion

branch: GraniteSWAForCausalLM
AI-usage: full (OpenCode + Qwen3.6-35b)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove unnecessary newline

AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Remove unnecessary tensor additions to GRANITE architecture

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Correctly handle naming for ffn gate inp

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Always default hparams.rope_pattern to 1s

This isn't strictly necessary, but it will allow other models to rely on
hparams.has_rope(il) without needting to prepopulate.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Move to has_rope for all granite model architectures

Now that we have a proper hparam for this, it's better to use it and not
require a hacky fallback in the hparam method itself.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: No hacky rope_finetuned fallback in has_rope

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fully remove rope hparam filling in granitemoe

There are no granitemoe models that use NoPE (it's not actually used in the
layer building below), so this was just dead code.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Save out rope_pattern in model-saver

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Set hparams.rope_finetuned for round trip

Since the value is _read_ from rope_finetuned, we need to persist it when
the model is saved with the saver.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Code review cleanup

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* refactor: Keep gate/up fused for MoE path

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Skip GRANITE_SWA in model saver

https://github.com/ggml-org/llama.cpp/pull/25505#discussion_r3773175651

Keeping is_swa_impl in the saver can break other models.

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* add sliding window pattern for model in test

* style: Fix indentation

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* fix: Fix \r\n

Thanks Claude!

Branch: GraniteSWAForCausalLM
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Keep shared expert fused

Branch: GraniteSWAForCausalLM
AI-usage: full (Claude + Sonnet 5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: More indentation fixes

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
 2026-08-19 16:53:31 +02:00