* tests: add backend option to test-llama-archs
* Update tests/test-llama-archs.cpp
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* remove extra space
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Restore get_cache_directory() as fs::path as string() can be lossy on Windows
Partially reverts #29125
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
ggml_conv_1d_dw builds its im2col as f32 when the kernel is bf16, then
multiplies the two, so a depthwise convolution over bf16 weights asks
for kernel_mul_mv_f32_bf16, which was never instantiated. The base, the
_4 and the _short families are filled in next to their bf16 neighbours,
inside the same runtime guard, so a device without bf16 support is
unaffected.
* CUDA: enable sparse-fa for dsv4 prefill (again)
* CUDA: unroll the query loop of the sparse mask scan
The query loop of flash_attn_mask_to_sparse_indices has a runtime trip
count, which keeps the unrolled scan over the values of a lane from
issuing its loads together. Template the kernel on ncols1 so the loop
is bounded at compile time: batch one decodes compile to straight line
code and the scan drops from 46 to 17 us at 49k columns on sparse
decode shapes.
* CUDA: pick the out of bounds check of the sparse mask scan in host code
The query loop of the ncols1 == 8 scan keeps a runtime bound and an
early exit, so it does not unroll past its first iteration. Template the
kernel on whether the last group of queries is partial, decided on the
host from n_queries, and hoist the column bound out of the loop: the
loop becomes straight line code and the batched sparse op at 49k
context drops from 586 to 244 us.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* jinja : parse unary +/- before variables
Lexer already emits unary_operator for -n / +n, and runtime executes
unary -. Parse them at multiplicative precedence so slices like
items[:-n] and GigaChat indent[:-indent_factor] work.
* jinja : keep filters/tests outside unary operands
Unary +/- must bind only the primary/postfix operand so -n|abs is
(-n)|abs, not -(n|abs). Add unary + and filter/test regression coverage.
Signed-off-by: sinksilk <785976238@qq.com>
---------
Signed-off-by: sinksilk <785976238@qq.com>
The OpenAI chat completions API specifies content part type "video_url"
with a {"url": ...} object, and clients typically send data: URIs
(e.g. data:video/mp4;base64,...). The llama-server only accepted the
non-standard "input_video" type and rejected data: URIs for video
(accept_base64_uri=false), so any OpenAI-conformant client failed with
"unsupported content[].type" or "Invalid uri format".
- accept "video_url" as an alias of "input_video"
- read the media object from whichever key was used
- allow data: URIs for video (data:video/*), as already done for images
This commit adds a new recipe/target to the Makefile which allows the
logits verification to be run on pre-existing model outputs.
The motivation for this is that for large models it can take a long time
to run them models, and especially for the original model which seldom
changes this is very time consuming. With this change we can run the
original model one which will store the tokens and logits, and then
manually run the converted model and the run use this recipe to verify
them against the orignal model.
* dspark: add Gemma 4 draft support
Add GGUF conversion and runtime support for full-attention and SWA Gemma 4
DSpark drafts, including tied output weights and boolean backbone metadata.
Assisted-by: Codex
* dflash: infer Gemma draft features from metadata
* vulkan: optimize IQ4_XS matmul kernels
Assisted-by: OpenAI Codex
* vulkan: address IQ4_XS review nits
- drop the dead LOAD_VEC_A != 8 branch in the IQ4_XS shmem load; iq4_xs is
in lut_load_vec_a()'s "8" list, so that path is never generated
- disable MMVQ for IQ4_XS on Intel (27.3% tg regression on A770)
- remove a stray empty line in types.glsl
Assisted-By: Claude Opus 5 <noreply@anthropic.com>
Since #28732 our internal symbols are exported. A duplicate copy dlopened and
dlclosed by ggml_backend_load_all() then interposes them, so its destructors
destroy the live vk_instance and later device queries hit the GGML_ASSERT on
vk_instance.device_indices. Hidden visibility exports only GGML_BACKEND_API,
as before #28732.
Fixes#29138
Assisted-by: henk:claude-fable-5
Avoid returning references through lambdas that hold a local cast pointer, which triggers -Werror=dangling-reference in some CI compilers. Reuse the precomputed select_expr pointer directly.
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp
* server: route every model load through the queue
A model loaded by the fast path has no queue entry, so tick() evicts
it at its LOADED transition before its own request is proxied. Every
load now joins the queue, whose entry protects the model until its
waiters leave.
* server: do not admit requests into a stopping model
A request for a model that is being stopped still sees it LOADED and
is proxied into the dying child. Such a request now joins the queue
and is served by the next instance. The stopping mark is cleared
under the same lock that sets UNLOADED, so no request can see a
model that is neither stopping nor unloaded while its child is gone.
* vulkan : Intel FA kernel optimization for split k path
* vulkan : Host code update for Intel split k FA kernel path selection, fix A770 Linux op test failures
* vulkan : use symmetric coopMatMulAdd() in flash_attn_decode_phase_1 shader to resolve test op failre on A770 Linux with 26.2.3 mesa driver
* vulkan : fix editorconfig issue in flash_attn_decode_phase_2.comp
---------
Co-authored-by: Liu, Russell <russell.liu@intel.com>
* metal : gate mul_mm_id src1 rescale behind ggml_prec
Assisted-by: Claude Fable 5.1
* ggml-webgpu: reject MUL_MAT_ID when src1 precision is F32
* cuda/vulkan: reject MUL_MAT_ID in supports_op when src1 prec is F32
fix `supports_op` to return false for failing backends when the specified src1 precision is f32
Assisted-by: Claude Fable 5.1
---------
Co-authored-by: yomaytk <yoshimura.masashi.frbs@gmail.com>
* model : add DFlash layer-input taps for HunyuanVL
DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:
GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")
Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.
The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.
Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft conversion against HunYuan targets
Converting a DFlash draft with a HunYuan target failed in two ways.
1. DFlashModel.set_vocab() reuses the target class' vocab handling by
calling it unbound with the draft instance, but HunYuanModel.set_vocab()
called self._fix_special_tokens(), a method that only exists on
HunYuanModel, so the conversion always aborted with
AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'
Make the vocab helpers module-level functions taking the model
explicitly, so they do not depend on the instance being a HunYuanModel.
They have no other callers, so the two id lookups are folded into
_fix_special_tokens().
2. The delegated call runs with self.dir_model pointed at the target but
keeps the draft's self.hparams, so config lookups inside the target's
vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
config instead of the target's. That aborts on targets with
pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
writes special token ids that disagree with the target.
Add _vocab_hparams(): it returns the target's config (with text_config
merged to the root, as TextModel does) when the model is a draft
converted with --target-model-dir, and the model's own hparams
otherwise, so a normal conversion is unaffected.
Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.
Co-authored-by: wendadawen <wendadawen@qq.com>
* convert : fix DFlash draft vocab against HunYuan targets
Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.
* convert : fix HunYuan special token ids for DFlash drafts
* convert : use load_hparams for HunYuan special token ids
* convert: add MiMo-V2.6 support
Hoist the K3 mxfp4 conversion repack into base.py so it can be reused
Remove decoder from mmproj convert
* Update conversion/mimo.py
* fix: use autoparser
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
