* tests: add model resolution test on synthetic repo listings
Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.
Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.
* tests: keep model resolution checks active and let the handler resolve
Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.
* tests: fix model resolution build on fatal warnings CI and Windows
The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.
* tests: fix winsock inclusion order for the model resolution test
WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.
* tests: link cpp-httplib to the model resolution test
The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.
* common_http_client
* common: finish the http client wrapper
Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.
* tests: rewrite model resolution on the http client stub
Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.
Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.
* tests: fix the model resolution on Windows and the builds without TLS
Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.
* tests: make the model resolution failures self explanatory
Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.
* common: hold the http client factory behind exported functions
The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.
* common: add the http client factory source missed in the previous commit
* common: typedef the http client factory callback
Address review from @ngxson
* tests: serve the model resolution repos over the loopback
Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.
* common: add portable common_get_env and common_set_env helpers
Address review from @ngxson
* common: drop the http client factory left without a caller
The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.
* common: read the model endpoint through the env helpers
* nit: drop the stub leftovers from the model resolution test
* common: align common_set_env and isolate the test cache per run
The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.
The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.
* tests: move the model resolution server into main
* tests: support the DSpark sidecar resolution
* common: revert the http client to the plain httplib client
address review from @ngxson
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* add a direct size condition for `large` weights; the original
dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
does not satisfy the dimension condition (ne0 >= 2048)
* sampling: enhance penalty handling in common_sampler_init
- Set default value for penalty_last_n based on model context if not specified.
- Ensure penalty_last_n and n_prev are non-negative.
- Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
- Implement backend initialization and application logic for penalties, including frequency and presence adjustments.
* tests: add backend penalties sampling tests and utility functions
- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness.
- Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties.
- Added `test_backend_penalties_sampling` to validate backend penalties with various configurations.
- Enhanced the test suite for better coverage of penalty handling in sampling.
* sampling: add support for top-k penalties in backend sampling
* sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.
* sampling: enhance penalty comparison tests with masking penalties logic
* add comments on padding
* sampling: add comments on modifications
* add the unit test to cover masked-out token as -INF
* validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values
* refactor: test functions to share logic and be less verbose
* add test to cover case where previously penalized token is not part of candidates
* remove comments
* remove redundant penalty_last_n initialization and validation in common_sampler_init
* add support for penalties in sampler chain with configurable positions
* add validation for penalty parameters and enhance tests for non-finite values
* add context parameter to common_sampler_init and set default for penalty_last_n
* add llama_n_ctx parameter to common_sampler_init for improved sampler initialization
* replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor
* add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64
* add test for backend penalties sampling after top-p with large history window
* remove as unused
* add is_disabled method, tensor logits reshape, add rest review suggestions
* clarify comment
* CUDA: Fix data-races when reusing block_reduce
block_reduce currently doesn't resync after reading from SMEM, causing
potential data-races when reusing SMEM for multiple reductions.
One may consider simply always adding this in block_reduce, but this
comes at a potential perf cost
* double-buffering for single-row softmax
* double-buffering for norm as well
* Add comment
* Add explanatory comment to block_reduce
* Specify need for + do memory barrier only in multi-warp scenario
* Implement review-suggestion from @gaugarg-nv
* llama : MTP support for DeepSeek V3.2
* model : no need to include MTP layers during DeepSeek V3.2 model type discovery
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* feat(silu_back): implemented silu_back op for f32
* fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.
- Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and
GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels.
- Add Metal dispatch and support plumbing and test the production Sinkhorn
iteration count and embedding width.
Assisted-by: Codex
Co-authored-by: Thiago Padilha <thiago@padilha.cc>
The dspark- files resolve like the other speculative sidecars: the
-hfd tag applies to them, a requested sidecar resolves without a full
model at the tag, and an explicit -md selection disables the discovery.
When no type is requested, dspark outranks dflash in the auto-selection
since its sidecar carries the extra Markov head.
Incrementing `ref_count` at the beginning is important later
in the `free()` method of the `ggml_backend_opencl_context` at program end.
If we do not increment the `ref_count`, the result would be -1 here,
and consequently, the profiling data would not be flushed and written.
( #ifdef GGML_OPENCL_PROFILING )
* cli : persist reasoning_content in chat history
llama-cli collected reasoning from the stream for display but only
stored assistant content in messages, so --reasoning-preserve could
not re-inject prior thoughts on later turns.
* vulkan : add pool1d push constants and pipeline field
Declared data structures needed for POOL1D OP, which are the vk_op_pool1d_push_constants struct and pipeline_pool1d_f32 field.
* vulkan : add pool1d compute shader
Added pool1d.comp for Vulkan backend mirroring the existing pool2d shader.
* vulkan : add full GGML_OP_POOL_1D support
Added pipeline creation and op dispatch for 1D pooling in the Vulkan backend.
* vulkan : fix pool1d shader logic
Registered pool1d_f32 in vulkan-shaders-gen.cpp and fixed tensor dimension indices and avg pool scale.
* vulkan : fix pool1d end boundary crash and expand test coverage
Fixed an issue where the shader crashed when the end boundary was negative when k0 < p0. Also, added more test cases related to this fix.
* Removed crash guard for Intel
Crash fixed from driver 32.0.101.8860
* Added driver version check for windows
* Change to convert from driverVersion rather than string
* No need to use signed
* Refactor
* allow GPU other than Xe2+
* adjusted function body position
* llama : load MTP tensors only if they are really used
* llama : skip loading MTP (if not used) in remaining models that support MTP
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* SYCL: add oneMKL GEMM flash attention for XMX-accelerated prompt processing
* fattn-mkl: fix interleaved dst layout in normalize kernel
- Fix mkl_fa_normalize_head: use interleaved dst layout
((query * n_q_heads + head) * DV) matching TILE's
flash_attn_combine_results. Previously used dense head-major
layout which wrote head outputs to wrong addresses, corrupting
attention for all models except Qwen3.6-27B (where GQA=6 heads
were sparse enough to avoid visible overlap).
- Remove 7 redundant stream->wait() calls — SYCL in-order queue
already serializes pure SYCL kernel dependencies. Retain only
the 4 MKL GEMM ↔ SYCL handshake barriers (oneMKL GEMM uses its
own internal queue that does not respect SYCL in-order).
- Remove unused dst_row_stride, diagnostic clutter, and dead
K/V hex dump (fa_diag block in fattn-mkl.cpp).
- Add MKL_FA_DISABLE=1 env var for A/B testing.
- Add FA-DISP watchdog (MKL_FA_DEBUG=1) and FA-DIAG output
fingerprint (MKL_FA_DIAG=1) in fattn.cpp.
Tested: Gemma-4-26B, Gemma-4-31B, Qwen3.6-27B, Qwen3.6-35B-A3B
Perf (B70/Battlemage, 32K, q8_0 KV):
Gemma-4-26B: 1473 t/s MKL vs 746 TILE (1.97x)
Qwen3.6-27B: 609 t/s MKL vs 330 TILE (1.85x)
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* Thank you for the review feedback: rename env vars, use GGML_LOG_INFO, document in SYCL.md
Completed the following:
- Rename MKL_FA_DISABLE → GGML_SYCL_ENABLE_MKL_FA (inverted: 0 to disable)
- Rename MKL_FA_DEBUG → GGML_SYCL_MKL_FA_DEBUG
- Rename MKL_FA_DIAG → GGML_SYCL_MKL_FA_DIAG
- Replace fprintf(stderr, ...) / fflush(stderr) with GGML_LOG_INFO() macro
- Document all three env vars in docs/backend/SYCL.md under Runtime
- Add comment explaining MKL FA activation trigger (flash-attn + quantized
KV cache + batch-size >= 1024 + n_kv >= 1024)
Resolves review feedback from arthw.
Again, thank you!!!
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* Thank you for the review feedback round 2: use ggml_sycl_get_env, remove dup waits, gate perf macros
- Replace raw getenv() with ggml_sycl_get_env() in all 4 env-var checks
(fattn.cpp: GGML_SYCL_ENABLE_MKL_FA, GGML_SYCL_MKL_FA_DEBUG,
GGML_SYCL_MKL_FA_DIAG; fattn-mkl.cpp: GGML_SYCL_MKL_FA_DEBUG)
- Remove duplicated stream->wait() before ev.wait_and_throw() in GEMM
KQ and GEMM VKQ — ev.wait_and_throw() already waits for completion
- Gate MKL_ACCUM macro behind do_print so timing accumulators are
no-ops in normal operation
- Remove redundant MIT/Intel copyright header from fattn-mkl.cpp
- Remove unused #include <cfloat>
- Expand SYCL.md MKL FA docs with step-by-step activation trigger
and example llama-cli command
Again, thank you!!!
Co-Authored-By: Claude Code on DeepSeek-v4-Pro
* fattn-mkl: enable MKL FA for all KV cache types
Remove the quantized-only restriction on MKL activation — the MKL
kernel converts any non-F16 K/V to F16 via to_fp16_sycl before GEMM,
so F16 (default), BF16, and F32 caches all benefit from XMX hardware
acceleration. The type restriction was an unnecessary gate.
Before (F16/BF16 default cache + FA on at 32K prefill): ~356 t/s (TILE path)
After: ~670 t/s (MKL path, matching quantized-cache baseline)
Minimal change: two conditions removed, one comment updated in fattn.cpp.
No kernel or conversion code changes — the dequant pipeline already
covers all types.
* fattn-mkl: rename mkl_disable -> mkl_enable for clarity
* fattn-mkl: refine MKL FA dispatch gates
Three changes:
1. Remove quantized-only restriction - MKL FA activates for all
KV cache types (F16 default, BF16, F32, quantized). The MKL
kernel converts non-F16 K/V via to_fp16_sycl before GEMM.
2. Rename mkl_disable -> mkl_enable to match env var
(GGML_SYCL_ENABLE_MKL_FA).
3. Replace batch-size threshold with Q->ne[1] >= 32 gate.
Keeps TG (Q=1) and MTP drafts (Q=3-8) on VEC path where
fused kernel beats MKL launch overhead. Routes all
multi-token prefill through XMX-accelerated GEMM.
Production data confirms Q patterns: 1-8 TG, 32-127 cache reuse,
128+ full reprocess. At 32K F16/BF16 FA-on: 356 -> 670 t/s.
* ggml-sycl: fix F16 cache + MKL FA multi-turn corruption; add gate guards
Two changes:
1. Always copy F16 K/V to dense row-major buffers before MKL GEMM.
Previously F16 was read in-place with raw tensor strides. During
multi-turn conversations, the accumulated KV cache had different
stride properties than a fresh prefill, producing corrupted outputs.
Now dense F16 gets a fast memcpy; interleaved (Gemma) gets a strided
copy kernel. This matches what the quantized paths already did through
to_fp16_sycl.
2. Gate MKL FA on unsupported op params (max_bias, logit_softcap, batch
dim mismatch) and pathological F16 strides (nb[1] not a multiple of
ne[0]*2). These conditions would previously crash inside the MKL
kernel. Pathological strides (test-only) and ALiBi/softcap fall
through to TILE/VEC which handle them correctly.
The stride check uses modulo rather than equality, so both dense
(nb1 == ne0*2) and interleaved (nb1 == H * ne0*2) pass — all real
models use these layouts. Only test cases with overlapping rows
(nb1=32 or nb1=75 for ne0=40) are blocked.
Thanks to hmscider for the oneDNN FA PR (#25222) which surfaced the
same insight: always normalize inputs to contiguous F16 before GEMM.
Co-Authored-By: Claude Code using DeepSeek-V4-Pro <noreply@anthropic.com>
* fattn-mkl: fix quant+GQA KV strides, tighten MKL gate, add K>=1024 tests
Adding K>=1024 flash-attn test cases surfaced several MKL bugs:
- Quant K/V with a padded seq-view (real KV cache) used the wrong
strides in the dequant path... only the true Gemma interleave
layout should reconstruct strides. nb[2] vs ne[1]*nb[1]
- Gate was firing on shapes the kernel doesn't handle: head_dim < 64
or not a multiple of 64, MHA, attention sinks, and
bf16 decode... fell through to vec which no bf16 case.
Gate MKL to the validated envelope: gqa>=2, head_dim 64 through 512
(has to be a multiple of 64) with matching K/V head size, mask,
no sinks/alibi/softcap... everything else falls back to tile.
Covers Qwen Dense/MoE and Gemma4 Dense/MoE
Ran test-backend-ops -o FLASH_ATTN_EXT: 3641/3641 pass.
Perplexity unchanged... 6.7267 MKL vs 6.7290 stock using
Qwen 27b q5_k_xl
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* fattn-mkl: bound attention scratch so it doesn't grow with batch or context... also dropped the bf16 comment in fattn.cpp per arthw review.
* Update ggml/src/ggml-sycl/fattn-mkl.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* Update ggml/src/ggml-sycl/fattn-mkl.cpp
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* apply arthw suggestions: enum for dequant modes, macro for wg_size, env-var one-liners
---------
Co-authored-by: Claude Code using DeepSeek-V4-Pro <noreply@anthropic.com>
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
* llama : enforce the same K and V cache types for DeepSeek V4; enable FA if V cache is quantized
* llama : enforce the same K and V cache types for MLA models
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
