* common : extract trie/ac to a separate file
* common : support multiple token sequences in the reasoning budget sampler
* common/trie : return matched word index
* common/trie : rename "word" to "pattern"
* common/reasoning-budget : expose matched end sequence
* common/sampling : replay end sequence when reasoning budget is done
* cont : update to use multiple end sequences
* cont : clean up
* performance harness - the empirical root
Assisted-by: Claude Opus 4.8
* 210.36ms -> 2.67ms per streamed token
Assisted-by: Claude Opus 4.8
* 11.58ms -> 0.62ms per streamed token
Assisted-by: Claude Opus 4.8
* 22.02ms -> 3.33ms per streamed token
Assisted-by: Claude Opus 4.8
* 3.07ms -> 1.36ms per streamed token at 40 messages
Assisted-by: Claude Opus 4.8
---------
Co-authored-by: Zach Winter <dmtommy@icloud.com>
* ui: remove viewport fade in and smooth autoscroll bottom snap
fadeInView mounted every message and markdown block at opacity 0 and
relied on an IntersectionObserver to reveal it. When the observer never
fires (blocks mounted offscreen during long agentic loops) the content
stays invisible forever while still present in the DOM. Remove the
action, its orphaned isElementInViewport util and all call sites:
blocks now render visible immediately.
AutoScrollController.scrollToBottom defaulted to behavior smooth and is
invoked every 100 ms while streaming. Each tick restarts an easing
animation toward a moving scrollHeight, producing a random elastic bump
of a few pixels when the user reaches the bottom and autoscroll
reengages. Default to instant scrolling; the user facing scroll down
button keeps its smooth behavior.
* ui: skip rendering of offscreen chat messages via content-visibility
Apply content-visibility auto with contain-intrinsic-size to chat
messages so the browser skips layout and paint for messages outside
the viewport. The DOM stays complete: component state, find-in-page,
text selection, and the mutation based autoscroll are unaffected, and
browsers without support simply ignore the properties.
* ui: remove conversation switch fade
Switching conversations faded the message list out and in over 500 ms
plus a 300 ms route delay, deferring the message refresh behind two
requestAnimationFrame calls. Remove the fade, its navigation hooks and
dead state, and refresh messages directly so switching is only bound
by actual render time.
* ui: describe present behavior in comments and drop unused parameter
* ui: anchor the context gauge popup to the form with plain CSS
The stats card was portaled to body and repositioned in script on
every ancestor scroll event, trailing the page by one frame while
streaming. Render it as an absolutely positioned sibling of the input
box inside the already relative form, so nothing runs during scroll.
The card sits just above the dial, centered on it and overlapping the
textarea, from a single measurement of the dial center and top taken
when it opens; the dial and the card share the same positioning frame,
so the values stay exact while the card is open. Mouse pointers open
on hover with a grace delay to reach the card, touch pointers toggle
on tap, any press outside the card and the dial closes it, and Enter
and Space toggle from the keyboard. The card lives outside the input
box because its overflow-hidden and backdrop-filter would clip any
positioned descendant.
* ui: extract context gauge popup constants and relocate its state store
Move the placement values and the close grace delay to
lib/constants/context-gauge-popup.ts, matching the auto-scroll
constants layout, and move the popup state module from the component
folder to lib/stores where runes modules live in this codebase.
* ui: declare the context gauge popup card ref as $state
* Start building graph - reuse deepseek32
* Enable kv cache and rotation for glm_dsa architecture
Just follow Deepseek 3.2 for now.
* Reuse prev_top_k for "shared" indexer layers
* GLM 5.2 uses LLAMA_ROPE_TYPE_NORM for the indexer.
This is transformers' `apply_rotary_pos_emb_interleave`
* Default indexer types to GLM pattern
Previous converted GGUFs like https://huggingface.co/unsloth/GLM-5.2-GGUF write indexer weights to _all_ layers, even if they are only required for "full" types. This PR relies on a new key "%s.attention.indexer.types"; if absent, it will use the default GLM 5.2 schedule as defined in https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json#L26.
Note that conversion is not saving this key yet.
* Save indexer types to gguf, restore on load
* Use ggml_lightning_indexer when cparams.fused_lid
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* GLM 5 and 5.1 use full indexers
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* Fix indentation
* Ensure array is zero-filled
* Prefer explicit std::fill
* Assert prev_top_k exists for shared indexer
---------
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
* ui: fix MCP server display name conflicts in tools lists
Tool groups were keyed by display label so two servers reporting
the same name broke the keyed each blocks and only one was visible.
Key rendering, expand state and toggles by the stable server id
instead, and suffix duplicate labels with a counter in config order.
* ui: customizable MCP server display name with autofill
Add a display name field to the MCP server form, add and edit alike.
The custom name takes precedence over the server-reported one, so two
servers reporting the same name can be told apart; clearing the field
returns to the automatic label. In the add dialog a debounced preview
handshake prefills the field with the server-reported name: a manual
edit freezes the autofill, stale responses are discarded, failures
stay silent, and an unedited prefill is not persisted so the label
keeps following the server.
* ui: fix recursive fetch passthrough in the client test setup
The original fetch was captured inside beforeEach, where it is the
previous test's spy since vi.spyOn returns the existing one, so the
default passthrough recursed on itself for any URL outside the
mocked set. Capture the real fetch once at module load.
* support "reasoning_effort": "none" in OAI API
* handle reasoning.effort: "none" in OAI responses API
* clarify non-"none" values of reasoning_effort have no effect
* use json_value instead of body.at
* hex-l2: use dirty ranges for flushing
* hex-l2: simplify range based flush logic
* hex-l2: optimize dirty range scans
* hex-hvx: support for reduce_max_i32
* hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits
* hex-mmid: optimize mmid row-mapping generation
* hex-mmid: optimize mmid row-mapping generation
* hex-mmid: optimize mmid row-mapping generation (round2)
* hmx-mm: optimize output proc by tiling (col-chunking)
* hex-fa: start the next q dmas a bit earlier
* hex-fa: prefetch Q even earlier
* hvx-fa: optimize softmax to keep things in hvx registers
* hex-fa: hoist const register init in softmax loop
* hmx-fa: kick off next-qkv DMAs before o-proc
* hmx-fa: hoist various checks out of the inner loop
* hmx-fa: adjust the cost model to better balance softmax work across hvx threads
* hmx-fa: overlap diag rescale build with last HMX task
* hmx-fa: optimize idx update in output proc
* hmx-fa: unroll the softmax loops for improved perf
* hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles
* hex-trace: double the default number of trace entries
* hex-trace: add trace events for opbatch and buffer mgmt
* hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats
* hex-trace: replace ascii timeline diagram with pipeline bubbles detector
* hex-trace: handle missing start/stop events
* hex-dma: always log stop/start trace events even for dummy dmas
* hex-scripts: fix flake warnings
* ui: add Default reasoning option that defers to the server
The webui always injected enable_thinking, overriding the chat template
default and the --reasoning flag, breaking models that reason
unconditionally (e.g. Gemma 4 E4B) on a fresh client.
Default sends nothing so the server decides, Off and effort levels
force the value as before. All choices are remembered.
Also remove the boolean thinking API from the conversations store and
drop ChatFormReasoningEffortSubmenu.svelte (dead code).
* ui: close the whole menu tree on reasoning level selection
The reasoning levels were raw buttons inside the SubContent, so
selecting one only closed the submenu via manual state while the root
dropdown stayed open. DropdownMenu.Item closes the full tree on select
like the sibling entries and brings native keyboard navigation.
* ui: prevent the add menu tooltip from flashing when the dropdown closes
* Squash history before conflict-resolution during rebase on master
WIP commit
Add 32-byte loads, restore per-block amax
Use nvfp4x4 intrinsic when available
Fuse per-channel amax and quantization kernels
Do pointer arithmetic only once on x
Remove unnecessary ternary in the load
We assert on host side that ne00 is 64-aligned
Add back scale-search, but optimize it with intrinsics
Code cleanup
Make scale in MMQ-epilogue NVFP4-specific/restrictive for now
Remove unneeded include, add comment
Fix trailing whitespace
Guard __builtin_align__(32) struct to NVIDIA
Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001
* compiler massaging to avoid unnecessary LDCs
* kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4
* Always pass in src1_scale.ptr
* Extract ggml_cuda_is_aligned helper
* hex-geglu: optimized all-in-one geglu microkernel
* hex-geglu: enable non-contiguous src and strided DMA
* hex-act: enable non-contiguous srs and strided DMA for rest of ACT ops
* hex-act: generalize GLU per-thread functions via DEFINE_GLU_PER_THREAD macro
* hexagon: move UNARY_SILU and UNARY_GELU to unary-ops
* hex-act: replace the generic ops_context scratchpad usage with a local htp_vtcm_layout computation per act op.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* feat(ui): add symbolic math support to JS sandbox via nerdamer
Preload nerdamer (with decimal.js) in the sandboxed worker,
exposing the `nerdamer` global for symbolic computation:
simplify, expand, factor, diff, integrate, solve, laplace,
ilt, limit, partfrac, gcd/lcm, roots, coefficients, and more.
Mirrors the math.js integration pattern from the
feature/sandbox-symbolic-math branch, but uses nerdamer
for a lighter, more focused symbolic math engine.
* Update sandbox-harness.ts
* docs(ui): update sandbox tool description with detailed nerdamer usage guide
* Clarify nerdamer usage in sandbox tool description
Updated the description of the sandbox tool to clarify usage of nerdamer.
* ui: build nerdamer sandbox prelude from vendored source
Replace the vendored all.min.js with the readable nerdamer-prime
source and its two bundled deps (big-integer, decimal.js), licenses
included. A vite plugin bundles and minifies them at build time with
the upstream esbuild flags, exposed as virtual:nerdamer and imported
lazily on first sandbox use. The vendors package.json pins commonjs
so the project level type: module does not break esbuild format
detection. The harness gains a CSP removing network egress from the
worker, and browser tests cover the prelude, exact arithmetic, the
fetch block and the timeout.
Upstream snapshot: together-science/nerdamer-prime@1936145
* feat(ui): make symbolic math (nerdamer) a user-toggleable setting
- Add SYMBOLIC_MATH_ENABLED setting key and registry entry (checkbox, default false)
- Convert SANDBOX_TOOL_DEFINITION to buildSandboxToolDefinition(includeSymbolicMath)
so the tool description includes/excludes nerdamer API docs dynamically
- Cache sandbox harness per variant ('nerdamer' / 'plain') for instant toggle
- Deprecate SANDBOX_TOOL_DEFINITION constant alias for backward compatibility
- Update tools store to pass symbolic math config into tool definition
* docs(ui): tell LLM to list nerdamer functions first, do not guess
* test(ui): enable symbolic math in sandbox tests via settingsStore config
* style(ui): fix formatting for tools.svelte.ts
---------
Co-authored-by: Pascal <admin@serveurperso.com>
With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars
and no --spec-type given, the draft resolved to a full model while the
sidecar was the intended draft.
When the speculative types are still at their default, discover the
sidecars of the draft repo, pick the first available following the
existing mtp > dflash > eagle3 priority, and set the corresponding
type, so this now works without any extra flag:
llama-server -hf repo:Q3_K_M -hfd repo:Q8_0
An explicit --spec-type disables the inference, and a draft repo
without sidecars keeps resolving to a full model as before.
* chat: fix DS4 template to explicitly follow reference behavior
* Support DeepSeekv4 flag (`drop_reasoning`).
* fix: hook DS3.2 parser for DS4 as well
* fix: add tool result reordering
* fix: post-merge
* ggml: enable PowerPC backend variants on AIX
Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior.
Also fix a missing semicolon in the PowerPC Q0 matmul implementation.
* Fix missing semicolon in sgemm.cpp
* webgpu : add CONV_2D_DW (depthwise conv2d) kernel
Implement GGML_OP_CONV_2D_DW for the WebGPU backend,
ported from the Vulkan backend's conv2d_dw.comp.
Assisted-by: Claude Opus-4.8
* Remove unnecessary comments in webgpu support
* update supported ops tables, triggered by adding webgpu CONV_2D_DW
* cuda: add k-quant support to GET_ROWS
Device-side embedding lookups require GET_ROWS to handle the k-quants
used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without
it the backend rejects the op and the scheduler falls back to the host,
copying the full embedding matrix back on every token in single-device
graphs.
Factor the super-block dequantizers out of the dequantize_block kernels
in convert.cu into shared device functions in dequantize.cuh and reuse
them from a new k_get_rows_kq kernel : one thread block dequantizes one
(dst row, super-block) pair with the existing thread layouts, 32 threads
for q4_K and 64 for the other k-quants.
Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left
as a TODO.
* cuda: add i-quant support to GET_ROWS
Extends the shared super-block dequantizers to the nine i-quants and
reuses them from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernels. supports_op gates the k-quant and i-quant path on
ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its
own (QK4_NL sub-blocks). mxfp4 is left as a TODO.
* cuda: add mxfp4 support to GET_ROWS
Moves the mxfp4 dequantizer into the shared super-block helpers and
reuses it from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since
its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own.
This closes GET_ROWS type coverage on CUDA: every quantized GGML type
now takes the direct device path.
* cuda: gate the GET_ROWS row size only for 32-value sub-block types
Address review from @pwilkin: the i-quant commit replaced the return
shared by the whole supported type cascade, so f16/f32/bf16/i32 and the
legacy quants also inherited the ne0 % QK_K == 0 gate and any row size
that is not a multiple of 256 fell back to the scheduler. Split the
cascade: unconditional support is restored everywhere, the gate stays
only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the
QK_K super-blocks the kernel iterates on.
The Qwen3-VL learned position embedding is interpolated to the runtime patch
grid with the default bilinear+antialias (align_corners=False) sampling, while
the transformers reference uses align_corners=True (torch.linspace(0, side-1, T)).
The mismatch scales grounding coordinates about the image center, growing with
image size and per-axis for non-square images (see #16880).
With -hfd pointing to a repo shipping speculative sidecars, the draft
resolved to the main model of that repo, since find_best_model()
excludes sidecar files, and the explicit draft plan suppressed the
sidecar discovery on the -hf repo.
The draft plan already discovers its sidecars, they were just never
consumed. Wire them as the draft, following the fallback pattern of
the main plan, so this now works as expected:
llama-server -hf repo -hfd repo --spec-type draft-dflash
* server: return 400 instead of 500 on validation error with X-Conversation-Id
set_req() attaches the spipe as soon as the header is present, before the request
body is parsed. When params validation throws, set_next() never runs and next_orig
stays empty, so on_complete() called it and crashed with std::bad_function_call,
turning the prepared 400 JSON into a generic 500.
on_complete() now treats an empty next_orig as "streaming never started" and evicts
the session installed by set_req(), so a failed request leaves nothing behind for
discovery or replay. This also covers valid requests that carry the header but do
not stream, which previously left an empty finalized session in the map until the
GC TTL.
* ui: do not send the backend_sampling placeholder
On a fresh profile the syncable settings hold the empty string placeholder meaning
"let the server decide". Every neighbor field goes through the hasValue() guard
that filters it, except backend_sampling, which sent the placeholder verbatim and
made every default settings completion fail validation.
Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an
explicit false still reaches the server and the intent of #18781 (send both true
and false) is preserved. Only the placeholder is filtered.
* Refactor vk_queue to use per-instance mutexes and unique handles
* integrates VK_KHR_internally_synchronized_queues, abstracting the queue submission into a polymorphic interface that completely bypasses host-side mutex locking when driver-side synchronization is supported
* fix compilation error
* fix duplicate pNext chain for VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR
* add fallback defines for VK_KHR_internally_synchronized_queues
* add null checks for queues in vk_device_struct destructor
* use unique_ptr for outer queues to enforce exclusive ownership and optimize lifetime
* use static constexpr for eInternallySynchronizedKHR
* add lock guard to ggml_vk_create_aliased_queue for thread safety
* initialize sync_query_features.internallySynchronizedQueues to VK_FALSE
* reuse sync_query_features for internallySynchronizedQueues and simplify chaining
* refactor internallySynchronizedQueues detection
* fix internallySynchronizedQueues query guard
* use eInternallySynchronizedKHR constant
* fix self-referential alias for eInternallySynchronizedKHR
* use macro for eInternallySynchronizedKHR fallback
* fix internallySynchronizedQueues query timing in ggml-vulkan.cpp to prevent device creation mismatch
* reset sync_query_features.pNext before reusing in device creation chain, also removed the redundant second probe call
* refactor internally synchronized queues detection to use chained feature query and avoid redundant API calls
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* rename sync_enable_features to internally_synchronized_queues_features
* queue_flags is still computed before has_internally_synchronized_queues is set
* fix trailing whitespace
* replace eInternallySynchronizedKHR macro with static constexpr
* preserve source queue semantics in single-queue aliased transfer queue
* vulkan: fix cmd_pool access via pointer for compute_queue unique_ptr
* vulkan: lock queue during debug label emission when not internally synchronized
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
k_get_rows_float did a scalar one-element-per-thread copy and recomputed the
row-invariant work (index load, fast_div_modulo, src/dst row pointers) for
every element. Hoist that out of the per-element loop, and add a vectorized
path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the
contiguous same-type (no-cast) case.
The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at
runtime on 16-byte alignment of the base pointers and all row strides and on
ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small
single-row gather can drop below the device CU count and regress; an
occupancy gate keeps those on the block-rich scalar path.
On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288)
drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan
backend, with no regression on the small conv-state gather; total get_rows
-27%. test-backend-ops GET_ROWS passes (47/47).
Assisted-by: Claude Opus 4.8
