Compare commits

..
57 Commits
Author SHA1 Message Date
Aleksander Grygier 7f106b5459 ui: Llama UI design system: tokens, surfaces and fonts (#32468)
* ui: bring @ggml-org/llama-ui into ui/lib/ui

The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI design system: tokens, surfaces and brand

The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts

Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.

ui/lib/ui
- the 17 base components the app had and the library did not (alert,
  alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
  drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
  skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
  app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
  the consumer imports before the theme, as it does Tailwind
- 0.3.0

ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
  @ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
  scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
  imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
  are removed; components.json moves to the library

Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
  rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
  as its source

The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: AGENTS.md places Llama UI in ui/lib/ui

Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code

The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open submenus outside the menu they come from

A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw submenus like the menu they open from

Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.

The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a late current-node write off a closed or switched conversation

updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.

Assisted-by: Claude (Fable 5.2)

* ui: have Llama Chat use the library's Model UI and app components

Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.

ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
  sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
  and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
  (actions, content, forms, misc) and @ggml-org/llama-ui/models

ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
  ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
  come from the library; Chat's copies and the two constants only they
  used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
  id and applies the user's display settings, ModelOrgAvatar hides the
  avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
  svelte/reactivity has no file of its own, so a library source importing
  it got its types from ui/lib/ui's own install when there was one

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI takes the brand's colours and logo from llama-brand

@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.

- primitives.css drops the 25 tokens it held copies of (the neutral
  greys, sand, the skies and cinder) and builds on llama-brand's
  css/colors.css instead; the skies' gradients point at llama-brand's.
  Apps import that file before theme.css, as they do Tailwind and
  tw-animate-css: the library's stylesheets import no package, so an app
  that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
  node_modules, like the library's other dependencies

The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat take the library's constants, enums and helpers

Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.

- the library gets entry points for them: @ggml-org/llama-ui/constants,
  @ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
  @ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
  the byte units, the memory-fit constants and minMemoryTierGb, the bit
  depth labels, the capability and modality icons, labels and order,
  ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
  formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
  ModelDownloadProgress are supersets of the library's (the latter extends
  it now), and formatFileSize counts in 1024s

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the chat form's add key and model selector a tighter shadow

In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.

- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
  the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
  selector trigger
- DESIGN.md names it next to the other elevations

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 14:39:03 +01:00
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Gaurav Garg 054a1babf3 Rotate K/V read order for better L2 cache hit rates (#32407) 2026-11-17 16:49:28 +05:30
Emanuil Rusev 513ad7ae07 ui: turn conversation tabs off by default (#32402)
Browsers already have tabs, so a second tab bar inside the page adds a layer most people don't need. The setting stays, so anyone who wants tabs can turn them on. Users with a saved config keep their current value, since the full config is stored in localStorage.
2026-11-17 11:44:13 +01:00
Aleksander Grygier 67cf8967eb ui: move Llama Chat to SvelteKit 3 and its built-in service worker (#32463)
* ui: replace deprecated packages and fix npm audit findings

Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker

Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.

@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: use SvelteKit 3's update detection and Svelte 5's current patterns

A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.

Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
  checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
  attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
  setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move to ESLint 10

ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.

ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: override minimatch to clear npm audit's last finding

ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 11:43:36 +01:00
cherry77-cloud 4df3f2959c qwen3next : apply output projection weight scale (#32393)
Assisted-by: Codex
2026-11-17 11:32:30 +01:00
Daniel Bevenius a49db739fb cuda : raise shared mem limit only for the launched kernel (#32392)
This commit removes a call of the CUDA_SET_SHARED_MEMORY_LIMIT macro as
the fallback decision was moved out of launch_mul_mat_q.
launch_mul_mat_q now has the fallback template parameter which was added
in Commit 6eddde06a4 ("CUDA: refactor MMQ
kernel configuration (#24127)").

This code was then refactored in Commit
c49edf0963 ("CUDA: fuse FFN gate/up
matmuls and GLU in MMQ with interleaved gate/up rows (#29948)") which
added the has_fusion template parameter but it kept both macro calls.

launch_mul_mat_q only launches one type of kernel now (this is now
decided earlier in the templated call chain) and each specialization
will set the limit itself.

The motivation for this is to simplify the code and improve readabilty.
2026-11-17 10:53:08 +01:00
Titaniumtown eada89c315 nix: expose a SYCL package (#31066) 2026-11-17 11:04:12 +02:00
Ruben Ortlam 8f647f1fdb vulkan: disable flip Flash Attention shader on AMD Windows due to a driver bug (#32415) 2026-11-17 10:55:28 +02:00
Neo Zhang dde9de82cb sycl : Fix FA mkl issue: a mask broadcast over heads or batches (#31536)
* fix the SYCL fa with mkl error of a mask broadcast over heads or batches; add argument cache-ram to start-svr.sh

* add argument cache-ram to win-start-svr.bat

* fix format issue, remove --cache-ram in script: move to another PR
2026-11-17 10:50:32 +02:00
Souleyman e0a9118fe9 metal : add implicit GEMM kernel for CONV_2D (#28069) 2026-11-17 09:46:24 +01:00
Nikhil Jain 140a279874 webgpu : Implement timestep embedding (#31695)
* Only run webgpu CI on my fork

* Add webgpu only workflow

* resolve merge conflicts

* save state

* save kernel

* add kernel back in

* fix merge issues with kernels

* updates

* update tests

* change invocation count and workgroup to be u32

* clang format

* clang format

* update docs

* format

* remove extra file
2026-11-17 10:41:37 +02:00
Takamichi Ohara c145db2728 webgpu : use f32 while dequantizing to avoid FTZ (#32140) 2026-11-17 10:40:23 +02:00
lhez ede6106780 opencl: support f16 and non-contig f32 silu (#32319)
* opencl: support non-contig silu

Assisted-by: Claude Opus 5.5

* opencl: support f16 silu

Assisted-by: Claude Opus 5.5
2026-11-17 10:38:07 +02:00
9214bdb554 sycl: use fast division in bin-bcast kernels (#32213)
Replace the per-element modulo of i0 with fast_div_modulo in the ravel and
unravel bin-bcast kernels, drop the now-unused ne10/ne20 kernel arguments,
and reject divisors that do not fit in 32 bits when the magic values are
computed, as the CUDA implementation does.

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-11-17 15:17:47 +08:00
shaofeiqi d3c3e242c1 opencl: add A7 Q8_0 non-MoE binary kernel (#31003) 2026-11-16 20:15:56 -08:00
Sidhartha Reddy Potu 9581154a22 mtmd : fix Muse Glimmer vision embedding order (#32433)
Assisted-by: Claude
2026-11-17 00:29:02 +01:00
Gabe GoodhartandSigbjørn Skjæret 1ef8c5d684 convert : respect max_position_embeddings for granite hybrid (#32436)
* fix(conversion): Respect max_position_embeddings for granite hybrid

Branch: GraniteHybridContextLengthConversion
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: shorter comment w/ HF reference

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-16 23:49:57 +01:00
Pascal e8fbcad855 tests: print the exception message in test-llama-context (#32428)
A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
2026-11-16 21:53:32 +01:00
Xuan-Son Nguyen 95e837aaf1 skills: explicitly specify paths to README-dev.md (#32431) 2026-11-16 21:52:12 +01:00
Bartowski d998c676ff convert: Fix inkling 2 mmproj conversion (#32430) 2026-11-16 20:39:32 +01:00
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00
shaofeiqi ad5cbfa911 opencl: add A7 Q4_K non-MoE binary kernel (#30948) 2026-11-16 10:58:29 -08:00
Hongqiang Wang d261e49e7e opencl: fix three defects of the bin flash-attention prefill path (#32110)
* opencl: release the WMM flash-attention mask fallback image

The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.

* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel

With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.

* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits

The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
2026-11-16 10:57:48 -08:00
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Xuan-Son Nguyen 4303cd94ae contributing: mark agents.md and code-review as required (#32416) 2026-11-16 19:26:10 +01:00
Beinsezii c7518dbec5 MMVQ: Tune ne11 Gate for RDNA3_5 (#32308) 2026-11-17 00:56:20 +08:00
Xuan-Son Nguyen a68c6136bb mtmd: support fish audio s2 (#32330)
* add fisk-audio-s2 support

* add voice cloning

* cont merge master

* nits

* nits

* clean up

* nits

* use new batch api
2026-11-16 17:32:27 +01:00
Aleksander Grygier 3a7660371d ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat

The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's package @ggml-org/llama-chat

It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the built-in web UI Llama Chat in docs, help text and comments

The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.

Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name tools/ui as a former path of Llama Chat in the code-review skill

Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's path in the security policy's exclusions

Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-16 17:23:36 +01:00
Shubham RoyandDeveloper e8698ac163 fix:- Misc. bug: Settings/Tools is lying about tool picks inside a ch… (#32223)
* fix:- Misc. bug: Settings/Tools is lying about tool picks inside a chat #32214

* checks run lint error

---------

Co-authored-by: Developer <dev@example.com>
2026-11-16 17:15:16 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel a06044b934 llama : deprecate old batch API (#32306) 2026-11-16 15:16:13 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
Jeff Bolz ef3848f9d7 vulkan: Support split_k for conv3d. (#29929)
This tries to use the 256x128 shape (coopmat2-only) with split_k when
it appears profitable. It reuses the mul_mat split_k reduce shader.
2026-11-16 15:25:12 +02:00
uvos 47fe133318 HIP/CUDA: enable GDN mma kernel on CDNA (#31327) 2026-11-16 15:24:19 +02:00
Ruben Ortlam 301db63370 vulkan: fix Nvidia fp32 FA shmem overflow, disable shmem staging if it takes too much memory (#32244) 2026-11-16 15:22:57 +02:00
BlivionIaG a001af915e hip: add gfx1013 to rdna1 (#32236) 2026-11-16 15:22:35 +02:00
Pascal 164ab018f9 webgpu: support per-head flash attention masks (#32394)
The flash attention shaders indexed the mask without its head
dimension, so masks with ne[2] > 1 gave wrong results. The mask offset
and the vec block states now include the head, broadcast as
head % mask->ne[2] like the other backends. The params buffer grows to
256 bytes to fit the two new mask fields.

The shared memory estimate dropped kv_shmem when only one of K and V
is read directly, while the shaders keep it unless both are, which made
the K f16 / V q8_0 cases exceed the workgroup storage limit. The vec kv
tile now takes both flags: the memory estimate drops kv_shmem only when
both are direct, and the tile still divides the KV padding when either
is. get_alloc_size and supports_op use the same rule as the pipelines,
and supports_op accounts for the register tile fallback.
2026-11-16 14:20:18 +01:00
Georgi Gerganov 5cb9516798 readme : fix line endings 2026-11-16 14:03:30 +02:00
8bfd64b5ac readme : update (#32382)
* readme : update

* authors : update

* readme : minor

* cont : minor

* cont : typos

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Niklas Wenzel <dev@nikwen.de>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Niklas Wenzel <dev@nikwen.de>
2026-11-16 13:28:31 +02:00
Xuan-Son Nguyen e1925b3695 jinja: support json sort_keys=true (#32366) 2026-11-16 12:28:19 +01:00
Georgi Gerganov 95db855bce ci : use vX.Y.Z tags as docker tags (#32388)
* ci : use vX.Y.Z tags as docker tags

When the Docker workflow is manually dispatched from a vX.Y.Z tag, use that tag verbatim for the image tags instead of appending the build number and commit hash.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)

* ci : require a tag for manual docker runs

Manual dispatches of the Docker workflow must now use a tag ref, so the image tags are derived from the selected tag. The check can be skipped with the new skip_ref_check input. Scheduled nightly builds are unchanged.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-16 13:27:10 +02:00
miyan 1d2e880bad vulkan : fix 32-bit compilation (#32275) 2026-11-16 11:50:31 +01:00
PascalandRuben Ortlam adf554d239 vulkan: make the radix top-k tie selection deterministic (#31931)
* vulkan: make the radix top-k tie selection deterministic

The radix select gathers the elements above the threshold and the ties
with an atomic counter, so with more ties than remaining slots the kept
columns depend on the thread scheduling.

Each subgroup now owns a contiguous chunk of the row with the same
column mapping in every pass, counts its kept elements, and writes them
at ranks from a scan of the subgroup totals; overflowing ties are ranked
in column order with a ballot, so the lowest tied columns win, as on
CUDA and Metal. The radix pipelines require subgroup arithmetic, ballot
and full subgroups. Enable the deterministic top-k check on Vulkan.

* optimize shader for AMD, Intel and Nvidia

---------

Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
2026-11-16 11:41:44 +01:00
Ruixiang Wang 6b01a4767f CI: trigger spec dec testing for relevant PRs (#32281)
* ci: trigger spec dec testing after refactoring

* add options to run spec dec ci nightly manually
2026-11-16 11:14:07 +01:00
liminfei-amd f477c1b90a cuda : fix MUL_MAT_ID abort with odd src0 columns on AMD (#32385)
On AMD devices MUL_MAT_ID with an F32, F16 or BF16 src0 and at most 8 tokens always used MMVF.
MMVF reads src0 in pairs of values and asserts an even ne[0], so a rank 1 LoRA on MoE expert weights (lora_b has ne[0] == 1) aborted.
Use MMVF there only for an even ne[0]; other shapes take the existing fallback.

Fixes #30576

Assisted-by: GitHub Copilot

Signed-off-by: AMD ROCm Contributor <91481003+liminfei-amd@users.noreply.github.com>
2026-11-16 17:53:06 +08:00
Titaniumtown 21fa9c043f kv-cells: avoid an O(n_cells) scan per sequence removal (#32380) 2026-11-16 10:52:09 +01:00
Pascal a946eb541a server: answer reasoning_effort levels the template rejects with 400 (#32332)
The reasoning effort probe already renders every candidate level, but
it dropped the ones the template raises on, so a request with such a
level (e.g. max on Qwen3.8) reached the template and came back as a
500 on /v1/chat/completions, /v1/responses and /v1/messages.

Keep the raising levels in reasoning_effort_rejected and reject them
in oaicompat_chat_params_parse with an invalid_argument, which the
server answers with a 400 listing the supported levels. Aliases and
levels a template ignores are not rejected, so high on Qwen3.8
still renders as xhigh.

Also trim the output_config.effort test to a single level.
2026-11-16 06:55:56 +01:00
thelittlefireman 05db09a4d3 ggml-cuda: fix MoE regression on HIP with #29948 (#32344) 2026-11-16 06:37:42 +01:00
BeinseziiandSigbjørn Skjæret 9fba51d2c9 convert: fix ripple dflash2 NoPE metadata (#32363)
* convert: fix ripple dflash2 NoPE metadata

was wrong

https://huggingface.co/ricursiveAI/Ripple-65B-Preview-DFlash2/blob/743c4669f7daa11d3ca350b9a057c059481b1afd/config.json#L45-L47

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-16 00:47:27 +01:00
Xuan-Son NguyenandAldehir Rojas b4c0e00a4f jinja: add 4 missing features (#32352)
* jinja: add 4 missing features

* apply suggestion

Co-authored-by: Aldehir Rojas <hello@alde.dev>

* fix error msg

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>
2026-11-16 00:40:44 +01:00
Evan Huus 1719dbb0ee server : skip MCP discovery for router children (#32359) 2026-11-15 23:55:33 +01:00
Xuan-Son Nguyen fc504f91d8 mtmd: support higgs v3 STT and vibevoice-ASR (#32329)
* mtmd: support higgs v3 and vibevoice-ASR

* use proper mapping

* address review comment

* nits

* rm hacky get_audio_length, add mtmd_helper_get_media_meta
2026-11-15 21:25:34 +01:00
Seungjun Bae 18b74fa41d model : read qwen3vl output head size from the tensor (#32349)
Allows Qwen3-ForcedAligner, whose output head (5000 timestamp classes) is smaller than the vocab.
2026-11-15 19:45:17 +01:00
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00
1304 changed files with 29471 additions and 12687 deletions
+7 -4
View File
@@ -19,12 +19,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${CANN_BASE_IMAGE} AS build
@@ -40,7 +43,7 @@ WORKDIR /app
# -- Copy project files --
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# -- Set CANN environment variables (required for compilation) --
# Using ENV instead of `source` allows environment variables to persist across the entire image layer
+7 -4
View File
@@ -9,12 +9,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -28,7 +31,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \
+7 -4
View File
@@ -17,12 +17,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_CUDA_DEV_CONTAINER} AS build
@@ -40,7 +43,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${CUDA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=${CUDA_DOCKER_ARCH}"; \
+7 -4
View File
@@ -11,12 +11,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS build
@@ -41,7 +44,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
echo "GGML_SYCL_F16 is set" \
+7 -4
View File
@@ -15,12 +15,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_MUSA_DEV_CONTAINER} AS build
@@ -45,7 +48,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${MUSA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DMUSA_ARCHITECTURES=${MUSA_DOCKER_ARCH}"; \
+138 -41
View File
@@ -22,12 +22,19 @@
spirv-headers,
nodejs,
importNpmLock,
syclToolkit ? null,
level-zero,
ocl-icd,
intel-graphics-compiler,
intel-compute-runtime,
makeBinaryWrapper,
useBlas ?
builtins.all (x: !x) [
useCuda
useMetalKit
useRocm
useVulkan
useSycl
]
&& blas.meta.available,
useCuda ? config.cudaSupport,
@@ -38,11 +45,28 @@
rocmGpuTargets ? builtins.concatStringsSep ";" rocmPackages.clr.gpuTargets,
useVulkan ? false,
useRpc ? false,
useSycl ? false,
# Rebuild, not a runtime switch. cmake default is off.
syclF16 ? false,
# null keeps JIT. AOT makes icpx run ocloc, which DPC++ does not ship.
syclDeviceArch ? null,
syclMaxParallelLinkJobs ? null,
llamaVersion ? "0.0.0", # Arbitrary version, substituted by the flake
# It's necessary to consistently use backendStdenv when building with CUDA support,
# otherwise we get libstdc++ errors downstream.
effectiveStdenv ? if useCuda then cudaPackages.backendStdenv else stdenv,
# CUDA needs its backend stdenv or libstdc++ mismatches downstream.
# SYCL needs icpx which is provided by the oneAPI stdenv
effectiveStdenv ?
if useCuda && useSycl then
throw "llama-cpp: useCuda and useSycl are mutually exclusive"
else if useCuda then
cudaPackages.backendStdenv
else if useSycl then
if syclToolkit == null then
throw "llama-cpp: useSycl requires syclToolkit, consider setting it to intel-oneapi-toolkit"
else
syclToolkit.stdenv
else
stdenv,
enableStatic ? effectiveStdenv.hostPlatform.isStatic,
precompileMetalShaders ? false,
useWebUi ? true,
@@ -65,7 +89,8 @@ let
++ lib.optionals useMetalKit [ "MetalKit" ]
++ lib.optionals useMpi [ "MPI" ]
++ lib.optionals useRocm [ "ROCm" ]
++ lib.optionals useVulkan [ "Vulkan" ];
++ lib.optionals useVulkan [ "Vulkan" ]
++ lib.optionals useSycl [ "SYCL" ];
pnameSuffix =
strings.optionalString (suffices != [ ])
@@ -108,6 +133,26 @@ let
shaderc
spirv-headers
];
syclAot = useSycl && syclDeviceArch != null && syclDeviceArch != "";
# source ocloc binary from intel-compute-runtime
oclocAot = runCommand "ocloc-aot" { nativeBuildInputs = [ makeBinaryWrapper ]; } ''
mkdir -p "$out/bin"
shopt -s nullglob
exact=(${lib.getBin intel-compute-runtime}/bin/ocloc)
vers=(${lib.getBin intel-compute-runtime}/bin/ocloc-*)
if [ ''${#exact[@]} -eq 1 ]; then
src=''${exact[0]}
elif [ ''${#vers[@]} -eq 1 ]; then
src=''${vers[0]}
else
echo "expected one ocloc binary from intel-compute-runtime" >&2
exit 1
fi
makeWrapper "$src" "$out/bin/ocloc" \
--prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath [ intel-graphics-compiler ]}
'';
in
effectiveStdenv.mkDerivation (finalAttrs: {
@@ -133,11 +178,13 @@ effectiveStdenv.mkDerivation (finalAttrs: {
src = lib.cleanSource ../../.;
};
# Builds the webui locally, taking care not to require updating any sha256 hash.
# Builds Llama Chat locally, taking care not to require updating any sha256 hash.
webui = stdenvNoCC.mkDerivation {
pname = "webui";
version = llamaVersion;
src = lib.cleanSource ../../tools/ui;
# the app compiles Llama UI from its source in ../../lib/ui, so the source is ui/ as a whole
src = lib.cleanSource ../../ui;
sourceRoot = "source/apps/chat";
nativeBuildInputs = [
nodejs
@@ -146,7 +193,7 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# no sha256 required when using buildNodeModules
npmDeps = importNpmLock.buildNodeModules {
npmRoot = ../../tools/ui;
npmRoot = ../../ui/apps/chat;
inherit nodejs;
};
@@ -156,8 +203,8 @@ effectiveStdenv.mkDerivation (finalAttrs: {
};
postPatch = lib.optionalString useWebUi ''
cp -r ${finalAttrs.webui} tools/ui/dist
chmod -R u+w tools/ui/dist
cp -r ${finalAttrs.webui} ui/apps/chat/dist
chmod -R u+w ui/apps/chat/dist
'';
# With PR#6015 https://github.com/ggml-org/llama.cpp/pull/6015,
@@ -168,6 +215,13 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# see https://github.com/ggml-org/llama.cpp/pull/6118 for discussion
__noChroot = effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders;
# Do not fortify under SYCL as it breaks compiled memcpy invocations
# _FORTIFY_SOURCE rewrites memcpy -> __memcpy_chk
hardeningDisable = optionals useSycl [
"fortify"
"fortify3"
];
nativeBuildInputs =
[
cmake
@@ -181,7 +235,12 @@ effectiveStdenv.mkDerivation (finalAttrs: {
autoAddDriverRunpath
]
++ optionals (effectiveStdenv.hostPlatform.isGnu && enableStatic) [ glibc.static ]
++ optionals (effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ];
++ optionals (effectiveStdenv.hostPlatform.isDarwin && useMetalKit && precompileMetalShaders) [ xcrunHost ]
++ optionals useSycl [
autoAddDriverRunpath
makeBinaryWrapper
]
++ optionals syclAot [ oclocAot ];
buildInputs =
optionals effectiveStdenv.hostPlatform.isDarwin darwinBuildInputs
@@ -190,39 +249,53 @@ effectiveStdenv.mkDerivation (finalAttrs: {
++ optionals useRocm rocmBuildInputs
++ optionals useBlas [ blas ]
++ optionals useVulkan vulkanBuildInputs
++ optionals useSycl [
level-zero
ocl-icd
intel-graphics-compiler
]
++ [ openssl ];
cmakeFlags =
[
(cmakeBool "LLAMA_BUILD_SERVER" true)
(cmakeBool "LLAMA_BUILD_WEBUI" useWebUi)
(cmakeBool "BUILD_SHARED_LIBS" (!enableStatic))
(cmakeBool "CMAKE_SKIP_BUILD_RPATH" true)
(cmakeBool "GGML_NATIVE" false)
(cmakeBool "GGML_BLAS" useBlas)
(cmakeBool "GGML_CUDA" useCuda)
(cmakeBool "GGML_HIP" useRocm)
(cmakeBool "GGML_METAL" useMetalKit)
(cmakeBool "GGML_VULKAN" useVulkan)
(cmakeBool "GGML_STATIC" enableStatic)
(cmakeBool "GGML_RPC" useRpc)
]
++ optionals useCuda [
(
with cudaPackages.flags;
cmakeFeature "CMAKE_CUDA_ARCHITECTURES" (
builtins.concatStringsSep ";" (map dropDot cudaCapabilities)
)
cmakeFlags = [
(cmakeBool "LLAMA_BUILD_SERVER" true)
(cmakeBool "LLAMA_BUILD_WEBUI" useWebUi)
(cmakeBool "BUILD_SHARED_LIBS" (!enableStatic))
(cmakeBool "CMAKE_SKIP_BUILD_RPATH" true)
(cmakeBool "GGML_NATIVE" false)
(cmakeBool "GGML_BLAS" useBlas)
(cmakeBool "GGML_CUDA" useCuda)
(cmakeBool "GGML_HIP" useRocm)
(cmakeBool "GGML_METAL" useMetalKit)
(cmakeBool "GGML_VULKAN" useVulkan)
(cmakeBool "GGML_STATIC" enableStatic)
(cmakeBool "GGML_RPC" useRpc)
]
++ optionals useCuda [
(
with cudaPackages.flags;
cmakeFeature "CMAKE_CUDA_ARCHITECTURES" (
builtins.concatStringsSep ";" (map dropDot cudaCapabilities)
)
]
++ optionals useRocm [
(cmakeFeature "CMAKE_HIP_COMPILER" "${rocmPackages.llvm.clang}/bin/clang")
(cmakeFeature "CMAKE_HIP_ARCHITECTURES" rocmGpuTargets)
]
++ optionals useMetalKit [
(lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1")
(cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders))
];
)
]
++ optionals useRocm [
(cmakeFeature "CMAKE_HIP_COMPILER" "${rocmPackages.llvm.clang}/bin/clang")
(cmakeFeature "CMAKE_HIP_ARCHITECTURES" rocmGpuTargets)
]
++ optionals useMetalKit [
(lib.cmakeFeature "CMAKE_C_FLAGS" "-D__ARM_FEATURE_DOTPROD=1")
(cmakeBool "GGML_METAL_EMBED_LIBRARY" (!precompileMetalShaders))
]
++ optionals useSycl [
(cmakeBool "GGML_SYCL" true)
(cmakeBool "GGML_SYCL_F16" syclF16)
]
++ optionals syclAot [
(cmakeFeature "GGML_SYCL_DEVICE_ARCH" syclDeviceArch)
]
++ optionals (syclAot && syclMaxParallelLinkJobs != null) [
(cmakeFeature "GGML_SYCL_MAX_PARALLEL_LINK_JOBS" (toString syclMaxParallelLinkJobs))
];
# Environment variables needed for ROCm
env = optionalAttrs useRocm {
@@ -237,11 +310,35 @@ effectiveStdenv.mkDerivation (finalAttrs: {
cp $src/include/llama.h $out/include/
'';
# Fix SYCL binary calling missing important libraries
# Also use Level Zero + sysman INSTEAD of OpenCL
# due to OpenCL reporting free memory as total memory,
# breaking `-fit` functionality
postFixup = lib.optionalString useSycl ''
driverLibs=${
lib.makeLibraryPath [
intel-graphics-compiler
ocl-icd
]
}
for b in "$out"/bin/*; do
[ -f "$b" ] && [ -x "$b" ] || continue
patchelf --print-interpreter "$b" >/dev/null 2>&1 || continue
patchelf --add-needed libOpenCL.so.1 --add-needed libigc.so.2 \
--add-rpath "$driverLibs" "$b"
wrapProgram "$b" \
--set-default ONEAPI_DEVICE_SELECTOR level_zero:gpu \
--set-default ZES_ENABLE_SYSMAN 1
done
'';
meta = {
# Configurations we don't want even the CI to evaluate. Results in the
# "unsupported platform" messages. This is mostly a no-op, because
# cudaPackages would've refused to evaluate anyway.
badPlatforms = optionals useCuda lib.platforms.darwin;
badPlatforms =
optionals useCuda lib.platforms.darwin
++ optionals useSycl (lib.subtractLists [ "x86_64-linux" ] lib.platforms.all);
# Configurations that are known to result in build failures. Can be
# overridden by importing Nixpkgs with `allowBroken = true`.
+7 -4
View File
@@ -28,12 +28,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
## Build Image
@@ -83,7 +86,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# Build Stage
RUN bash -c "source ${OpenVINO_DIR}/setupvars.sh && \
+7 -4
View File
@@ -17,12 +17,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
### Build image
@@ -52,7 +55,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
+7 -4
View File
@@ -9,12 +9,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -31,7 +34,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -B build -DGGML_NATIVE=OFF -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON && \
cmake --build build --config Release -j$(nproc)
+7 -4
View File
@@ -9,12 +9,15 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
# the app compiles Llama UI from its source next to it
COPY ui/lib/ui/package.json /app/ui/lib/ui/package.json
COPY ui/lib/ui/src/lib/ /app/ui/lib/ui/src/lib/
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -28,7 +31,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_ZENDNN=ON && \
cmake --build build -j $(nproc)
+1 -1
View File
@@ -10,7 +10,7 @@
build*/
tools/ui/node_modules/
ui/apps/chat/node_modules/
models/*
+1 -1
View File
@@ -48,7 +48,7 @@ insert_final_newline = unset
[vendor/**.patch]
trim_trailing_whitespace = unset
[tools/ui/**]
[ui/**]
indent_style = unset
indent_size = unset
end_of_line = unset
+3 -1
View File
@@ -14,7 +14,9 @@ runs:
run: |
BUILD_NUMBER="$(git rev-list --count HEAD)"
SHORT_HASH="$(git rev-parse --short=7 HEAD)"
if [[ "${{ env.BRANCH_NAME }}" == "master" || "${{ env.BRANCH_NAME }}" == "b${BUILD_NUMBER}" ]]; then
if [[ "$GITHUB_REF_TYPE" == "tag" && "${{ env.BRANCH_NAME }}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "name=${{ env.BRANCH_NAME }}" >> $GITHUB_OUTPUT
elif [[ "${{ env.BRANCH_NAME }}" == "master" || "${{ env.BRANCH_NAME }}" == "b${BUILD_NUMBER}" ]]; then
echo "name=b${BUILD_NUMBER}" >> $GITHUB_OUTPUT
else
SAFE_NAME=$(echo "${{ env.BRANCH_NAME }}" | tr '/' '-')
+1 -2
View File
@@ -48,7 +48,6 @@ examples:
- all-globs-to-all-files:
- '!tools/server/**'
- '!tools/mtmd/**'
- '!tools/ui/**'
testing:
- changed-files:
- any-glob-to-any-file:
@@ -72,7 +71,7 @@ android:
server/ui:
- changed-files:
- any-glob-to-any-file:
- tools/ui/**
- ui/**
server:
- changed-files:
- any-glob-to-any-file:
+13 -1
View File
@@ -16,6 +16,10 @@ on:
description: "Skip the s390x build target (useful for fast test runs that do not need the IBM Z runner)"
type: boolean
default: false
skip_ref_check:
description: "Skip the requirement to run from a tag (allows branch refs)"
type: boolean
default: false
schedule:
# Rebuild daily rather than on every push because it is expensive
- cron: '12 4 * * *'
@@ -38,6 +42,14 @@ jobs:
source_tag: ${{ steps.srctag.outputs.name }}
steps:
- name: Check manual run ref
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.skip_ref_check }}
run: |
if [[ "$GITHUB_REF_TYPE" != "tag" ]]; then
echo "::error::Manual runs of this workflow must be started from a tag"
exit 1
fi
- name: Clone
id: checkout
uses: actions/checkout@v6
@@ -170,7 +182,7 @@ jobs:
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
# - name: Set up QEMU
# if: ${{ contains(matrix.config.platforms, 'linux/amd64') }}
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
BODY: ${{ github.event.pull_request.body }}
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'video demo') }}
run: |
# sections written by demo:pr (see tools/ui/demo/cli.ts): the managed comment
# sections written by demo:pr (see ui/apps/chat/demo/cli.ts): the managed comment
# markers, or a preview linking to a demo video uploaded for this PR
has_demo=false
if { grep -qF '<!-- demo:start -->' <<< "$BODY" && grep -qF '<!-- demo:end -->' <<< "$BODY"; } ||
+12 -12
View File
@@ -130,7 +130,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -196,7 +196,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -274,7 +274,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -381,7 +381,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -479,7 +479,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Set up JDK
uses: actions/setup-java@v5
@@ -571,7 +571,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -624,7 +624,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -672,7 +672,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -894,7 +894,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Install Ninja
run: |
@@ -1460,7 +1460,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -1532,7 +1532,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Free up disk space
uses: ggml-org/free-disk-space@v1.3.1
@@ -1647,7 +1647,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
+37 -2
View File
@@ -7,6 +7,40 @@ on:
description: 'Commit SHA1 to build'
required: false
type: string
spec_methods:
description: 'Speculative methods to test'
required: true
default: 'compact'
type: choice
options:
- compact
- nightly
- all
push:
branches:
- master
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
pull_request:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/server-speculative.yml',
'common/speculative.*',
'src/models/dflash*.cpp',
'src/models/dspark.cpp',
'src/models/eagle3.cpp',
'src/models/lilicorr.cpp',
'src/models/qwen35.cpp',
'tools/server/tests/unit/test_spec_methods.py'
]
cache-mode: none
permissions:
@@ -56,8 +90,9 @@ jobs:
- name: Test speculative decoding
env:
SPEC_METHODS: compact
SPEC_MODEL_DIR: ${{ vars.SPEC_METHOD_MODEL_DIR || format('{0}/spec-models', runner.temp) }}
SPEC_METHODS: ${{ github.event.inputs.spec_methods || 'compact' }}
# pull requests from forks do not get repository variables
SPEC_MODEL_DIR: ${{ vars.SPEC_METHOD_MODEL_DIR || '/mnt/cache/spec-models' }}
N_GPU_LAYERS: 99
run: |
cd tools/server/tests
+4 -4
View File
@@ -23,20 +23,20 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+5 -5
View File
@@ -37,26 +37,26 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
env:
LLAMA_BUILD_NUMBER: ${{ steps.version.outputs.ui_version }}
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run PWA unit tests (versioned build output)
run: npx vitest --project=unit --run tests/unit/pwa.spec.ts
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v7
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+5 -5
View File
@@ -33,15 +33,15 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
run-id: ${{ inputs.run_id }}
github-token: ${{ github.token }}
- name: Create distribution archive
run: |
tar -czf dist.tar.gz -C tools/ui/dist .
tar -czf dist.tar.gz -C ui/apps/chat/dist .
sha256sum dist.tar.gz > dist.tar.gz.sha256
mv dist.tar.gz dist.tar.gz.sha256 tools/ui/dist/
mv dist.tar.gz dist.tar.gz.sha256 ui/apps/chat/dist/
- name: Install Hugging Face Hub CLI
run: pip install -U huggingface_hub
@@ -52,12 +52,12 @@ jobs:
- name: Sync built files to Hugging Face bucket (version tag)
run: |
# Upload the built files to the Hugging Face bucket under the release version
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
- name: Sync built files to Hugging Face bucket (latest)
run: |
# Also upload to the 'latest' directory for fallback downloads
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
- name: Verify upload
run: |
+15 -13
View File
@@ -17,7 +17,8 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -25,7 +26,8 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/ui/**.*',
'tools/server/tests/**.*'
]
@@ -67,33 +69,33 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
e2e-tests:
name: E2E Tests
@@ -109,25 +111,25 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Build Storybook
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+46 -17
View File
@@ -13,7 +13,8 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -21,7 +22,8 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'ui/lib/**.*',
'tools/server/tests/**.*'
]
@@ -66,46 +68,73 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
lib-checks:
name: Library checks
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v6
with:
ref: ${{ github.event.inputs.sha || github.event.pull_request.head.sha || github.sha || github.head_ref || github.ref_name }}
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: "24"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: ui/lib/ui
- name: Run type checking
run: npm run check
working-directory: ui/lib/ui
- name: Build the package
run: npm run prepack
working-directory: ui/lib/ui
e2e-tests:
name: E2E Tests
@@ -124,38 +153,38 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts (reuses ui-build)
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build Storybook
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+54 -1
View File
@@ -1,4 +1,4 @@
# date: Wed Nov 4 12:03:31 EET 2026
# date: Mon Nov 16 10:05:56 EET 2026
# this file is auto-generated by scripts/gen-authors.sh
Нияз Гарифзянов <112617865+garrnizon@users.noreply.github.com>
@@ -43,7 +43,9 @@ Aarni Koskela <akx@iki.fi>
Aaron Miller <apage43@ninjawhale.com>
Aaron Teo <57927438+taronaeo@users.noreply.github.com>
Aaron Teo <aaron.teo1@ibm.com>
Aaron Thomas <84493696+arxhr007@users.noreply.github.com>
Aaryaman Vasishta <aaryaman.vasishta@amd.com>
Aayan Karmali <44980366+StockerMC@users.noreply.github.com>
Abheek Gulati <abheekg@hotmail.com>
abhijain1204fujitsu <139222713+abhijain1204fujitsu@users.noreply.github.com>
Abhijit Ramesh <abhijitramesh2k@gmail.com>
@@ -112,10 +114,12 @@ Aleksander Grygier <admin@serveurperso.com>
Aleksander Grygier <aleksander.grygier@gmail.com>
Aleksander Grygier <thichthat@gmail.com>
Aleksei Nikiforov <103434461+AlekseiNikiforovIBM@users.noreply.github.com>
Alen Požun <kalyx.solutions@proton.me>
Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@opensuse.org>
Alessandro98-git <61804547+Alessandro98-git@users.noreply.github.com>
Alex <18387287+wadealexc@users.noreply.github.com>
Alex <59368173+AlexGabbia@users.noreply.github.com>
Alex <alexzkhr@gmail.com>
Alex <awhill19@icloud.com>
Alex Azarov <alex@azarov.by>
Alex Azarov <alexander.azarov@mapbox.com>
@@ -132,6 +136,7 @@ Alex Trotta <44127594+Ahajha@users.noreply.github.com>
Alex Tuddenham <61622354+AlexsCode@users.noreply.github.com>
Alex von Gluck IV <kallisti5@unixzen.com>
Alex Wu <dindinw@users.noreply.github.com>
Alex-JP-93 <Jianpeng.Li@amd.com>
alex-spacemit <jinghui.huang@spacemit.com>
Alexander Batischev <eual.jp@gmail.com>
Alexander Heisler <126129661+heislera763@users.noreply.github.com>
@@ -143,6 +148,7 @@ Alexis Williams <typedrat@users.noreply.github.com>
alexpinel <93524949+alexpinel@users.noreply.github.com>
Alfred <zxu3@clemson.edu>
Ali Chraghi <63465728+alichraghi@users.noreply.github.com>
Ali F. Al-Ramadan <55710790+aalramadan@users.noreply.github.com>
Ali Nehzat <ali.nehzat@thanks.dev>
Ali Tariq <ali.tariq@10xengineers.ai>
Ali Tariq <alitariq4589@gmail.com>
@@ -158,6 +164,7 @@ Aman Karki <amankarki151@gmail.com>
Aman Karki <itsamankarki@gmail.com>
amd-dwang <dong.wang@amd.com>
amd-lalithnc <lalithnc@amd.com>
AMIR <127450356+amirthn6533@users.noreply.github.com>
Amir <amir_zia@outlook.com>
amirai21 <89905406+amirai21@users.noreply.github.com>
AmirAli Mirian <37371367+amiralimi@users.noreply.github.com>
@@ -213,6 +220,7 @@ Antoine Viallon <antoine@lesviallon.fr>
Anton I <inflyer@hotmail.com>
Anton Mitkov <anton_b_mitkov@abv.bg>
Anton Mitkov <anton.mitkov@codeplay.com>
Anton Sokolchenko <wsevendays@gmail.com>
Antonis Makropoulos <benuix@gmail.com>
anubhavfujitsu <Anubhav.Srivastava@fujitsu.com>
Anudit Nagar <nagaranudit@gmail.com>
@@ -233,10 +241,12 @@ arlo-phoenix <140345165+arlo-phoenix@users.noreply.github.com>
Armen Kaleshian <kriation@users.noreply.github.com>
Arsen Arutunan <58118221+limloop@users.noreply.github.com>
Arseniy Obolenskiy <good.doog.2012@gmail.com>
ArshM17-NV <amaknojia@nvidia.com>
Artem <guinmoon@gmail.com>
Artem Zinnatullin <ceo@abstractny.gay>
Artyom Lebedev <vagran.ast@gmail.com>
aryantandon01 <80969509+aryantandon01@users.noreply.github.com>
Asahi-Prv <246092086+Asahi-Prv@users.noreply.github.com>
Asahi-Prv <asahihori7@gmail.com>
asbelin <71226900+asbelin@users.noreply.github.com>
Asbjørn Olling <asbjornolling@gmail.com>
@@ -258,6 +268,7 @@ AustinMroz <austinmroz@utexas.edu>
AUTOMATIC1111 <16777216c@gmail.com>
automaticcat <daogiatuank54@gmail.com>
awatuna <23447591+awatuna@users.noreply.github.com>
Awss <gitj.18@alt.awss1i.com>
b4b4o <zwbao@foxmail.com>
Bach Le <bach@bullno1.com>
BADR <contact@pythops.com>
@@ -351,6 +362,7 @@ CarterLi999 <664681047@qq.com>
Casey Primozic <casey@cprimozic.net>
Casey Primozic <me@ameo.link>
CausalLM <148736309+CausalLM@users.noreply.github.com>
CC <65151926+realMoana@users.noreply.github.com>
ccbinn <ccbinn@163.com>
cduk <19917266+cduk@users.noreply.github.com>
cebtenzzre <cebtenzzre@gmail.com>
@@ -379,6 +391,7 @@ Cheng Shao <terrorjack@type.dance>
Chenguang Li <757486878@qq.com>
Chenguang Li <87689256+noemotiovon@users.noreply.github.com>
chiheb ben cheikh <100644963+ChihebBENCHEIKH1@users.noreply.github.com>
chinmaykamjith001 <chinmaykamjith@gmail.com>
Chipmunk <101038159+CHIPMUNK-T0T@users.noreply.github.com>
chiranko <96988916+chiranko@users.noreply.github.com>
Chris Danis <cdanis@gmail.com>
@@ -434,6 +447,7 @@ crasm <crasm@git.vczf.us>
crat0z <11581854+crat0z@users.noreply.github.com>
CRD716 <crd716@gmail.com>
CrispStrobe <154636388+CrispStrobe@users.noreply.github.com>
Cristian Adam <cristian.adam@gmail.com>
Cristiano Pinto <140563307+crowmoed@users.noreply.github.com>
crsawyer <7572190+crsawyer@users.noreply.github.com>
Csaba Kecskemeti <csaba.kecskemeti@gmail.com>
@@ -527,6 +541,7 @@ Djip007 <3705339+Djip007@users.noreply.github.com>
Djip007 <djip.perois@free.fr>
dm4 <dm4@secondstate.io>
dm4 <sunrisedm4@gmail.com>
Dmitrii Panchev <dmnobunaga@users.noreply.github.com>
Dmitry Atamanov <data-man@users.noreply.github.com>
Dmytro Minochkin <dmytro.minochkin@gmail.com>
Dmytro Romanov <casteldazur@gmail.com>
@@ -567,6 +582,7 @@ Ed Addario <29247825+EAddario@users.noreply.github.com>
Ed Lee <edilee@mozilla.com>
Ed Lepedus <ed.lepedus@googlemail.com>
Eddie-Wang <wangjinheng1120@163.com>
eddiechen <Echen1246@gmail.com>
Eden <aa9736195201@gmail.com>
eduardopessin <100053075+eduardopessin@users.noreply.github.com>
Edward Taylor <edeetee@gmail.com>
@@ -598,6 +614,7 @@ Erik Winter <93544977+ewintr@users.noreply.github.com>
Ervin Áron Tasnádi <etasnadi@protonmail.com>
Esko Toivonen <eskot98@gmail.com>
Ethan Guo <ethanguo.dev@gmail.com>
Ethan Lin <linsen458@gmail.com>
Ethan Turner <eturner64@gmail.com>
Ettore Di Giacinto <mudler@users.noreply.github.com>
EugeoSynthesisThirtyTwo <gabriel.dhimoila@gmail.com>
@@ -625,6 +642,7 @@ fanyang <fanyang89@outlook.com>
Farbod Bijary <110523279+farbodbj@users.noreply.github.com>
Fathi Boudra <fathi.boudra@linaro.org>
Fattire <528174+fat-tire@users.noreply.github.com>
Feiteng <lifeiteng0422@gmail.com>
felix <felix314159@users.noreply.github.com>
Felix <stenbackfelix@gmail.com>
Felix Ye <felixyjs@gmail.com>
@@ -632,6 +650,7 @@ fengerhu1 <2748250768@qq.com>
feni6 <fenissova@proton.me>
fidoriel <49869342+fidoriel@users.noreply.github.com>
fiesh <fiesh@zefix.tv>
figkcoci-debug <figkcoci@gmail.com>
FINAL-Bench <kkms1116@koreacu.ac.kr>
Finn Voorhees <finnvoorhees@gmail.com>
Firat <firatkiral@gmail.com>
@@ -646,6 +665,7 @@ Foad Abo Dahood <32059146+masterFoad@users.noreply.github.com>
Folko-Ven <71110216+Folko-Ven@users.noreply.github.com>
forforever73 <63285796+forforever73@users.noreply.github.com>
Foul-Tarnished <107711110+Foul-Tarnished@users.noreply.github.com>
fox5611 <66219952+fox5611@users.noreply.github.com>
Francisco Herrera <ppaanncchhoo507@gmail.com>
Francisco Melo <43780565+francis2tm@users.noreply.github.com>
Francois Dugast <francois.dugast@intel.com>
@@ -709,6 +729,7 @@ GiviMAD <GiviMAD@users.noreply.github.com>
gliptic <gliptic@users.noreply.github.com>
gn64 <yukikaze.jp@gmail.com>
goerch <jhr.walter@t-online.de>
Gotya-OFFCIAL <zhengfreeman@gmail.com>
Govlzkoy <gotope@users.noreply.github.com>
grahameth <96447521+grahameth@users.noreply.github.com>
Gregor Jasny <gjasny@googlemail.com>
@@ -749,6 +770,7 @@ Haowei Wu <breadcyanide@icloud.com>
Haoxiang Fei <tonyfettes@tonyfettes.com>
Harald Fernengel <harald.fernengel@here.com>
Harapan Rachman <harapanrachman@gmail.com>
Harkanwal Bedi <bedi1887@gmail.com>
Harkirat Gill <harkirat.gill@amd.com>
HarrisonSec <gzxharrison@gmail.com>
Hatsune Miku <129688334+at8u@users.noreply.github.com>
@@ -768,6 +790,7 @@ Henrik Forstén <henrik.forsten@gmail.com>
Henry Linjamäki <henry.linjamaki@gmail.com>
Henry Linjamäki <henry.mikael.linjamaki@intel.com>
Henry147147 <44851451+Henry147147@users.noreply.github.com>
Hera <58063257+hera2019@users.noreply.github.com>
Herman Semenoff <GermanAizek@yandex.ru>
Herman Semenov <GermanAizek@yandex.ru>
Hesen Peng <hesen.peng@gmail.com>
@@ -846,6 +869,7 @@ Imad Saddik <79410781+ImadSaddik@users.noreply.github.com>
iMil <imil@NetBSD.org>
Incarnas <119618389+bit-incarnas@users.noreply.github.com>
InflexCZE <martin.pavlicek11@gmail.com>
Iñigo Imaz <49730431+inimaz@users.noreply.github.com>
Intel AI Get-to Market Customer Success and Solutions <ai.gtm.css@gmail.com>
intelmatt <61025942+intelmatt@users.noreply.github.com>
iohub <rickyang.pro@gmail.com>
@@ -857,7 +881,9 @@ IsaacDynamo <61521674+IsaacDynamo@users.noreply.github.com>
Ishaan Gandhi <Ishaangandhi@gmail.com>
iSma <ismail.senhaji@gmail.com>
Ismail <115064057+AlrIsmail@users.noreply.github.com>
IsNot <209144869@qq.com>
issixx <46835150+issixx@users.noreply.github.com>
Iswarya Alex <47045679+iswaryaalex@users.noreply.github.com>
itsnotoger <19309683+itsnotoger@users.noreply.github.com>
itterative <190138728+itterative@users.noreply.github.com>
Ivan <nekotekina@gmail.com>
@@ -888,6 +914,7 @@ James Francis <6763899+JamesFranc@users.noreply.github.com>
James O'Leary <65884233+jpohhhh@users.noreply.github.com>
James Reynolds <magnusviri@users.noreply.github.com>
jameswu2014 <545426914@qq.com>
JamieBordoni <jamie.bordoni@protonmail.com>
Jan Boon <jan.boon@kaetemi.be>
Jan Boon <kaetemi@gmail.com>
Jan Ekström <jeebjp@gmail.com>
@@ -956,8 +983,10 @@ Jinyang He <hejinyang@loongson.cn>
jinzihao <jinzihao1996@gmail.com>
Jiří Podivín <66251151+jpodivin@users.noreply.github.com>
Jiří Sejkora <Sejseloid@gmail.com>
Jiří Svítil <32205472+mistrjirka@users.noreply.github.com>
Jiwoong Song <55657581+jwsong98@users.noreply.github.com>
JJJYmmm <92386084+JJJYmmm@users.noreply.github.com>
jjperry69 <jjperry69@gmail.com>
jklincn <985765408@qq.com>
jklincn <jklincn@outlook.com>
JM Robles <roblesjm@gmail.com>
@@ -970,6 +999,7 @@ Joe Rowell <joerowell4@gmail.com>
Joe Todd <joe.todd@codeplay.com>
joecryptotoo <80373433+joecryptotoo@users.noreply.github.com>
Joel <agnel.joel@gmail.com>
Joel Fernandes <joelagnelf@nvidia.com>
Johan <JohanAR@users.noreply.github.com>
Johannes Gäßler <johannesg@5d6.de>
Johannes Rudolph <johannes.rudolph@gmail.com>
@@ -988,6 +1018,7 @@ Jonas J <111707981+John-194@users.noreply.github.com>
Jonas Jankaitis <111707981+John-194@users.noreply.github.com>
Jonas Wunderlich <32615971+jonas-w@users.noreply.github.com>
Jonathan <47618606+jbuchananr@users.noreply.github.com>
Jonathan Buck <jonathan.j.buck@gmail.com>
Jonathan Clohessy <jonathan.clohessy@arm.com>
Jonathan Graehl <99024+graehl@users.noreply.github.com>
Jorge A <161275481+jorgealias@users.noreply.github.com>
@@ -1097,6 +1128,7 @@ klosax <131523366+klosax@users.noreply.github.com>
KnightYao <KnightYao@users.noreply.github.com>
KokerZhou <111279477+KokerZhou@users.noreply.github.com>
Kolen Cheung <ickc@users.noreply.github.com>
Kong Jiale <2049447146@qq.com>
kononnable <kononnable@gmail.com>
Konrad Moren <kmoren@nvidia.com>
konradmb <konradmb@o2.pl>
@@ -1167,6 +1199,7 @@ li-lizhe <82501526+li-lizhe@users.noreply.github.com>
liangliangchang <lichang@amd.com>
liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
limitedAtonement <limitedAtonement@users.noreply.github.com>
Lines <linellazatin@gmail.com>
lingyezhixing <144504450+lingyezhixing@users.noreply.github.com>
linsen458-spec <linsen458@gmail.com>
Linux User <linuxusergd@outlook.com>
@@ -1227,6 +1260,7 @@ makomk <makosoft@googlemail.com>
manayang <jackmanayang@gmail.com>
manikbhandari <mbbhandarimanik2@gmail.com>
Manuel <44313466+makuche@users.noreply.github.com>
ManuelPDM <153872964+ManuelPDM@users.noreply.github.com>
maor-ps <154728172+maor-ps@users.noreply.github.com>
Maphist0 <28743569+Maphist0@users.noreply.github.com>
Marc Köhlbrugge <subscriptions@marckohlbrugge.com>
@@ -1336,6 +1370,7 @@ Michael Hueschen <m@mhueschen.dev>
Michael Kesper <mkesper@schokokeks.org>
Michael Klimenko <mklimenko29@gmail.com>
Michael Lamothe <michael.lamothe@gmail.com>
Michael Paulauski <mike10010100@gmail.com>
Michael Podvitskiy <podvitskiymichael@gmail.com>
Michael Potter <NanoTekGuy@Gmail.com>
Michael Taylor <162068037+mctylr-gh@users.noreply.github.com>
@@ -1455,6 +1490,7 @@ Nikolay Popov <131475237+npopov-vst@users.noreply.github.com>
Nils Gladitz <nilsgladitz@gmail.com>
Nindaleth <Nindaleth@users.noreply.github.com>
ningshanwutuobang <ningshanwutuobang@gmail.com>
Nisbenz <benazzaanis783@gmail.com>
Noah <99681487+NoahOksuz@users.noreply.github.com>
nold <Nold360@users.noreply.github.com>
nopperl <54780682+nopperl@users.noreply.github.com>
@@ -1462,6 +1498,7 @@ nullname <chraac@gmail.com>
Nuno <rare-magma@posteo.eu>
nuri <yoonuri1@gmail.com>
nusu-github <29514220+nusu-github@users.noreply.github.com>
nv-vankit <162085734+nv-vankit@users.noreply.github.com>
nwyin <tommynguyen0512@gmail.com>
o7si <32285332+o7si@users.noreply.github.com>
Oğuzhan Akkaya <oakkaya@ymail.com>
@@ -1485,6 +1522,7 @@ oobabooga <oobabooga4@gmail.com>
opparco <parco.opaai@gmail.com>
Ori Pekelman <ori@pekelman.com>
Oscar Barenys <rtfss1@gmail.com>
Oscar M. Herrera <oscar@kss-inc.com>
OSecret <135510162+OLSecret@users.noreply.github.com>
ostix360 <55257054+ostix360@users.noreply.github.com>
Ouadie EL FAROUKI <ouadie.elfarouki@codeplay.com>
@@ -1611,6 +1649,7 @@ Raj Hammeer Singh Hada <hammeerraj@gmail.com>
Rajendra Matcha <matcraje@qti.qualcomm.com>
Ralph Soika <ralph.soika@imixs.com>
Raman Shinde <raman.shinde15@gmail.com>
Ramzi <146260554+rmz-24@users.noreply.github.com>
Rand Xie <randxiexyy29@gmail.com>
Randall Fitzgerald <randall@dasaku.net>
Random Fly <renfei8@live.cn>
@@ -1651,6 +1690,7 @@ Rich Dougherty <rich@rd.nz>
Richard <r-burton@hotmail.co.uk>
Richard Davison <richard.davison1@gmail.com>
Richard Kiss <him@richardkiss.com>
Richard Palethorpe <io@richiejp.com>
Richard Roberson <richardr1126@gmail.com>
RichardScottOZ <rnmscott@netspace.net.au>
Rick G <26732651+TheFlipbook@users.noreply.github.com>
@@ -1695,6 +1735,7 @@ Roni <sulpher@gmx.net>
Ronny Brendel <ronnyb@nvidia.com>
Ronny Brendel <ronnybrendel@gmail.com>
Ronsor <ronsor@ronsor.pw>
Roshan Kumar <52444089+roshankumarg529@users.noreply.github.com>
Rotem Dan <rotemdan@gmail.com>
Rowan Hart <rowanbhart@gmail.com>
rspOverflow <217881046+rspOverflow@users.noreply.github.com>
@@ -1727,9 +1768,11 @@ Saba Fallah <10401143+sfallah@users.noreply.github.com>
Saba Fallah <sabafallah@gmail.com>
Sachin Desai <smdesai@gmail.com>
Sachin Sharma <sachin@zettabolt.com>
Saejin Kim-Heinert <saejinmh@gmail.com>
Safi Ullah <safiullah.3915@gmail.com>
safranowith <bsh155762@gmail.com>
Said Taghadouini <84044788+staghado@users.noreply.github.com>
SaidSultani <62264302+SaidSultani@users.noreply.github.com>
Sait Furkan Teke <35101659+stfurkan@users.noreply.github.com>
SakuraUmi <yukinon244@gmail.com>
Salvador E. Tropea <stropea@inti.gob.ar>
@@ -1754,6 +1797,7 @@ Sang-Kil Park <sang.park@42dot.ai>
Sanjay Ahari <sanjayahari1704@gmail.com>
Sarah Wu <sarahwu185@gmail.com>
Sarah Wu <sarwu@nvidia.com>
sargentmajord <jasonlather13@gmail.com>
Sascha Rogmann <59577610+srogmann@users.noreply.github.com>
sasha0552 <admin@sasha0552.org>
Satinder Grewal <grewal.satinder@gmail.com>
@@ -1786,6 +1830,7 @@ Sergiu <8598216+mzsergiu@users.noreply.github.com>
serhii-nakon <57632032+serhii-nakon@users.noreply.github.com>
Sertaç Özercan <852750+sozercan@users.noreply.github.com>
seryogakovalyov <seryogakovalyov@gmail.com>
Seungjun Bae <linkedlatch@gmail.com>
Seungmin Kim <8457324+ehfd@users.noreply.github.com>
SeungWon Jeong <65549245+redlion0929@users.noreply.github.com>
Seyoung Jeong <seyoungjeong@gmail.com>
@@ -1871,6 +1916,7 @@ SmartestWashingMachine <ottobizness@gmail.com>
smugman-dot <wbsmoke101@gmail.com>
SnA1lGo <44647694+skrandy@users.noreply.github.com>
snadampal <87143774+snadampal@users.noreply.github.com>
Sofian Elmotiem <sofianelmotiem@gmail.com>
SoftwareRenderer <138734813+SoftwareRenderer@users.noreply.github.com>
Someone <sergei.kozlukov@aalto.fi>
Someone Serge <sergei.kozlukov@aalto.fi>
@@ -1886,6 +1932,7 @@ sprayandwipe <spray.and.wipe@gmail.com>
Sreeram <sreeram.sivasubramony@amd.com>
SRHMorris <69468379+SRHMorris@users.noreply.github.com>
Srihari-mcw <96763064+Srihari-mcw@users.noreply.github.com>
Srijan Keshri <212402043+arcusbuilds@users.noreply.github.com>
Srinivas Billa <nivibilla@gmail.com>
srkizer <github.soreepeong.prqovzj1d@srkizer.net>
ssweens <1149151+ssweens@users.noreply.github.com>
@@ -1919,6 +1966,7 @@ Sutou Kouhei <kou@cozmixng.org>
Svetlozar Georgiev <55534064+sgeor255@users.noreply.github.com>
Swigler <124839156+Swigler@users.noreply.github.com>
swittk <switt1995@gmail.com>
sxq <18856096915@163.com>
SXX <song_xiaoxi@126.com>
SXX <sxx1136965276@gmail.com>
Tai Duc Nguyen <taiducnguyen.drexel@gmail.com>
@@ -2044,6 +2092,7 @@ Ulrich Drepper <drepper@gmail.com>
unbounded <haakon@likedan.net>
Unbug Lee <tidelgl@gmail.com>
unraido <127105806+unraido@users.noreply.github.com>
usrlocalben <benjamin@rqdq.com>
uvos <carl@uvos.xyz>
uvos <devnull@uvos.xyz>
uvos <philipp@uvos.xyz>
@@ -2084,6 +2133,7 @@ vk <89937361+itsvedantkumar@users.noreply.github.com>
Vlad <spitfireage@gmail.com>
Vladimir <bogdad@gmail.com>
Vladimir Malyutin <first-leon@yandex.ru>
Vladimir Sotnikov <bubert9@gmail.com>
Vladimir Vuksanovic <109677816+vvuksanovic@users.noreply.github.com>
Vladimir Zorin <vladimir@deviant.guru>
Vladislav <vladplotnikov34@gmail.com>
@@ -2139,6 +2189,7 @@ Wroclaw <wroclaw223@outlook.com>
wsbagnsv1 <sclumpfpapa36@gmail.com>
Wu Jian Ping <wujjpp@hotmail.com>
Wu Jian Ping <wujp@greatld.com>
WuXin <apuravmanhas123@gmail.com>
wwoodsTM <104587230+wwoodsTM@users.noreply.github.com>
Wyatt Caldwell <218154709+Detensable@users.noreply.github.com>
wzy <32936898+Freed-Wu@users.noreply.github.com>
@@ -2162,6 +2213,7 @@ Xinpeng Dou <15529241576@163.com>
Xinpeng Dou <81913537+Dou-Git@users.noreply.github.com>
xloem <0xloem@gmail.com>
xris99 <79798089+xris99@users.noreply.github.com>
Xuan Son Nguyen <son@huggingface.co>
Xuan Son Nguyen <thichthat@gmail.com>
Xuan-Son Nguyen <son@huggingface.co>
Xuan-Son Nguyen <thichthat@gmail.com>
@@ -2247,6 +2299,7 @@ zhangrunda <zhangrunda1234@outlook.com>
zhangtao2-1 <478679312@qq.com>
ZHAOKAI WANG <sanxianwei@163.com>
Zhaolun Yin <129580161+ZhaolunYin@users.noreply.github.com>
zhaoyshine <30611065+zhaoyshine@users.noreply.github.com>
Zheng.Deng <32841220+dengzheng-cloud@users.noreply.github.com>
zhentaoyu <zhentao.yu@intel.com>
Zhenwei Jin <109658203+kylo5aby@users.noreply.github.com>
+1 -1
View File
@@ -134,7 +134,7 @@ option(LLAMA_BUILD_TOOLS "llama: build tools"
option(LLAMA_BUILD_EXAMPLES "llama: build examples" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_SERVER "llama: build server example" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_APP "llama: build the unified binary" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_UI "llama: build the embedded Web UI for server" OFF)
option(LLAMA_BUILD_UI "llama: build Llama Chat, the embedded chat app, for server" OFF)
option(LLAMA_USE_PREBUILT_UI "llama: use prebuilt UI from HF Bucket when available" ON)
option(LLAMA_CONNECT "llama: include the llama-connect tool (downloads a prebuilt binary)" OFF)
+1 -1
View File
@@ -102,7 +102,7 @@
/tools/rpc/ @ggml-org/ggml-rpc
/tools/server/* @ggml-org/llama-server # no subdir
/tools/server/tests/ @ggml-org/llama-server
/tools/ui/ @ggml-org/llama-ui
/ui/ @ggml-org/llama-ui
/tools/tokenize/ @ggerganov
/tools/tts/ @ggerganov
/vendor/ @ggerganov
+12
View File
@@ -16,6 +16,8 @@ The project differentiates between 3 levels of contributors:
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
@@ -24,6 +26,16 @@ If AI is used to generate any portion of the code, contributors must adhere to t
4. Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
# Pull requests (for contributors & collaborators)
+9 -41
View File
@@ -13,10 +13,16 @@
[![Docker](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/docker.yml?label=Docker)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
[![Winget](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/winget.yml?label=Winget)](https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml)
[ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Asanmai%20OR%20author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Athelittlefireman%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3AmasterFoad%20OR%20author%3Aallozaur%20OR%20author%3AAndreas-Krebbel%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Achaxu01%20OR%20author%3Alhez%20OR%20author%3AAlekseiNikiforovIBM%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming%20OR%20author%3Aynankani)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
[v1.0.0](https://huggingface.co/blog/ggml-org/llama-cpp-v1) / [ggml](https://github.com/ggml-org/ggml) / [ops](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) / [maintainer PRs](https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr%20is%3Aopen%20draft%3AFalse%20(author%3Asanmai%20OR%20author%3Argerganov%20OR%20author%3AKitaitiMakoto%20OR%20author%3Adanbev%20OR%20author%3Aaldehir%20OR%20author%3Amax-krasnyansky%20OR%20author%3ACISC%20OR%20author%3Aggerganov%20OR%20author%3Aam17an%20OR%20author%3Ajhen0409%20OR%20author%3Abartowski1182%20OR%20author%3Anikwen%20OR%20author%3Ahipudding%20OR%20author%3Aravi9%20OR%20author%3AServeurpersoCom%20OR%20author%3Athelittlefireman%20OR%20author%3Apwilkin%20OR%20author%3Areeselevine%20OR%20author%3Angxson%20OR%20author%3Ajeffbolznv%20OR%20author%3Amarty1885%20OR%20author%3A0cc4m%20OR%20author%3ATitaniumtown%20OR%20author%3Aangt%20OR%20author%3AIMbackK%20OR%20author%3Aarthw%20OR%20author%3AJohannesGaessler%20OR%20author%3AORippler%20OR%20author%3Aruixiang63%20OR%20author%3Axctan%20OR%20author%3AmasterFoad%20OR%20author%3Aallozaur%20OR%20author%3AAndreas-Krebbel%20OR%20author%3Ayomaytk%20OR%20author%3Aaendk%20OR%20author%3Awine99%20OR%20author%3Agaugarg-nv%20OR%20author%3Ataronaeo%20OR%20author%3Aforforever73%20OR%20author%3Achaxu01%20OR%20author%3Alhez%20OR%20author%3AAlekseiNikiforovIBM%20OR%20author%3Anetrunnereve%20OR%20author%3Afairydreaming%20OR%20author%3Aynankani)%20sort%3Aupdated-desc) / [dev stats](https://github.com/ggml-org/llama.cpp-dev) / [lib llama API](https://github.com/ggml-org/llama.cpp/issues/9289) / [llama-server REST API](https://github.com/ggml-org/llama.cpp/issues/9291)
</div>
## Description
The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
## Quick start
A few options to get `llama.cpp` installed on your machine:
@@ -51,50 +57,12 @@ llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
<i>VLM session with <b>llama cli</b></i>
</td>
<td align="center">
<img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Built-in web UI against <b>llama serve</b></i>
<img width="1392" height="958" alt="Llama Chat, the built-in chat app, against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Llama Chat, the built-in chat app, against <b>llama serve</b></i>
</td>
</tr>
<table>
## Description
The main goal of `llama.cpp` is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The `llama.cpp` project is build on top of the [ggml](https://github.com/ggml-org/ggml) library.
## Supported backends
| Backend | Target devices |
| --- | --- |
| [BLAS](docs/build.md#blas-build) | All |
| [BLIS](docs/backend/BLIS.md) | All |
| [CANN](docs/build.md#cann) | Huawei Ascend NPU |
| [CUDA](docs/build.md#cuda) | Nvidia GPU |
| [HIP](docs/build.md#hip) | AMD GPU |
| [Hexagon](docs/backend/snapdragon/README.md) | Snapdragon |
| [IBM zDNN](docs/backend/zDNN.md) | IBM Z & LinuxONE |
| [MUSA](docs/build.md#musa) | Moore Threads GPU |
| [Metal](docs/build.md#metal-build) | Apple Silicon |
| [OpenCL](docs/backend/OPENCL.md) | Adreno GPU |
| [OpenVINO [In Progress]](docs/backend/OPENVINO.md) | Intel CPUs, GPUs, and NPUs |
| [RPC](https://github.com/ggml-org/llama.cpp/tree/master/tools/rpc) | All |
| [SYCL](docs/backend/SYCL.md) | Intel GPU |
| [VirtGPU](docs/backend/VirtGPU.md) | VirtGPU APIR |
| [Vulkan](docs/build.md#vulkan) | GPU |
| [WebGPU](docs/build.md#webgpu) | All |
| [ZenDNN](docs/build.md#zendnn) | AMD CPU |
## Documentation
#### Tools
+1 -1
View File
@@ -46,7 +46,7 @@ Only vulnerabilities that fall within these parts of the project are considered
- `ggml/**/*`
- `gguf-py/**/*`
- `tools/server/*`, **excluding** the following topics:
- Web UI
- Llama Chat, the built-in chat app (`ui/apps/chat`)
- Features marked as experimental
- Features not recommended for use in untrusted environments (e.g., router, MCP)
- Bugs that can lead to Denial-of-Service attack
+12 -4
View File
@@ -2829,7 +2829,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -2895,7 +2895,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -3416,7 +3416,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui-mcp-proxy", "--webui-mcp-proxy"},
{"--no-ui-mcp-proxy", "--no-webui-mcp-proxy"},
"experimental: whether to enable MCP CORS proxy, for the Web UI to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)\n"
"experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, bool value) {
params.ui_mcp_proxy = value;
@@ -3485,7 +3485,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui", "--webui"},
{"--no-ui", "--no-webui"},
string_format("whether to enable the Web UI (default: %s)", params.ui ? "enabled" : "disabled"),
string_format("whether to enable Llama Chat, the built-in chat app (default: %s)", params.ui ? "enabled" : "disabled"),
[](common_params & params, bool value) {
params.ui = value;
}
@@ -4522,6 +4522,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.tts_speaker_file = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
add_opt(common_arg(
{"--tts-speaker-text"}, "TEXT",
"transcript of --tts-speaker-file, for models that need it to clone the voice\n"
"see docs/multimodal/tts.md for per-model usage notes",
[](common_params & params, const std::string & value) {
params.tts_speaker_text = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
//
// diffusion params
+29 -1
View File
@@ -2,6 +2,7 @@
#include "jinja/runtime.h"
#include "jinja/value.h"
#include "json.h"
#include <algorithm>
#include <climits>
@@ -857,8 +858,35 @@ static value tts_norm_builtin_sub(const func_args & args) {
return mk_val<value_string>(string(res, text_val.all_parts_are_input()));
}
void common_chat_tts_norm_add_builtins(context & ctx) {
bool common_tts_dict_find(const common_tts_dict & dict, const std::string & key, std::string & value) {
const auto it = std::lower_bound(dict.keys.begin(), dict.keys.end(), key);
if (it == dict.keys.end() || *it != key) {
return false;
}
const size_t idx = it - dict.keys.begin();
if (idx >= dict.values.size()) {
throw std::runtime_error("tts_dict: malformed dictionary, a key has no value");
}
value = dict.values[idx];
return true;
}
void common_chat_tts_norm_add_builtins(context & ctx, std::shared_ptr<const common_tts_dicts> dicts) {
ctx.set_val("tts_norm_scan", mk_val<value_func>("tts_norm_scan", tts_norm_builtin_scan));
ctx.set_val("tts_norm_sub", mk_val<value_func>("tts_norm_sub", tts_norm_builtin_sub));
ctx.set_val("tts_dict", mk_val<value_func>("tts_dict", [dicts](const func_args & args) -> value {
args.ensure_count(2, 2);
args.ensure_vals<value_string, value_string>();
const std::string name = args.get_pos(0)->as_string().str();
const auto it = dicts ? dicts->find(name) : common_tts_dicts::const_iterator();
if (!dicts || it == dicts->end()) {
throw std::runtime_error("tts_dict: unknown dictionary '" + name + "'");
}
std::string doc;
if (!common_tts_dict_find(it->second, args.get_pos(1)->as_string().str(), doc)) {
return mk_val<value_none>();
}
return value_from_json(common_json::parse(doc), false);
}));
}
+16 -3
View File
@@ -1,6 +1,7 @@
#pragma once
#include <cstdint>
#include <map>
#include <memory>
#include <string>
#include <utility>
@@ -11,9 +12,10 @@
// it is NOT a regex engine: the semantics are defined by llama.cpp (see docs/tts-norm-patterns.md), so that this
// implementation and the python reference (scripts/chat_tts_norm.py) give the same result for every input
//
// two functions are exposed to task templates (e.g. "tts") via common_chat_tts_norm_add_builtins():
// functions exposed to task templates (e.g. "tts") via common_chat_tts_norm_add_builtins():
// tts_norm_scan(text, rules) -> list of segments {rule, text, groups}
// tts_norm_sub(text, pattern, repl) -> string
// tts_dict(name, key) -> value of key in a dictionary of the model, or none
namespace jinja {
struct context;
@@ -57,6 +59,17 @@ std::vector<common_tts_norm_segment> common_tts_norm_scan(const std::string & te
// repl references: %0 (whole match), %1..%9 (groups), %% (literal percent)
std::string common_tts_norm_sub(const std::string & text, const common_tts_norm_pattern & pattern, const std::string & repl);
// add tts_norm_scan() and tts_norm_sub() to the context of a task template
// a dictionary of the model, e.g. a pronunciation lexicon
struct common_tts_dict {
std::vector<std::string> keys; // sorted by their bytes
std::vector<std::string> values; // the JSON document of each key
};
using common_tts_dicts = std::map<std::string, common_tts_dict>;
// binary search of key in a dictionary; on success, value is set to the JSON document
bool common_tts_dict_find(const common_tts_dict & dict, const std::string & key, std::string & value);
// add tts_norm_scan(), tts_norm_sub() and tts_dict() to the context of a task template
// they are not jinja global builtins on purpose: chat templates are not affected
void common_chat_tts_norm_add_builtins(jinja::context & ctx);
void common_chat_tts_norm_add_builtins(jinja::context & ctx, std::shared_ptr<const common_tts_dicts> dicts = nullptr);
+49 -8
View File
@@ -318,6 +318,7 @@ struct common_chat_templates {
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
std::unique_ptr<common_chat_template> template_tool_use;
std::unique_ptr<common_chat_template> template_tts; // the prompt of a text to speak
std::shared_ptr<common_tts_dicts> tts_dicts; // dictionaries of the model for tts_dict(), by name
};
common_chat_tool_choice common_chat_tool_choice_parse_oaicompat(const std::string & tool_choice) {
@@ -806,10 +807,10 @@ bool common_chat_templates_was_explicit(const struct common_chat_templates * tmp
}
// renders a task template (e.g. "tts") with the given variables
static std::string common_chat_template_render(const common_chat_template & tmpl, const json & vars) {
static std::string common_chat_template_render(const common_chat_template & tmpl, const json & vars, const std::shared_ptr<common_tts_dicts> & tts_dicts) {
jinja::context ctx(tmpl.source());
// text normalization helpers: tts_norm_scan() and tts_norm_sub(), see docs/tts-norm-patterns.md
common_chat_tts_norm_add_builtins(ctx);
// text normalization helpers: tts_norm_scan(), tts_norm_sub() and tts_dict(), see docs/tts-norm-patterns.md
common_chat_tts_norm_add_builtins(ctx, tts_dicts);
jinja::global_from_json(ctx, vars, false);
ctx.set_val("bos_token", jinja::mk_val<jinja::value_string>(tmpl.bos_token()));
ctx.set_val("eos_token", jinja::mk_val<jinja::value_string>(tmpl.eos_token()));
@@ -827,6 +828,11 @@ static std::string string_lower(std::string s) {
return s;
}
// VibeVoice-ASR: default system prompt of the template written by the converter
static bool is_vibevoice_asr_template(const std::string & src) {
return src.find("transcribes audio input into text output in JSON format") != std::string::npos;
}
// LFM3-Audio: language code -> name
static std::string lfm3_language_name(const std::string & language) {
static const std::map<std::string, std::string> names = {
@@ -847,7 +853,8 @@ static std::string lfm3_asr_system_prompt(const std::string & language) {
return "Perform ASR in " + lfm3_language_name(language) + ".";
}
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language) {
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language,
double audio_length, const std::string & context) {
common_chat_prompt_preset asr_preset;
asr_preset.system = "";
asr_preset.user = "Transcribe audio to text";
@@ -861,6 +868,17 @@ common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates
asr_preset.system = lfm3_asr_system_prompt(language);
asr_preset.user = "";
asr_preset.has_language = true;
} else if (is_vibevoice_asr_template(src)) {
// without the audio length the model does not know where the audio ends and loops
const char * keys = "transcribe it with these keys: Start time, End time, Speaker ID, Content";
if (context.empty()) {
asr_preset.user = string_format("\nThis is a %.2f seconds audio, please %s", audio_length, keys);
} else {
asr_preset.user = string_format("\nThis is a %.2f seconds audio, with extra info: %s\n\nPlease %s", audio_length, context.c_str(), keys);
asr_preset.has_context = true;
}
asr_preset.media_first = true;
asr_preset.needs_audio_length = true;
}
}
@@ -876,7 +894,7 @@ std::string common_chat_get_tts_prompt(const common_chat_templates * chat_templa
{"messages", json::array({{{"role", "user"}, {"content", text}}})},
{"language", language},
{"voice", voice},
});
}, chat_templates->tts_dicts);
}
std::string common_chat_get_speech_system_prompt(const common_chat_templates * chat_templates,
@@ -1007,6 +1025,25 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
LOG_ERR("%s: failed to parse the tts chat template (ignoring it): %s\n", __func__, e.what());
}
}
// dictionaries for tts_dict(), listed by name in tokenizer.tts_dicts
tmpls->tts_dicts = std::make_shared<common_tts_dicts>();
for (int32_t i = 0; i < llama_model_meta_arr_n(model, "tokenizer.tts_dicts"); i++) {
const std::string name = llama_model_meta_arr_str(model, "tokenizer.tts_dicts", i);
const std::string key = "tokenizer.tts_dict." + name + ".keys";
const std::string val_key = "tokenizer.tts_dict." + name + ".values";
const int32_t n = llama_model_meta_arr_n(model, key.c_str());
if (n < 0 || llama_model_meta_arr_n(model, val_key.c_str()) != n) {
LOG_ERR("%s: the tts dictionary '%s' has no value for each of its keys (ignoring it)\n", __func__, name.c_str());
continue;
}
auto & dict = (*tmpls->tts_dicts)[name];
dict.keys.reserve(n);
dict.values.reserve(n);
for (int32_t j = 0; j < n; j++) {
dict.keys.emplace_back(llama_model_meta_arr_str(model, key.c_str(), j));
dict.values.emplace_back(llama_model_meta_arr_str(model, val_key.c_str(), j));
}
}
}
return tmpls;
}
@@ -1427,6 +1464,12 @@ static common_chat_params_init_fn common_chat_template_detect_params_init(const
return common_chat_params_init_qwen3_coder;
}
// Inkling 1 and 2
if (is_inkling_template(src)) {
LOG_DBG("Using specialized template: Inkling\n");
return common_chat_params_init_inkling;
}
return nullptr;
}
@@ -1609,9 +1652,6 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
auto_params.thinking_end_tags = {std::move(end_tag)};
}
}
if (is_inkling2_template(src)) {
common_chat_inkling2_split_media(auto_params.prompt);
}
const auto & arena = auto_params.parser;
LOG_DBG("%s: generated parser:\n%s\n\nparser generation prompt: %s\n", __func__, arena.dump(arena.root()).c_str(), auto_params.generation_prompt.c_str());
return auto_params;
@@ -1962,5 +2002,6 @@ common_json common_chat_templates_get_caps(const common_chat_templates * chat_te
: chat_templates->template_default->caps;
common_json result = caps.to_map();
result["reasoning_effort_levels"] = caps.reasoning_effort_levels;
result["reasoning_effort_rejected"] = caps.reasoning_effort_rejected;
return result;
}
+8 -2
View File
@@ -491,10 +491,16 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
struct common_chat_prompt_preset {
std::string system;
std::string user;
bool has_language = false; // the preset already names the requested language
bool has_language = false; // the preset already names the requested language
bool has_context = false; // the user text already holds the context
bool media_first = false; // the media goes before the user text
bool needs_audio_length = false; // the user text depends on the audio length, call again with it
};
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language = "");
// audio_length: in seconds, 0 if unknown
// context: extra info from the user (e.g. names, hotwords), only used by some presets, see has_context
common_chat_prompt_preset common_chat_get_asr_prompt(const common_chat_templates * chat_templates, const std::string & language = "",
double audio_length = 0.0, const std::string & context = "");
// prompt of the model's "tts" chat template for the text to speak, empty if the model has none
// language and voice are optional, the template defines their defaults
+1
View File
@@ -796,6 +796,7 @@ struct common_params {
std::string tts_lang = "";
std::string tts_voice = "";
std::string tts_speaker_file = "";
std::string tts_speaker_text = "";
bool is_gen_docs = false; // whether we are running inside llama-gen-docs
};
+11 -1
View File
@@ -627,9 +627,14 @@ caps caps_get(jinja::program & prog) {
// levels sharing a render are aliases and the highest one is kept
const auto unknown = render("REASONING_EFFORT_UNKNOWN");
std::vector<std::string> renders;
std::vector<std::string> rejected;
for (const auto & level : candidates) {
const auto rendered = render(level);
if (!rendered || rendered == unknown) {
if (!rendered) {
rejected.push_back(level);
continue;
}
if (rendered == unknown) {
continue;
}
const auto it = std::find(renders.begin(), renders.end(), *rendered);
@@ -640,6 +645,11 @@ caps caps_get(jinja::program & prog) {
result.reasoning_effort_levels.push_back(level);
}
}
// a template that renders no level fails the probe for another reason, so its failures say nothing about the levels
if (!result.reasoning_effort_levels.empty()) {
result.reasoning_effort_rejected = rejected;
}
}
JJ_DEBUG("%s\n", result.to_string().c_str());
+3
View File
@@ -23,6 +23,9 @@ struct caps {
// reasoning effort levels the template renders distinctly, from lowest to highest
std::vector<std::string> reasoning_effort_levels;
// reasoning effort levels the template raises on
std::vector<std::string> reasoning_effort_rejected;
// one of the 2 content capabilities must be true
bool supports_string_content = true;
bool supports_typed_content = false;
-35
View File
@@ -263,41 +263,6 @@ lexer_result lexer::tokenize(const std::string & source) {
char ch = src[pos];
bool is_closing_block = ch == '-' && next_pos_is( {'%', '}'} );
// Check for unary operators
if (!is_closing_block && (ch == '-' || ch == '+')) {
start_pos = pos;
token::type last_token_type = tokens.empty() ? token::eof : tokens.back().t;
if (last_token_type == token::text || last_token_type == token::eof) {
throw lexer_exception(std::string("unexpected character: ") + ch, source, pos);
}
switch (last_token_type) {
case token::identifier:
case token::numeric_literal:
case token::string_literal:
case token::close_paren:
case token::close_square_bracket:
// Part of a binary operator
// a - 1, 1 - 1, true - 1, "apple" - 1, (1) - 1, a[1] - 1
// Continue parsing normally
break;
default: {
// Is part of a unary operator
// (-1), [-1], (1 + -1), not -1, -apple
++pos; // Consume the operator
// Check for numbers following the unary operator
std::string num = consume_numeric();
std::string value = std::string(1, ch) + num;
token::type t = num.empty() ? token::unary_operator : token::numeric_literal;
// JJ_DEBUG("consumed unary operator or numeric literal: '%s'", value.c_str());
tokens.push_back({t, value, start_pos});
continue;
}
}
}
// Try to match one of the tokens in the mapping table
bool matched = false;
for (const auto & [seq, typ] : ordered_mapping_table) {
+1 -2
View File
@@ -39,7 +39,6 @@ struct token {
multiplicative_binary_operator, // * / %
power_operator, // **
comparison_binary_operator, // < > <= >= == !=
unary_operator, // ! - +
comment, // {# ... #}
};
type t;
@@ -74,7 +73,6 @@ static std::string type_to_string(token::type t) {
case token::multiplicative_binary_operator: return "multiplicative_binary_operator";
case token::power_operator: return "power_operator";
case token::comparison_binary_operator: return "comparison_binary_operator";
case token::unary_operator: return "unary_operator";
case token::comment: return "comment";
default: return "unknown";
}
@@ -141,6 +139,7 @@ struct lexer {
{"~", token::additive_binary_operator},
{"**", token::power_operator},
{"*", token::multiplicative_binary_operator},
{"//", token::multiplicative_binary_operator},
{"/", token::multiplicative_binary_operator},
{"%", token::multiplicative_binary_operator},
// Assignment operator
+2 -1
View File
@@ -482,7 +482,8 @@ private:
}
statement_ptr parse_unary_expression() {
if (is(token::unary_operator)) {
// a sign where an operand is expected is unary: -x, 1 + -x, x if c else -1
if (is(token::additive_binary_operator) && peek().value != "~") {
depth_guard guard(*this);
size_t start_pos = current;
auto op = next();
+35 -10
View File
@@ -191,11 +191,30 @@ static value exec_binary_op(context & ctx, const token & op, value left_val, val
if (is_float) {
return mk_val<value_float>(res);
} else {
return mk_val<value_int>(static_cast<int64_t>(res));
return mk_val<value_int>(float_to_int(res));
}
} else if (op.value == "/") {
JJ_DEBUG("Division operation: %f / %f", a, b);
return mk_val<value_float>(a / b);
} else if (op.value == "//") {
// python semantics: round toward -inf
if (is_val<value_int>(left_val) && is_val<value_int>(right_val)) {
const int64_t x = left_val->as_int();
const int64_t y = right_val->as_int();
if (y == 0) {
throw std::runtime_error("integer division by zero");
}
if (y == -1) {
// INT64_MIN / -1 is UB, wrap around instead
return mk_val<value_int>((int64_t) (0 - (uint64_t) x));
}
int64_t q = x / y;
if (x % y != 0 && (x < 0) != (y < 0)) {
q -= 1;
}
return mk_val<value_int>(q);
}
return mk_val<value_float>(std::floor(a / b));
} else if (op.value == "%") {
double rem = std::fmod(a, b);
JJ_DEBUG("Modulo operation: %f %% %f = %f", a, b, rem);
@@ -203,7 +222,7 @@ static value exec_binary_op(context & ctx, const token & op, value left_val, val
if (is_float) {
return mk_val<value_float>(rem);
} else {
return mk_val<value_int>(static_cast<int64_t>(rem));
return mk_val<value_int>(float_to_int(rem));
}
} else if (op.value == "<") {
JJ_DEBUG("Comparison operation: %f < %f is %d", a, b, a < b);
@@ -274,12 +293,17 @@ static value exec_binary_op(context & ctx, const token & op, value left_val, val
// String membership
if (is_val<value_string>(left_val) && is_val<value_string>(right_val)) {
// case: "a" in "abc"
bool member = test_is_in();
if (op.value == "in") {
return mk_val<value_bool>(member);
} else if (op.value == "not in") {
return mk_val<value_bool>(!member);
if (op.value == "in" || op.value == "not in") {
bool member = test_is_in();
return mk_val<value_bool>(op.value == "in" ? member : !member);
}
// case: "a" < "b"; UTF-8 byte order is the same as codepoint order, like python
const std::string a = left_val->as_string().str();
const std::string b = right_val->as_string().str();
if (op.value == "<") { return mk_val<value_bool>(a < b); }
if (op.value == ">") { return mk_val<value_bool>(a > b); }
if (op.value == "<=") { return mk_val<value_bool>(a <= b); }
if (op.value == ">=") { return mk_val<value_bool>(a >= b); }
}
// Value key in object
@@ -787,13 +811,14 @@ value call_statement::execute_impl(context & ctx) const {
}
auto * callee_func = cast_val<value_func>(callee_val);
context caller_ctx(ctx); // new scope for caller execution
// caller() can be saved and called after this scope ended, it owns a snapshot of the variables
auto caller_ctx = std::make_shared<context>(ctx.snapshot());
const func_handler func = [this, caller_ctx = std::move(caller_ctx)](const func_args & args) -> value {
const func_handler func = [this, caller_ctx](const func_args & args) -> value {
if (args.ctx.macro_depth >= JINJA_RECURSIVE_LIMIT) {
throw std::runtime_error("macro calls are nested more than " + std::to_string(JINJA_RECURSIVE_LIMIT) + " levels deep");
}
context block_ctx(caller_ctx); // new scope for block execution
context block_ctx(*caller_ctx); // new scope for block execution
block_ctx.macro_depth = args.ctx.macro_depth + 1;
bind_parameters("caller", this->caller_args, args, block_ctx);
+30 -3
View File
@@ -81,12 +81,38 @@ struct context {
current_time(parent.current_time),
is_get_stats(parent.is_get_stats),
macro_depth(parent.macro_depth),
// inherit variables (for example, when entering a new scope)
env(mk_val<value_object>(*parent.env)) {}
// child scope: own variables only, lookups fall back to the parent (which must outlive this scope)
parent(&parent),
env(mk_val<value_object>()) {
env->has_builtins = false;
}
context(context &&) = default;
// copy with all visible variables flattened in, for a scope that can outlive its parents (caller() of a call block)
context snapshot() const {
std::vector<const context *> chain;
for (const context * c = this; c; c = c->parent) {
chain.push_back(c);
}
context res(*this);
res.parent = nullptr;
for (auto it = chain.rbegin(); it != chain.rend(); ++it) {
for (const auto & kv : (*it)->env->as_ordered_object()) {
res.env->insert(kv.first, kv.second);
}
}
return res;
}
// returns nullptr if not found
value find_val(const value & key) {
return env->find(key);
for (const context * c = this; c; c = c->parent) {
if (value v = c->env->find(key)) {
return v;
}
}
return nullptr;
}
value get_val(const std::string & name) {
@@ -107,6 +133,7 @@ struct context {
}
private:
const context * parent = nullptr;
value_object env;
};
+49 -18
View File
@@ -281,14 +281,12 @@ static value tojson(const func_args & args) {
if (is_val<value_int>(val_indent)) {
indent = static_cast<int>(val_indent->as_int());
}
if (val_sort->as_bool()) { // undefined == false
throw not_implemented_exception("tojson sort_keys=true not implemented");
}
const bool sort_keys = val_sort->as_bool(); // undefined == false
const bool ensure_ascii = val_ascii->as_bool(); // undefined == false
auto separators = (is_val<value_array>(val_separators) ? val_separators : mk_val<value_array>())->as_array();
std::string item_sep = separators.size() > 0 ? separators[0]->as_string().str() : (indent < 0 ? ", " : ",");
std::string key_sep = separators.size() > 1 ? separators[1]->as_string().str() : ": ";
std::string json_str = value_to_json(args.get_pos(0), indent, item_sep, key_sep);
std::string json_str = value_to_json(args.get_pos(0), indent, item_sep, key_sep, sort_keys);
if (ensure_ascii) {
json_str = json_ensure_ascii_preserving_format(json_str);
}
@@ -713,7 +711,7 @@ const func_builtins & value_float_t::get_builtins() const {
}},
{"int", [](const func_args & args) -> value {
args.ensure_vals<value_float>();
int64_t val = static_cast<int64_t>(args.get_pos(0)->as_float());
int64_t val = float_to_int(args.get_pos(0)->as_float());
return mk_val<value_int>(val);
}},
{"float", [](const func_args & args) -> value {
@@ -1101,8 +1099,28 @@ const func_builtins & value_bool_t::get_builtins() const {
return builtins;
}
[[noreturn]] static value array_sum_not_implemented(const func_args &) {
throw not_implemented_exception("Array sum builtin not implemented");
static value array_sum(const func_args & args) {
args.ensure_kwargs({"attribute", "start"});
args.ensure_count(1, 3);
args.ensure_vals<value_array>();
value undef = mk_val<value_undefined>();
value attribute = args.get_kwarg_or_pos("attribute", 1);
value start = args.get_kwarg_or_pos("start", 2);
// same as the + operator: compute in double, int result if all operands are int
bool is_float = is_val<value_float>(start);
double sum = start->is_undefined() ? 0.0 : start->as_float();
for (const auto & item : args.get_pos(0)->as_array()) {
value v = attribute->is_undefined() ? item : get_attribute(item, attribute, undef);
if (v->is_undefined() && !attribute->is_undefined()) {
throw raised_exception("sum: " + item->type() + " has no attribute '" + attribute->as_string().str() + "'");
}
if (!is_val<value_int>(v) && !is_val<value_float>(v)) {
throw raised_exception("sum: unsupported operand type " + v->type());
}
is_float = is_float || is_val<value_float>(v);
sum += v->as_float();
}
return is_float ? value(mk_val<value_float>(sum)) : value(mk_val<value_int>(float_to_int(sum)));
}
const func_builtins & value_array_t::get_builtins() const {
@@ -1171,7 +1189,7 @@ const func_builtins & value_array_t::get_builtins() const {
{"select", select_or_reject<false, false>},
{"rejectattr", select_or_reject<true, true>},
{"reject", select_or_reject<true, false>},
{"sum", array_sum_not_implemented},
{"sum", array_sum},
{"join", [](const func_args & args) -> value {
args.ensure_kwargs({"d", "attribute"});
args.ensure_count(1, 3);
@@ -1656,6 +1674,11 @@ bool value_compare(const value & a, const value & b, value_compare_op op) {
return result;
}
template<>
value value_from_json(const common_json & json_val, bool mark_input) {
return from_json(json_val, mark_input);
}
template<>
void global_from_json(context & ctx, const common_json & json_obj, bool mark_input) {
// printf("global_from_json: %s\n" , json_obj.dump(2).c_str());
@@ -1741,7 +1764,7 @@ static std::string float_json(double val) {
// recursively convert value to JSON string
// TODO: avoid circular references
static void value_to_json_internal(std::ostringstream & oss, const value & val, int curr_lvl, int indent, const std::string_view item_sep, const std::string_view key_sep) {
static void value_to_json_internal(std::ostringstream & oss, const value & val, int curr_lvl, int indent, const std::string_view item_sep, const std::string_view key_sep, bool sort_keys) {
auto indent_str = [indent, curr_lvl]() -> std::string {
return (indent > 0) ? std::string(curr_lvl * indent, ' ') : "";
};
@@ -1788,7 +1811,7 @@ static void value_to_json_internal(std::ostringstream & oss, const value & val,
oss << newline();
for (size_t i = 0; i < arr.size(); ++i) {
oss << indent_str() << (indent > 0 ? std::string(indent, ' ') : "");
value_to_json_internal(oss, arr[i], curr_lvl + 1, indent, item_sep, key_sep);
value_to_json_internal(oss, arr[i], curr_lvl + 1, indent, item_sep, key_sep, sort_keys);
if (i < arr.size() - 1) {
oss << item_sep;
}
@@ -1799,17 +1822,25 @@ static void value_to_json_internal(std::ostringstream & oss, const value & val,
oss << "]";
} else if (is_val<value_object>(val)) {
const auto & obj = val->as_ordered_object(); // IMPORTANT: need to keep exact order
std::vector<std::pair<std::string, value>> items;
items.reserve(obj.size());
for (const auto & pair : obj) {
items.emplace_back(is_val<value_float>(pair.first) ? float_json(pair.first->as_float()) : pair.first->as_string().str(), pair.second);
}
if (sort_keys) {
// same order as python json.dumps(sort_keys=True): utf-8 byte order is code point order
std::stable_sort(items.begin(), items.end(), [](const auto & a, const auto & b) { return a.first < b.first; });
}
oss << "{";
if (!obj.empty()) {
if (!items.empty()) {
oss << newline();
size_t i = 0;
for (const auto & pair : obj) {
for (const auto & [key, item] : items) {
oss << indent_str() << (indent > 0 ? std::string(indent, ' ') : "");
const std::string key = is_val<value_float>(pair.first) ? float_json(pair.first->as_float()) : pair.first->as_string().str();
value_to_json_internal(oss, mk_val<value_string>(key), curr_lvl + 1, indent, item_sep, key_sep);
value_to_json_internal(oss, mk_val<value_string>(key), curr_lvl + 1, indent, item_sep, key_sep, sort_keys);
oss << key_sep;
value_to_json_internal(oss, pair.second, curr_lvl + 1, indent, item_sep, key_sep);
if (i < obj.size() - 1) {
value_to_json_internal(oss, item, curr_lvl + 1, indent, item_sep, key_sep, sort_keys);
if (i < items.size() - 1) {
oss << item_sep;
}
oss << newline();
@@ -1823,9 +1854,9 @@ static void value_to_json_internal(std::ostringstream & oss, const value & val,
}
}
std::string value_to_json(const value & val, int indent, const std::string_view item_sep, const std::string_view key_sep) {
std::string value_to_json(const value & val, int indent, const std::string_view item_sep, const std::string_view key_sep, bool sort_keys) {
std::ostringstream oss;
value_to_json_internal(oss, val, 0, indent, item_sep, key_sep);
value_to_json_internal(oss, val, 0, indent, item_sep, key_sep, sort_keys);
JJ_DEBUG("value_to_json: result=%s", oss.str().c_str());
return oss.str();
}
+21 -3
View File
@@ -105,6 +105,10 @@ struct context; // forward declaration
template<typename T_JSON>
void global_from_json(context & ctx, const T_JSON & json_obj, bool mark_input);
// convert a single JSON value, same rules as global_from_json()
template<typename T_JSON>
value value_from_json(const T_JSON & json_val, bool mark_input);
//
// base value type
//
@@ -199,7 +203,7 @@ value find_builtin_func(context & ctx, const std::string & name, value & input);
value try_builtin_func(context & ctx, const std::string & name, value & input, bool undef_on_missing = false);
value apply_filter_by_name(context & ctx, std::string filter_id, value input, const func_args & args);
std::string value_to_json(const value & val, int indent = -1, const std::string_view item_sep = ", ", const std::string_view key_sep = ": ");
std::string value_to_json(const value & val, int indent = -1, const std::string_view item_sep = ", ", const std::string_view key_sep = ": ", bool sort_keys = false);
// Note: only used for debugging purposes
std::string value_to_string_repr(const value & val);
@@ -233,6 +237,20 @@ struct value_equality {
// primitive value types
//
// double to int64, saturating and with nan as 0; a plain cast is undefined outside the int64 range
inline int64_t float_to_int(double v) {
if (std::isnan(v)) {
return 0;
}
if (v >= 9223372036854775808.0) {
return INT64_MAX;
}
if (v <= -9223372036854775808.0) {
return INT64_MIN;
}
return static_cast<int64_t>(v);
}
struct value_int_t : public value_t {
static constexpr auto kind_id = value_kind::INTEGER;
int64_t val_int;
@@ -240,7 +258,7 @@ struct value_int_t : public value_t {
value_int_t(int64_t v) : value_t(kind_id) {
val_int = v;
val_flt = static_cast<double>(v);
if (static_cast<int64_t>(val_flt) != v) {
if (float_to_int(val_flt) != v) {
val_flt = v < 0 ? -INFINITY : INFINITY;
}
}
@@ -276,7 +294,7 @@ struct value_float_t : public value_t {
double val_flt;
value_float_t(double v) : value_t(kind_id) {
val_flt = v;
val_int = std::isfinite(v) ? static_cast<int64_t>(v) : 0;
val_int = std::isfinite(v) ? float_to_int(v) : 0;
}
virtual std::string type() const override { return "Float"; }
virtual double as_float() const override { return val_flt; }
+165
View File
@@ -0,0 +1,165 @@
#include "parsers.h"
bool is_inkling_template(const std::string & src) {
return src.find("<|message_user|>") != std::string::npos &&
src.find("<|content_image|>") != std::string::npos;
}
// the template makes one message block per content part, but a media marker comes inside the text of a message
// <header><|content_text|>{media}text<|end_message|> --> <header>{media}<|end_message|><header><|content_text|>text<|end_message|>
static void inkling_split_media(common_chat_prompt & prompt) {
const std::string TEXT = "<|content_text|>";
const std::string END = "<|end_message|>";
const std::string MEDIA_BEG = "<__media_";
const std::string MEDIA_END = "__>";
std::vector<common_chat_prompt_part> out;
for (size_t i = 0; i < prompt.parts.size(); i++) {
const auto & part = prompt.parts[i];
const bool in_text_block = part.is_input && part.text.find(MEDIA_BEG) != std::string::npos &&
!out.empty() && !out.back().is_input && string_ends_with(out.back().text, TEXT) &&
i + 1 < prompt.parts.size() && !prompt.parts[i + 1].is_input && string_starts_with(prompt.parts[i + 1].text, END);
const size_t header_pos = in_text_block ? out.back().text.rfind("<|message_") : std::string::npos;
if (header_pos == std::string::npos) {
out.push_back(part);
continue;
}
const std::string header = out.back().text.substr(header_pos, out.back().text.size() - TEXT.size() - header_pos);
std::string rest = part.text;
for (bool first = true; !rest.empty(); first = false) {
const size_t beg = rest.find(MEDIA_BEG);
const size_t end = beg == std::string::npos ? beg : rest.find(MEDIA_END, beg);
const bool is_media = beg == 0 && end != std::string::npos;
const size_t len = is_media ? end + MEDIA_END.size() : (end == std::string::npos ? rest.size() : beg);
if (!first) {
out.push_back({ END + header + (is_media ? "" : TEXT), false });
} else if (is_media) {
// mtmd opens the media block
out.back().text.resize(out.back().text.size() - TEXT.size());
}
out.push_back({ rest.substr(0, len), true });
rest.erase(0, len);
}
}
prompt.parts = std::move(out);
}
// Inkling 1 and 2 - the assistant turn is a list of blocks, all closed by <|end_message|>:
// <|message_model|><|content_thinking|>...<|end_message|>
// <|message_model|><|content_text|>...<|end_message|>
// <|message_model|>NAME<|content_invoke_tool_json|>{"name":"NAME","args":{...}}<|end_message|>
common_chat_params common_chat_params_init_inkling(const common_chat_template & tmpl,
const autoparser::generation_params & inputs) {
common_chat_params data;
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.supports_thinking = true;
inkling_split_media(data.prompt);
const std::string HEADER = "<|message_model|>";
const std::string THINK = "<|content_thinking|>";
const std::string TEXT = "<|content_text|>";
const std::string TOOL_JSON = "<|content_invoke_tool_json|>";
const std::string END = "<|end_message|>";
data.preserved_tokens = { HEADER, THINK, TEXT, TOOL_JSON, END };
data.thinking_start_tag = THINK;
data.thinking_end_tags = { END };
data.message_delimiters = {
{ COMMON_CHAT_ROLE_ASSISTANT, HEADER },
{ COMMON_CHAT_ROLE_USER, "<|message_user|>" },
{ COMMON_CHAT_ROLE_SYSTEM, "<|message_system|>" },
{ COMMON_CHAT_ROLE_TOOL, "<|message_tool|>" },
};
if (inputs.has_continuation()) {
const auto & msg = inputs.continue_msg;
data.generation_prompt.clear();
if (!msg.reasoning_content.empty()) {
data.generation_prompt = HEADER + THINK + msg.reasoning_content;
}
if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
if (!data.generation_prompt.empty()) {
data.generation_prompt += END;
}
data.generation_prompt += HEADER + TEXT + msg.render_content();
}
data.prompt.append(data.generation_prompt);
}
auto has_tools = inputs.tools.is_array() && !inputs.tools.empty();
auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
auto use_tools = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto header = p.literal(HEADER);
auto end = p.literal(END);
auto think = p.literal(THINK);
auto thinking = p.rule("thinking", extract_reasoning ? think + p.reasoning(p.until(END)) + end :
p.atomic(p.content(think)) + p.content(p.until(END) + end));
auto text = p.rule("text", p.literal(TEXT) + p.content(p.until(END)) + end);
if (has_response_format) {
auto response_format = p.rule("response-format", p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)));
return p.zero_or_more(header + thinking) + header + p.literal(TEXT) + response_format + end;
}
auto blocks = p.zero_or_more(header + (thinking | text));
if (!use_tools) {
return blocks;
}
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
const auto & function = tool.at("function");
std::string name = function.at("name");
auto open = p.tool_name(p.literal(name)) + p.literal(TOOL_JSON) + p.literal("{\"name\":" + json(name).dump() + ",\"args\":");
auto args = p.tool_args(p.json());
tool_choice |= p.rule("tool-" + std::to_string(tool_index), p.tool(p.tool_open(open) + args + p.tool_close(p.literal("}")) + end));
});
// tool calls are the last blocks of the turn
auto calls = inputs.parallel_tool_calls ? tool_choice + p.zero_or_more(header + tool_choice) : tool_choice;
auto tool_calls = header + p.trigger_rule("tool-calls", calls);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return blocks + tool_calls;
}
return blocks + p.optional(tool_calls);
});
if (has_response_format || use_tools) {
data.grammar_lazy = !has_response_format && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_REQUIRED;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
});
if (data.grammar_lazy) {
// a header followed by text and not by a content token is a tool call, the grammar starts at the name
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|message_model\\|>([^<])" },
};
// Inkling 1: the first header is in the generation prompt, and a lazy grammar does not get the prompt.
// an output that starts with no content token is a tool call. not for a continuation, it starts with text
if (!inputs.has_continuation()) {
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN_FULL, "[^<]+" });
}
}
}
return data;
}
-46
View File
@@ -1,46 +0,0 @@
#include "parsers.h"
bool is_inkling2_template(const std::string & src) {
return src.find("<|message_user|>") != std::string::npos &&
src.find("<|content_image|>") != std::string::npos;
}
// the template makes one message block per content part, but a media marker comes inside the text of a message
// <header><|content_text|>{media}text<|end_message|> --> <header>{media}<|end_message|><header><|content_text|>text<|end_message|>
void common_chat_inkling2_split_media(common_chat_prompt & prompt) {
const std::string TEXT = "<|content_text|>";
const std::string END = "<|end_message|>";
const std::string MEDIA_BEG = "<__media_";
const std::string MEDIA_END = "__>";
std::vector<common_chat_prompt_part> out;
for (size_t i = 0; i < prompt.parts.size(); i++) {
const auto & part = prompt.parts[i];
const bool in_text_block = part.is_input && part.text.find(MEDIA_BEG) != std::string::npos &&
!out.empty() && !out.back().is_input && string_ends_with(out.back().text, TEXT) &&
i + 1 < prompt.parts.size() && !prompt.parts[i + 1].is_input && string_starts_with(prompt.parts[i + 1].text, END);
const size_t header_pos = in_text_block ? out.back().text.rfind("<|message_") : std::string::npos;
if (header_pos == std::string::npos) {
out.push_back(part);
continue;
}
const std::string header = out.back().text.substr(header_pos, out.back().text.size() - TEXT.size() - header_pos);
std::string rest = part.text;
for (bool first = true; !rest.empty(); first = false) {
const size_t beg = rest.find(MEDIA_BEG);
const size_t end = beg == std::string::npos ? beg : rest.find(MEDIA_END, beg);
const bool is_media = beg == 0 && end != std::string::npos;
const size_t len = is_media ? end + MEDIA_END.size() : (end == std::string::npos ? rest.size() : beg);
if (!first) {
out.push_back({ END + header + (is_media ? "" : TEXT), false });
} else if (is_media) {
// mtmd opens the media block
out.back().text.resize(out.back().text.size() - TEXT.size());
}
out.push_back({ rest.substr(0, len), true });
rest.erase(0, len);
}
}
prompt.parts = std::move(out);
}
+3 -4
View File
@@ -41,10 +41,7 @@ std::string common_chat_template_generation_prompt_impl(
bool is_lfm2_template(const std::string & src);
bool is_inkling2_template(const std::string & src);
// give each media marker of a text block its own message block
void common_chat_inkling2_split_media(common_chat_prompt & prompt);
bool is_inkling_template(const std::string & src);
common_chat_params common_chat_params_init_cohere2moe(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
@@ -60,6 +57,8 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
common_chat_params common_chat_params_init_harmony_xml(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_inkling(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_k2_horizon(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_kimi_k2(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
+1 -1
View File
@@ -10,7 +10,7 @@ set(LLAMA_CHAT_PARSERS_SOURCES
${CMAKE_CURRENT_LIST_DIR}/gigachat-v3.cpp
${CMAKE_CURRENT_LIST_DIR}/gpt-oss.cpp
${CMAKE_CURRENT_LIST_DIR}/harmony-xml.cpp
${CMAKE_CURRENT_LIST_DIR}/inkling2.cpp
${CMAKE_CURRENT_LIST_DIR}/inkling.cpp
${CMAKE_CURRENT_LIST_DIR}/k2-horizon.cpp
${CMAKE_CURRENT_LIST_DIR}/kimi-k2.cpp
${CMAKE_CURRENT_LIST_DIR}/kimi-k3.cpp
+12
View File
@@ -138,6 +138,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"Grok1ForCausalLM": "grok",
"GrokForCausalLM": "grok",
"GroveMoeForCausalLM": "grovemoe",
"HiggsAudio3Model": "higgs_audio",
"HunYuanDenseV1ForCausalLM": "hunyuan",
"HunYuanMoEV1ForCausalLM": "hunyuan",
"HunYuanVLForConditionalGeneration": "hunyuan",
@@ -282,6 +283,9 @@ TEXT_MODEL_MAP: dict[str, str] = {
"Qwen3NextForCausalLM": "qwen",
"Qwen3OmniMoeForConditionalGeneration": "qwen3vl",
"PocketTTSModel": "pockettts",
"Zonos3Model": "zonos3",
"YuE2ForCausalLM": "yue2",
"MiniMaxMusic3ForConditionalGeneration": "minimax_music3",
"Qwen3TTSForConditionalGeneration": "qwen3tts",
"Qwen3VLForConditionalGeneration": "qwen3vl",
"Qwen3VLMoeForConditionalGeneration": "qwen3vl",
@@ -324,6 +328,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"UMT5Model": "t5",
"UltravoxModel": "ultravox",
"UnlimitedOCRForCausalLM": "deepseek",
"VibeVoiceAsrForConditionalGeneration": "vibevoice_asr",
"VLlama3ForCausalLM": "llama",
"VllmDecision2Model": "vllm_decision2",
"VoxtralForConditionalGeneration": "llama",
@@ -333,6 +338,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"XverseForCausalLM": "xverse",
"YoutuForCausalLM": "deepseek",
"YoutuVLForConditionalGeneration": "deepseek",
"fish_qwen3_omni": "fish_s2",
"modeling_grove_moe.GroveMoeForCausalLM": "grovemoe",
"modeling_sarvam_moe.SarvamMoEForCausalLM": "bailingmoe",
}
@@ -356,6 +362,9 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"EmbeddingGemma2Model": "gemma",
"Exaone4_5_ForConditionalGeneration": "exaone",
"FishOmniModel": "drama3",
"Zonos3Model": "zonos3",
"YuE2ForCausalLM": "yue2",
"MiniMaxMusic3ForConditionalGeneration": "minimax_music3",
"Gemma3ForConditionalGeneration": "gemma",
"Gemma3nForConditionalGeneration": "gemma",
"Gemma4ForConditionalGeneration": "gemma",
@@ -370,6 +379,7 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"Granite4VisionForConditionalGeneration": "granite",
"GraniteSpeechForConditionalGeneration": "granite",
"GraniteSpeechPlusForConditionalGeneration": "granite",
"HiggsAudio3Model": "higgs_audio",
"HunYuanVLForConditionalGeneration": "hunyuan",
"Idefics3ForConditionalGeneration": "smolvlm",
"Inkling2ForConditionalGeneration": "inkling2",
@@ -429,9 +439,11 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"Step3p7ForConditionalGeneration": "step3",
"UltravoxModel": "ultravox",
"UnlimitedOCRForCausalLM": "deepseek",
"VibeVoiceAsrForConditionalGeneration": "vibevoice_asr",
"VoxtralForConditionalGeneration": "ultravox",
"VoxtralRealtimeForConditionalGeneration": "ultravox",
"YoutuVLForConditionalGeneration": "youtuvl",
"fish_qwen3_omni": "fish_s2",
}
+8 -3
View File
@@ -1986,8 +1986,10 @@ class TextModel(ModelBase):
logger.info(f"gguf: rope theta swa = {local_rope_theta}")
# Add rope_pattern from no_rope_layers
if no_rope_layers := self.hparams.get("no_rope_layers"):
# Convert 1/0 to bool (1 = use RoPE, 0 = NoPE)
rope_pattern = [bool(x) for x in no_rope_layers]
if len(no_rope_layers) == self.block_count: # "no_rope_layers": [0, 0, 0, 1]
rope_pattern = [bool(x) for x in no_rope_layers]
else: # "no_rope_layers": [3]
rope_pattern = [i not in no_rope_layers for i in range(self.block_count)]
self.gguf_writer.add_rope_pattern(rope_pattern)
logger.info(f"gguf: rope_pattern = {sum(rope_pattern)} RoPE layers / {len(rope_pattern)} total")
if (f_rms_eps := self.find_hparam(["rms_norm_eps", "norm_eps"], optional=True)) is not None:
@@ -3475,7 +3477,7 @@ def get_model_architecture(hparams: dict[str, Any], model_type: ModelType) -> st
# Kimi-K3's text_config reports "KimiLinearForCausalLM", which is the older
# Kimi-Linear-48B architecture and cannot load K3 (no attention residuals,
# latent MoE, situ, ...). Route on the top-level architecture instead.
if model_type == ModelType.TEXT and arch in ("NEOChatModel", "StepVLForConditionalGeneration", "Sarashina2VisionForCausalLM", "Exaone4_5_ForConditionalGeneration", "Step3p7ForConditionalGeneration", "KimiK3ForConditionalGeneration", "KimiK31ForConditionalGeneration", "Nemotron3_5TranscribeForConditionalGeneration"):
if model_type == ModelType.TEXT and arch in ("NEOChatModel", "StepVLForConditionalGeneration", "Sarashina2VisionForCausalLM", "Exaone4_5_ForConditionalGeneration", "Step3p7ForConditionalGeneration", "KimiK3ForConditionalGeneration", "KimiK31ForConditionalGeneration", "Nemotron3_5TranscribeForConditionalGeneration", "HiggsAudio3Model"):
return arch
# if "architectures" is found in the sub-config, use that instead
@@ -3486,6 +3488,9 @@ def get_model_architecture(hparams: dict[str, Any], model_type: ModelType) -> st
# some models (llava_onevision2) only declare the architecture inside text_config
if arch is None and text_config.get("architectures") is not None:
arch = text_config["architectures"][0]
# some configs only have a model_type (fishaudio/s2-pro)
if arch is None:
arch = hparams.get("model_type")
if arch is None:
raise ValueError("Failed to detect model architecture")
return arch
+32 -19
View File
@@ -18,6 +18,30 @@ _TOWER_SHARED = "audio_decoder_mtp_shared.layers."
_TOWER_RESIDUAL = "audio_decoder_mtp.layers."
def load_codec(dir_model) -> dict[str, Tensor]:
sd = torch.load(dir_model / "codec.pth", map_location="cpu", mmap=True, weights_only=True)
return sd.get("state_dict", sd)
def fold_conv(sd: dict[str, Tensor], prefix: str) -> Tensor:
# fold weight norm
if prefix + ".conv.weight" in sd:
return sd[prefix + ".conv.weight"].float()
g = sd[prefix + ".conv.parametrizations.weight.original0"].float()
v = sd[prefix + ".conv.parametrizations.weight.original1"].float()
return g * v / v.norm(dim=(1, 2), keepdim=True)
def fold_wn(g: Tensor, v: Tensor) -> Tensor:
# fold old-style weight norm of a 1x1 conv, returns [out, in]
return (g.float() * v.float() / v.float().norm(dim=(1, 2), keepdim=True)).squeeze(-1)
def fold_snake(alpha: Tensor) -> tuple[Tensor, Tensor]:
alpha = alpha.float().flatten()
return alpha, 1.0 / (alpha + 1e-9)
@ModelBase.register("FishOmniModel")
@ModelBase.example("fishaudio/drama-3")
class Drama3TextModel(Qwen3_5MoeTextModel):
@@ -36,6 +60,7 @@ class Drama3TextModel(Qwen3_5MoeTextModel):
class Drama3MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = False
gen_projector_type = gguf.VisionProjectorType.DRAMA3_GEN
_LAYER_MAP = {
"enorm.weight": (gguf.MODEL_TENSOR.A_GEN_CODE_ENORM, True),
@@ -69,7 +94,7 @@ class Drama3MmprojModel(MmprojModel):
cfg = self.hparams
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.DRAMA3_GEN)
self.gguf_writer.add_clip_gen_audio_projector_type(self.gen_projector_type)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(cfg["dim"])
self.gguf_writer.add_gen_audio_feed_forward_length(cfg["intermediate_size"])
@@ -153,32 +178,20 @@ class Drama3MmprojModel(MmprojModel):
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
# only the DAC decoder, the encoder is for voice cloning (not supported yet)
sd = torch.load(self.dir_model / "codec.pth", map_location="cpu", mmap=True, weights_only=True)
sd = sd.get("state_dict", sd)
sd = load_codec(self.dir_model)
T = gguf.MODEL_TENSOR
def get(name: str) -> Tensor:
return sd[name].float()
def conv(prefix: str) -> Tensor:
# fold weight norm
if prefix + ".conv.weight" in sd:
return get(prefix + ".conv.weight")
g = get(prefix + ".conv.parametrizations.weight.original0")
v = get(prefix + ".conv.parametrizations.weight.original1")
return g * v / v.norm(dim=(1, 2), keepdim=True)
return fold_conv(sd, prefix)
def quant_table(prefix: str) -> Tensor:
# fold out_proj into the codebook, so decoding is a row lookup
g = get(prefix + ".out_proj.weight_g")
v = get(prefix + ".out_proj.weight_v")
w = (g * v / v.norm(dim=(1, 2), keepdim=True)).squeeze(-1)
w = fold_wn(sd[prefix + ".out_proj.weight_g"], sd[prefix + ".out_proj.weight_v"])
return get(prefix + ".codebook.weight") @ w.T + get(prefix + ".out_proj.bias")
def snake(alpha: Tensor) -> tuple[Tensor, Tensor]:
alpha = alpha.flatten()
return alpha, 1.0 / (alpha + 1e-9)
q = "quantizer."
yield (self.format_tensor_name(T.A_GEN_WAV_QUANT_FIRST_CB), quant_table(q + "semantic_quantizer.quantizers.0"))
n_rest = sum(1 for k in sd if k.startswith(q + "quantizer.quantizers.") and k.endswith(".codebook.weight"))
@@ -228,7 +241,7 @@ class Drama3MmprojModel(MmprojModel):
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_ENTRY, suffix=".bias"), get(d + "0.conv.bias"))
for bid in range(4):
p = f"{d}{bid + 1}.block."
a, b = snake(get(p + "0.alpha"))
a, b = fold_snake(get(p + "0.alpha"))
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_SNAKE, bid, suffix=".alpha"), a)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_SNAKE, bid, suffix=".beta"), b)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_CONV, bid, suffix=".weight"), conv(p + "1"))
@@ -236,7 +249,7 @@ class Drama3MmprojModel(MmprojModel):
for xid in range(3):
r = f"{p}{xid + 2}.block."
for act, tensor in (("0", T.A_GEN_WAV_DAC_RES_ACT1), ("2", T.A_GEN_WAV_DAC_RES_ACT2)):
a, b = snake(get(r + act + ".alpha"))
a, b = fold_snake(get(r + act + ".alpha"))
name = gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
yield (name + ".alpha", a)
yield (name + ".beta", b)
@@ -244,7 +257,7 @@ class Drama3MmprojModel(MmprojModel):
name = gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
yield (name + ".weight", conv(r + cv))
yield (name + ".bias", get(r + cv + ".conv.bias"))
a, b = snake(get(d + "5.alpha"))
a, b = fold_snake(get(d + "5.alpha"))
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_SNAKE, suffix=".alpha"), a)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_SNAKE, suffix=".beta"), b)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_CONV, suffix=".weight"), conv(d + "6"))
+276
View File
@@ -0,0 +1,276 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, gguf
from .drama3 import Drama3MmprojModel, fold_conv, fold_snake, fold_wn, load_codec
from .qwen import Qwen3Model
# Fish Audio S2 Pro (dual-AR)
# text model: Qwen3 backbone (slow AR), samples the semantic code of each frame
# mmproj: fast AR code predictor (other codebooks) and Fish DAC decoder (codec.pth, same as drama3)
# residual codebook size of codec.pth, the fast AR head is cut to it
_N_RESIDUAL_CODES = 1024
# codec encoder strides (encoder_rates of the codec config), the quantizer then downsamples 2x2
_CODEC_ENC_STRIDES = (2, 4, 8, 8)
# codec encoder transformers, both 1024 wide with 16 heads of 64
# generic audio layers a.blk.0..3: last encoder block, a.blk.4..11: quantizer pre_module
_N_ENC_TFM_LAYERS = 4
_N_PRE_TFM_LAYERS = 8
_CODEC_HEAD_DIM = 64
def _load_hparams(dir_model: Path) -> dict[str, Any]:
hparams = ModelBase.load_hparams(dir_model, is_mistral_format=False)
cfg = hparams["text_config"]
hparams["text_config"] = {
"hidden_size": cfg["dim"],
"num_hidden_layers": cfg["n_layer"],
"num_attention_heads": cfg["n_head"],
"num_key_value_heads": cfg["n_local_heads"],
"head_dim": cfg["head_dim"],
"intermediate_size": cfg["intermediate_size"],
"rms_norm_eps": cfg["norm_eps"],
"rope_theta": cfg["rope_base"],
"max_position_embeddings": cfg["max_seq_len"],
"vocab_size": cfg["vocab_size"],
"tie_word_embeddings": cfg["tie_word_embeddings"],
}
return hparams
def _neox_rows(w: Tensor, head_dim: int) -> Tensor:
# checkpoint uses interleaved RoPE, reorder the rows of each head so NEOX RoPE gives the same result
order = torch.cat([torch.arange(0, head_dim, 2), torch.arange(1, head_dim, 2)])
return w.reshape(-1, head_dim, *w.shape[1:])[:, order].reshape(w.shape)
@ModelBase.register("fish_qwen3_omni")
@ModelBase.example("fishaudio/s2-pro")
class FishS2TextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
super().__init__(dir_model, *args, hparams=_load_hparams(dir_model), **kwargs)
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# the rest is mapped by tensor_mapping.py
name = name.removeprefix("text_model.")
T = gguf.MODEL_TENSOR
head_dim = self.hparams["head_dim"]
if name.endswith(".attention.wqkv.weight"):
n_q = self.hparams["num_attention_heads"] * head_dim
n_kv = self.hparams["num_key_value_heads"] * head_dim
q, k, v = data_torch.split([n_q, n_kv, n_kv], dim=0)
yield (self.format_tensor_name(T.ATTN_Q, bid), _neox_rows(q, head_dim))
yield (self.format_tensor_name(T.ATTN_K, bid), _neox_rows(k, head_dim))
yield (self.format_tensor_name(T.ATTN_V, bid), v)
return
if name.endswith(".attention.q_norm.weight"):
yield (self.format_tensor_name(T.ATTN_Q_NORM, bid), _neox_rows(data_torch, head_dim))
return
if name.endswith(".attention.k_norm.weight"):
yield (self.format_tensor_name(T.ATTN_K_NORM, bid), _neox_rows(data_torch, head_dim))
return
yield from super().modify_tensors(data_torch, name, bid)
@ModelBase.register("fish_qwen3_omni")
@ModelBase.example("fishaudio/s2-pro")
class FishS2MmprojModel(Drama3MmprojModel):
has_audio_encoder = True # codec encoder, for voice cloning
gen_projector_type = gguf.VisionProjectorType.FISH_S2_GEN
_FAST_AR_MAP = {
"attention_norm.weight": gguf.MODEL_TENSOR.A_GEN_CODE_ATTN_NORM,
"attention.wo.weight": gguf.MODEL_TENSOR.A_GEN_CODE_ATTN_OUT,
"ffn_norm.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_NORM,
"feed_forward.w1.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_GATE,
"feed_forward.w3.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_UP,
"feed_forward.w2.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_DOWN,
}
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
super().__init__(dir_model, *args, hparams=_load_hparams(dir_model), **kwargs)
def get_audio_config(self) -> dict[str, Any] | None:
cfg = dict(self.global_config["audio_decoder_config"])
cfg["num_hidden_layers"] = cfg["n_layer"]
return cfg
def set_gguf_parameters(self):
super().set_gguf_parameters()
# codec encoder: raw 44.1 kHz waveform -> backbone input rows, other hparams are hardcoded in tools/mtmd
self.gguf_writer.add_clip_has_audio_encoder(True)
self.gguf_writer.add_clip_audio_projector_type(gguf.VisionProjectorType.FISH_S2_SPKENC)
self.gguf_writer.add_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_audio_num_mel_bins(1)
self.gguf_writer.add_audio_block_count(_N_ENC_TFM_LAYERS + _N_PRE_TFM_LAYERS)
self.gguf_writer.add_audio_embedding_length(1024)
self.gguf_writer.add_audio_feed_forward_length(3072)
self.gguf_writer.add_audio_head_count(1024 // _CODEC_HEAD_DIM)
self.gguf_writer.add_audio_attention_layernorm_eps(1e-5)
def tensor_force_quant(self, name, new_name, bid, n_dims):
# the VQ picks the nearest codebook entry, small errors in the encoder change the codes
if new_name.startswith("a.codec."):
return gguf.GGMLQuantizationType.F32
# codec transformers: F16 gives codes closer to the reference than BF16
if new_name.startswith("a.blk.") and n_dims == 2:
return gguf.GGMLQuantizationType.F16
return super().tensor_force_quant(name, new_name, bid, n_dims)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name = item[0]
if name.startswith("audio_decoder.") or name == "text_model.model.embeddings.weight":
return item
return None
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
T = gguf.MODEL_TENSOR
cfg = self.hparams
# already named by generate_extra_tensors()
if name.startswith("a."):
yield (name, data_torch)
return
if name == "text_model.model.embeddings.weight":
# semantic band of the backbone embeddings
begin = self.global_config["semantic_start_token_id"]
yield (self.format_tensor_name(T.A_GEN_CODE_OUT_EMBD), data_torch[begin:begin + cfg["vocab_size"]])
return
if name == "audio_decoder.codebook_embeddings.weight":
# backbone input of a speech row, one table per codebook
yield (self.format_tensor_name(T.A_GEN_CODE_EMBD), data_torch.reshape(cfg["num_codebooks"], -1, data_torch.shape[-1]))
return
if name == "audio_decoder.embeddings.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_TOK_EMBD), data_torch)
return
if name == "audio_decoder.output.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_HEAD), data_torch[:_N_RESIDUAL_CODES])
return
if name == "audio_decoder.norm.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_OUTPUT_NORM), data_torch)
return
assert bid is not None and name.startswith(f"audio_decoder.layers.{bid}.")
key = name.removeprefix(f"audio_decoder.layers.{bid}.")
if key == "attention.wqkv.weight":
head_dim = cfg["head_dim"]
n_q = cfg["n_head"] * head_dim
n_kv = cfg["n_local_heads"] * head_dim
q, k, v = data_torch.split([n_q, n_kv, n_kv], dim=0)
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_Q, bid), _neox_rows(q, head_dim))
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_K, bid), _neox_rows(k, head_dim))
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_V, bid), v)
return
yield (self.format_tensor_name(self._FAST_AR_MAP[key], bid), data_torch)
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
yield from super().generate_extra_tensors()
sd = load_codec(self.dir_model)
T = gguf.MODEL_TENSOR
def get(name: str) -> Tensor:
return sd[name].float()
def tn(tensor: gguf.MODEL_TENSOR, bid: int | None = None, xid: int | None = None) -> str:
return gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
def conv_wb(tensor: gguf.MODEL_TENSOR, prefix: str, bid: int | None = None, xid: int | None = None, stride: int = 1):
# taps-major [K / stride, OC, stride * IC], see conv() in tools/mtmd/models/fish-s2-spkenc.cpp
w = fold_conv(sd, prefix)
n_out, n_in, k = w.shape
w = w.reshape(n_out, n_in, k // stride, stride).permute(2, 0, 3, 1).reshape(k // stride, n_out, stride * n_in)
yield (tn(tensor, bid, xid) + ".weight", w)
yield (tn(tensor, bid, xid) + ".bias", get(prefix + ".conv.bias"))
def snake_ab(tensor: gguf.MODEL_TENSOR, name: str, bid: int | None = None, xid: int | None = None):
a, b = fold_snake(get(name))
yield (tn(tensor, bid, xid) + ".alpha", a)
yield (tn(tensor, bid, xid) + ".beta", b)
def tfm_layers(prefix: str, n_layer: int, offset: int):
for il in range(n_layer):
p = f"{prefix}layers.{il}."
bid = offset + il
wq, wk, wv = get(p + "attention.wqkv.weight").chunk(3, dim=0)
yield (self.format_tensor_name(T.A_ENC_ATTN_Q, bid), _neox_rows(wq, _CODEC_HEAD_DIM))
yield (self.format_tensor_name(T.A_ENC_ATTN_K, bid), _neox_rows(wk, _CODEC_HEAD_DIM))
yield (self.format_tensor_name(T.A_ENC_ATTN_V, bid), wv)
yield (self.format_tensor_name(T.A_ENC_OUTPUT, bid), get(p + "attention.wo.weight"))
yield (self.format_tensor_name(T.A_ENC_INPUT_NORM, bid), get(p + "attention_norm.weight"))
yield (self.format_tensor_name(T.A_ENC_ATTN_SCALE, bid), get(p + "attention_layer_scale.gamma"))
yield (self.format_tensor_name(T.A_ENC_OUTPUT_NORM, bid), get(p + "ffn_norm.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_GATE, bid), get(p + "feed_forward.w1.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_UP, bid), get(p + "feed_forward.w3.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_DOWN, bid), get(p + "feed_forward.w2.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_SCALE_LS, bid), get(p + "ffn_layer_scale.gamma"))
# encoder: entry conv, 4x (3 residual units, Snake, strided conv), Snake, post conv
e = "encoder.block."
yield from conv_wb(T.A_CODEC_ENC_ENTRY, e + "0")
for bid, stride in enumerate(_CODEC_ENC_STRIDES):
p = f"{e}{bid + 1}.block."
for xid in range(3):
r = f"{p}{xid}.block."
yield from snake_ab(T.A_CODEC_ENC_RES_ACT1, r + "0.alpha", bid, xid)
yield from conv_wb(T.A_CODEC_ENC_RES_CONV1, r + "1", bid, xid)
yield from snake_ab(T.A_CODEC_ENC_RES_ACT2, r + "2.alpha", bid, xid)
yield from conv_wb(T.A_CODEC_ENC_RES_CONV2, r + "3", bid, xid)
yield from snake_ab(T.A_CODEC_ENC_SNAKE, p + "3.alpha", bid)
yield from conv_wb(T.A_CODEC_ENC_CONV, p + "4", bid, stride=stride)
# the last block ends with a windowed transformer
yield from tfm_layers(f"{e}4.block.5.", _N_ENC_TFM_LAYERS, 0)
yield (self.format_tensor_name(T.A_POST_NORM), get(f"{e}4.block.5.norm.weight"))
yield from snake_ab(T.A_CODEC_ENC_POST_SNAKE, e + "5.alpha")
yield from conv_wb(T.A_CODEC_ENC_POST_CONV, e + "6")
# quantizer: 2x (strided conv + ConvNeXt), windowed transformer, then semantic + residual VQ
q = "quantizer."
for bid in range(2):
p = f"{q}downsample.{bid}."
yield from conv_wb(T.A_CODEC_DOWN_CONV, p + "0", bid, stride=2)
# depthwise: one [C] row per tap
yield (tn(T.A_CODEC_DOWN_DWCONV, bid) + ".weight", fold_conv(sd, p + "1.dwconv").squeeze(1).T.contiguous())
yield (tn(T.A_CODEC_DOWN_DWCONV, bid) + ".bias", get(p + "1.dwconv.conv.bias"))
for key, tensor in (("norm", T.A_CODEC_DOWN_NORM), ("pwconv1", T.A_CODEC_DOWN_PW1), ("pwconv2", T.A_CODEC_DOWN_PW2)):
yield (tn(tensor, bid) + ".weight", get(f"{p}1.{key}.weight"))
yield (tn(tensor, bid) + ".bias", get(f"{p}1.{key}.bias"))
yield (tn(T.A_CODEC_DOWN_GAMMA, bid), get(p + "1.gamma"))
yield from tfm_layers(q + "pre_module.", _N_PRE_TFM_LAYERS, _N_ENC_TFM_LAYERS)
yield (self.format_tensor_name(T.A_CODEC_PRE_NORM), get(q + "pre_module.norm.weight"))
# nearest codebook entry by cosine similarity, see VectorQuantize.decode_latents()
def vq(prefix: str) -> tuple[Tensor, Tensor, Tensor]:
w = fold_wn(sd[prefix + ".in_proj.weight_g"], sd[prefix + ".in_proj.weight_v"])
cb = torch.nn.functional.normalize(get(prefix + ".codebook.weight"), dim=-1)
return w, get(prefix + ".in_proj.bias"), cb
w, b, cb = vq(q + "semantic_quantizer.quantizers.0")
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_IN, suffix=".weight"), w)
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_IN, suffix=".bias"), b)
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_CB), cb)
n_rest = self.hparams["num_codebooks"] - 1
rest = [vq(f"{q}quantizer.quantizers.{i}") for i in range(n_rest)]
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_IN, suffix=".weight"), torch.stack([r[0] for r in rest]))
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_IN, suffix=".bias"), torch.stack([r[1] for r in rest]))
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_CB), torch.stack([r[2] for r in rest]))
+3 -1
View File
@@ -552,7 +552,9 @@ class GraniteHybridModel(Mamba2Model, GraniteMoeModel):
or not self._ssm_layers
)
self.gguf_writer.add_rope_scaling_finetuned(use_rope)
if not use_rope:
# Set context length to an arbitrary large value if rope is not used AND no previous value is present.
# https://huggingface.co/ibm-granite/granite-4.0-h-micro-GGUF/discussions/1
if not use_rope and "max_position_embeddings" not in self.hparams:
self.gguf_writer.add_context_length(2**20)
## Validation ##
+74
View File
@@ -0,0 +1,74 @@
from __future__ import annotations
from typing import Any, Callable, TYPE_CHECKING
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, gguf
from .qwen import Qwen3Model
from .ultravox import WhisperEncoderModel
# plain ChatML as built by transcribe.py upstream, the Qwen3 template renders past assistant turns differently from the last one
HIGGS_AUDIO_CHAT_TEMPLATE = (
"{%- for message in messages -%}"
"{{- '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>\\n' -}}"
"{%- endfor -%}"
"{%- if add_generation_prompt -%}"
"{{- '<|im_start|>assistant\\n' -}}"
"{%- if enable_thinking is defined and enable_thinking is false -%}"
"{{- '<think>\\n\\n</think>\\n\\n' -}}"
"{%- endif -%}"
"{%- endif -%}"
)
@ModelBase.register("HiggsAudio3Model")
@ModelBase.example("bosonai/higgs-audio-v3-stt")
class HiggsAudioModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
def set_vocab(self):
super().set_vocab()
self.gguf_writer.add_chat_template(HIGGS_AUDIO_CHAT_TEMPLATE)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, gen = item
# the audio_* tensors are dropped by TextModel, rename the text head before that
if name == "audio_decoder_proj.text_lm_head.weight":
name = "lm_head.weight"
else:
name = "model." + name
return super().filter_tensors((name, gen))
@ModelBase.register("HiggsAudio3Model")
class HiggsAudioWhisperEncoderModel(WhisperEncoderModel):
has_vision_encoder = False
has_audio_encoder = True
def get_audio_config(self) -> dict[str, Any] | None:
return self.global_config["audio_encoder_config"]
def tensor_force_quant(self, name, new_name, bid, n_dims):
# depthwise conv kernel, same as the whisper convs
if name == "audio_encoder_proj.temporal.weight":
return gguf.GGMLQuantizationType.F16
return super().tensor_force_quant(name, new_name, bid, n_dims)
def set_gguf_parameters(self):
assert self.global_config["projector_type"] == "mlp" and self.global_config["projector_temporal_downsample"] == 2
assert self.global_config["audio_adapter_type"] == "stack" and not self.global_config.get("duplicate_audio_triplet", False)
super().set_gguf_parameters()
self.gguf_writer.add_clip_projector_type(gguf.VisionProjectorType.HIGGS_AUDIO)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, _ = item
if not name.startswith(("audio_tower.", "audio_encoder_proj.")):
return None
return super().filter_tensors(item)
+4
View File
@@ -302,6 +302,10 @@ class Inkling2VisionModel(MmprojModel):
w.add_vision_attention_layernorm_eps(config["layer_norm_eps"])
w.add_vision_spatial_merge_size(config["spatial_merge_size"])
def dequant_model(self):
# vision tensors are BF16, only the text experts are quantized
return
@classmethod
def filter_tensors(cls, item):
name, gen = item
+285
View File
@@ -0,0 +1,285 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, MmprojModel, gguf
from .drama3 import fold_snake
from .qwen import Qwen3Model
# MiniMax Music 3, diffusers layout: one checkpoint per component
# text model: the global LLM (language_model/), samples the semantic code of each frame
# mmproj: RVQ depth decoder, condition encoder, flow-matching DiT and DAC decoder (vocoder)
LM_DIR = "language_model"
TOK_DIR = "tokenizer"
MMPROJ_DIRS = ("rvq_depth_decoder", "condition_encoder", "transformer", "vocoder")
# token ids of the checkpoint, see encoders.py of the diffusers pipeline
AUDIO_START = 151669
AUDIO_END = 151670
AUDIO_CODE_OFFSET = 151675
N_SEMANTIC = 16384
def _load_hparams(dir_model: Path) -> dict[str, Any]:
return ModelBase.load_hparams(dir_model / LM_DIR, is_mistral_format=False)
def _index_dirs(model: ModelBase, dirs: Iterable[str], keep: Callable[[str], bool]) -> dict[str, Callable[[], Tensor]]:
# the names get the component dir as prefix
root = model.dir_model
tensors: dict[str, Callable[[], Tensor]] = {}
try:
for d in dirs:
model.dir_model = root / d
prefix = "model" if d == LM_DIR else "diffusion_pytorch_model"
for name, gen in model.index_local_tensors(prefix).items():
if keep(f"{d}.{name}"):
tensors[f"{d}.{name}"] = gen
finally:
model.dir_model = root
return tensors
def _fold_wn(g: Tensor, v: Tensor) -> Tensor:
# torch weight norm, over every dim but 0
g, v = g.float(), v.float()
return g * v / v.flatten(1).norm(dim=1).view(-1, *[1] * (v.dim() - 1))
@ModelBase.register("MiniMaxMusic3ForConditionalGeneration")
class MiniMaxMusic3TextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
super().__init__(dir_model, *args, hparams=_load_hparams(dir_model), **kwargs)
def index_tensors(self, remote_hf_model_id: str | None = None) -> dict[str, Callable[[], Tensor]]:
tensors = _index_dirs(self, [LM_DIR], lambda _: True)
return {name.removeprefix(LM_DIR + "."): gen for name, gen in tensors.items()}
def set_gguf_parameters(self):
super().set_gguf_parameters()
# the codes are sampled with classifier-free guidance, the uncond prompt runs as a second sequence
self.gguf_writer.add_classifier_free_guidance(True)
def set_vocab(self):
root = self.dir_model
self.dir_model = root / TOK_DIR
try:
tokens, toktypes, tokpre = self.get_vocab_base()
special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=True)
finally:
self.dir_model = root
# the semantic codes are not in the tokenizer, name them for debugging
for i in range(N_SEMANTIC):
tokens[AUDIO_CODE_OFFSET + i] = f"<|audio_code_{i}|>"
toktypes[AUDIO_CODE_OFFSET + i] = gguf.TokenType.CONTROL
# the generation ends with <|audio_end|>, the chat template of the tokenizer is not used
template_path = Path(__file__).parent.parent / "models" / "templates" / "MiniMaxAI-MiniMax-Music3-tts.jinja"
with open(template_path, encoding="utf-8") as f:
special_vocab.chat_template = [{"name": "tts", "template": f.read()}]
special_vocab.special_token_ids["eos"] = AUDIO_END
self.gguf_writer.add_tokenizer_model("gpt2")
self.gguf_writer.add_tokenizer_pre(tokpre)
self.gguf_writer.add_token_list(tokens)
self.gguf_writer.add_token_types(toktypes)
special_vocab.add_to_gguf(self.gguf_writer)
@ModelBase.register("MiniMaxMusic3ForConditionalGeneration")
class MiniMaxMusic3MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = False
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
hparams = ModelBase.load_hparams(dir_model, is_mistral_format=False)
hparams["text_config"] = _load_hparams(dir_model)
for d in MMPROJ_DIRS:
hparams[d] = ModelBase.load_hparams(dir_model / d, is_mistral_format=False)
super().__init__(dir_model, *args, hparams=hparams, **kwargs)
def get_audio_config(self) -> dict[str, Any] | None:
# the generic gen layers are the depth decoder
return self.global_config["rvq_depth_decoder"]
def index_tensors(self, remote_hf_model_id: str | None = None) -> dict[str, Callable[[], Tensor]]:
# from the backbone, only the embeddings of the semantic codes
return _index_dirs(self, (LM_DIR, *MMPROJ_DIRS),
lambda name: not name.startswith(LM_DIR + ".") or name == f"{LM_DIR}.model.embed_tokens.weight")
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
return item
def set_gguf_parameters(self):
hp = self.hparams
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.MINIMAX_MUSIC3_GEN)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(hp["hidden_size"])
self.gguf_writer.add_gen_audio_feed_forward_length(hp["intermediate_size"])
self.gguf_writer.add_gen_audio_block_count(hp["num_layers"])
self.gguf_writer.add_gen_audio_head_count(hp["num_attention_heads"])
self.gguf_writer.add_gen_audio_attention_layernorm_eps(1e-6)
# the DiT, the condition encoder and the vocoder hparams come from the tensor shapes or are hard-coded in tools/mtmd
def tensor_force_quant(self, name, new_name, bid, n_dims):
# only the transformer layers follow --outtype, the convs and the small projections stay F32
if n_dims >= 2 and new_name.startswith(("a.gen.code.blk.", "a.gen.flow.blk.")):
return super().tensor_force_quant(name, new_name, bid, n_dims)
if new_name.startswith(("a.gen.code.head", "a.gen.code.embd", "a.gen.code.out_embd", "a.gen.code.proj_in")):
return super().tensor_force_quant(name, new_name, bid, n_dims)
return gguf.GGMLQuantizationType.F32
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
if name.startswith("a."):
yield (name, data_torch)
return
comp, key = name.split(".", 1)
if comp == LM_DIR:
yield ("a.gen.code.out_embd.weight", data_torch[AUDIO_CODE_OFFSET:AUDIO_CODE_OFFSET + N_SEMANTIC])
elif comp == "rvq_depth_decoder":
yield from self._depth(data_torch, key)
elif comp == "condition_encoder":
yield from self._cond(data_torch, key)
elif comp == "transformer":
yield from self._dit(data_torch, key)
# the vocoder is folded in generate_extra_tensors()
def _depth(self, t: Tensor, key: str) -> Iterable[tuple[str, Tensor]]:
p = "a.gen.code"
n_cb = self.hparams["num_codebooks"] - 1
if key == "audio_embeddings.weight":
# one table per residual codebook
yield (f"{p}.embd.weight", t.reshape(n_cb, -1, t.shape[-1]))
elif key.startswith("audio_heads."):
if key == "audio_heads.0.weight":
heads = [self._tensor(f"rvq_depth_decoder.audio_heads.{i}.weight") for i in range(n_cb)]
yield (f"{p}.head.weight", torch.stack(heads))
elif key == "projection.weight":
yield (f"{p}.proj_in.weight", t)
elif key == "pos_embedding.weight":
yield (f"{p}.position_embd.weight", t)
elif key == "norm.weight":
yield (f"{p}.post_ln.weight", t)
else:
_, il, rest = key.split(".", 2)
layer = {
"input_layernorm.weight": "ln1.weight",
"attn.to_q.weight": "attn_q.weight",
"attn.to_k.weight": "attn_k.weight",
"attn.to_v.weight": "attn_v.weight",
"attn.to_out.weight": "attn_out.weight",
"post_attention_layernorm.weight": "ln2.weight",
"gate_proj.weight": "ffn_gate.weight",
"up_proj.weight": "ffn_up.weight",
"down_proj.weight": "ffn_down.weight",
}
yield (f"{p}.blk.{il}.{layer[rest]}", t)
def _cond(self, t: Tensor, key: str) -> Iterable[tuple[str, Tensor]]:
p = "a.gen.cond"
if key == "layer_weight_logits":
# softmax mix of the 8 hidden states of a frame, with the scale folded in
scale = self._tensor("condition_encoder.layer_scale").float()
yield (f"{p}.layer_w", torch.softmax(t.float(), dim=0) * scale)
elif key == "proj.weight":
yield (f"{p}.proj.weight", t)
elif key == "proj.bias":
yield (f"{p}.proj.bias", t)
def _dit(self, t: Tensor, key: str) -> Iterable[tuple[str, Tensor]]:
p = "a.gen.flow"
if key.startswith("transformer_blocks."):
_, il, rest = key.split(".", 2)
q = f"{p}.blk.{il}"
if rest.startswith("ff_in."):
# value half first, then the silu gate half
suffix = rest.removeprefix("ff_in.")
up, gate = t.chunk(2, dim=0)
yield (f"{q}.ffn_up.{suffix}", up)
yield (f"{q}.ffn_gate.{suffix}", gate)
return
layer = {
"norm1.weight": "ln1.weight",
"norm1.bias": "ln1.bias",
"attn.to_q.weight": "attn_q.weight",
"attn.to_k.weight": "attn_k.weight",
"attn.to_v.weight": "attn_v.weight",
"attn.to_out.0.weight": "attn_out.weight",
"norm2.weight": "ln2.weight",
"norm2.bias": "ln2.bias",
"ff_out.weight": "ffn_down.weight",
"ff_out.bias": "ffn_down.bias",
}
yield (f"{q}.{layer[rest]}", t)
return
if key == "time_proj.weight":
# random Fourier features, 2 * pi folded in
yield (f"{p}.time_freq", t.float().flatten() * 2.0 * torch.pi)
return
out = {
"time_embed.linear_1.weight": "time_in.weight",
"time_embed.linear_1.bias": "time_in.bias",
"time_embed.linear_2.weight": "time_out.weight",
"time_embed.linear_2.bias": "time_out.bias",
"preprocess_conv.weight": "pre.weight",
"proj_in.weight": "proj_in.weight",
"proj_out.weight": "proj_out.weight",
"postprocess_conv.weight": "post.weight",
}[key]
yield (f"{p}.{out}", t.squeeze(-1) if key.endswith("conv.weight") else t)
def _tensor(self, name: str) -> Tensor:
# read directly, without the dtype handling of prepare_tensors()
return self.model_tensors[name]().float()
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
# DAC decoder, same layout as the yue2 VAE (a.gen.vae.*), with the snake and the weight norm folded
def get(key: str) -> Tensor:
return self._tensor(f"vocoder.{key}")
def wn(key: str) -> Tensor:
return _fold_wn(get(key + ".weight_g"), get(key + ".weight_v"))
def snake(dst: str, key: str) -> Iterable[tuple[str, Tensor]]:
a, b = fold_snake(get(key + ".alpha"))
yield (f"{dst}.alpha", a)
yield (f"{dst}.beta", b)
p = "a.gen.vae"
yield (f"{p}.in_proj.weight", get("dec_in_proj.weight"))
yield (f"{p}.in_proj.bias", get("dec_in_proj.bias"))
yield (f"{p}.conv_in.weight", wn("conv_in"))
yield (f"{p}.conv_in.bias", get("conv_in.bias"))
n_blk = len(self.global_config["vocoder"]["upsampling_ratios"])
for b in range(n_blk):
s = f"blocks.{b}"
yield from snake(f"{p}.blk.{b}.snake", f"{s}.snake1")
yield (f"{p}.blk.{b}.up.weight", wn(f"{s}.conv_t1"))
yield (f"{p}.blk.{b}.up.bias", get(f"{s}.conv_t1.bias"))
for r in range(3):
rs = f"{s}.res_unit{r + 1}"
yield from snake(f"{p}.blk.{b}.res.{r}.snake1", f"{rs}.snake1")
yield (f"{p}.blk.{b}.res.{r}.conv1.weight", wn(f"{rs}.conv1"))
yield (f"{p}.blk.{b}.res.{r}.conv1.bias", get(f"{rs}.conv1.bias"))
yield from snake(f"{p}.blk.{b}.res.{r}.snake2", f"{rs}.snake2")
yield (f"{p}.blk.{b}.res.{r}.conv2.weight", wn(f"{rs}.conv2"))
yield (f"{p}.blk.{b}.res.{r}.conv2.bias", get(f"{rs}.conv2.bias"))
yield from snake(f"{p}.post_snake", "snake_out")
yield (f"{p}.conv_out.weight", wn("conv_out"))
yield (f"{p}.conv_out.bias", get("conv_out.bias"))
+157
View File
@@ -0,0 +1,157 @@
from __future__ import annotations
from typing import Any, Callable, Iterable, TYPE_CHECKING
if TYPE_CHECKING:
from torch import Tensor
from .base import MmprojModel, ModelBase, gguf
from .qwen import Qwen2Model
# the upstream chat_template.jinja is a processor template (list content, audio parts), use plain ChatML with its default system prompt
VIBEVOICE_ASR_CHAT_TEMPLATE = (
"{%- if messages[0]['role'] == 'system' -%}"
"{{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' -}}"
"{%- set loop_messages = messages[1:] -%}"
"{%- else -%}"
"{{- '<|im_start|>system\\nYou are a helpful assistant that transcribes audio input into text output in JSON format.<|im_end|>\\n' -}}"
"{%- set loop_messages = messages -%}"
"{%- endif -%}"
"{%- for message in loop_messages -%}"
"{{- '<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>\\n' -}}"
"{%- endfor -%}"
"{%- if add_generation_prompt -%}"
"{{- '<|im_start|>assistant\\n' -}}"
"{%- endif -%}"
)
@ModelBase.register("VibeVoiceAsrForConditionalGeneration")
@ModelBase.example("microsoft/VibeVoice-ASR-HF")
class VibeVoiceAsrModel(Qwen2Model):
model_arch = gguf.MODEL_ARCH.QWEN2
def set_vocab(self):
super().set_vocab()
self.gguf_writer.add_chat_template(VIBEVOICE_ASR_CHAT_TEMPLATE)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, _ = item
if not name.startswith("language_model."):
return None
return super().filter_tensors(item)
@ModelBase.register("VibeVoiceAsrForConditionalGeneration")
class VibeVoiceAsrAudioModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = True
# values hard-coded in tools/mtmd/models/vibevoice-asr.cpp
_encoder_consts = {"channels": 1, "kernel_size": 7, "num_filters": 32, "ffn_expansion": 4, "hidden_act": "gelu", "rms_norm_eps": 1e-5}
_encoders = ("acoustic_tokenizer_encoder", "semantic_tokenizer_encoder")
def get_audio_config(self) -> dict[str, Any] | None:
c = self.global_config["acoustic_tokenizer_encoder_config"]
return {
**c,
"intermediate_size": c["ffn_expansion"] * c["hidden_size"],
"num_attention_heads": 1,
"num_hidden_layers": sum(c["depths"]),
}
def set_gguf_parameters(self):
super().set_gguf_parameters()
for enc in self._encoders:
c = self.global_config[f"{enc}_config"]
for k, v in self._encoder_consts.items():
assert c[k] == v, f"{enc}.{k} = {c[k]}, expected {v}"
assert c["depths"] == self.hparams["depths"] and c["downsampling_ratios"] == self.hparams["downsampling_ratios"]
self.gguf_writer.add_clip_projector_type(gguf.VisionProjectorType.VIBEVOICE_ASR)
# the input is the raw waveform, one channel takes the place of the mel bins
self.gguf_writer.add_audio_num_mel_bins(1)
self.gguf_writer.add_audio_attention_layernorm_eps(self.hparams["rms_norm_eps"])
self.gguf_writer.add_audio_conv_kernel_size(self.hparams["kernel_size"])
# stride of each stage, the stem has none
self.gguf_writer.add_audio_patch_sizes([1] + [int(r) for r in self.hparams["downsampling_ratios"]])
self.gguf_writer.add_audio_depths([int(d) for d in self.hparams["depths"]])
def tensor_force_quant(self, name, new_name, bid, n_dims):
if ".dw." in new_name:
return gguf.GGMLQuantizationType.F32 # ggml_ssm_conv
if ".conv." in new_name and new_name.endswith(".weight"):
return gguf.GGMLQuantizationType.F16
return super().tensor_force_quant(name, new_name, bid, n_dims)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, _ = item
if not name.startswith(cls._encoders + ("multi_modal_projector.",)):
return None
return super().filter_tensors(item)
_block_tensors = {
"norm.weight": gguf.MODEL_TENSOR.A_ENC_CNX_NORM,
"mixer.conv.weight": gguf.MODEL_TENSOR.A_ENC_CNX_DW,
"mixer.conv.bias": gguf.MODEL_TENSOR.A_ENC_CNX_DW,
"gamma": gguf.MODEL_TENSOR.A_ENC_CNX_LS1,
"ffn_norm.weight": gguf.MODEL_TENSOR.A_ENC_CNX_FFN_NORM,
"ffn.linear1.weight": gguf.MODEL_TENSOR.A_ENC_CNX_FFN_UP,
"ffn.linear1.bias": gguf.MODEL_TENSOR.A_ENC_CNX_FFN_UP,
"ffn.linear2.weight": gguf.MODEL_TENSOR.A_ENC_CNX_FFN_DOWN,
"ffn.linear2.bias": gguf.MODEL_TENSOR.A_ENC_CNX_FFN_DOWN,
"ffn_gamma": gguf.MODEL_TENSOR.A_ENC_CNX_LS2,
}
# mm.a.mlp.{0,1} + mm.a.mlp_norm.0 for the acoustic connector, {2,3} + 2 for the semantic one
_proj_tensors = {
"acoustic_linear_1": (gguf.MODEL_TENSOR.A_MMPROJ, 0),
"acoustic_norm": (gguf.MODEL_TENSOR.A_MMPROJ_NORM, 0),
"acoustic_linear_2": (gguf.MODEL_TENSOR.A_MMPROJ, 1),
"semantic_linear_1": (gguf.MODEL_TENSOR.A_MMPROJ, 2),
"semantic_norm": (gguf.MODEL_TENSOR.A_MMPROJ_NORM, 2),
"semantic_linear_2": (gguf.MODEL_TENSOR.A_MMPROJ, 3),
}
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# the layer scales (gamma) have no suffix upstream
suffix = "." + name.rsplit(".", 1)[-1] if name.endswith((".weight", ".bias")) else ".weight"
if name.startswith("multi_modal_projector."):
key, idx = self._proj_tensors[name.split(".")[1]]
yield (self.format_tensor_name(key, idx, suffix=suffix), data_torch)
return
enc, _, rest = name.partition(".")
enc_id = self._encoders.index(enc)
parts = rest.split(".")
depths = [int(d) for d in self.hparams["depths"]]
# causal convs of the stages: 0 = stem, 1..6 = downsample, 7 = head
if rest.startswith("stem.conv."):
conv_id = 0
elif parts[0] == "conv_layers" and parts[2] == "conv":
conv_id = int(parts[1]) + 1
elif parts[0] == "head":
conv_id = len(depths)
else:
conv_id = None
if conv_id is not None:
yield (gguf.TENSOR_NAMES[gguf.MODEL_TENSOR.A_ENC_CNX_CONV].format(bid=enc_id, xid=conv_id) + suffix, data_torch)
return
# ConvNeXt blocks are numbered per stage upstream and flat here
if parts[0] == "stem":
stage, blk, tail = 0, int(parts[2]), ".".join(parts[3:])
else:
assert parts[0] == "conv_layers" and parts[2] == "stage", name
stage, blk, tail = int(parts[1]) + 1, int(parts[3]), ".".join(parts[4:])
xid = sum(depths[:stage]) + blk
key = self._block_tensors[tail]
if key == gguf.MODEL_TENSOR.A_ENC_CNX_DW and tail.endswith(".weight"):
data_torch = data_torch.squeeze(1) # [C, 1, K] -> [C, K] for ggml_ssm_conv
yield (gguf.TENSOR_NAMES[key].format(bid=enc_id, xid=xid) + suffix, data_torch)
+234
View File
@@ -0,0 +1,234 @@
from __future__ import annotations
import base64
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, MmprojModel, gguf, logger
from .qwen import QwenModel, Qwen3Model
# token ids of the yue2_infer protocol (protocol.py)
EOD = 151643
ABC_START, ABC_END = 151847, 151848
MUSIC_START, MUSIC_END = 151851, 151852
CODEC_OFFSET, CODEC_SIZE = 151853, 32768
LATENT_START = 184621
# the VAE is a separate repo (m-a-p/YuE2-Vae), downloaded into this subdir of the model dir
VAE_DIR = "vae"
INSTRUCTIONS = {
"off": "Generate music with codec tokens from the given conditions.",
"melody": "Generate a melody-only ABC transcription without chord symbols, then generate music with codec tokens from the given conditions.",
"full": "Generate a chord-annotated ABC transcription, then generate music with codec tokens from the given conditions.",
}
# the text is "[Tags]\n<style>\n[Lyrics]\n<lyrics>", optionally followed by "\n[ABC]\n<score>" (cover mode)
# voice picks the planning mode: full (default), melody or off
TTS_TEMPLATE = (
"{%- set text = messages[0]['content'] -%}"
"{%- set mode = (voice | lower) if voice is defined and voice and (voice | lower) in ['full', 'melody', 'off'] else 'full' -%}"
"{%- set parts = text.split('\\n[ABC]\\n') -%}"
"{%- set instructions = {"
+ ", ".join(f"'{k}': '{v}'" for k, v in INSTRUCTIONS.items())
+ "} -%}"
"{{- '<|endoftext|>' + instructions[mode] + '\\n' + parts[0].rstrip() + '\\n' -}}"
"{%- if mode == 'off' -%}"
"{{- '<abc></abc><|music_start|>' -}}"
"{%- elif parts | length > 1 -%}"
"{{- '<abc>' + parts[1] + '</abc><|music_start|>' -}}"
"{%- else -%}"
"{{- '<abc>' -}}"
"{%- endif -%}"
)
def _is_nar_tensor(name: str) -> bool:
return ".nar_" in name or name.startswith(("llm2vae.", "vae2llm.", "time_embedder.", "latent_pos_embed."))
@ModelBase.register("YuE2ForCausalLM")
class YuE2TextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
# the NAR path goes to the mmproj
if _is_nar_tensor(item[0]):
return None
return super().filter_tensors(item)
def set_gguf_parameters(self):
super().set_gguf_parameters()
# the codes are sampled with classifier-free guidance, the uncond prompt runs as a second sequence
self.gguf_writer.add_classifier_free_guidance(True)
def set_vocab(self):
# checkpoint-native tiktoken file, see tokenization_yue2.py
ranks: dict[bytes, int] = {}
with open(self.dir_model / "qwen.tiktoken", "rb") as f:
for line in f.read().splitlines():
if line:
tok, rank = line.split()
ranks[base64.b64decode(tok)] = int(rank)
if len(ranks) != EOD:
raise ValueError(f"YuE2: expected {EOD} BPE tokens, got {len(ranks)}")
vocab_size = self.hparams["vocab_size"]
tokens: list[str] = [""] * vocab_size
toktypes: list[int] = [gguf.TokenType.UNUSED] * vocab_size
merges: list[str] = []
for tok, rank in ranks.items():
tokens[rank] = QwenModel.token_bytes_to_string(tok)
toktypes[rank] = gguf.TokenType.NORMAL
if len(tok) > 1:
merged = QwenModel.bpe(ranks, tok, max_rank=rank)
assert len(merged) == 2
merges.append(" ".join(map(QwenModel.token_bytes_to_string, merged)))
specials = ["<|endoftext|>", "<|im_start|>", "<|im_end|>", "<R>", "<S>", "<X>", "<mask>", "<sep>"]
specials += [f"<extra_{i}>" for i in range(200)]
specials[204:206] = ["<abc>", "</abc>"]
named = {EOD + i: s for i, s in enumerate(specials)}
named[MUSIC_START] = "<|music_start|>"
named[MUSIC_END] = "<|music_end|>"
for i in range(CODEC_SIZE):
named[CODEC_OFFSET + i] = f"<|codec_{i}|>"
for i, s in enumerate(["<|latent_start|>", "<|latent_end|>", "<|latent_pad|>"]):
named[LATENT_START + i] = s
for i in range(EOD, vocab_size):
tokens[i] = named.get(i, f"[PAD{i}]")
toktypes[i] = gguf.TokenType.CONTROL if i in named else gguf.TokenType.UNUSED
self.gguf_writer.add_tokenizer_model("gpt2")
self.gguf_writer.add_tokenizer_pre("qwen2")
self.gguf_writer.add_token_list(tokens)
self.gguf_writer.add_token_types(toktypes)
self.gguf_writer.add_token_merges(merges)
self.gguf_writer.add_bos_token_id(EOD)
self.gguf_writer.add_eos_token_id(MUSIC_END)
self.gguf_writer.add_pad_token_id(EOD)
self.gguf_writer.add_add_bos_token(False)
self.gguf_writer.add_chat_template([{"name": "tts", "template": TTS_TEMPLATE}])
@ModelBase.register("YuE2ForCausalLM")
class YuE2MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = False
def get_audio_config(self) -> dict[str, Any] | None:
# the NAR path: same shapes as the backbone layers
return {k: self.global_config[k] for k in ("num_hidden_layers", "hidden_size", "intermediate_size",
"num_attention_heads", "num_key_value_heads", "rms_norm_eps")}
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name = item[0]
if name == "latent_pos_embed.pe":
return None # sinusoidal, computed at runtime
if _is_nar_tensor(name) or name == "model.norm.weight":
return item
return None
def set_gguf_parameters(self):
hp = self.hparams
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.YUE2_GEN)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(hp["hidden_size"])
self.gguf_writer.add_gen_audio_feed_forward_length(hp["intermediate_size"])
self.gguf_writer.add_gen_audio_block_count(hp["num_hidden_layers"])
self.gguf_writer.add_gen_audio_head_count(hp["num_attention_heads"])
self.gguf_writer.add_gen_audio_head_count_kv(hp["num_key_value_heads"])
self.gguf_writer.add_gen_audio_attention_layernorm_eps(hp["rms_norm_eps"])
# rope theta, the ODE solver and the VAE hparams are hard-coded in tools/mtmd
def tensor_force_quant(self, name, new_name, bid, n_dims):
# read on the host, or small convs and projections of the flow / VAE graphs
if not new_name.startswith("a.gen.nar.blk.") or n_dims < 2:
return gguf.GGMLQuantizationType.F32
return super().tensor_force_quant(name, new_name, bid, n_dims)
@staticmethod
def _wn(sd: dict[str, Tensor], prefix: str) -> Tensor:
# torch weight norm, over every dim but 0
g, v = sd[prefix + ".weight_g"].float(), sd[prefix + ".weight_v"].float()
return g * v / v.flatten(1).norm(dim=1).view(-1, *[1] * (v.dim() - 1))
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
from safetensors.torch import load_file
path = self.dir_model / VAE_DIR / "model.safetensors"
if not path.is_file():
raise FileNotFoundError(f"YuE2: VAE not found at {path}, download m-a-p/YuE2-Vae into it")
logger.info(f"gguf: loading VAE decoder from '{path}'")
sd = {k[len("decoder.layers."):]: v for k, v in load_file(str(path)).items() if k.startswith("decoder.layers.")}
p = "a.gen.vae"
# SnakeBeta with log-scale params: x + 1/(exp(beta) + 1e-9) * sin(exp(alpha) * x)^2
def snake(dst: str, src: str) -> Iterable[tuple[str, Tensor]]:
yield (f"{dst}.alpha", torch.exp(sd[src + ".alpha"].float()))
yield (f"{dst}.beta", 1.0 / (torch.exp(sd[src + ".beta"].float()) + 1e-9))
yield (f"{p}.conv_in.weight", self._wn(sd, "0"))
yield (f"{p}.conv_in.bias", sd["0.bias"])
n_blk = sum(1 for k in sd if k.endswith(".layers.1.weight_g") and k.count(".") == 3)
for b in range(n_blk):
s = f"{b + 1}.layers"
yield from snake(f"{p}.blk.{b}.snake", f"{s}.0")
yield (f"{p}.blk.{b}.up.weight", self._wn(sd, f"{s}.1"))
yield (f"{p}.blk.{b}.up.bias", sd[f"{s}.1.bias"])
for r in range(3):
rs = f"{s}.{r + 2}.layers"
yield from snake(f"{p}.blk.{b}.res.{r}.snake1", f"{rs}.0")
yield (f"{p}.blk.{b}.res.{r}.conv1.weight", self._wn(sd, f"{rs}.1"))
yield (f"{p}.blk.{b}.res.{r}.conv1.bias", sd[f"{rs}.1.bias"])
yield from snake(f"{p}.blk.{b}.res.{r}.snake2", f"{rs}.2")
yield (f"{p}.blk.{b}.res.{r}.conv2.weight", self._wn(sd, f"{rs}.3"))
yield (f"{p}.blk.{b}.res.{r}.conv2.bias", sd[f"{rs}.3.bias"])
yield from snake(f"{p}.post_snake", f"{n_blk + 1}")
yield (f"{p}.conv_out.weight", self._wn(sd, f"{n_blk + 2}"))
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
if name.startswith("a.gen.vae."):
yield (name, data_torch)
return
p = "a.gen.nar"
aux = {
"model.norm.weight": f"{p}.output_norm.weight",
"vae2llm.weight": f"{p}.inp.weight",
"vae2llm.bias": f"{p}.inp.bias",
"llm2vae.weight": f"{p}.out.weight",
"llm2vae.bias": f"{p}.out.bias",
"time_embedder.mlp.0.weight": f"{p}.time.0.weight",
"time_embedder.mlp.0.bias": f"{p}.time.0.bias",
"time_embedder.mlp.2.weight": f"{p}.time.2.weight",
"time_embedder.mlp.2.bias": f"{p}.time.2.bias",
}
if name in aux:
yield (aux[name], data_torch)
return
# the NAR layers keep the libllama names, mtmd loads them as a text model
assert bid is not None, name
layer = {
"nar_input_layernorm.weight": "attn_norm.weight",
"nar_self_attn.q_proj.weight": "attn_q.weight",
"nar_self_attn.k_proj.weight": "attn_k.weight",
"nar_self_attn.v_proj.weight": "attn_v.weight",
"nar_self_attn.o_proj.weight": "attn_output.weight",
"nar_self_attn.q_norm.weight": "attn_q_norm.weight",
"nar_self_attn.k_norm.weight": "attn_k_norm.weight",
"nar_pre_mlp_layernorm.weight": "ffn_norm.weight",
"nar_mlp.gate_proj.weight": "ffn_gate.weight",
"nar_mlp.up_proj.weight": "ffn_up.weight",
"nar_mlp.down_proj.weight": "ffn_down.weight",
}
key = name.split(".", 3)[3] # strip "model.layers.{bid}."
yield (f"{p}.blk.{bid}.{layer[key]}", data_torch)
+575
View File
@@ -0,0 +1,575 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any, Callable, Iterable, TYPE_CHECKING
import numpy as np
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, TextModel, MmprojModel, LazyTorchTensor, gguf, logger
LM_DIR = Path("lm") / "zonos3_mini"
# codec decoder: quantizers and semantic head from the generator, the rest from the head checkpoint
CODEC_GENERATOR = Path("autoencoder") / "generator.pt"
CODEC_VARIANT = "keep2_plain"
CODEC_HEAD = Path("autoencoder") / "heads" / CODEC_VARIANT / "step_0050000.pt"
CODEC_STAGES = [0, 2, 4, 6, 8] # transformer stages of the decoder
DEFAULT_VOICE = "Savannah_en-American_F_1"
# speaker encoder (ownmect_s2), MECT_B2 blocks: (cnn depth, out channels, (stride f, stride t), expansion)
SPK_CKPT = Path("speaker_encoders") / "ownmect_s2" / "ckpt_0037.pth"
SPK_STAGES = [
[(3, 32, (1, 1), 4)] * 2,
[(3, 64, (2, 1), 2)] + [(3, 64, (1, 2), 2)] * 3,
[(3, 128, (2, 1), 2)] + [(3, 128, (1, 2), 2)] * 4,
[(3, 256, (2, 1), 1)] + [(2, 256, (1, 2), 1)] * 2,
]
VOICE_MAX_FRAMES = 900 # reference_max_frames of the reference
# GDN geometry, hard-coded in the reference (zonos3.reference.lm)
GDN_HEADS, GDN_HEAD_DIM, GDN_KERNEL = 4, 128, 4
# the text table keeps the tokenizer ids shifted by this offset, ids 2 and 3 are BOS and EOS
TEXT_OFFSET = 4
TEXT_BOS, TEXT_EOS = 2, 3
def _round_up(value: float, multiple: int) -> int:
return multiple * ((int(value) + multiple - 1) // multiple)
def _load_params(dir_model: Path) -> dict[str, Any] | None:
path = dir_model / LM_DIR / "params.json"
if not path.is_file():
return None
with open(path, encoding="utf-8") as f:
params = json.load(f).get("model", {})
return params if params.get("model_type") == "zonos2" else None
def _is_zonos3_checkpoint(dir_model: Path) -> bool:
return _load_params(dir_model) is not None
@ModelBase.register_hparams_loader(_is_zonos3_checkpoint)
def _load_zonos3_hparams(dir_model: Path) -> dict[str, Any]:
logger.info("gguf: detected ZONOS3 checkpoint")
params = _load_params(dir_model)
assert params is not None
for key, want in (("attention_pattern", "gdn3_global1"), ("residual_type", "full_attnres"),
("moe_router_type", "linear"), ("moe_router_use_eda", False), ("use_xsa", False),
("moe_end_from_layer", 0), ("speaker_enabled", True)):
if params.get(key) != want:
raise NotImplementedError(f"ZONOS3: unsupported {key} = {params.get(key)!r}, expected {want!r}")
text_config = dict(params, hidden_size=params["dim"])
return {"architectures": ["Zonos3Model"], "text_config": text_config, "audio_config": {"num_hidden_layers": 0}}
def _index_lm_tensors(model: ModelBase) -> dict[str, Callable[[], Tensor]]:
path = model.dir_model / LM_DIR / "model.pth"
logger.info(f"gguf: indexing model part '{LM_DIR / 'model.pth'}'")
state = torch.load(str(path), map_location="cpu", mmap=True, weights_only=True)
tensors: dict[str, Callable[[], Tensor]] = {}
for name, data in state.items():
gen = (lambda data=data: LazyTorchTensor.from_eager(data)) if model.lazy else (lambda data=data: data)
titem = model.filter_tensors((name, gen))
model._source_tensor_names[name] = titem[0] if titem else None
if titem:
tensors[titem[0]] = titem[1]
return tensors
@ModelBase.register("Zonos3Model")
class Zonos3TextModel(TextModel):
model_arch = gguf.MODEL_ARCH.ZONOS3
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
hp = self.hparams
self.n_embd = hp["dim"]
self.n_codebooks = hp["n_codebooks"]
self.audio_vocab = hp["codebook_size"] + 2 # codes, EOA, pad
self.audio_pad_id = hp["audio_pad_id"]
self.text_rows = hp["text_vocab"] + 1 # + the padding row, used by the generated rows
self.cond_counts = [n + 1 for n in hp["condition_bucket_counts"]] # + the unknown bucket
self.gdn_period = int(hp["attention_pattern"][len("gdn"):].split("_")[0]) + 1
self.n_ff = _round_up(hp["ffn_dim_multiplier"] * hp["dim"], hp["multiple_of"])
self.n_ff_exp = _round_up(hp["moe_ffn_dim_multiplier"] * hp["dim"], hp["moe_multiple_of"])
self.n_latent = int(hp["dim"] * hp["moe_alpha"])
self._embd: dict[str, Tensor] = {}
self._router: dict[int, dict[str, Tensor]] = {}
def index_tensors(self, remote_hf_model_id: str | None = None) -> dict[str, Callable[[], Tensor]]:
return _index_lm_tensors(self)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, _ = item
# audio heads and speaker projection go to the mmproj
if name.startswith(("multi_output.", "speaker_projection.")):
return None
return item
def set_vocab(self):
with open(self.dir_model / LM_DIR / "tokenizer" / "superbpe8192.json", encoding="utf-8") as f:
tok = json.load(f)
model = tok["model"]
if model["type"] != "BPE" or model.get("byte_fallback") or tok.get("normalizer") is not None:
raise NotImplementedError("ZONOS3: unexpected tokenizer type")
vocab: dict[str, int] = model["vocab"]
n_bpe = len(vocab)
if n_bpe + TEXT_OFFSET != self.hparams["text_vocab"]:
raise ValueError(f"ZONOS3: tokenizer has {n_bpe} tokens, text_vocab is {self.hparams['text_vocab']}")
added = {t["id"] for t in tok.get("added_tokens", [])}
tokens: list[str] = ["<|unused_0|>", "<|unused_1|>", "<|bos|>", "<|eos|>"]
toktypes: list[int] = [gguf.TokenType.CONTROL] * TEXT_OFFSET
inv = {i: t for t, i in vocab.items()}
for i in range(n_bpe):
tokens.append(inv[i])
toktypes.append(gguf.TokenType.CONTROL if i in added else gguf.TokenType.NORMAL)
# padding row of the text table (the text column of every generated row)
tokens.append("<|text_pad|>")
toktypes.append(gguf.TokenType.CONTROL)
# condition rows, the last bucket of each table is "unknown"
for cond, n in zip(self.hparams["condition_names"], self.cond_counts):
tokens += [f"<|{cond}_{i}|>" for i in range(n - 1)] + [f"<|{cond}_unk|>"]
toktypes += [gguf.TokenType.CONTROL] * n
merges = [" ".join(m) if isinstance(m, list) else m for m in model["merges"]]
self.gguf_writer.add_tokenizer_model("gpt2")
self.gguf_writer.add_tokenizer_pre("zonos3")
self.gguf_writer.add_token_list(tokens)
self.gguf_writer.add_token_types(toktypes)
self.gguf_writer.add_token_merges(merges)
self.gguf_writer.add_bos_token_id(TEXT_BOS)
self.gguf_writer.add_eos_token_id(TEXT_EOS)
self.gguf_writer.add_pad_token_id(TEXT_OFFSET + 1) # <|pad|> of the tokenizer
# the reference wraps the text as [BOS] + ids + [EOS]
self.gguf_writer.add_add_bos_token(True)
self.gguf_writer.add_add_eos_token(True)
# the reference normalizes the text with NeMo, the template covers the English numbers
template_path = Path(__file__).parent.parent / "models" / "templates" / "Zyphra-ZONOS3-Mini-tts.jinja"
with open(template_path, "r", encoding="utf-8") as f:
self.gguf_writer.add_chat_template([{"name": "tts", "template": f.read()}])
def set_gguf_parameters(self):
hp = self.hparams
n_layer = self.block_count
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_block_count(n_layer)
self.gguf_writer.add_context_length(hp["max_seqlen"])
self.gguf_writer.add_embedding_length(self.n_embd)
self.gguf_writer.add_feed_forward_length(self.n_ff)
self.gguf_writer.add_head_count(hp["n_heads"])
self.gguf_writer.add_head_count_kv(hp["n_kv_heads"])
self.gguf_writer.add_key_length(hp["head_dim"])
self.gguf_writer.add_value_length(hp["head_dim"])
self.gguf_writer.add_rope_dimension_count(hp["head_dim"])
self.gguf_writer.add_rope_freq_base(hp["rope_theta"])
self.gguf_writer.add_layer_norm_rms_eps(hp["norm_eps"])
self.gguf_writer.add_full_attention_interval(self.gdn_period)
# gated delta net
self.gguf_writer.add_ssm_conv_kernel(GDN_KERNEL)
self.gguf_writer.add_ssm_inner_size(GDN_HEADS * GDN_HEAD_DIM)
self.gguf_writer.add_ssm_state_size(GDN_HEAD_DIM)
self.gguf_writer.add_ssm_time_step_rank(GDN_HEADS)
self.gguf_writer.add_ssm_group_count(GDN_HEADS)
# latent MoE, the gating (softmax over the selected logits) is hard-coded in libllama
self.gguf_writer.add_leading_dense_block_count(hp["moe_start_from_layer"])
self.gguf_writer.add_expert_count(hp["moe_n_experts"])
self.gguf_writer.add_expert_used_count(hp["moe_router_topk"])
self.gguf_writer.add_expert_feed_forward_length(self.n_ff_exp)
self.gguf_writer.add_moe_latent_size(self.n_latent)
def tensor_force_quant(self, name, new_name, bid, n_dims):
# router logits pick the experts, keep them exact
if new_name.endswith(("ffn_gate_inp.weight", "exp_probs_b.bias")):
return gguf.GGMLQuantizationType.F32
return super().tensor_force_quant(name, new_name, bid, n_dims)
def _token_embd(self) -> Tensor:
n_cb = self.n_codebooks
pad = sum(self._embd[f"multi_embedder.embedders.{cb}.weight"][self.audio_pad_id].float() for cb in range(n_cb))
text = self._embd[f"multi_embedder.embedders.{n_cb}.weight"].float()
if text.shape[0] != self.text_rows:
raise ValueError(f"ZONOS3: text table has {text.shape[0]} rows, expected {self.text_rows}")
conds = [self._embd[f"condition_embeddings.{i}.weight"].float() for i in range(len(self.cond_counts))]
for t, n in zip(conds, self.cond_counts):
assert t.shape[0] == n
# a text row is [pad x n_codebooks, text_id], condition rows replace the summed row
return torch.cat([text + pad] + conds, dim=0)
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
T = gguf.MODEL_TENSOR
hp = self.hparams
n_layer = self.block_count
if name.startswith(("multi_embedder.", "condition_embeddings.")):
self._embd[name] = data_torch
if len(self._embd) == self.n_codebooks + 1 + len(self.cond_counts):
yield (self.format_tensor_name(T.TOKEN_EMBD), self._token_embd())
self._embd.clear()
return
if name == "block_attnres_queries":
# one query per routing point: 2*il before the mixer, 2*il+1 before the FFN, the last one before out_norm
assert data_torch.shape[0] == 2 * n_layer + 1
for il in range(n_layer):
yield (self.format_tensor_name(T.ATTN_RES_SCORE, il), data_torch[2 * il])
yield (self.format_tensor_name(T.FFN_RES_SCORE, il), data_torch[2 * il + 1])
yield (self.format_tensor_name(T.OUTPUT_RES_SCORE), data_torch[2 * n_layer])
return
if name == "out_norm.weight":
yield (self.format_tensor_name(T.OUTPUT_NORM), data_torch)
return
assert bid is not None, name
key = name.split(".", 2)[2] # strip "layers.{bid}."
if key == "attention_norm.weight":
yield (self.format_tensor_name(T.ATTN_NORM, bid), data_torch)
return
if key == "ffn_norm.weight":
yield (self.format_tensor_name(T.FFN_NORM, bid), data_torch)
return
# gated delta net
if key == "attention.in_proj.weight":
# split the fused [qkv | z | a | b] weight into the qwen35 GDN tensors
# attn_gate holds z, per-head output gate in the attention layers
inner = GDN_HEADS * GDN_HEAD_DIM
qkv, z, a, b = data_torch.split([3 * inner, inner, GDN_HEADS, GDN_HEADS], dim=0)
yield (self.format_tensor_name(T.ATTN_QKV, bid), qkv)
yield (self.format_tensor_name(T.ATTN_GATE, bid), z)
yield (self.format_tensor_name(T.SSM_ALPHA, bid), a)
yield (self.format_tensor_name(T.SSM_BETA, bid), b)
return
if key == "attention.conv.weight":
yield (self.format_tensor_name(T.SSM_CONV1D, bid), data_torch.squeeze(1))
return
if key == "attention.A_log":
yield (self.format_tensor_name(T.SSM_A, bid, suffix=""), -torch.exp(data_torch.float()))
return
if key == "attention.dt_bias":
yield (self.format_tensor_name(T.SSM_DT, bid, suffix=".bias"), data_torch)
return
if key == "attention.out_weight":
yield (self.format_tensor_name(T.SSM_NORM, bid), data_torch)
return
if key == "attention.wo.weight" and (bid + 1) % self.gdn_period != 0:
yield (self.format_tensor_name(T.SSM_OUT, bid), data_torch)
return
# global attention
if key == "attention.wq.weight":
yield (self.format_tensor_name(T.ATTN_Q, bid), data_torch)
return
if key == "attention.wkv.weight":
yield (self.format_tensor_name(T.ATTN_K, bid), data_torch[0])
yield (self.format_tensor_name(T.ATTN_V, bid), data_torch[1])
return
if key == "attention.wo.weight":
yield (self.format_tensor_name(T.ATTN_OUT, bid), data_torch)
return
if key == "attention.gater.weight":
# one sigmoid gate per head, same tensor name as the GDN z gate
yield (self.format_tensor_name(T.ATTN_GATE, bid), data_torch)
return
if key == "attention.temp":
# Q = rms_norm(q) * |temp|, the per-head scale is stored as the Q norm weight
temp = data_torch.float().abs().reshape(hp["n_heads"], 1).expand(hp["n_heads"], hp["head_dim"])
yield (self.format_tensor_name(T.ATTN_Q_NORM, bid), temp.contiguous())
return
# dense FFN (layer 0): out = w_out(w_in[0](x) * silu(w_in[1](x)))
if key == "feed_forward.w_in.weight":
yield (self.format_tensor_name(T.FFN_UP, bid), data_torch[0])
yield (self.format_tensor_name(T.FFN_GATE, bid), data_torch[1])
return
if key == "feed_forward.w_out.weight":
yield (self.format_tensor_name(T.FFN_DOWN, bid), data_torch)
return
# latent MoE
if key in ("feed_forward.router.router_linear.weight", "feed_forward.router.rmsnorm_eda.weight"):
# logits = router_linear(rms_norm(x) * w), the norm weight is folded into the router columns
r = self._router.setdefault(bid, {})
r[key] = data_torch
if len(r) == 2:
w = r["feed_forward.router.router_linear.weight"].float() * r["feed_forward.router.rmsnorm_eda.weight"].float()[None, :]
yield (self.format_tensor_name(T.FFN_GATE_INP, bid), w)
del self._router[bid]
return
if key == "feed_forward.router.balancing_biases":
# the reference selects on logits - bias
yield (self.format_tensor_name(T.FFN_EXP_PROBS_B, bid, suffix=".bias"), -data_torch.float())
return
if key == "feed_forward.latent_down.weight":
yield (self.format_tensor_name(T.MOE_LATENT_DOWN, bid), data_torch)
return
if key == "feed_forward.latent_up.weight":
yield (self.format_tensor_name(T.MOE_LATENT_UP, bid), data_torch)
return
if key == "feed_forward.experts.gate_up_proj":
gate, up = data_torch.chunk(2, dim=1)
yield (self.format_tensor_name(T.FFN_GATE_EXP, bid), gate)
yield (self.format_tensor_name(T.FFN_UP_EXP, bid), up)
return
if key == "feed_forward.experts.down_proj":
yield (self.format_tensor_name(T.FFN_DOWN_EXP, bid), data_torch)
return
raise ValueError(f"ZONOS3: unexpected tensor {name}")
@ModelBase.register("Zonos3Model")
class Zonos3MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = True
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
hp = self.global_config["text_config"]
self.n_codebooks = hp["n_codebooks"]
self.text_vocab = hp["text_vocab"]
self._embd: dict[str, Tensor] = {}
def index_tensors(self, remote_hf_model_id: str | None = None) -> dict[str, Callable[[], Tensor]]:
return _index_lm_tensors(self)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name, _ = item
if name.startswith(("multi_output.", "multi_embedder.", "speaker_projection.")):
return item
return None
def set_gguf_parameters(self):
self.gguf_writer.add_file_type(self.ftype)
# speaker encoder for voice cloning, the front-end hparams are hard-coded in tools/mtmd
self.gguf_writer.add_clip_has_audio_encoder(True)
self.gguf_writer.add_clip_audio_projector_type(gguf.VisionProjectorType.ZONOS3_SPKENC)
self.gguf_writer.add_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_audio_num_mel_bins(80)
self.gguf_writer.add_audio_block_count(0)
self.gguf_writer.add_audio_embedding_length(192)
self.gguf_writer.add_audio_head_count(1)
self.gguf_writer.add_audio_feed_forward_length(192)
self.gguf_writer.add_audio_attention_layernorm_eps(1e-5)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.ZONOS3_GEN)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(self.n_embd_text)
self.gguf_writer.add_gen_audio_feed_forward_length(0)
self.gguf_writer.add_gen_audio_block_count(0)
self.gguf_writer.add_gen_audio_head_count(0)
self.gguf_writer.add_gen_audio_attention_layernorm_eps(self.global_config["text_config"]["norm_eps"])
self.gguf_writer.add_gen_audio_model_variant(CODEC_VARIANT)
if any(self.dir_model.glob(f"gguf/voices/*/{DEFAULT_VOICE}.codes.npy")):
self.gguf_writer.add_gen_audio_default_voice(DEFAULT_VOICE)
# softcap and the codec hparams are hard-coded in tools/mtmd
def tensor_force_quant(self, name, new_name, bid, n_dims):
# read on the host, or used by get_rows / ggml_ssm_conv / the conv ops of the speaker encoder
if new_name.startswith(("voice.", "a.spk.")) or new_name.endswith((".table", ".dw.weight", ".mixer.weight", ".resample.weight")):
return gguf.GGMLQuantizationType.F32
return super().tensor_force_quant(name, new_name, bid, n_dims)
@staticmethod
def _wn(sd: dict[str, Tensor], prefix: str) -> Tensor:
# torch weight norm, over every dim but 0
g, v = sd[prefix + ".weight_g"].float(), sd[prefix + ".weight_v"].float()
return (g * v / v.flatten(1).norm(dim=1).view(-1, *[1] * (v.dim() - 1))).squeeze(-1)
def _speaker_encoder(self) -> Iterable[tuple[str, Tensor]]:
# same layout as the reference ggml runtime (tools/convert_gguf.py speaker-encoder)
sd = torch.load(self.dir_model / SPK_CKPT, map_location="cpu", weights_only=True)["modules"]["spk_model"]
sd = {k[len("model."):]: v.double() for k, v in sd.items() if not k.endswith("num_batches_tracked")}
p = "a.spk"
def bn(dst: str, src: str) -> Iterable[tuple[str, Tensor]]:
scale = sd[src + ".weight"] / torch.sqrt(sd[src + ".running_var"] + 1e-5)
yield (f"{p}.{dst}.scale", scale)
yield (f"{p}.{dst}.shift", sd[src + ".bias"] - sd[src + ".running_mean"] * scale)
def put(dst: str, t: Tensor) -> tuple[str, Tensor]:
return (f"{p}.{dst}", t)
fe = "feature_extractor."
yield put("fe.head.conv.weight", sd[fe + "head.0.weight"])
yield put("fe.head.conv.bias", sd[fe + "head.0.bias"])
yield from bn("fe.head.bn", fe + "head.1")
c_in, f_in = 32, 80
for si, stage in enumerate(SPK_STAGES):
for k, (depth, out_ch, (s_f, s_t), expansion) in enumerate(stage):
src, dst = f"{fe}mect_block{si + 1}.{k}.", f"fe.b{si + 1}.{k}."
for r in range(depth):
cs, cd = f"{src}cnn_block.{r}.", f"{dst}cnn.{r}."
# grouped 3x3 convs, one depthwise kernel per input slot of a group
for conv in ("conv1", "conv2"):
w = sd[cs + conv + ".weight"]
for j in range(w.shape[1]):
yield put(f"{cd}{conv}.w{j}", w[:, j:j + 1])
for i in (1, 2):
yield put(f"{cd}pw{i}.weight", sd[f"{cs}conv_pw{i}.weight"])
yield put(f"{cd}pw{i}.bias", sd[f"{cs}conv_pw{i}.bias"])
yield from bn(f"{cd}bn{i}", f"{cs}bn{i}")
if cs + "shortcut.0.weight" in sd:
yield put(cd + "sc.weight", sd[cs + "shortcut.0.weight"])
yield from bn(cd + "sc.bn", cs + "shortcut.1")
if expansion > 1:
yield put(dst + "reduce.weight", sd[src + "reduce_conv.0.weight"])
yield put(dst + "reduce.bias", sd[src + "reduce_conv.0.bias"])
yield from bn(dst + "reduce.bn", src + "reduce_conv.1")
yield put(dst + "rproj.weight", sd[src + "reduce_project.0.weight"].squeeze(-1))
yield put(dst + "rproj.bias", sd[src + "reduce_project.0.bias"])
yield put(dst + "rproj.ln.weight", sd[src + "reduce_project.1.weight"])
yield put(dst + "rproj.ln.bias", sd[src + "reduce_project.1.bias"])
yield put(dst + "pos.dw.weight", sd[src + "pos_conv.pos_conv.0.weight"].squeeze(1))
yield put(dst + "pos.dw.bias", sd[src + "pos_conv.pos_conv.0.bias"])
yield from bn(dst + "pos.bn", src + "pos_conv.pos_conv.1")
yield put(dst + "pos.pw.weight", sd[src + "pos_conv.pos_conv.3.weight"].squeeze(-1))
yield put(dst + "pos.pw.bias", sd[src + "pos_conv.pos_conv.3.bias"])
tr = src + "transformer."
for a, b in (("q", "q_proj"), ("k", "k_proj"), ("v", "v_proj"), ("o", "out_proj")):
yield put(f"{dst}attn.{a}.weight", sd[f"{tr}attention.{b}.weight"])
yield put(f"{dst}attn.{a}.bias", sd[f"{tr}attention.{b}.bias"])
yield put(dst + "ln1.weight", sd[tr + "layer_norm.weight"])
yield put(dst + "ln1.bias", sd[tr + "layer_norm.bias"])
moe = tr + "feed_forward.intermediate_dense.token_moe."
yield put(dst + "moe.gate.weight", sd[moe + "gate.weight"])
yield put(dst + "moe.experts.weight", sd[moe + "experts.weight"])
yield put(dst + "ffn_out.weight", sd[tr + "feed_forward.output_dense.weight"])
yield put(dst + "ffn_out.bias", sd[tr + "feed_forward.output_dense.bias"])
yield put(dst + "ln2.weight", sd[tr + "final_layer_norm.weight"])
yield put(dst + "ln2.bias", sd[tr + "final_layer_norm.bias"])
yield put(dst + "eproj.weight", sd[src + "expansion_project.0.weight"].squeeze(-1))
yield put(dst + "eproj.bias", sd[src + "expansion_project.0.bias"])
c_in, f_in = c_in * s_f, f_in // s_f
# the stage outputs are mixed by softmax weights
yield put("fe.dense_w", torch.softmax(sd[fe + "dense3.inputs_weights"], dim=1)[0, :, :, 0])
yield put("pool.l1.weight", sd["encoding_layers.linear1.weight"].squeeze(-1))
yield put("pool.l1.bias", sd["encoding_layers.linear1.bias"])
yield put("pool.l2.weight", sd["encoding_layers.linear2.weight"].squeeze(-1))
yield put("pool.l2.bias", sd["encoding_layers.linear2.bias"])
yield from bn("bn", "bn")
yield put("linear.weight", sd["linear.weight"])
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
yield from self._speaker_encoder()
logger.info(f"gguf: loading codec decoder from '{CODEC_GENERATOR}' and '{CODEC_HEAD}'")
gen = torch.load(self.dir_model / CODEC_GENERATOR, map_location="cpu", mmap=True, weights_only=True)["generator"]
head = torch.load(self.dir_model / CODEC_HEAD, map_location="cpu", mmap=True, weights_only=True)["trainable"]
p = "a.gen.wav"
def convnext(dst: str, src: dict[str, Tensor], prefix: str) -> Iterable[tuple[str, Tensor]]:
yield (f"{dst}.dw.weight", src[f"{prefix}.dwconv.weight"].squeeze(1))
yield (f"{dst}.dw.bias", src[f"{prefix}.dwconv.bias"])
yield (f"{dst}.norm.weight", src[f"{prefix}.norm.weight"])
yield (f"{dst}.norm.bias", src[f"{prefix}.norm.bias"])
yield (f"{dst}.pw1.weight", src[f"{prefix}.pwconv1.weight"])
yield (f"{dst}.pw1.bias", src[f"{prefix}.pwconv1.bias"])
yield (f"{dst}.pw2.weight", src[f"{prefix}.pwconv2.weight"])
yield (f"{dst}.pw2.bias", src[f"{prefix}.pwconv2.bias"])
# RVQ codebooks folded through their out_proj, the biases are added once
def rvq(prefix: str) -> tuple[Tensor, Tensor]:
return gen[prefix + ".codebook.weight"].float() @ self._wn(gen, prefix + ".out_proj").T, gen[prefix + ".out_proj.bias"].float()
t, b = rvq("quantizer.semantic_quantizer.quantizers.0")
yield (f"{p}.sem.table", t)
yield (f"{p}.sem.bias", b)
res = [rvq(f"quantizer.quantizer.quantizers.{i}") for i in range(self.n_codebooks - 1)]
yield (f"{p}.res.table", torch.cat([t for t, _ in res], dim=0))
# one bias per residual codebook, a frame decoded before all its codes are known leaves the missing ones out
yield (f"{p}.res.bias", torch.stack([b for _, b in res]))
n_sem = sum(1 for k in gen if k.startswith("semantic_head.") and k.endswith(".dwconv.weight"))
for i in range(n_sem):
yield from convnext(f"{p}.sem.{i}", gen, f"semantic_head.{i}")
yield (f"{p}.sem.out.weight", self._wn(gen, f"semantic_head.{n_sem}"))
yield (f"{p}.sem.out.bias", gen[f"semantic_head.{n_sem}.bias"])
names = {"norm1.weight": "norm1.weight", "norm1.bias": "norm1.bias", "qkv.weight": "qkv.weight",
"attention_out.weight": "wo.weight", "attention_scale": "attn_scale", "norm2.weight": "norm2.weight",
"norm2.bias": "norm2.bias", "ffn.0.weight": "ffn1.weight", "ffn.3.weight": "ffn2.weight", "ffn_scale": "ffn_scale"}
for si, s in enumerate(CODEC_STAGES):
yield (f"{p}.stage.{si}.in.weight", head[f"stages.{s}.input_proj.weight"])
if f"stages.{s}.output_proj.weight" in head:
yield (f"{p}.stage.{si}.out.weight", head[f"stages.{s}.output_proj.weight"])
n_layer = sum(1 for k in head if k.startswith(f"stages.{s}.layers.") and k.endswith(".qkv.weight"))
for il in range(n_layer):
for src, dst in names.items():
yield (f"{p}.stage.{si}.blk.{il}.{dst}", head[f"stages.{s}.layers.{il}.{src}"])
n_head = sum(1 for k in head if k.startswith("head.blocks.") and k.endswith(".dwconv.weight"))
for i in range(n_head):
yield from convnext(f"{p}.head.{i}", head, f"head.blocks.{i}")
yield (f"{p}.head.norm.weight", head["head.norm.weight"])
yield (f"{p}.head.norm.bias", head["head.norm.bias"])
# both iSTFT branches in one projection
yield (f"{p}.head.out.weight", torch.cat([head["head.outs.0.weight"], head["head.outs.1.weight"]], dim=0))
yield (f"{p}.head.out.bias", torch.cat([head["head.outs.0.bias"], head["head.outs.1.bias"]], dim=0))
# read on the host
yield (f"{p}.mixer.weight", head["head.mixer.weight"].squeeze(0))
yield (f"{p}.resample.weight", gen["to_external.kernel"].squeeze(1))
# "reference" speaker mode, the default of the reference: the voice is a prefix of delayed code rows
def _voices(self, tables: list[Tensor]) -> Iterable[tuple[str, Tensor]]:
n_cb, pad = self.n_codebooks, self.global_config["text_config"]["audio_pad_id"]
for path in sorted((self.dir_model / "gguf" / "voices").glob("*/*.codes.npy")):
codes = torch.from_numpy(np.load(path, allow_pickle=False).astype(np.int64))
if codes.ndim != 2 or codes.shape[1] != n_cb or codes.min() < 0 or codes.max() >= pad - 1:
logger.warning(f"voice {path.name}: skipped, expected [frames, {n_cb}] codes")
continue
codes = codes[:VOICE_MAX_FRAMES]
n_frames = codes.shape[0]
# codebook cb of frame t is in row t + cb, the other cells are pad
rows = torch.full((n_frames + n_cb - 1, n_cb), pad, dtype=torch.int64)
for cb in range(n_cb):
rows[cb:cb + n_frames, cb] = codes[:, cb]
embd = sum(tables[cb][rows[:, cb]] for cb in range(n_cb))
yield (f"voice.{path.name[:-len('.codes.npy')]}", embd)
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
T = gguf.MODEL_TENSOR
if name.startswith(("a.gen.wav.", "a.spk.")):
yield (name, data_torch)
return
# the speaker encoder ends with the speaker projection of the backbone
if name.startswith("speaker_projection."):
yield ("a.spk.proj." + name.split(".")[-1], data_torch)
return
if name == "multi_output.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_HEAD), data_torch)
return
assert name.startswith("multi_embedder.")
self._embd[name] = data_torch
if len(self._embd) < self.n_codebooks + 1:
return
# every row built by the mmproj has text id = text_vocab, so that row is folded into codebook 0
tables = [self._embd[f"multi_embedder.embedders.{cb}.weight"].float() for cb in range(self.n_codebooks)]
tables[0] = tables[0] + self._embd[f"multi_embedder.embedders.{self.n_codebooks}.weight"][self.text_vocab].float()
yield (self.format_tensor_name(T.A_GEN_CODE_OUT_EMBD), torch.cat(tables, dim=0))
yield from self._voices([LazyTorchTensor.to_eager(t) for t in tables])
self._embd.clear()
+1 -1
View File
@@ -692,7 +692,7 @@ llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_
# Or Using llama-server executable
./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024
# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.
# Option 1: Open your browser to http://localhost:8080 to open Llama Chat, the built-in chat app of the llama.cpp server.
# Option 2: In a NEW terminal, test the server with curl
# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
+172
View File
@@ -11,6 +11,10 @@ Supported models:
- [LFM3-Audio](#lfm3-audio)
- [Chatterbox](#chatterbox)
- [Drama 3](#drama-3)
- [ZONOS3](#zonos3)
- [Fish Audio S2 Pro](#fish-audio-s2-pro)
- [YuE2](#yue2) (songs)
- [MiniMax Music 3](#minimax-music-3) (songs)
The sections below list the params supported by each model, with `llama-tts` examples. In `llama-server`, the same options are set with `--tts-lang` and `--tts-voice`, or per request (see the [`/tts` endpoint](../../tools/server/README.md#post-tts-generate-speech-audio-from-text)).
@@ -180,3 +184,171 @@ The routed experts of the checkpoint are FP8, they are dequantized at conversion
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
```
## ZONOS3
[Zyphra ZONOS3 Mini](https://huggingface.co/Zyphra/ZONOS3-Mini): a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.
Available params:
- `--tts-voice` picks one of the predefined voices of the release (`Savannah_en-American_F_1` by default); `llama-server` lists them in `GET /props`. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference
- The language is the one of the text; `--tts-lang` (`lang` in `llama-server`) only selects the text normalization
- Numbers are spelled out by the `tts` chat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (`1st`, `1er`, `1º`), English and German years. A 4-digit number is read as a year only after a word like `in`, `since`, a month name, or `im Jahr`, `seit` in German (`in 1990` -> "in nineteen ninety", `1234 people` -> "one thousand two hundred thirty-four people"). Languages: `en`, `fr`, `de`, `es`, `it`, `pt`, `ja` and `zh` (or `cmn`), the region is ignored (`fr_fr`, `pt_br`); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g. `1.234` is a thousand in German
- Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
- `--tts-speaker-file` clones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the reference
- `llama-server` also takes `speed` (relative to 15 UTF-8 bytes per second) and `loudness` (LUFS), see the `/tts` endpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown
- Note: `loudness` has a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip
- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference: temperature 0.75 (`--temp` changes it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5
- Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code
Example usage:
```sh
llama-tts -m zonos3-mini.gguf \
-mm mmproj-zonos3-mini.gguf \
-p "The old lighthouse keeper climbed the spiral stairs every evening." \
--tts-voice Imogen_en-British_F_1 \
--output out.wav
```
**Note for GGUF conversion:**
Convert the root of the release repository, not the `lm/` subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the `keep2_plain` codec decoder, it is best kept in F32:
```sh
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf
```
## Fish Audio S2 Pro
[Fish Audio S2 Pro](https://huggingface.co/fishaudio/s2-pro): a Qwen3 backbone (4B) samples each frame's semantic token, a 4-layer fast AR predicts the 9 other codes, and the Fish DAC (same as Drama 3) decodes them to 44.1 kHz audio.
Available params:
- Speaker tags `<|speaker:0|>`, `<|speaker:1|>`, ... in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>`
- Inline instructions in brackets control the delivery, for example `[laughing]`, `[whisper]` or `[pause]`
- The language is the one of the text, `--tts-lang` is not used
- Voice cloning: `--tts-speaker-file` with the reference clip, optionally `--tts-speaker-text` with its transcript (it is put in the prompt as the reference does). The clip is encoded by the codec encoder half of `codec.pth`, it is truncated to the first 30 s
- Note: `--temp`, `--top-k` and `--top-p` are not used (only `--seed`), frames are sampled with the settings of the reference (semantic token: repetition penalty 1.1 over the last 16 frames, top-k 30, top-p 0.8, temperature 0.8, or top-p 0.9 and temperature 1.0 when the last 4 frames repeat a token; other codes: greedy)
- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
Example usage:
```sh
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
-p "<|speaker:0|>Did you remember to buy the bread? <|speaker:1|>Of course I did [laughing] it is on the kitchen table." \
-n 1000 \
--output out.wav
```
Voice cloning (`--tts-speaker-text` can be left out):
```sh
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
--tts-speaker-file reference.wav \
--tts-speaker-text "The words spoken in reference.wav." \
-p "Hello, this is a cloned voice." \
--output out.wav
```
**Note for GGUF conversion:**
The mmproj holds the fast AR and `codec.pth` (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16. The codec encoder is always stored in F16/F32, small errors in it change the codes picked by its VQ:
```sh
python convert_hf_to_gguf.py path/to/s2-pro --outtype q8_0 --outfile s2-pro.gguf
python convert_hf_to_gguf.py path/to/s2-pro --mmproj --outtype bf16 --outfile mmproj-s2-pro.gguf
```
## YuE2
[m-a-p/YuE2-3B](https://huggingface.co/m-a-p/YuE2-3B) generates songs (vocals and accompaniment) from a style prompt and lyrics, as 48 kHz stereo. The backbone (a Qwen3 text model) writes an ABC score, then one semantic code per 40 ms frame; the NAR half of the checkpoint turns the codes into VAE latents by flow matching, attending to the keys and values of the backbone, and the [YuE2-Vae](https://huggingface.co/m-a-p/YuE2-Vae) decoder makes the audio.
The prompt is the text of the reference request:
```
[Tags]
City Pop, upbeat, danceable, groovy bass, electric guitar, synth
[Lyrics]
[Verse]
...
```
Available params:
- `--tts-voice` picks the planning mode of the reference (`cot`): `full` (default, a chord-annotated score), `melody` (a melody-only score) or `off` (no score)
- Cover mode: append the score to the prompt as `\n[ABC]\n<score>`, with `--tts-voice melody` or `full`; the model then goes straight to the codes
- The generation stops at the limits of the reference (4096 score tokens, then 9000 codes, 6 minutes); `-n` sets a lower limit on the score tokens and the codes together
- The context must keep the default size (24576 positions, the one of the reference): a smaller `-c` is rejected when the model is loaded
- Note: `--top-k` and `--top-p` are not used, both phases are sampled with the settings of the reference (`yue2_generation_config.json`); `--temp` changes the temperature of the codes
- Note: the backbone context must use an F16 KV cache (default): the NAR pass loads its keys and values
- The codes are sampled with classifier-free guidance like the reference: scale 1.01 with `--tts-voice off`, off (1.0) otherwise. `llama-server` takes `cfg_scale` to change it, the reference suggests 1.2 for a stronger text guidance. The uncond prompt runs as a second sequence, so the context must also hold it (the instruction, the score and the codes once more)
- The NAR pass solves the flow-matching ODE in 32 midpoint steps like the reference; `llama-server` takes `flow_steps` to change it. The NAR time is about proportional to it: 16 steps are about 35% faster for the NAR and the VAE together, with a small deviation from the reference (about 0.2 dB of spectral difference); it grows with fewer steps (about 0.4 dB at 8, 0.8 dB at 4)
- The NAR pass runs on chunks of codes like the reference: one chunk for a song that fits in 24576 positions, so the audio arrives after the last code. When streaming, each 30 s of codes is a chunk, processed during the generation, so the audio arrives in 30 s steps; the chunks are independent (each sees the prompt, the score and its own codes), which may be audible at the boundaries
Example usage:
```sh
llama-tts -m yue2-3b.gguf \
-mm mmproj-yue2-3b.gguf \
-f song.txt \
--output song.wav
```
**Note for GGUF conversion:**
The VAE is a separate repository, download it into the `vae` subdirectory of the model:
```sh
hf download m-a-p/YuE2-3B --local-dir YuE2-3B
hf download m-a-p/YuE2-Vae --local-dir YuE2-3B/vae
python convert_hf_to_gguf.py YuE2-3B --outfile yue2-3b.gguf
python convert_hf_to_gguf.py YuE2-3B --mmproj --outfile mmproj-yue2-3b.gguf
```
The mmproj holds the NAR transformer layers (loaded with libllama as a Qwen3 text model, in the `--outtype` type), and the latent projections and the VAE decoder, always in F32.
## MiniMax Music 3
[MiniMaxAI/MiniMax-Music3](https://huggingface.co/MiniMaxAI/MiniMax-Music3) generates songs (vocals and accompaniment) from a music description and lyrics, as 44.1 kHz stereo. For each 40 ms frame, the backbone (a Qwen3 text model) samples the semantic code and the RVQ depth decoder the 7 other codes; the hidden states of both go to a flow-matching DiT, then a DAC decoder makes the audio. Both use the settings of the reference (the diffusers pipeline).
The prompt has the same format as YuE2, the description goes under `[Tags]`:
```
[Tags]
Genre: acoustic pop. BPM: 96. Key: C major. Vocals: soft female lead. Arrangement: fingerpicked guitar and soft piano.
[Lyrics]
[verse]
Morning light filtering through the pine
[chorus]
Softly the world begins to breathe
```
Available params:
- The description and the lyrics are cleaned up like the reference: markdown is removed, a line that starts with structure tags (e.g. `[Verse]`) keeps only the tags
- The generation stops at the end-of-audio token or at the limit of the reference (9000 frames, 6 minutes); `-n` sets a lower limit in frames (25 per second)
- `--temp` changes the temperature of the codes (default 1.0); `--top-k` and `--top-p` are not used, the codes are sampled with top-k 50 like the reference
- The codes are sampled with classifier-free guidance, scale 1.5 like the reference; `llama-server` takes `cfg_scale` to change it (1.0 disables it). The uncond prompt runs as a second sequence that shares the context, so a song needs about 2 x (prompt tokens + frames) positions: the default context (10240) holds about 3 minutes with a short description, use a larger `-c` (e.g. 20480) for longer songs
- The flow matching runs 30 Euler steps like the reference; `llama-server` takes `flow_steps` to change it
- The audio is made in windows of 200 frames (8 s) every 100 frames, like the reference: a window is processed during the generation once the frame after it exists, so with streaming the audio arrives in steps of about 4 s
Example usage:
```sh
llama-tts -m minimax-music3.gguf \
-mm mmproj-minimax-music3.gguf \
-f song.txt \
--output song.wav
```
**Note for GGUF conversion:**
The conversion reads the diffusers layout of the repository, the `qwen_7B` directory and the `.pth` files are not needed:
```sh
hf download MiniMaxAI/MiniMax-Music3 --local-dir MiniMax-Music3 --exclude "qwen_7B/*" --exclude "*.pth"
python convert_hf_to_gguf.py MiniMax-Music3 --outfile minimax-music3.gguf
python convert_hf_to_gguf.py MiniMax-Music3 --mmproj --outfile mmproj-minimax-music3.gguf
```
The mmproj holds the depth decoder and the DiT layers (in the `--outtype` type), and the condition encoder, the small DiT projections and the DAC decoder, always in F32.
+1 -1
View File
@@ -127,7 +127,7 @@ Legend:
| SWIGLU_CLAMP | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ |
| SWIGLU_OAI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TANH | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| TIMESTEP_EMBEDDING | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| TIMESTEP_EMBEDDING | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| TOP_K | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
| TOP_K_MASK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
| TRI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
+6 -6
View File
@@ -19,8 +19,8 @@
"OpenCL","GELU","type=f16,ne_a=[5,7,11,13],v=0","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f16,ne_a=[128,2,2,2],v=0","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f16,ne_a=[5,7,11,13],v=0","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[128,2,2,2],v=0","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[5,7,11,13],v=0","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[128,2,2,2],v=0","support","1","yes","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[5,7,11,13],v=0","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f16,ne_a=[128,2,2,2],v=0","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f16,ne_a=[5,7,11,13],v=0","support","1","yes","OpenCL"
"OpenCL","HARDSIGMOID","type=f16,ne_a=[128,2,2,2],v=0","support","1","yes","OpenCL"
@@ -61,8 +61,8 @@
"OpenCL","GELU","type=f16,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f16,ne_a=[128,2,2,2],v=1","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f16,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[128,2,2,2],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
"OpenCL","SILU","type=f16,ne_a=[5,7,11,13],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f16,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f16,ne_a=[5,7,11,13],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSIGMOID","type=f16,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
@@ -145,8 +145,8 @@
"OpenCL","GELU","type=f32,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f32,ne_a=[128,2,2,2],v=1","support","0","no","OpenCL"
"OpenCL","GELU_QUICK","type=f32,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f32,ne_a=[128,2,2,2],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f32,ne_a=[5,7,11,13],v=1","support","0","no","OpenCL"
"OpenCL","SILU","type=f32,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
"OpenCL","SILU","type=f32,ne_a=[5,7,11,13],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f32,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSWISH","type=f32,ne_a=[5,7,11,13],v=1","support","1","yes","OpenCL"
"OpenCL","HARDSIGMOID","type=f32,ne_a=[128,2,2,2],v=1","support","1","yes","OpenCL"
Can't render this file because it is too large.
+1 -1
View File
@@ -12351,7 +12351,7 @@
"WebGPU: WebGPU","ARANGE","type=f32,start=-1.000000,stop=255.000000,step=0.250000","support","1","yes","WebGPU"
"WebGPU: WebGPU","ARANGE","type=f32,start=-1.000000,stop=255.250000,step=0.250000","support","1","yes","WebGPU"
"WebGPU: WebGPU","ARANGE","type=f32,start=0.000000,stop=1048576.000000,step=1.000000","support","1","yes","WebGPU"
"WebGPU: WebGPU","TIMESTEP_EMBEDDING","type=f32,ne_a=[2,1,1,1],dim=320,max_period=10000","support","0","no","WebGPU"
"WebGPU: WebGPU","TIMESTEP_EMBEDDING","type=f32,ne_a=[2,1,1,1],dim=320,max_period=10000","support","1","yes","WebGPU"
"WebGPU: WebGPU","LEAKY_RELU","type=f32,ne_a=[10,5,4,3],negative_slope=0.100000","support","0","no","WebGPU"
"WebGPU: WebGPU","CUMSUM","type=f32,ne=[10,5,4,3]","support","1","yes","WebGPU"
"WebGPU: WebGPU","CUMSUM","type=f32,ne=[127,5,4,3]","support","1","yes","WebGPU"
Can't render this file because it is too large.
+18
View File
@@ -7,6 +7,8 @@ Jinja has no regex support, and exposing `std::regex` / Python `re` would not he
- `tts_norm_scan(text, rules)`: split text into matched and unmatched segments
- `tts_norm_sub(text, pattern, repl)`: replace every match
A third function, `tts_dict(name, key)`, gives the template access to dictionaries stored in the model (e.g. a pronunciation lexicon).
Implementations:
- C++: [common/chat-tts-norm.cpp](../common/chat-tts-norm.cpp), injected into task templates by `common_chat_template_render()` in [common/chat.cpp](../common/chat.cpp). They are not available to chat templates.
- Python reference: [scripts/chat_tts_norm.py](../scripts/chat_tts_norm.py), for use with `jinja2` (`env.globals["tts_norm_scan"] = chat_tts_norm.tts_norm_scan`)
@@ -69,6 +71,22 @@ Replaces every match of `pattern`, with the same scanning rules as `tts_norm_sca
Any other `%` sequence in `repl`, or a reference to a group that does not exist, is an error.
### `tts_dict(name, key)`
Looks up `key` in the dictionary `name` of the model and returns its value, or `none` if the key is missing. An unknown dictionary name is an error. A model can ship any number of dictionaries, for example a pronunciation lexicon per language:
```jinja
{%- set e = tts_dict('lexicon_en', word) -%}
{%- if e is none -%}{{ spell(word) }}
{%- elif e is mapping -%}{{ e[tag] if tag in e else e.DEFAULT }}
{%- else -%}{{ e }}
{%- endif -%}
```
The lookup is exact: no case folding and no normalization, do them in the template.
The GGUF lists the names of the dictionaries in `tokenizer.tts_dicts`. Each one is stored as two arrays of strings of the same length: `tokenizer.tts_dict.<name>.keys`, sorted by the UTF-8 bytes of the key, and `tokenizer.tts_dict.<name>.values`. A value is a JSON document (string, number, list, object, ...) and is returned decoded. `GGUFWriter.add_tts_dicts(dicts)` writes all of this.
## Pattern syntax
Patterns operate on Unicode codepoints. Invalid UTF-8 bytes (C++ only) are seen as U+FFFD, one per byte, and preserved in the output.
+29 -31
View File
@@ -87,36 +87,32 @@ for id: llama_token in tokens {
print("\n")
var batch = llama_batch_init(max(Int32(tokens.count), Int32(n_parallel)), 0, 1)
let batch = llama_batch_ext_init(context)
defer {
llama_batch_free(batch)
llama_batch_ext_free(batch)
}
// evaluate the initial prompt
batch.n_tokens = Int32(tokens.count)
for (i, token) in tokens.enumerated() {
batch.token[i] = token
batch.pos[i] = Int32(i)
batch.n_seq_id[i] = 1
// batch.seq_id[i][0] = 0
// TODO: is this the proper way to do this?
if let seq_id = batch.seq_id[i] {
seq_id[0] = 0
let idx = llama_batch_ext_add_token(batch, 0, token)
if idx < 0 {
print("llama_batch_ext_add_token() failed: \(idx)")
exit(1)
}
batch.logits[i] = 0
var pos = llama_pos(i)
llama_batch_ext_set_pos(batch, idx, &pos)
}
// llama_decode will output logits only for the last token of the prompt
batch.logits[Int(batch.n_tokens) - 1] = 1
// llama_process will output logits only for the last token of the prompt
llama_batch_ext_set_output_logits(batch, Int32(tokens.count) - 1, true)
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
exit(1)
}
for i in 1 ..< n_parallel {
llama_memory_seq_cp(llama_get_memory(context), 0, Int32(i), 0, batch.n_tokens)
llama_memory_seq_cp(llama_get_memory(context), 0, Int32(i), 0, Int32(tokens.count))
}
if n_parallel > 1 {
@@ -125,16 +121,17 @@ if n_parallel > 1 {
var streams: [String] = .init(repeating: "", count: n_parallel)
var streamBuffers: [[CChar]] = .init(repeating: [], count: n_parallel)
var i_batch = [Int32](repeating: batch.n_tokens - 1, count: n_parallel)
var i_batch = [Int32](repeating: Int32(tokens.count) - 1, count: n_parallel)
var n_cur = batch.n_tokens
var n_cur = Int32(tokens.count)
var n_decode = 0
let t_main_start = ggml_time_us()
while n_cur <= n_len {
// prepare the next batch
batch.n_tokens = 0
llama_batch_ext_clear(batch)
var n_tokens: Int32 = 0
// sample the next token for each parallel sequence / stream
for i in 0 ..< n_parallel {
@@ -165,31 +162,32 @@ while n_cur <= n_len {
streams[i] += nextStringPiece
// push this new token for next evaluation
batch.token[Int(batch.n_tokens)] = new_token_id
batch.pos[Int(batch.n_tokens)] = n_cur
batch.n_seq_id[Int(batch.n_tokens)] = 1
if let seq_id = batch.seq_id[Int(batch.n_tokens)] {
seq_id[0] = Int32(i)
let idx = llama_batch_ext_add_token(batch, Int32(i), new_token_id)
if idx < 0 {
print("llama_batch_ext_add_token() failed: \(idx)")
exit(1)
}
batch.logits[Int(batch.n_tokens)] = 1
var pos = n_cur
llama_batch_ext_set_pos(batch, idx, &pos)
llama_batch_ext_set_output_logits(batch, idx, true)
i_batch[i] = batch.n_tokens
i_batch[i] = idx
batch.n_tokens += 1
n_tokens += 1
n_decode += 1
}
// all streams are finished
if batch.n_tokens == 0 {
if n_tokens == 0 {
break
}
n_cur += 1
// evaluate the current batch with the transformer model
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
exit(1)
}
}
@@ -5,20 +5,16 @@ enum LlamaError: Error {
case couldNotInitializeContext
}
func llama_batch_clear(_ batch: inout llama_batch) {
batch.n_tokens = 0
}
func llama_batch_add(_ batch: inout llama_batch, _ id: llama_token, _ pos: llama_pos, _ seq_ids: [llama_seq_id], _ logits: Bool) {
batch.token [Int(batch.n_tokens)] = id
batch.pos [Int(batch.n_tokens)] = pos
batch.n_seq_id[Int(batch.n_tokens)] = Int32(seq_ids.count)
for i in 0..<seq_ids.count {
batch.seq_id[Int(batch.n_tokens)]![Int(i)] = seq_ids[i]
@discardableResult
func llama_batch_add(_ batch: OpaquePointer, _ id: llama_token, _ pos: llama_pos, _ seq_id: llama_seq_id, _ logits: Bool) -> Int32 {
let idx = llama_batch_ext_add_token(batch, seq_id, id)
if idx < 0 {
return idx
}
batch.logits [Int(batch.n_tokens)] = logits ? 1 : 0
batch.n_tokens += 1
var pos = pos
llama_batch_ext_set_pos(batch, idx, &pos)
llama_batch_ext_set_output_logits(batch, idx, logits)
return idx
}
actor LlamaContext {
@@ -26,7 +22,7 @@ actor LlamaContext {
private var context: OpaquePointer
private var vocab: OpaquePointer
private var sampling: UnsafeMutablePointer<llama_sampler>
private var batch: llama_batch
private var batch: OpaquePointer
private var tokens_list: [llama_token]
var is_done: Bool = false
@@ -42,7 +38,7 @@ actor LlamaContext {
self.model = model
self.context = context
self.tokens_list = []
self.batch = llama_batch_init(512, 0, 1)
self.batch = llama_batch_ext_init(context)
self.temporary_invalid_cchars = []
let sparams = llama_sampler_chain_default_params()
self.sampling = llama_sampler_chain_init(sparams)
@@ -53,7 +49,7 @@ actor LlamaContext {
deinit {
llama_sampler_free(sampling)
llama_batch_free(batch)
llama_batch_ext_free(batch)
llama_model_free(model)
llama_free(context)
llama_backend_free()
@@ -110,10 +106,6 @@ actor LlamaContext {
return SwiftString
}
func get_n_tokens() -> Int32 {
return batch.n_tokens;
}
func completion_init(text: String) {
print("attempting to complete \"\(text)\"")
@@ -133,25 +125,27 @@ actor LlamaContext {
print(String(cString: token_to_piece(token: id) + [0]))
}
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
for i1 in 0..<tokens_list.count {
let i = Int(i1)
llama_batch_add(&batch, tokens_list[i], Int32(i), [0], false)
}
batch.logits[Int(batch.n_tokens) - 1] = 1 // true
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
let idx = llama_batch_add(batch, tokens_list[i], Int32(i), 0, i == tokens_list.count - 1)
if idx < 0 {
print("llama_batch_add() failed: \(idx)")
}
}
n_cur = batch.n_tokens
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
}
n_cur = Int32(tokens_list.count)
}
func completion_loop() -> String {
var new_token_id: llama_token = 0
new_token_id = llama_sampler_sample(sampling, context, batch.n_tokens - 1)
new_token_id = llama_sampler_sample(sampling, context, -1)
if llama_vocab_is_eog(vocab, new_token_id) || n_cur == n_len {
print("\n")
@@ -178,13 +172,13 @@ actor LlamaContext {
print(new_token_str)
// tokens_list.append(new_token_id)
llama_batch_clear(&batch)
llama_batch_add(&batch, new_token_id, n_cur, [0], true)
llama_batch_ext_clear(batch)
llama_batch_add(batch, new_token_id, n_cur, 0, true)
n_decode += 1
n_cur += 1
if llama_decode(context, batch) != 0 {
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("failed to evaluate llama!")
}
@@ -201,21 +195,20 @@ actor LlamaContext {
for _ in 0..<nr {
// bench prompt processing
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
let n_tokens = pp
for i in 0..<n_tokens {
llama_batch_add(&batch, 0, Int32(i), [0], false)
llama_batch_add(batch, 0, Int32(i), 0, i == n_tokens - 1)
}
batch.logits[Int(batch.n_tokens) - 1] = 1 // true
llama_memory_clear(llama_get_memory(context), false)
let t_pp_start = DispatchTime.now().uptimeNanoseconds / 1000;
if llama_decode(context, batch) != 0 {
print("llama_decode() failed during prompt")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed during prompt")
}
llama_synchronize(context)
@@ -228,14 +221,14 @@ actor LlamaContext {
let t_tg_start = DispatchTime.now().uptimeNanoseconds / 1000;
for i in 0..<tg {
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
for j in 0..<pl {
llama_batch_add(&batch, 0, Int32(i), [Int32(j)], true)
llama_batch_add(batch, 0, Int32(i), Int32(j), true)
}
if llama_decode(context, batch) != 0 {
print("llama_decode() failed during text generation")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed during text generation")
}
llama_synchronize(context)
}
+5
View File
@@ -164,6 +164,11 @@
}
// lib.optionalAttrs (system == "x86_64-linux") {
rocm = config.legacyPackages.llamaPackagesRocm.llama-cpp;
sycl = config.packages.default.override {
useSycl = true;
syclToolkit = pkgs.intel-oneapi-toolkit;
};
};
# Packages exposed in `.#checks` will be built by the CI and by
+64 -16
View File
@@ -639,6 +639,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_iter(
float * const __restrict__ KQ_rowsum,
const int jt,
const int kb0,
const int kb0_next,
const int k_VKQ_sup) {
#if defined(VOLTA_MMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE) || defined(AMD_MFMA_AVAILABLE)
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
@@ -1030,10 +1031,10 @@ static __device__ __forceinline__ void flash_attn_ext_f16_iter(
if (!last_iter) {
if (ncols2 > 1 || mask_h) {
flash_attn_ext_f16_load_mask<ncols1, nwarps, nbatch_fa, use_cp_async, oob_check, use_sparse>
(mask_h, tile_mask, stride_mask, k_VKQ_0 + nbatch_fa, k_VKQ_sup, jt*ncols1, ne01, nullptr);
(mask_h, tile_mask, stride_mask, kb0_next*nbatch_fa, k_VKQ_sup, jt*ncols1, ne01, nullptr);
}
flash_attn_ext_f16_load_tile<stride_tile_K, nwarps, nbatch_fa, use_cp_async, oob_check, use_sparse>
(K_h2, tile_K, min(nbatch_K2, DKQ/2), stride_K, k_VKQ_0 + nbatch_fa, k_VKQ_sup, indices);
(K_h2, tile_K, min(nbatch_K2, DKQ/2), stride_K, kb0_next*nbatch_fa, k_VKQ_sup, indices);
}
}
@@ -1108,7 +1109,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_iter(
scale, slope, logit_softcap, ne01, ne02,
stride_K, stride_V, stride_mask,
tile_Q, tile_K, tile_V, tile_mask,
Q_B, VKQ_C, KQ_max, KQ_rowsum, kb0);
Q_B, VKQ_C, KQ_max, KQ_rowsum, kb0, kb0_next);
NO_DEVICE_CODE;
#endif // defined(VOLTA_MMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE) || defined(AMD_MFMA_AVAILABLE)
}
@@ -1285,7 +1286,8 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
const int jt,
const int zt_gqa,
const int kb0_start,
const int kb0_stop) {
const int kb0_stop,
const int kb0_max) {
#if defined(VOLTA_MMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE) || defined(AMD_MFMA_AVAILABLE)
//In this kernel Q, K, V are matrices while i, j, k are matrix indices.
@@ -1438,7 +1440,13 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
__syncthreads();
}
int kb0 = kb0_start;
// With multi-stage loading the K/V block order is rotated, see flash_attn_ext_f16.
// Then K/V blocks >= iter_k wrap around to 0 and the fully masked K/V blocks >= kb0_max are skipped.
const int iter_k = (ne11 + nbatch_fa - 1) / nbatch_fa;
const int nkb_pre = max(0, min(kb0_stop, kb0_max) - kb0_start);
const int nkb_post = max(0, min(kb0_stop - iter_k, kb0_max));
const int nkb = max(1, nkb_pre + nkb_post);
int kb0 = nstages > 1 && nkb_pre == 0 && nkb_post > 0 ? 0 : kb0_start;
// Preload mask and K data for first iteration when using cp_async with multiple stages:
if constexpr (nstages > 1) {
@@ -1456,7 +1464,29 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
}
// kb0_start is always < kb0_stop so the last iter can be executed unconditionally.
if constexpr (ncols2 == 1 || use_sparse) {
if constexpr (nstages > 1) {
constexpr bool oob_check = false;
for (int ikb = 1; ikb < nkb; ++ikb) {
const int kb0_next = kb0 + 1 == kb0_max ? 0 : kb0 + 1;
constexpr bool last_iter = false;
constexpr int k_VKQ_sup = nbatch_fa;
flash_attn_ext_f16_iter
<DKQ, DV, ncols1, ncols2, nwarps, use_logit_softcap, V_is_K_view, use_sparse, needs_fixup, is_fixup, last_iter, oob_check,
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, kb0_next, k_VKQ_sup);
kb0 = kb0_next;
}
constexpr bool last_iter = true;
constexpr int k_VKQ_sup = nbatch_fa;
flash_attn_ext_f16_iter
<DKQ, DV, ncols1, ncols2, nwarps, use_logit_softcap, V_is_K_view, use_sparse, needs_fixup, is_fixup, last_iter, oob_check,
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, kb0, k_VKQ_sup);
} else if constexpr (ncols2 == 1 || use_sparse) {
constexpr bool oob_check = true;
for (; kb0 < kb0_stop-1; ++kb0) {
constexpr bool last_iter = false;
@@ -1466,7 +1496,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, k_VKQ_sup);
KQ_max, KQ_rowsum, jt, kb0, kb0 + 1, k_VKQ_sup);
}
constexpr bool last_iter = true;
const int k_VKQ_sup = ne11 - kb0*nbatch_fa;
@@ -1475,7 +1505,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, k_VKQ_sup);
KQ_max, KQ_rowsum, jt, kb0, kb0, k_VKQ_sup);
} else {
constexpr bool oob_check = false;
for (; kb0 < kb0_stop-1; ++kb0) {
@@ -1486,7 +1516,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, k_VKQ_sup);
KQ_max, KQ_rowsum, jt, kb0, kb0 + 1, k_VKQ_sup);
}
constexpr bool last_iter = true;
constexpr int k_VKQ_sup = nbatch_fa;
@@ -1495,7 +1525,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
T_A_KQ, T_B_KQ, T_C_KQ, T_A_VKQ, T_B_VKQ, T_C_VKQ>
(Q_f2, K_h2, V_h2, mask_h, indices, dstk, dstk_fixup, scale, slope, logit_softcap,
ne01, ne02, stride_K, stride_V, stride_mask, tile_Q, tile_K, tile_V, tile_mask, Q_B, VKQ_C,
KQ_max, KQ_rowsum, jt, kb0, k_VKQ_sup);
KQ_max, KQ_rowsum, jt, kb0, kb0, k_VKQ_sup);
}
// With multi-stage loading there is no __syncthreads at the end of the iter,
@@ -1893,7 +1923,7 @@ static __device__ __forceinline__ void flash_attn_ext_f16_process_tile(
GGML_UNUSED_VARS(Q_f2, K_h2, V_h2, mask_h, indices, sinks_f, dstk, dstk_fixup,
scale, slope, logit_softcap, ne01, ne02, gqa_ratio,
stride_Q1, stride_Q2, stride_K, stride_V, stride_mask,
jt, kb0_start, kb0_stop);
jt, kb0_start, kb0_stop, kb0_max);
NO_DEVICE_CODE;
#endif // defined(VOLTA_MMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE) || defined(AMD_MFMA_AVAILABLE)
}
@@ -2021,6 +2051,18 @@ static __global__ void flash_attn_ext_f16(
int kb0_start = kbc % iter_k;
int kb0_stop = min(iter_k, kb0_start + kbc_stop - kbc);
// Rotate the K/V block order: kbc uses K/V block (kbc - kbc_tile) % iter_k instead of kbc % iter_k, kbc_tile is the start of the CUDA block that started the tile.
// Then concurrent CUDA blocks use the same K/V blocks and can share them in L2. Only with multi-stage loading, else they all wait for the same K/V data.
constexpr bool rotate = ggml_cuda_fattn_mma_get_nstages(DKQ, DV, ncols1, ncols2, use_sparse) > 1;
const int kbc0 = kbc;
int kbc_tile = kbc;
if (rotate && kb0_start != 0) {
// Get the CUDA block that started the tile, inverse of the kbc formula above:
const int64_t nkbc = int64_t(iter_k)*iter_j*iter_z_gqa*ne12*ne03;
const int64_t b_first = ((kbc - kb0_start + 1)*int64_t(gridDim.x) - 1) / nkbc;
kbc_tile = b_first*nkbc / gridDim.x;
}
while (kbc < kbc_stop && kb0_stop == iter_k) {
// z_KV == K/V head index, zt_gqa = Q head start index per K/V head, jt = token position start index
const int sequence = kbc /(iter_k*iter_j*iter_z_gqa*ne12);
@@ -2043,26 +2085,30 @@ static __global__ void flash_attn_ext_f16(
const float slope = ncols2 == 1 ? get_alibi_slope(max_bias, zt_Q, n_head_log2, m0, m1) : 1.0f;
// With rotation kb0_stop is not cut, process_tile skips the fully masked K/V blocks >= kb0_max.
if (use_sparse) {
kb0_stop = min(kb0_stop, (KV_max[(sequence % ne33)*iter_j + jt] + nbatch_fa - 1) / nbatch_fa);
} else if (KV_max) {
} else if (KV_max && !rotate) {
kb0_stop = min(kb0_stop, KV_max[sequence*iter_j + jt] / nbatch_fa);
}
const int kb0_max = rotate && KV_max ? KV_max[sequence*iter_j + jt] / nbatch_fa : iter_k;
const int kb0_rot = rotate ? (kbc - kbc_tile) % iter_k : kb0_start;
constexpr bool is_fixup = false; // All but (potentially) the last iterations write their data to dst rather than the fixup buffer.
if (kb0_start == 0) {
constexpr bool needs_fixup = false; // CUDA block is working on an entire tile.
flash_attn_ext_f16_process_tile<DKQ, DV, ncols1, ncols2, nwarps, use_logit_softcap, V_is_K_view, use_sparse, needs_fixup, is_fixup>
(Q_f2, K_h2, V_h2, mask_h, indices, sinks_f, dstk, dst_meta, scale, slope, logit_softcap,
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_start, kb0_stop);
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_rot, kb0_rot + kb0_stop - kb0_start, kb0_max);
} else {
constexpr bool needs_fixup = true; // CUDA block is missing the beginning of a tile.
flash_attn_ext_f16_process_tile<DKQ, DV, ncols1, ncols2, nwarps, use_logit_softcap, V_is_K_view, use_sparse, needs_fixup, is_fixup>
(Q_f2, K_h2, V_h2, mask_h, indices, sinks_f, dstk, dst_meta, scale, slope, logit_softcap,
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_start, kb0_stop);
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_rot, kb0_rot + kb0_stop - kb0_start, kb0_max);
}
kbc += iter_k;
kbc -= kbc % iter_k;
kbc_tile = kbc0;
kb0_start = 0;
kb0_stop = min(iter_k, kbc_stop - kbc);
@@ -2095,15 +2141,17 @@ static __global__ void flash_attn_ext_f16(
if (use_sparse) {
kb0_stop = min(kb0_stop, (KV_max[(sequence % ne33)*iter_j + jt] + nbatch_fa - 1) / nbatch_fa);
} else if (KV_max) {
} else if (KV_max && !rotate) {
kb0_stop = min(kb0_stop, KV_max[sequence*iter_j + jt] / nbatch_fa);
}
const int kb0_max = rotate && KV_max ? KV_max[sequence*iter_j + jt] / nbatch_fa : iter_k;
const int kb0_rot = rotate ? (kbc - kbc_tile) % iter_k : kb0_start;
constexpr bool is_fixup = true; // Last index writes its data to fixup buffer to avoid data races with other blocks.
constexpr bool needs_fixup = false;
flash_attn_ext_f16_process_tile<DKQ, DV, ncols1, ncols2, nwarps, use_logit_softcap, V_is_K_view, use_sparse, needs_fixup, is_fixup>
(Q_f2, K_h2, V_h2, mask_h, indices, sinks_f, dstk, dst_meta, scale, slope, logit_softcap,
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_start, kb0_stop);
ne01, ne02, gqa_ratio, ne11, stride_Q1, stride_Q2, stride_K, stride_V, stride_mask, jt, zt_gqa, kb0_rot, kb0_rot + kb0_stop - kb0_start, kb0_max);
#else
GGML_UNUSED_VARS(Q_ptr, K_ptr, V_ptr, mask_ptr, sinks_ptr, KV_max_ptr, KV_rows_ptr, dst_ptr, dst_meta_ptr, scale,
max_bias, m0, m1, n_head_log2, logit_softcap,
+139 -72
View File
@@ -9,11 +9,16 @@ constexpr int D = 128;
using bf16 = nv_bfloat16;
template <int R, int C> struct matrix {
bf16 hi[R * C], lo[R * C];
#ifdef GGML_USE_HIP
static constexpr int alloc = C + 4;
#else
static constexpr int alloc = C;
#endif // GGML_USE_HIP
bf16 hi[R * alloc], lo[R * alloc];
static __device__ __forceinline__ int index(int r, int c) {
#ifdef GGML_USE_HIP
return r * C + c;
return r * alloc + c;
#else
return 2 * ggml_cuda_mma::swizzle<C / 2, nv_bfloat162>(r * (C / 2) + c / 2, r) + c % 2;
#endif // GGML_USE_HIP
@@ -56,11 +61,11 @@ template <int C, int V> struct alignas(128) shared {
static __device__ __forceinline__ int lower_index(int r, int c) { return r * C + c; }
};
#if defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3))
#if defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3)) || defined(AMD_MFMA_AVAILABLE)
using namespace ggml_cuda_mma;
#ifdef GGML_USE_HIP
using acc = tile<16, 16, float, DATA_LAYOUT_J_MAJOR>;
using a_tile = tile<16, 8, nv_bfloat162, DATA_LAYOUT_I_MAJOR_MIRRORED>;
using a_tile = tile<16, 8, nv_bfloat162, get_input_data_layout()>;
using b_tile = a_tile;
#else
using acc = tile<16, 8, float>;
@@ -73,6 +78,23 @@ __device__ __forceinline__ tile_t load(const matrix<R, C> & m, int row, int col,
tile_t t;
const bf16 * data = low ? m.lo : m.hi;
#ifdef GGML_USE_HIP
#ifdef AMD_MFMA_AVAILABLE
if constexpr (!TRANS) {
#pragma unroll
for (int i = 0; i < tile_t::ne; ++i) {
const int r = row + tile_t::get_i(i), c = col + 2 * tile_t::get_j(i);
ggml_cuda_memcpy_1<4>(t.x + i, data + m.index(r, c));
}
} else {
#pragma unroll
for (int i = 0; i < tile_t::ne; ++i) {
const int r = row + tile_t::get_i(i), c = col + 2 * tile_t::get_j(i);
const bf16 x = data[m.index(c, r)];
const bf16 y = data[m.index(c + 1, r)];
t.x[i] = __float22bfloat162_rn(make_float2(__bfloat162float(x), __bfloat162float(y)));
}
}
#else
#pragma unroll
for (int i = 0; i < tile_t::ne; ++i) {
const int r = row + tile_t::get_i(i), c = col + 2 * tile_t::get_j(i);
@@ -80,6 +102,7 @@ __device__ __forceinline__ tile_t load(const matrix<R, C> & m, int row, int col,
const bf16 y = data[TRANS ? m.index(c + 1, r) : m.index(r, c + 1)];
t.x[i] = __float22bfloat162_rn(make_float2(__bfloat162float(x), __bfloat162float(y)));
}
#endif // AMD_MFMA_AVAILABLE
#else
const auto * packed = reinterpret_cast<const nv_bfloat162 *>(data);
if constexpr (TRANS) {
@@ -118,15 +141,17 @@ template <int R, int C> __device__ __forceinline__ void store(matrix<R, C> & m,
}
#endif // GGML_USE_HIP
}
#endif // defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3))
#endif // defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3)) || defined(AMD_MFMA_AVAILABLE)
// Each block owns V independent value columns and advances by C tokens.
// Keep recurrent state in FP32; pack high/residual BF16 operands inside the block.
template <int C, int V, int BLOCKS = 1>
static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_mma_args a) {
#if defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3))
static_assert(ggml_cuda_get_physical_warp_size() == 32, "GDN MMA requires 32-lane warps");
constexpr int WARPS = 8;
#if defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3)) || defined(AMD_MFMA_AVAILABLE)
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
static_assert(256 % warp_size == 0, "block must span a whole number of waves");
constexpr int warps = 256 / warp_size;
constexpr int input_rows = warp_size / 32; // input rows a wave stages per step
#ifdef GGML_USE_HIP
constexpr int N = 16;
#else
@@ -134,34 +159,37 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
#endif // GGML_USE_HIP
static_assert(C >= 16 && (C & (C - 1)) == 0 && V % N == 0 && D % V == 0, "invalid GDN tile");
extern __shared__ __align__(128) unsigned char bytes[];
auto & s = *reinterpret_cast<shared<C, V> *>(bytes);
const int lane = threadIdx.x, warp = threadIdx.y, tid = warp * 32 + lane;
const int slice = blockIdx.x % (D / V);
const int h = (blockIdx.x / (D / V)) % a.H;
const int seq = blockIdx.x / ((D / V) * a.H);
const int v0 = slice * V;
const int64_t soff = ((int64_t) seq * a.H + h) * D * D;
const int64_t qoff = (seq / a.rq3) * a.sq3 + (h % a.H_k) * a.sq1;
const int64_t voff = seq * a.sv3 + h * a.sv1 + v0;
const int64_t goff = seq * a.sb3 + h * a.sb1;
float * out = a.dst + ((int64_t) seq * a.n_tokens * a.H + h) * D + v0;
constexpr int STATE_TILES = (D / 16) * (V / N);
constexpr int PER_WARP = STATE_TILES / WARPS;
static_assert(STATE_TILES % WARPS == 0, "state must divide across warps");
auto & s = *reinterpret_cast<shared<C, V> *>(bytes);
const int lane = threadIdx.x;
const int warp = threadIdx.y;
const int tid = warp * warp_size + lane;
const int sub = warp_size > 32 ? lane / 32 : 0;
const int slane = warp_size > 32 ? lane % 32 : lane;
const int slice = blockIdx.x % (D / V);
const int h = (blockIdx.x / (D / V)) % a.H;
const int seq = blockIdx.x / ((D / V) * a.H);
const int v0 = slice * V;
const int64_t soff = ((int64_t) seq * a.H + h) * D * D;
const int64_t qoff = (seq / a.rq3) * a.sq3 + (h % a.H_k) * a.sq1;
const int64_t voff = seq * a.sv3 + h * a.sv1 + v0;
const int64_t goff = seq * a.sb3 + h * a.sb1;
float * out = a.dst + ((int64_t) seq * a.n_tokens * a.H + h) * D + v0;
constexpr int state_tiles = (D / 16) * (V / N);
constexpr int PER_WARP = state_tiles / warps;
static_assert(state_tiles % warps == 0, "state must divide across warps");
acc state[PER_WARP];
#pragma unroll
for (int j = 0; j < PER_WARP; ++j) {
const int tile_index = warp + j * WARPS, r = (tile_index / (V / N)) * 16, c = (tile_index % (V / N)) * N;
const int tile_index = warp + j * warps, r = (tile_index / (V / N)) * 16, c = (tile_index % (V / N)) * N;
#pragma unroll
for (int i = 0; i < acc::ne; ++i) {
state[j].x[i] = a.state[soff + (v0 + c + acc::get_j(i)) * D + r + acc::get_i(i)];
}
}
auto read_qk = [&](int64_t t, float4 & q, float4 & k) {
t += warp;
q = k = make_float4(0, 0, 0, 0);
if (t < a.n_tokens) {
const int64_t off = qoff + t * a.sq2 + 4 * lane;
const int64_t off = qoff + t * a.sq2 + 4 * slane;
if ((((uintptr_t) (a.q + off) | (uintptr_t) (a.k + off)) & 15) == 0) {
q = *reinterpret_cast<const float4 *>(a.q + off);
k = *reinterpret_cast<const float4 *>(a.k + off);
@@ -172,22 +200,26 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
}
};
// Carry the first eight input rows across chunks to hide global-load latency.
constexpr int QK_ROWS = warps * input_rows;
constexpr bool PREFETCH = C == 16 && V == 32;
float4 next_q, next_k;
float4 next_q;
float4 next_k;
if constexpr (PREFETCH) {
read_qk(0, next_q, next_k);
read_qk(warp * input_rows + sub, next_q, next_k);
}
for (int64_t t0 = 0; t0 < a.n_tokens; t0 += C) {
const int valid = min((int64_t) C, a.n_tokens - t0);
#pragma unroll
for (int j = 0; j < C / 8; ++j) {
const int t = warp + 8 * j, d = 4 * lane;
float4 q, k;
for (int j = 0; j < C / QK_ROWS; ++j) {
const int t = warp * input_rows + sub + QK_ROWS * j;
const int d = 4 * slane;
float4 q;
float4 k;
if (PREFETCH && j == 0) {
q = next_q;
k = next_k;
} else {
read_qk(t0 + 8 * j, q, k);
read_qk(t0 + t, q, k);
}
s.q.store2(t, d, make_float2(q.x * a.scale, q.y * a.scale));
s.q.store2(t, d + 2, make_float2(q.z * a.scale, q.w * a.scale));
@@ -197,13 +229,13 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
// Load gates in parallel before the FP32 prefix scan.
if (warp == 0) {
float carry = 0.f;
for (int base = 0; base < C; base += 32) {
for (int base = 0; base < C; base += warp_size) {
const int t = base + lane;
float g = t < valid ? a.g[goff + (t0 + t) * a.sb2] : 0.f;
const float beta = t < valid ? a.beta[goff + (t0 + t) * a.sb2] : 0.f;
#pragma unroll
for (int offset = 1; offset < 32; offset *= 2) {
const float previous = ggml_cuda_shfl_up(g, offset, 32);
for (int offset = 1; offset < warp_size; offset *= 2) {
const float previous = ggml_cuda_shfl_up(g, offset, warp_size);
if (lane >= offset) {
g += previous;
}
@@ -213,23 +245,23 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
s.prefix[t] = g;
s.beta[t] = beta;
}
carry = ggml_cuda_shfl(g, 31, 32);
carry = ggml_cuda_shfl(g, warp_size - 1, warp_size);
}
}
#pragma unroll
for (int j = 0; j < PER_WARP; ++j) {
const int ti = warp + j * WARPS;
const int ti = warp + j * warps;
store(s.scratch.state, state[j], (ti / (V / N)) * 16, (ti % (V / N)) * N);
}
__syncthreads();
if constexpr (PREFETCH) {
if (t0 + C < a.n_tokens) {
read_qk(t0 + C, next_q, next_k);
read_qk(t0 + C + warp * input_rows + sub, next_q, next_k);
}
}
// V32 overlaps Gram work with one output tile per projection warp.
constexpr bool SPLIT_PROJECTION = C == 16 && V == 32;
acc base_output[((C / 16) * (V / N) + WARPS - 1) / WARPS];
acc base_output[((C / 16) * (V / N) + warps - 1) / warps];
// delta = beta * (v - decay * K S), base output = decay * Q S
auto project = [&]() {
if constexpr (V == 32) {
@@ -237,23 +269,26 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
return;
}
}
for (int ti = (V == 32 ? warp - 4 : warp); ti < (C / 16) * (V / N); ti += (V == 32 ? 4 : WARPS)) {
const int r = (ti / (V / N)) * 16, c = (ti % (V / N)) * N;
acc qs, ks;
for (int ti = (V == 32 ? warp - 4 : warp); ti < (C / 16) * (V / N); ti += (V == 32 ? 4 : warps)) {
const int r = (ti / (V / N)) * 16;
const int c = (ti % (V / N)) * N;
acc qs;
acc ks;
product<D, false, true>(qs, s.q, s.scratch.state, r, c);
product<D, false, true>(ks, s.k, s.scratch.state, r, c);
#pragma unroll
for (int i = 0; i < acc::ne; ++i) {
const int t = r + acc::get_i(i), v = c + acc::get_j(i);
const int t = r + acc::get_i(i);
const int v = c + acc::get_j(i);
const float decay = expf(s.prefix[t]);
const float vv = t < valid ? a.v[voff + (t0 + t) * a.sv2 + v] : 0.f;
s.delta.store(t, v, s.beta[t] * (vv - decay * ks.x[i]));
base_output[ti / WARPS].x[i] = decay * qs.x[i];
base_output[ti / warps].x[i] = decay * qs.x[i];
}
}
};
// A = beta * strict_tril(K K^T * decay), P = tril(Q K^T * decay)
for (int ti = warp; ti < 2 * (C / 16) * (C / N) && (V != 32 || warp < 4); ti += (V == 32 ? 4 : WARPS)) {
for (int ti = warp; ti < 2 * (C / 16) * (C / N) && (V != 32 || warp < 4); ti += (V == 32 ? 4 : warps)) {
const bool gram = ti < (C / 16) * (C / N);
const int t = ti % ((C / 16) * (C / N)), r = (t / (C / N)) * 16, c = (t % (C / N)) * N;
acc x;
@@ -280,27 +315,27 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
}
__syncthreads();
// inverse = (I + A)^-1 by forward substitution
for (int row = warp; row < C; row += WARPS) {
float x[(C + 31) / 32];
for (int row = warp; row < C; row += warps) {
float x[(C + warp_size - 1) / warp_size];
#pragma unroll
for (int j = 0; j < (C + 31) / 32; ++j) {
const int col = lane + 32 * j;
x[j] = col == row ? 1.f : (col < row ? -s.lower[s.lower_index(row, col)] : 0.f);
for (int j = 0; j < (C + warp_size - 1) / warp_size; ++j) {
const int col = lane + warp_size * j;
x[j] = col == row ? 1.f : (col < row ? -s.lower[s.lower_index(row, col)] : 0.f);
}
for (int pivot = row - 1; pivot > 0; --pivot) {
const float xp = ggml_cuda_shfl(x[pivot / 32], pivot % 32, 32);
const float xp = ggml_cuda_shfl(x[pivot / warp_size], pivot % warp_size, warp_size);
#pragma unroll
for (int j = 0; j < (C + 31) / 32; ++j) {
const int col = lane + 32 * j;
for (int j = 0; j < (C + warp_size - 1) / warp_size; ++j) {
const int col = lane + warp_size * j;
if (col < pivot) {
x[j] -= xp * s.lower[s.lower_index(pivot, col)];
}
}
}
#pragma unroll
for (int j = 0; j < (C + 31) / 32; ++j) {
if (lane + 32 * j < C) {
s.inverse.store(row, lane + 32 * j, x[j]);
for (int j = 0; j < (C + warp_size - 1) / warp_size; ++j) {
if (lane + warp_size * j < C) {
s.inverse.store(row, lane + warp_size * j, x[j]);
}
}
}
@@ -309,8 +344,9 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
}
__syncthreads();
// solved = inverse * delta
for (int ti = warp; ti < (C / 16) * (V / N); ti += WARPS) {
const int r = (ti / (V / N)) * 16, c = (ti % (V / N)) * N;
for (int ti = warp; ti < (C / 16) * (V / N); ti += warps) {
const int r = (ti / (V / N)) * 16;
const int c = (ti % (V / N)) * N;
acc x;
product<C, false, true>(x, s.inverse, s.delta, r, c);
store(s.scratch.solved, x, r, c);
@@ -318,21 +354,25 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
__syncthreads();
// out = base output + P * solved
for (int ti = (SPLIT_PROJECTION ? warp - 4 : warp); ti < (C / 16) * (V / N) && (!SPLIT_PROJECTION || warp >= 4);
ti += WARPS) {
const int r = (ti / (V / N)) * 16, c = (ti % (V / N)) * N;
ti += warps) {
const int r = (ti / (V / N)) * 16;
const int c = (ti % (V / N)) * N;
acc x;
product<C, false, true>(x, s.p, s.scratch.solved, r, c);
#pragma unroll
for (int i = 0; i < acc::ne; ++i) {
const int t = r + acc::get_i(i), v = c + acc::get_j(i);
const int t = r + acc::get_i(i);
const int v = c + acc::get_j(i);
if (t < valid) {
out[(t0 + t) * a.H * D + v] = base_output[ti / WARPS].x[i] + x.x[i];
out[(t0 + t) * a.H * D + v] = base_output[ti / warps].x[i] + x.x[i];
}
}
}
// S = decay(chunk) * S + K^T * (solved * decay to chunk end)
for (int i = tid; i < C * V; i += 256) {
const int t = i / V, v = i % V, j = s.scratch.solved.index(t, v);
for (int i = tid; i < C * V; i += warps * warp_size) {
const int t = i / V;
const int v = i % V;
const int j = s.scratch.solved.index(t, v);
const float x = __bfloat162float(s.scratch.solved.hi[j]) + __bfloat162float(s.scratch.solved.lo[j]);
s.delta.store(t, v, x * expf(s.prefix[valid - 1] - s.prefix[t]));
}
@@ -340,7 +380,9 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
const float decay = expf(s.prefix[valid - 1]);
#pragma unroll
for (int j = 0; j < PER_WARP; ++j) {
const int ti = warp + j * WARPS, r = (ti / (V / N)) * 16, c = (ti % (V / N)) * N;
const int ti = warp + j * warps;
const int r = (ti / (V / N)) * 16;
const int c = (ti % (V / N)) * N;
#pragma unroll
for (int i = 0; i < acc::ne; ++i) {
state[j].x[i] *= decay;
@@ -351,7 +393,9 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
}
#pragma unroll
for (int j = 0; j < PER_WARP; ++j) {
const int ti = warp + j * WARPS, r = (ti / (V / N)) * 16, c = (ti % (V / N)) * N;
const int ti = warp + j * warps;
const int r = (ti / (V / N)) * 16;
const int c = (ti % (V / N)) * N;
#pragma unroll
for (int i = 0; i < acc::ne; ++i) {
a.state_out[soff + (v0 + c + acc::get_j(i)) * D + r + acc::get_i(i)] = state[j].x[i];
@@ -360,10 +404,11 @@ static __global__ __launch_bounds__(256, BLOCKS) void gdn_single(ggml_cuda_gdn_m
#else
GGML_UNUSED_VARS(a);
NO_DEVICE_CODE;
#endif // defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3))
#endif // defined(AMPERE_MMA_AVAILABLE) || (defined(AMD_WMMA_AVAILABLE) && defined(RDNA3)) || defined(AMD_MFMA_AVAILABLE)
}
template <int C, int V, int BLOCKS = 1> bool init(int device) {
template <int C, int V, int BLOCKS = 1>
static bool init(int device) {
static std::once_flag once[GGML_CUDA_MAX_DEVICES];
static bool available[GGML_CUDA_MAX_DEVICES] = {};
std::call_once(once[device], [device] {
@@ -381,7 +426,13 @@ template <int C, int V, int BLOCKS = 1> bool init(int device) {
return available[device];
}
enum class gdn_path { ar, slice32_one_block, slice32_two_blocks, slice64, whole_head };
enum class gdn_path {
ar,
slice32_one_block,
slice32_two_blocks,
slice64,
whole_head
};
static gdn_path select_gdn_mma_path(int device, const ggml_cuda_gdn_mma_args & a) {
GGML_ASSERT(device >= 0 && device < GGML_CUDA_MAX_DEVICES);
@@ -391,10 +442,25 @@ static gdn_path select_gdn_mma_path(int device, const ggml_cuda_gdn_mma_args & a
}
const auto & info = ggml_cuda_info().devices[device];
#ifdef GGML_USE_HIP
if (!GGML_CUDA_CC_IS_RDNA3_5(info.cc) || a.n_tokens < 2048) {
return gdn_path::ar;
if (GGML_CUDA_CC_IS_RDNA3_5(info.cc)) {
if (a.n_tokens < 2048) {
return gdn_path::ar;
}
return init<16, 64>(device) ? gdn_path::slice64 : gdn_path::ar;
}
return init<16, 64>(device) ? gdn_path::slice64 : gdn_path::ar;
if (GGML_CUDA_CC_IS_CDNA(info.cc)) {
if (!amd_mfma_available(info.cc)) {
return gdn_path::ar;
}
const int64_t blocks = a.H * a.n_seqs * 2;
const int64_t blocks_nwaves = (blocks + info.nsm - 1) / info.nsm;
const int64_t blocks_efficiency = 100 * blocks / (info.nsm * blocks_nwaves);
if (blocks_nwaves > 1 && blocks_efficiency < 60) {
return gdn_path::ar;
}
return init<16, 64>(device) ? gdn_path::slice64 : gdn_path::ar;
}
return gdn_path::ar;
#elif defined(GGML_USE_MUSA)
GGML_UNUSED_VARS(info);
return gdn_path::ar;
@@ -428,7 +494,8 @@ bool ggml_cuda_gdn_mma_launch(int device, const ggml_cuda_gdn_mma_args & a, cuda
return false;
}
#ifdef GGML_USE_HIP
gdn_single<16, 64><<<a.H * a.n_seqs * 2, dim3(32, 8), sizeof(shared<16, 64>), stream>>>(a);
const int warp_size = ggml_cuda_info().devices[device].warp_size;
gdn_single<16, 64><<<a.H * a.n_seqs * 2, dim3(warp_size, 256 / warp_size), sizeof(shared<16, 64>), stream>>>(a);
#else
if (path == gdn_path::slice32_two_blocks) {
gdn_single<16, 32, 2><<<a.H * a.n_seqs * 4, dim3(32, 8), sizeof(shared<16, 32>), stream>>>(a);
+3 -2
View File
@@ -2526,7 +2526,7 @@ static bool ggml_cuda_mul_mat_id_needs_sync(const ggml_tensor * dst, const int c
} else if (src0->type == GGML_TYPE_F8_E4M3 &&
ggml_cuda_should_use_mmvf(src0, cc, warp_size, dst->ne[2])) {
return false;
} else if (src0->type != GGML_TYPE_F8_E4M3 && GGML_CUDA_CC_IS_AMD(cc)) {
} else if (src0->type != GGML_TYPE_F8_E4M3 && GGML_CUDA_CC_IS_AMD(cc) && src0->ne[0] % 2 == 0) {
return false;
}
}
@@ -2570,7 +2570,8 @@ static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor *
ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst);
return;
} else {
if (src0->type != GGML_TYPE_F8_E4M3 && GGML_CUDA_CC_IS_AMD(cc)) {
// MMVF reads src0 in pairs of values, so it needs an even number of columns
if (src0->type != GGML_TYPE_F8_E4M3 && GGML_CUDA_CC_IS_AMD(cc) && ne00 % 2 == 0) {
ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst);
return;
}
+17 -7
View File
@@ -1079,8 +1079,14 @@ static __global__ void mul_mat_q(
const uint2 tmp2 = fast_div_modulo(blockIdx.z, nchannels_y);
const int wt = tmp2.x;
int zt = tmp2.y;
#if defined(GGML_USE_HIP)
// Row tiles innermost for MoE, see launch_mul_mat_q.
const int jt = ids_dst ? blockIdx.y : blockIdx.x;
const int it = ids_dst ? blockIdx.x : blockIdx.y;
#else
const int jt = blockIdx.x;
const int it = blockIdx.y;
#endif // defined(GGML_USE_HIP)
// Defaults for regular matrix multiplication:
int col_low = jt*J;
@@ -1475,16 +1481,20 @@ static void launch_mul_mat_q(ggml_backend_cuda_context & ctx, const mmq_args & a
const dim3 block_dims(warp_size, nwarps, 1);
CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, /*fallback =*/ false, prec_src1, has_fusion>), nbytes_shared);
if constexpr (!has_fusion) {
CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, /*fallback =*/ true, prec_src1, false>), nbytes_shared);
}
CUDA_SET_SHARED_MEMORY_LIMIT((mul_mat_q<type, J, fallback, prec_src1, has_fusion>), nbytes_shared);
const int I_dst = has_fusion ? config.I/2 : config.I; // Rows of dst per tile, see mul_mat_q.
const int nti = (args.nrows_x + I_dst - 1) / I_dst;
const int ntzw = args.nchannels_y * args.nsamples_y;
// Column tiles innermost so that the concurrently running blocks share the same tile of x and read it from L2.
const dim3 block_nums_xy_tiling(args.ntj, nti, ntzw);
#if defined(GGML_USE_HIP)
// For MoE the column tiles are tied to experts, so the column-innermost order does not improve the reuse of x in L2.
// It also groups blocks with very different runtimes which schedules poorly on AMD GPUs, see #31599.
// Therefore the row tiles are innermost for MoE. The column-innermost order is kept for dense multiplication.
const bool rows_innermost = args.ids_dst != nullptr;
#else
constexpr bool rows_innermost = false;
#endif // defined(GGML_USE_HIP)
const dim3 block_nums_xy_tiling = rows_innermost ? dim3(nti, args.ntj, ntzw) : dim3(args.ntj, nti, ntzw);
GGML_ASSERT(args.ids_dst || args.nchannels_y % args.nchannels_x == 0);
GGML_ASSERT( args.nsamples_y % args.nsamples_x == 0);
@@ -1499,7 +1509,7 @@ static void launch_mul_mat_q(ggml_backend_cuda_context & ctx, const mmq_args & a
const uint3 sample_ratio_fd = init_fastdiv_values(sample_ratio);
if (has_fusion || !config.stream_k) {
GGML_ASSERT(nti <= 65535); // gridDim.y limit
GGML_ASSERT((rows_innermost ? args.ntj : nti) <= 65535); // gridDim.y limit
mul_mat_q<type, J, fallback, prec_src1, has_fusion><<<block_nums_xy_tiling, block_dims, nbytes_shared, stream>>>
(args.x, (const char *) args.fusion.gate, args.y, args.ids_dst, args.tile_spec, args.dst, nullptr,
args.y_scale, args.fusion.glu_op, args.fusion.glu_limit, (const float *) args.fusion.x_scale, (const float *) args.fusion.gate_scale,
+8
View File
@@ -378,6 +378,14 @@ bool ggml_cuda_should_use_mmvq(enum ggml_type type, int cc, int64_t ne11) {
return ne11 <= MMVQ_MAX_BATCH_SIZE;
}
}
if (GGML_CUDA_CC_IS_RDNA3_5(cc)) {
switch (type) { // tuned on Radeon 8060S (gfx1151)
case GGML_TYPE_Q8_0:
return ne11 <= 6;
default:
return ne11 <= MMVQ_MAX_BATCH_SIZE;
}
}
if (GGML_CUDA_CC_IS_CDNA(cc)) {
if (GGML_CUDA_CC_IS_CDNA1(cc)) {
switch (type) {
+2 -2
View File
@@ -240,9 +240,9 @@ typedef uint64_t ggml_cuda_thread_mask_t;
#define RDNA2
#endif
#if defined(__gfx1010__) || defined(__gfx1012__)
#if defined(__gfx1010__) || defined(__gfx1012__) || defined(__gfx1013__)
#define RDNA1
#endif // defined(__gfx1010__) || defined(__gfx1012__)
#endif // defined(__gfx1010__) || defined(__gfx1012__) || defined(__gfx1013__)
#if defined(RDNA4) || defined(RDNA4_M) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1)
#define RDNA // For the entire family
+31
View File
@@ -2540,6 +2540,37 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d(ggml_met
return res;
}
ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm(ggml_metal_library_t lib, const ggml_tensor * op) {
assert(op->op == GGML_OP_CONV_2D);
GGML_ASSERT(ggml_is_contiguous(op->src[0]));
GGML_ASSERT(op->src[0]->type == GGML_TYPE_F16 || op->src[0]->type == GGML_TYPE_F32);
GGML_ASSERT(op->src[1]->type == GGML_TYPE_F32);
GGML_ASSERT(op->type == GGML_TYPE_F32);
char base[256];
char name[256];
const int nr0 = op->src[0]->ne[3] <= 32 ? 32 : 64;
const int nr1 = 2048/nr0;
snprintf(base, 256, "kernel_conv_2d_mm_%s_%s_%dx%d", ggml_type_name(op->src[0]->type), ggml_type_name(op->src[1]->type), nr0, nr1);
snprintf(name, 256, "%s", base);
ggml_metal_pipeline_with_params res = ggml_metal_library_get_pipeline(lib, name);
if (!res.pipeline) {
res = ggml_metal_library_compile_pipeline(lib, base, name, nullptr);
}
res.nr0 = nr0;
res.nr1 = nr1;
res.nsg = 4;
res.smem = nr0*nr1*sizeof(float);
return res;
}
ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_dw(ggml_metal_library_t lib, const ggml_tensor * op, bool tiled) {
assert(op->op == GGML_OP_CONV_2D_DW);
+1
View File
@@ -169,6 +169,7 @@ struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_tran
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_col2im_1d (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_snake (ggml_metal_library_t lib, enum ggml_type type);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_dw (ggml_metal_library_t lib, const struct ggml_tensor * op, bool tiled);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_3d (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_upscale (ggml_metal_library_t lib, const struct ggml_tensor * op);
+25
View File
@@ -5014,6 +5014,31 @@ int ggml_metal_op_conv_2d(ggml_metal_op_t ctx, int idx) {
/*.d1 =*/ d1,
};
const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx->dev);
if (props_dev->has_simdgroup_mm) {
auto pipeline = ggml_metal_library_get_pipeline_conv_2d_mm(lib, op);
const int nr0 = pipeline.nr0;
const int nr1 = pipeline.nr1;
const int nsg = pipeline.nsg;
const int64_t NP = ne3*ne1*ne0;
const int64_t OC = ne03;
ggml_metal_encoder_set_pipeline(enc, pipeline);
ggml_metal_encoder_set_bytes (enc, &args, sizeof(args), 0);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op->src[0]), 1);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op->src[1]), 2);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op), 3);
ggml_metal_encoder_set_threadgroup_memory_size(enc, pipeline.smem, 0);
ggml_metal_encoder_dispatch_threadgroups(enc, (NP + nr1 - 1)/nr1, (OC + nr0 - 1)/nr0, 1, 32, nsg, 1);
return 1;
}
auto pipeline = ggml_metal_library_get_pipeline_conv_2d(lib, op);
int nth = ggml_metal_pipeline_max_theads_per_threadgroup(pipeline);
+181
View File
@@ -274,6 +274,187 @@ kernel void kernel_conv_2d<half>(
uint3 tpitg[[thread_position_in_threadgroup]],
uint3 ntg[[threads_per_threadgroup]]);
template <typename TK, short NR0, short NR1>
kernel void kernel_conv_2d_mm(
constant ggml_metal_kargs_conv_2d & args,
device const char * weights,
device const char * src,
device char * dst,
threadgroup char * shmem [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
ushort tiitg[[thread_index_in_threadgroup]],
ushort tiisg[[thread_index_in_simdgroup]],
ushort sgitg[[simdgroup_index_in_threadgroup]]) {
constexpr short NSG = 4;
constexpr short NTH = 32*NSG;
constexpr short NK = 32;
constexpr short NKB = NK/8;
constexpr short NB = 72;
constexpr short SGY = NR0/32;
threadgroup half * sa = (threadgroup half *) shmem;
threadgroup half * sb = (threadgroup half *) shmem + (NR0/8)*NKB*NB;
const int K = args.IC*args.KH*args.KW;
const int NP = args.N*args.OH*args.OW;
const int r0 = tgpig.y*NR0;
const int r1 = tgpig.x*NR1;
constexpr short NPL = NR1/32;
const short lb_n = 8*sgitg + tiisg%8;
const short lb_k = 8*(tiisg/8);
int ix0[NPL];
int iy0[NPL];
bool ip_ok[NPL];
device const char * srcb[NPL];
FOR_UNROLL (short t = 0; t < NPL; t++) {
const int ip = r1 + 32*t + lb_n;
const int iow = ip % args.OW;
const int ioh = (ip / args.OW) % args.OH;
const int in = ip / (args.OW*args.OH);
ix0[t] = iow*args.s0 - args.p0;
iy0[t] = ioh*args.s1 - args.p1;
ip_ok[t] = ip < NP;
srcb[t] = src + in*args.nb13;
}
simdgroup_half8x8 ma[4];
simdgroup_half8x8 mb[2];
simdgroup_float8x8 mc[8];
for (short i = 0; i < 8; i++) {
mc[i] = make_filled_simdgroup_matrix<float, 8>(0.f);
}
for (int k0 = 0; k0 < K; k0 += NK) {
threadgroup_barrier(mem_flags::mem_threadgroup);
{
const short k = tiisg;
const int kk = k0 + k;
FOR_UNROLL (short j = 0; j < NR0/NSG; j++) {
const short row = (NR0/NSG)*sgitg + j;
const int oc = r0 + row;
half v = 0;
if (kk < K && oc < args.OC) {
v = *((device const TK *)(weights + oc*args.nb03) + kk);
}
const short ib = NKB*(row/8) + k/8;
sa[NB*ib + 8*(row%8) + k%8] = v;
}
}
{
int kk = k0 + lb_k;
int kw = kk % args.KW;
int kh = (kk / args.KW) % args.KH;
int ic = kk / (args.KW*args.KH);
FOR_UNROLL (short i = 0; i < 8; i++) {
const short k = lb_k + i;
FOR_UNROLL (short t = 0; t < NPL; t++) {
const short n = 32*t + lb_n;
const int ix = ix0[t] + kw*args.d0;
const int iy = iy0[t] + kh*args.d1;
half v = 0;
if (ip_ok[t] && kk < K && ix >= 0 && ix < args.IW && iy >= 0 && iy < args.IH) {
v = *(device const float *)(srcb[t] + ic*args.nb12 + iy*args.nb11 + ix*args.nb10);
}
const short ib = NKB*(n/8) + k/8;
sb[NB*ib + 8*(k%8) + n%8] = v;
}
kk++;
if (++kw == args.KW) {
kw = 0;
if (++kh == args.KH) {
kh = 0;
ic++;
}
}
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
const short nkb = min((K - k0 + 7)/8, (int) NKB);
for (short ik = 0; ik < nkb; ik++) {
FOR_UNROLL (short i = 0; i < 4; i++) {
simdgroup_load(ma[i], sa + NB*(NKB*(4*(sgitg%SGY) + i) + ik), 8, 0, false);
}
FOR_UNROLL (short j = 0; j < 2; j++) {
simdgroup_load(mb[j], sb + NB*(NKB*(2*(sgitg/SGY) + j) + ik), 8, 0, false);
}
FOR_UNROLL (short i = 0; i < 8; i++) {
simdgroup_multiply_accumulate(mc[i], ma[i%4], mb[i/4], mc[i]);
}
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
threadgroup float * sc = (threadgroup float *) shmem;
for (short i = 0; i < 8; i++) {
simdgroup_store(mc[i], sc + NR1*(32*(sgitg%SGY) + 8*(i%4)) + 16*(sgitg/SGY) + 8*(i/4), NR1, 0, false);
}
threadgroup_barrier(mem_flags::mem_threadgroup);
const short cn = tiitg%NR1;
const int p = r1 + cn;
if (p >= NP) {
return;
}
const int ow = p % args.OW;
const int oh = (p / args.OW) % args.OH;
const int n = p / (args.OW*args.OH);
device char * dstp = dst + n*args.nb3 + oh*args.nb1 + ow*args.nb0;
for (short j = 0; j < NR0*NR1/NTH; j++) {
const short row = tiitg/NR1 + (NTH/NR1)*j;
const int oc = r0 + row;
if (oc < args.OC) {
*(device float *)(dstp + oc*args.nb2) = sc[NR1*row + cn];
}
}
}
typedef decltype(kernel_conv_2d_mm<float, 64, 32>) kernel_conv_2d_mm_t;
template [[host_name("kernel_conv_2d_mm_f32_f32_64x32")]] kernel kernel_conv_2d_mm_t kernel_conv_2d_mm<float, 64, 32>;
template [[host_name("kernel_conv_2d_mm_f16_f32_64x32")]] kernel kernel_conv_2d_mm_t kernel_conv_2d_mm<half, 64, 32>;
template [[host_name("kernel_conv_2d_mm_f32_f32_32x64")]] kernel kernel_conv_2d_mm_t kernel_conv_2d_mm<float, 32, 64>;
template [[host_name("kernel_conv_2d_mm_f16_f32_32x64")]] kernel kernel_conv_2d_mm_t kernel_conv_2d_mm<half, 32, 64>;
typedef void (conv_transpose_1d_t)(
constant ggml_metal_kargs_conv_transpose_1d & args,
device const float * src0,
+244 -5
View File
@@ -712,6 +712,9 @@ struct ggml_backend_opencl_context {
size_t image_max_buffer_size;
size_t image2d_max_width;
size_t image2d_max_height;
size_t image3d_max_width;
size_t image3d_max_height;
size_t image3d_max_depth;
cl_device_svm_capabilities svm_caps;
@@ -866,7 +869,8 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_sqr_cont_f32, kernel_sqr_cont_f32_4, kernel_sqr_cont_f16, kernel_sqr_cont_f16_4;
cl_kernel kernel_sqrt_cont_f32, kernel_sqrt_cont_f32_4, kernel_sqrt_cont_f16, kernel_sqrt_cont_f16_4;
cl_kernel kernel_mean_f32, kernel_mean_f32_4;
cl_kernel kernel_silu, kernel_silu_4;
cl_kernel kernel_silu, kernel_silu_4, kernel_silu_nc;
cl_kernel kernel_silu_f16, kernel_silu_f16_4, kernel_silu_f16_nc;
cl_kernel kernel_gelu, kernel_gelu_4;
cl_kernel kernel_gelu_erf, kernel_gelu_erf_4;
cl_kernel kernel_gelu_quick, kernel_gelu_quick_4;
@@ -1409,6 +1413,7 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_gemv_noshuffle_q4_1_f32_mc3; // multi-column (N=3) verify GEMV (spec/MTP)
cl_kernel kernel_gemm_noshuffle_q4_1_f32;
cl_kernel kernel_gemm_noshuffle_q8_0_f32, kernel_gemm_noshuffle_q8_0_f32_bin;
cl_kernel kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7_bin = nullptr;
cl_kernel kernel_gemm_noshuffle_q8_0_q8_1_dp4a = nullptr; // dp4a (int8) dense q8_0 prefill GEMM (opt-in)
cl_kernel kernel_gemm_noshuffle_q8_0_q8_1_dp4a_act_vload4 = nullptr;
cl_kernel kernel_gemm_noshuffle_q8_0_q8_1_dp4a_wimg = nullptr; // q8_0 dense dp4a, weights via texture (opt-in)
@@ -1431,6 +1436,7 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_gemv_noshuffle_q4_k_f32_mc3; // multi-column (N=3) verify GEMV
cl_kernel kernel_gemm_noshuffle_q4_k_f32;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin;
cl_kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin;
cl_kernel kernel_gemv_noshuffle_q4_k_f32_32b_trans;
cl_kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a = nullptr; // dp4a (int8) dense prefill GEMM
@@ -4149,6 +4155,10 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
CL_CHECK((backend_ctx->kernel_silu = clCreateKernel(backend_ctx->program_silu, "kernel_silu", &err), err));
CL_CHECK((backend_ctx->kernel_silu_4 = clCreateKernel(backend_ctx->program_silu, "kernel_silu_4", &err), err));
CL_CHECK((backend_ctx->kernel_silu_nc = clCreateKernel(backend_ctx->program_silu, "kernel_silu_nc", &err), err));
CL_CHECK((backend_ctx->kernel_silu_f16 = clCreateKernel(backend_ctx->program_silu, "kernel_silu_f16", &err), err));
CL_CHECK((backend_ctx->kernel_silu_f16_4 = clCreateKernel(backend_ctx->program_silu, "kernel_silu_f16_4", &err), err));
CL_CHECK((backend_ctx->kernel_silu_f16_nc = clCreateKernel(backend_ctx->program_silu, "kernel_silu_f16_nc", &err), err));
GGML_LOG_CONT(".");
}
@@ -6424,6 +6434,7 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
{
size_t bin_size = 0;
backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin = nullptr;
backend_ctx->kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7_bin = nullptr;
if (use_adreno_bin_kernels(backend_ctx)) {
const char * kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_noshuffle_q8_0_f32_ila", &bin_size);
@@ -6435,6 +6446,17 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_noshuffle_q8_0_f32_32b_trans_ila_a7", &bin_size);
if (kernel_bin && bin_size > 0) {
cl_program bin_prog =
build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, "", bin_size);
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7_bin =
clCreateKernel(bin_prog, "kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7", &err), err));
CL_CHECK(clReleaseProgram(bin_prog));
GGML_LOG_CONT(".");
}
}
}
@@ -6701,6 +6723,7 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin = nullptr;
{
if (backend_ctx->has_vector_subgroup_broadcast) {
@@ -6745,6 +6768,17 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
CL_CHECK(clReleaseProgram(bin_prog));
GGML_LOG_CONT(".");
}
kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_noshuffle_q4_k_f32_32b_trans_ila_a7", &bin_size);
if (kernel_bin && bin_size > 0) {
cl_program bin_prog =
build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, "", bin_size);
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin =
clCreateKernel(bin_prog, "kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7", &err), err));
CL_CHECK(clReleaseProgram(bin_prog));
GGML_LOG_CONT(".");
}
}
}
@@ -9258,6 +9292,9 @@ static ggml_backend_opencl_context * ggml_cl_init(ggml_backend_dev_t dev) {
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE_MAX_BUFFER_SIZE, sizeof(size_t), &backend_ctx->image_max_buffer_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE2D_MAX_WIDTH, sizeof(size_t), &backend_ctx->image2d_max_width, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE2D_MAX_HEIGHT, sizeof(size_t), &backend_ctx->image2d_max_height, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_WIDTH, sizeof(size_t), &backend_ctx->image3d_max_width, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_HEIGHT, sizeof(size_t), &backend_ctx->image3d_max_height, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_DEPTH, sizeof(size_t), &backend_ctx->image3d_max_depth, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_MAX_WORK_GROUP_SIZE, sizeof(size_t), &backend_ctx->max_workgroup_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_LOCAL_MEM_SIZE, sizeof(cl_ulong), &backend_ctx->local_mem_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_SVM_CAPABILITIES, sizeof(cl_device_svm_capabilities), &backend_ctx->svm_caps, 0));
@@ -11660,6 +11697,26 @@ static bool use_fa_bin_kernels_prefill(const ggml_backend_opencl_context * backe
constexpr bool prefill_only = true;
// The repacked Q/K/V are 3D images: Q {n_q, n_batch*n_head, dk/4}, K {dk, n_kv/4, n_batch*n_head_kv},
// V {n_kv, dv/4, n_batch*n_head_kv}. Past the device's 3D image limits clCreateImage fails and the
// call asserts (X2-90: V is n_kv wide, so any prefill past ~16k tokens of context), so decline those shapes.
const size_t n_kv = (size_t) k->ne[1];
const size_t n_bh = (size_t) q->ne[3] * (size_t) q->ne[2];
const size_t n_bhkv = (size_t) q->ne[3] * (size_t) k->ne[2];
if ((size_t) n_q > backend_ctx->image3d_max_width || n_bh > backend_ctx->image3d_max_height ||
(size_t) dk / 4 > backend_ctx->image3d_max_depth ||
(size_t) dk > backend_ctx->image3d_max_width || (n_kv + 3) / 4 > backend_ctx->image3d_max_height ||
n_kv > backend_ctx->image3d_max_width || (size_t) dv / 4 > backend_ctx->image3d_max_height ||
n_bhkv > backend_ctx->image3d_max_depth) {
return false;
}
// q8_0 K/V at DK=512 gives wrong results here when V differs from K (f16 K/V is correct).
// Decline it, as this backend does without the kernel library.
if (is_q8_0 && dk == 512) {
return false;
}
return (backend_ctx->gpu_family == GPU_FAMILY::ADRENO &&
(is_mixed || is_q8_0) && (dk == dv)
&& (dk == 64 || dk == 128 || dk == 256 || dk == 512)
@@ -11775,7 +11832,8 @@ inline bool use_q6_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
inline bool use_q4_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx, const ggml_tensor *tensor) {
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans ||
!backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin) {
!(backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -11924,8 +11982,10 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
ggml_is_contiguous(op->src[0]);
case GGML_OP_UNARY:
switch (ggml_get_unary_op(op)) {
case GGML_UNARY_OP_GELU:
case GGML_UNARY_OP_SILU:
return op->src[0]->type == op->type &&
(op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16);
case GGML_UNARY_OP_GELU:
case GGML_UNARY_OP_RELU:
case GGML_UNARY_OP_GELU_ERF:
case GGML_UNARY_OP_GELU_QUICK:
@@ -19579,15 +19639,42 @@ static void ggml_cl_silu(ggml_backend_t backend, const ggml_tensor * src0, const
cl_ulong offset0 = extra0->offset + src0->view_offs;
cl_ulong offsetd = extrad->offset + dst->view_offs;
if (!ggml_is_contiguous(src0) || !ggml_is_contiguous(dst)) {
GGML_TENSOR_LOCALS(int, ne0, src0, ne);
GGML_TENSOR_LOCALS(cl_ulong, nb0, src0, nb);
GGML_TENSOR_LOCALS(cl_ulong, nb, dst, nb);
cl_kernel kernel = src0->type == GGML_TYPE_F32 ? backend_ctx->kernel_silu_nc : backend_ctx->kernel_silu_f16_nc;
CL_CHECK(clSetKernelArg(kernel, 0, sizeof(cl_mem), &extra0->data_device));
CL_CHECK(clSetKernelArg(kernel, 1, sizeof(cl_ulong), &offset0));
CL_CHECK(clSetKernelArg(kernel, 2, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, 3, sizeof(cl_ulong), &offsetd));
CL_CHECK(clSetKernelArg(kernel, 4, sizeof(int), &ne00));
CL_CHECK(clSetKernelArg(kernel, 5, sizeof(cl_ulong), &nb00));
CL_CHECK(clSetKernelArg(kernel, 6, sizeof(cl_ulong), &nb01));
CL_CHECK(clSetKernelArg(kernel, 7, sizeof(cl_ulong), &nb02));
CL_CHECK(clSetKernelArg(kernel, 8, sizeof(cl_ulong), &nb03));
CL_CHECK(clSetKernelArg(kernel, 9, sizeof(cl_ulong), &nb0));
CL_CHECK(clSetKernelArg(kernel, 10, sizeof(cl_ulong), &nb1));
CL_CHECK(clSetKernelArg(kernel, 11, sizeof(cl_ulong), &nb2));
CL_CHECK(clSetKernelArg(kernel, 12, sizeof(cl_ulong), &nb3));
size_t global_work_size[] = {(size_t)ne01 * 64, (size_t)ne02, (size_t)ne03};
size_t local_work_size[] = {64, 1, 1};
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
return;
}
cl_kernel kernel;
int n = ggml_nelements(dst);
if (n % 4 == 0) {
kernel = backend_ctx->kernel_silu_4;
kernel = src0->type == GGML_TYPE_F32 ? backend_ctx->kernel_silu_4 : backend_ctx->kernel_silu_f16_4;
n /= 4;
} else {
kernel = backend_ctx->kernel_silu;
kernel = src0->type == GGML_TYPE_F32 ? backend_ctx->kernel_silu : backend_ctx->kernel_silu_f16;
}
CL_CHECK(clSetKernelArg(kernel, 0, sizeof(cl_mem), &extra0->data_device));
@@ -22794,6 +22881,7 @@ static void ggml_cl_flash_attn_prefill_bin(ggml_backend_t backend, const ggml_te
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
CL_CHECK(clReleaseMemObject(mem_tex_matrixO_1dbuf));
CL_CHECK(clReleaseMemObject(mem_tex_mask_fallback_1dbuf));
CL_CHECK(clReleaseMemObject(img_q_wmm));
CL_CHECK(clReleaseMemObject(img_k_wmm));
CL_CHECK(clReleaseMemObject(img_v_wmm));
@@ -27338,6 +27426,87 @@ static void ggml_cl_mul_mat_q8_0_f32_adreno(ggml_backend_t backend, const ggml_t
return;
}
const bool have_a8 = (backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin != nullptr);
if (!have_a8 && backend_ctx->kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7_bin && M >= 64) {
const int gemm_tile_n = 32;
int N_pad = CEIL_DIV(N, gemm_tile_n) * gemm_tile_n;
cl_mem a_img_a7 = nullptr;
cl_mem b_src_sub = nullptr;
cl_mem b_pad_sub = nullptr;
cl_mem b_buf = nullptr;
img_fmt = { CL_R, CL_FLOAT };
memset(&img_desc, 0, sizeof(img_desc));
img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
img_desc.image_width = (size_t)M * K / 4;
img_desc.buffer = extra0_q8_0->q;
CL_CHECK((a_img_a7 = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
region.origin = offset1;
region.size = (size_t)K * N * sizeof(float);
CL_CHECK((b_src_sub = clCreateSubBuffer(extra1->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
if (N_pad == N) {
b_buf = b_src_sub;
} else {
size_t padded_bytes = (size_t)K * N_pad * sizeof(float);
backend_ctx->prealloc_act_trans.allocate(context, padded_bytes);
region.origin = 0;
region.size = padded_bytes;
CL_CHECK((b_pad_sub = clCreateSubBuffer(backend_ctx->prealloc_act_trans.buffer, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
const float b_pad_zero = 0.0f;
CL_CHECK(clEnqueueFillBuffer(backend_ctx->queue, b_pad_sub, &b_pad_zero, sizeof(float),
0, padded_bytes, 0, NULL, NULL));
CL_CHECK(clEnqueueCopyBuffer(backend_ctx->queue, b_src_sub, b_pad_sub, 0, 0, (size_t)K * N * sizeof(float), 0, NULL, NULL));
b_buf = b_pad_sub;
}
img_fmt = { CL_R, CL_FLOAT };
memset(&img_desc, 0, sizeof(img_desc));
img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
img_desc.image_width = (size_t)K * N_pad;
img_desc.buffer = b_buf;
cl_mem b_img_a7;
CL_CHECK((b_img_a7 = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
const int c_offset_for_kernel = (int)offsetd;
int line_stride_B = K * (int)sizeof(float);
int line_stride_C = M * (int)sizeof(float);
kernel = backend_ctx->kernel_gemm_noshuffle_q8_0_f32_32b_trans_ila_a7_bin;
cl_uint k_arg = 0;
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &a_img_a7));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q8_0->d));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &b_img_a7));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &offset1));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &c_offset_for_kernel));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne00));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne01));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_B));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_C));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &N));
size_t local_work_size[3] = { 64, 2, 4 };
size_t m_tiles = (size_t)CEIL_DIV(M, 64);
size_t global_y = ((m_tiles + 1) / 2) * 2;
size_t global_work_size[3] = { 64, global_y, (size_t)CEIL_DIV(N_pad, 32) };
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
CL_CHECK(clReleaseMemObject(a_img_a7));
CL_CHECK(clReleaseMemObject(b_img_a7));
if (b_src_sub) {
CL_CHECK(clReleaseMemObject(b_src_sub));
}
if (b_pad_sub) {
CL_CHECK(clReleaseMemObject(b_pad_sub));
}
return;
}
// use bin kernel if available
if (backend_ctx->kernel_gemm_noshuffle_q8_0_f32_bin) {
int K_pad = K;
@@ -27675,6 +27844,8 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno_ila(ggml_backend_t backend, const gg
const int gemm_tile_n = 64;
int N_pad = CEIL_DIV(N, gemm_tile_n) * gemm_tile_n;
const bool have_a8 = backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin != nullptr;
static const char * q4_k_bin_dp4a_env = getenv("GGML_OPENCL_Q4_K_BIN_DP4A");
bool q4_k_bin_dp4a_on = q4_k_bin_dp4a_env
? (atoi(q4_k_bin_dp4a_env) != 0)
@@ -27744,6 +27915,74 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno_ila(ggml_backend_t backend, const gg
return;
}
if (!have_a8) {
cl_mem b_src_sub = nullptr;
cl_mem b_pad_sub = nullptr;
cl_mem b_buf = nullptr;
region.origin = offset1;
region.size = (size_t)K * N * sizeof(float);
CL_CHECK((b_src_sub = clCreateSubBuffer(extra1->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
if (N_pad == N) {
b_buf = b_src_sub;
} else {
size_t padded_bytes = (size_t)K * N_pad * sizeof(float);
backend_ctx->prealloc_act_trans.allocate(context, padded_bytes);
region.origin = 0;
region.size = padded_bytes;
CL_CHECK((b_pad_sub = clCreateSubBuffer(backend_ctx->prealloc_act_trans.buffer, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
const float b_pad_zero = 0.0f;
CL_CHECK(clEnqueueFillBuffer(backend_ctx->queue, b_pad_sub, &b_pad_zero, sizeof(float),
0, padded_bytes, 0, NULL, NULL));
CL_CHECK(clEnqueueCopyBuffer(backend_ctx->queue, b_src_sub, b_pad_sub, 0, 0, (size_t)K * N * sizeof(float), 0, NULL, NULL));
b_buf = b_pad_sub;
}
img_fmt = { CL_R, CL_FLOAT };
memset(&img_desc, 0, sizeof(img_desc));
img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
img_desc.image_width = (size_t)K * N_pad;
img_desc.buffer = b_buf;
cl_mem b_img_a7;
CL_CHECK((b_img_a7 = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
const int c_offset_for_kernel = (int)offsetd;
int line_stride_B = K * (int)sizeof(float);
int line_stride_C = M * (int)sizeof(float);
kernel = backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin;
cl_uint k_arg = 0;
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->q_img));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->d));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->dm));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->s));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &b_img_a7));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &offset1));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &c_offset_for_kernel));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne00));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne01));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_B));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_C));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &N));
size_t local_work_size[3] = { 64, 2, 4 };
size_t m_tiles = (size_t)CEIL_DIV(M, 64);
size_t global_y = ((m_tiles + 1) / 2) * 2;
size_t global_work_size[3] = { 64, global_y, (size_t)CEIL_DIV(N_pad, 32) };
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
CL_CHECK(clReleaseMemObject(b_img_a7));
if (b_src_sub) {
CL_CHECK(clReleaseMemObject(b_src_sub));
}
if (b_pad_sub) {
CL_CHECK(clReleaseMemObject(b_pad_sub));
}
return;
}
cl_mem b_sub_buf = nullptr;
cl_mem b_padded = nullptr;
cl_mem b_buf = nullptr;
+85
View File
@@ -28,3 +28,88 @@ kernel void kernel_silu_4(
float4 x = src0[get_global_id(0)];
dst[get_global_id(0)] = x / (1.0f + exp(-x));
}
kernel void kernel_silu_nc(
global const char * src0,
ulong offset0,
global char * dst,
ulong offsetd,
int ne00,
ulong nb00,
ulong nb01,
ulong nb02,
ulong nb03,
ulong nb0,
ulong nb1,
ulong nb2,
ulong nb3
) {
src0 = src0 + offset0;
dst = dst + offsetd;
const int i3 = get_group_id(2);
const int i2 = get_group_id(1);
const int i1 = get_group_id(0);
for (int i0 = get_local_id(0); i0 < ne00; i0 += get_local_size(0)) {
global const float * x = (global const float *)(src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
global float * y = (global float *)(dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0);
*y = *x / (1.0f + exp(-*x));
}
}
kernel void kernel_silu_f16(
global half * src0,
ulong offset0,
global half * dst,
ulong offsetd
) {
src0 = (global half*)((global char*)src0 + offset0);
dst = (global half*)((global char*)dst + offsetd);
float x = src0[get_global_id(0)];
dst[get_global_id(0)] = x / (1.0f + exp(-x));
}
kernel void kernel_silu_f16_4(
global half4 * src0,
ulong offset0,
global half4 * dst,
ulong offsetd
) {
src0 = (global half4*)((global char*)src0 + offset0);
dst = (global half4*)((global char*)dst + offsetd);
float4 x = convert_float4(src0[get_global_id(0)]);
dst[get_global_id(0)] = convert_half4(x / (1.0f + exp(-x)));
}
kernel void kernel_silu_f16_nc(
global const char * src0,
ulong offset0,
global char * dst,
ulong offsetd,
int ne00,
ulong nb00,
ulong nb01,
ulong nb02,
ulong nb03,
ulong nb0,
ulong nb1,
ulong nb2,
ulong nb3
) {
src0 = src0 + offset0;
dst = dst + offsetd;
const int i3 = get_group_id(2);
const int i2 = get_group_id(1);
const int i1 = get_group_id(0);
for (int i0 = get_local_id(0); i0 < ne00; i0 += get_local_size(0)) {
float x = *(global const half *)(src0 + i3*nb03 + i2*nb02 + i1*nb01 + i0*nb00);
*(global half *)(dst + i3*nb3 + i2*nb2 + i1*nb1 + i0*nb0) = x / (1.0f + exp(-x));
}
}
+32 -24
View File
@@ -10,10 +10,11 @@
template<float (*bin_op)(const float, const float), typename src0_t, typename src1_t, typename dst_t>
static void k_bin_bcast(const src0_t * src0, const src1_t * src1, dst_t * dst,
int ne0, int ne1, int ne2, int ne3,
int ne10, int ne11, int ne12, int ne13,
int ne11, int ne12, int ne13,
/*int s0, */ int s1, int s2, int s3,
int s00, int s01, int s02, int s03,
int s10, int s11, int s12, int s13,
const sycl::uint3 ne10_fd,
const sycl::nd_item<3> &item_ct1) {
const int i0s = item_ct1.get_local_range(2) * item_ct1.get_group(2) +
item_ct1.get_local_id(2);
@@ -44,7 +45,7 @@ static void k_bin_bcast(const src0_t * src0, const src1_t * src1, dst_t * dst,
for (int i0 = i0s; i0 < ne0;
i0 += item_ct1.get_local_range(2) * item_ct1.get_group_range(2)) {
const int i10 = i0 % ne10;
const int i10 = (int) fast_div_modulo((uint32_t) i0, ne10_fd).y();
dst_row[i0] = (dst_t)bin_op(src0 ? (float)src0_row[i0*s00] : 0.0f, (float)src1_row[i10*s10]);
}
}
@@ -52,10 +53,11 @@ static void k_bin_bcast(const src0_t * src0, const src1_t * src1, dst_t * dst,
template<float (*bin_op)(const float, const float), typename src0_t, typename src1_t, typename dst_t>
static void k_bin_bcast_unravel(const src0_t * src0, const src1_t * src1, dst_t * dst,
int ne0, int ne1, int ne2, int ne3,
int ne10, int ne11, int ne12, int ne13,
int ne11, int ne12, int ne13,
/*int s0, */ int s1, int s2, int s3,
int s00, int s01, int s02, int s03,
int s10, int s11, int s12, int s13,
const sycl::uint3 ne10_fd,
const sycl::nd_item<3> &item_ct1) {
const int i = item_ct1.get_local_range(2) * item_ct1.get_group(2) +
@@ -82,7 +84,7 @@ static void k_bin_bcast_unravel(const src0_t * src0, const src1_t * src1, dst_t
const src1_t * src1_row = src1 + i_src1;
dst_t * dst_row = dst + i_dst;
const int i10 = i0 % ne10;
const int i10 = (int) fast_div_modulo((uint32_t) i0, ne10_fd).y();
dst_row[i0] = (dst_t)bin_op(src0 ? (float)src0_row[i0*s00] : 0.0f, (float)src1_row[i10*s10]);
}
@@ -146,7 +148,8 @@ struct bin_bcast_sycl {
int64_t ne2 = cne[2];
int64_t ne3 = cne[3];
int64_t ne10 = cne1[0];
const sycl::uint3 ne10_fd = init_fastdiv_values((uint32_t) cne1[0]);
int64_t ne11 = cne1[1];
int64_t ne12 = cne1[2];
int64_t ne13 = cne1[3];
@@ -231,8 +234,8 @@ struct bin_bcast_sycl {
[=](sycl::nd_item<3> item_ct1) {
k_bin_bcast_unravel<bin_op>(
src0_dd, src1_dd, dst_dd, ne0, ne1, ne2, ne3,
ne10, ne11, ne12, ne13, s1, s2, s3, s00, s01, s02,
s03, s10, s11, s12, s13, item_ct1);
ne11, ne12, ne13, s1, s2, s3, s00, s01, s02,
s03, s10, s11, s12, s13, ne10_fd, item_ct1);
});
}
} else {
@@ -249,9 +252,9 @@ struct bin_bcast_sycl {
sycl::nd_range<3>(block_nums * block_dims, block_dims),
[=](sycl::nd_item<3> item_ct1) {
k_bin_bcast<bin_op>(src0_dd, src1_dd, dst_dd, ne0, ne1,
ne2, ne3, ne10, ne11, ne12, ne13,
ne2, ne3, ne11, ne12, ne13,
s1, s2, s3, s00, s01, s02, s03, s10, s11, s12, s13,
item_ct1);
ne10_fd, item_ct1);
});
}
}
@@ -362,12 +365,13 @@ void ggml_sycl_repeat(ggml_backend_sycl_context & ctx, ggml_tensor * dst) {
template<float (*bin_op)(const float, const float), typename src0_t, typename src1_t, typename src2_t, typename dst_t>
static void k_bin_bcast3(const src0_t * src0, const src1_t * src1, const src2_t * src2, dst_t * dst,
int ne0, int ne1, int ne2, int ne3,
int ne10, int ne11, int ne12, int ne13,
int ne20, int ne21, int ne22, int ne23,
int ne11, int ne12, int ne13,
int ne21, int ne22, int ne23,
int s1, int s2, int s3,
int s00, int s01, int s02, int s03,
int s10, int s11, int s12, int s13,
int s20, int s21, int s22, int s23,
const sycl::uint3 ne10_fd, const sycl::uint3 ne20_fd,
const sycl::nd_item<3> & item_ct1) {
const int i0s = item_ct1.get_local_range(2) * item_ct1.get_group(2) +
item_ct1.get_local_id(2);
@@ -403,8 +407,8 @@ static void k_bin_bcast3(const src0_t * src0, const src1_t * src1, const src2_t
for (int i0 = i0s; i0 < ne0;
i0 += item_ct1.get_local_range(2) * item_ct1.get_group_range(2)) {
const int i10 = i0 % ne10;
const int i20 = i0 % ne20;
const int i10 = (int) fast_div_modulo((uint32_t) i0, ne10_fd).y();
const int i20 = (int) fast_div_modulo((uint32_t) i0, ne20_fd).y();
const float acc = bin_op((float) src0_row[i0 * s00], (float) src1_row[i10 * s10]);
dst_row[i0] = (dst_t) bin_op(acc, (float) src2_row[i20 * s20]);
}
@@ -413,12 +417,13 @@ static void k_bin_bcast3(const src0_t * src0, const src1_t * src1, const src2_t
template<float (*bin_op)(const float, const float), typename src0_t, typename src1_t, typename src2_t, typename dst_t>
static void k_bin_bcast3_unravel(const src0_t * src0, const src1_t * src1, const src2_t * src2, dst_t * dst,
int ne0, int ne1, int ne2, int ne3,
int ne10, int ne11, int ne12, int ne13,
int ne20, int ne21, int ne22, int ne23,
int ne11, int ne12, int ne13,
int ne21, int ne22, int ne23,
int s1, int s2, int s3,
int s00, int s01, int s02, int s03,
int s10, int s11, int s12, int s13,
int s20, int s21, int s22, int s23,
const sycl::uint3 ne10_fd, const sycl::uint3 ne20_fd,
const sycl::nd_item<3> & item_ct1) {
const int i = item_ct1.get_local_range(2) * item_ct1.get_group(2) +
item_ct1.get_local_id(2);
@@ -444,8 +449,8 @@ static void k_bin_bcast3_unravel(const src0_t * src0, const src1_t * src1, const
const size_t i_src2 = i23 * s23 + i22 * s22 + i21 * s21;
const size_t i_dst = i3 * s3 + i2 * s2 + i1 * s1;
const int i10 = i0 % ne10;
const int i20 = i0 % ne20;
const int i10 = (int) fast_div_modulo((uint32_t) i0, ne10_fd).y();
const int i20 = (int) fast_div_modulo((uint32_t) i0, ne20_fd).y();
const float acc = bin_op((float) src0[i_src0 + i0 * s00], (float) src1[i_src1 + i10 * s10]);
dst[i_dst + i0] = (dst_t) bin_op(acc, (float) src2[i_src2 + i20 * s20]);
}
@@ -547,6 +552,9 @@ static void launch_bin_bcast3(ggml_backend_sycl_context & ctx, const ggml_tensor
GGML_ASSERT(cnb2[0] % sizeof(src2_t) == 0 && cnb2[1] % sizeof(src2_t) == 0 && cnb2[2] % sizeof(src2_t) == 0 &&
cnb2[3] % sizeof(src2_t) == 0);
const sycl::uint3 ne10_fd = init_fastdiv_values((uint32_t) ne10);
const sycl::uint3 ne20_fd = init_fastdiv_values((uint32_t) ne20);
const src0_t * src0_dd = (const src0_t *) src0->data;
const src1_t * src1_dd = (const src1_t *) src1->data;
const src2_t * src2_dd = (const src2_t *) src2->data;
@@ -575,17 +583,18 @@ static void launch_bin_bcast3(ggml_backend_sycl_context & ctx, const ggml_tensor
sycl::nd_range<3>(sycl::range<3>(1, 1, block_num) * sycl::range<3>(1, 1, block_size),
sycl::range<3>(1, 1, block_size)),
[=](sycl::nd_item<3> item_ct1) {
k_bin_bcast3_unravel<bin_op>(src0_dd, src1_dd, src2_dd, dst_dd, ne0, ne1, ne2, ne3, ne10, ne11,
ne12, ne13, ne20, ne21, ne22, ne23, s1, s2, s3, s00, s01, s02, s03,
s10, s11, s12, s13, s20, s21, s22, s23, item_ct1);
k_bin_bcast3_unravel<bin_op>(src0_dd, src1_dd, src2_dd, dst_dd, ne0, ne1, ne2, ne3,
ne11, ne12, ne13, ne21, ne22, ne23, s1, s2, s3, s00, s01, s02,
s03, s10, s11, s12, s13, s20, s21, s22, s23, ne10_fd, ne20_fd,
item_ct1);
});
} else {
stream->parallel_for(sycl::nd_range<3>(block_nums * block_dims, block_dims),
[=](sycl::nd_item<3> item_ct1) {
k_bin_bcast3<bin_op>(src0_dd, src1_dd, src2_dd, dst_dd, ne0, ne1, ne2, ne3, ne10,
ne11, ne12, ne13, ne20, ne21, ne22, ne23, s1, s2, s3, s00,
k_bin_bcast3<bin_op>(src0_dd, src1_dd, src2_dd, dst_dd, ne0, ne1, ne2, ne3,
ne11, ne12, ne13, ne21, ne22, ne23, s1, s2, s3, s00,
s01, s02, s03, s10, s11, s12, s13, s20, s21, s22, s23,
item_ct1);
ne10_fd, ne20_fd, item_ct1);
});
}
}
@@ -647,4 +656,3 @@ void ggml_sycl_op_add_add_fused(ggml_backend_sycl_context & ctx, ggml_tensor * a
GGML_ABORT("fatal error");
}
}
+6 -2
View File
@@ -16,6 +16,7 @@
#include <cstddef>
#include <fstream>
#include <iostream>
#include <limits>
#include <optional>
#include <string>
#include <type_traits>
@@ -909,8 +910,11 @@ static __dpct_inline__ float get_alibi_slope(const float max_bias,
return dpct::pow(base, exph);
}
static const sycl::uint3 init_fastdiv_values(uint32_t d) {
GGML_ASSERT(d != 0);
static const sycl::uint3 init_fastdiv_values(uint64_t d_64) {
GGML_ASSERT(d_64 != 0);
GGML_ASSERT(d_64 <= std::numeric_limits<uint32_t>::max());
const uint32_t d = (uint32_t) d_64;
uint32_t L = 0;
while (L < 32 && (uint32_t{ 1 } << L) < d) {
+2 -2
View File
@@ -160,7 +160,7 @@ static void mkl_fa_online_softmax_chunk(
int64_t m_stride = 0;
if (mask_data) {
int m_head = (mask_n_heads > 1)
? (kvh_head + gqa_group) : 0;
? (kvh_head + gqa_group) % mask_n_heads : 0;
mask_h = mask_data + (int64_t)m_head * mask_head_stride;
m_stride = mask_row_stride;
}
@@ -537,7 +537,7 @@ void ggml_sycl_flash_attn_ext_mkl(ggml_backend_sycl_context & ctx, ggml_tensor *
const sycl::half * mask_batch = nullptr;
if (mask) {
int m_batch = (mask->ne[3] > 1) ? ib : 0;
int m_batch = ib % (int)mask->ne[3];
mask_batch = (const sycl::half *)mask->data
+ m_batch * (mask->nb[3] / 2); // 2 = actual fp16 device size
}
+2 -2
View File
@@ -335,7 +335,7 @@ static int ggml_vk_comm_sfd_export(vk_comm_device & d) {
int fd = -1;
VkSemaphoreGetFdInfoKHR gi{};
gi.sType = VK_STRUCTURE_TYPE_SEMAPHORE_GET_FD_INFO_KHR;
gi.semaphore = d.sfd.sem;
gi.semaphore = static_cast<VkSemaphore>(d.sfd.sem);
gi.handleType = VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_SYNC_FD_BIT;
const VkResult result = d.sfd.get_fd((VkDevice) d.ctx->device->device, &gi, &fd);
if (result != VK_SUCCESS) {
@@ -348,7 +348,7 @@ static int ggml_vk_comm_sfd_export(vk_comm_device & d) {
static void ggml_vk_comm_sfd_import(vk_comm_device & d, vk::Semaphore sem, int fd) {
VkImportSemaphoreFdInfoKHR isi{};
isi.sType = VK_STRUCTURE_TYPE_IMPORT_SEMAPHORE_FD_INFO_KHR;
isi.semaphore = sem;
isi.semaphore = static_cast<VkSemaphore>(sem);
isi.flags = VK_SEMAPHORE_IMPORT_TEMPORARY_BIT;
isi.handleType = VK_EXTERNAL_SEMAPHORE_HANDLE_TYPE_SYNC_FD_BIT;
isi.fd = fd;
+1 -1
View File
@@ -70,7 +70,7 @@ void ggml_vk_buffer_memset(vk_buffer& dst, size_t offset, uint32_t c, size_t siz
vk_buffer ggml_vk_buffer_from_host_ptr(vk_device & device, void * ptr, size_t size);
// pipelines
uint64_t vk_tensor_offset(const ggml_tensor * tensor);
size_t vk_tensor_offset(const ggml_tensor * tensor);
uint32_t get_misalign_bytes(const ggml_backend_vk_context * ctx, const ggml_tensor * t);
void ggml_vk_wait_for_fence(ggml_backend_vk_context * ctx);
void ggml_pipeline_request_descriptor_sets(ggml_backend_vk_context *ctx, vk_pipeline& pipeline, uint32_t n);
+5 -5
View File
@@ -466,15 +466,15 @@ struct vk_conv2d_pipeline_state {
struct vk_conv3d_pipeline_state {
vk_conv3d_pipeline_state(uint32_t s0, uint32_t s1, uint32_t s2, uint32_t p0, uint32_t p1, uint32_t p2,
uint32_t d0, uint32_t d1, uint32_t d2, uint32_t KW, uint32_t KH, uint32_t KD, uint32_t aligned)
: s0(s0), s1(s1), s2(s2), p0(p0), p1(p1), p2(p2), d0(d0), d1(d1), d2(d2), KW(KW), KH(KH), KD(KD), aligned(aligned) {}
uint32_t d0, uint32_t d1, uint32_t d2, uint32_t KW, uint32_t KH, uint32_t KD, uint32_t aligned, uint32_t split_k)
: s0(s0), s1(s1), s2(s2), p0(p0), p1(p1), p2(p2), d0(d0), d1(d1), d2(d2), KW(KW), KH(KH), KD(KD), aligned(aligned), split_k(split_k) {}
uint32_t s0, s1, s2, p0, p1, p2, d0, d1, d2, KW, KH, KD;
uint32_t aligned;
uint32_t aligned, split_k;
bool operator<(const vk_conv3d_pipeline_state &b) const {
return std::tie(s0, s1, s2, p0, p1, p2, d0, d1, d2, KW, KH, KD, aligned) <
std::tie(b.s0, b.s1, b.s2, b.p0, b.p1, b.p2, b.d0, b.d1, b.d2, b.KW, b.KH, b.KD, b.aligned);
return std::tie(s0, s1, s2, p0, p1, p2, d0, d1, d2, KW, KH, KD, aligned, split_k) <
std::tie(b.s0, b.s1, b.s2, b.p0, b.p1, b.p2, b.d0, b.d1, b.d2, b.KW, b.KH, b.KD, b.aligned, b.split_k);
}
};
+103 -25
View File
@@ -259,7 +259,7 @@ void ggml_vk_print_device_lost_info(const vk_device& device) {
}
void * const vk_ptr_base = (void *)(uintptr_t) 0x1000; // NOLINT
uint64_t vk_tensor_offset(const ggml_tensor * tensor) {
size_t vk_tensor_offset(const ggml_tensor * tensor) {
if (tensor->view_src) {
return (uint8_t *) tensor->view_src->data - (uint8_t *) vk_ptr_base;
}
@@ -1258,10 +1258,14 @@ static vk_fa_tuning_params get_fa_tuning_params_scalar(const vk_device& device,
result.shmem_staging = (device->vendor_id == VK_VENDOR_ID_NVIDIA && hsk < 256 && hsv < 256) ? 1 : 0;
if (!reduce_block_rows && !ggml_vk_flash_attn_scalar_shmem_support(device, result, hsk, hsv, f32acc, k_type, v_type)) {
if (!reduce_block_rows && result.block_rows > 1 && !ggml_vk_flash_attn_scalar_shmem_support(device, result, hsk, hsv, f32acc, k_type, v_type)) {
result.block_rows /= 2;
}
if (result.shmem_staging && !ggml_vk_flash_attn_scalar_shmem_support(device, result, hsk, hsv, f32acc, k_type, v_type)) {
result.shmem_staging = 0;
}
// With head sizes above 128, 16 rows per workgroup spill registers on Qualcomm and run about
// 40x slower; at large KV a single dispatch can then exceed the Windows GPU timeout.
if (device->vendor_id == VK_VENDOR_ID_QUALCOMM && (hsk > 128 || hsv > 128)) {
@@ -1326,7 +1330,9 @@ static vk_fa_tuning_params get_fa_tuning_params_coopmat1(const vk_device& device
static const bool disable_flip = getenv("GGML_VK_DISABLE_FA_FLIP") != nullptr;
static const char * force_flip = getenv("GGML_VK_FORCE_FA_FLIP");
const bool flip_arch = force_flip != nullptr || device->vendor_id == VK_VENDOR_ID_NVIDIA ||
(device->vendor_id == VK_VENDOR_ID_AMD && (device->architecture == AMD_RDNA3 || device->architecture == AMD_RDNA4));
(device->vendor_id == VK_VENDOR_ID_AMD && (device->architecture == AMD_RDNA3 || device->architecture == AMD_RDNA4) &&
// AMD proprietary driver crashes compiling the flip shader
device->driver_id != vk::DriverId::eAmdProprietary);
if (allow_flip && !disable_flip && flip_arch && n_rows >= 64 &&
(result.subgroup_size == 32 || result.subgroup_size == 64) &&
hsk % 16 == 0 && hsv % 16 == 0 && hsk <= 256 && hsv <= 256 &&
@@ -3886,8 +3892,16 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
// variant (spec constant 1) additionally gathers the qwen4 indexer input on the fly.
{
const uint32_t BLOCK_SIZE = 1u << std::min(10u, device->max_workgroup_size_log2);
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_f32, "topk_radix_f32", topk_radix_select_f32_len, topk_radix_select_f32_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 0}, 1, true);
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_qsa, "topk_radix_qsa", topk_radix_select_f32_len, topk_radix_select_f32_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 1}, 1, true);
// the bucket scan needs one invocation per radix bucket
if (BLOCK_SIZE >= 256) {
if (device->subgroup_arithmetic && device->subgroup_require_full_support) {
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_f32, "topk_radix_f32", topk_radix_select_f32_subgroup_len, topk_radix_select_f32_subgroup_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 0, device->subgroup_size}, 1, true, true, device->subgroup_size);
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_qsa, "topk_radix_qsa", topk_radix_select_f32_subgroup_len, topk_radix_select_f32_subgroup_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 1, device->subgroup_size}, 1, true, true, device->subgroup_size);
} else {
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_f32, "topk_radix_f32", topk_radix_select_f32_len, topk_radix_select_f32_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 0}, 1, true);
ggml_vk_create_pipeline2(device, device->pipeline_topk_radix_qsa, "topk_radix_qsa", topk_radix_select_f32_len, topk_radix_select_f32_data, "main", 5, sizeof(vk_op_topk_radix_push_constants), {BLOCK_SIZE, 1, 1}, {BLOCK_SIZE, 1}, 1, true);
}
}
}
ggml_vk_create_pipeline(device, device->pipeline_argmax_f32, "argmax_f32", argmax_f32_len, argmax_f32_data, "main", 2, sizeof(vk_op_push_constants), {1, 1, 1}, { device->subgroup_size }, 1);
@@ -4282,6 +4296,7 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
spec_constants_cpy.push_back(conv2d_csh_store); \
spec_constants_cpy.push_back(conv2d_WM); \
spec_constants_cpy.push_back(conv2d_WN); \
spec_constants_cpy.push_back(state.split_k); \
ggml_vk_create_pipeline( \
device, c.second, "conv3d" #type_suffix, \
conv3d##type_suffix##spv_suffix##_len, conv3d##type_suffix##spv_suffix##_data, "main", 3, \
@@ -8520,7 +8535,9 @@ bool ggml_vk_flash_attn_scalar_shmem_support(const vk_device& device, const vk_f
kblocksh_size = 0;
}
const uint32_t total_size = tmpsh + tmpshv4 + masksh + iq_shmem + Qf + kvsh + kblocksh_size;
const uint32_t occupancy_limiter = std::max(params.limit_occupancy_shmem, 1u) * 4 * sizeof(float);
const uint32_t total_size = tmpsh + tmpshv4 + masksh + iq_shmem + Qf + kvsh + kblocksh_size + occupancy_limiter;
const bool supported = total_size <= device->properties.limits.maxComputeSharedMemorySize;
VK_LOG_DEBUG("ggml_vk_flash_attn_scalar_shmem_support(HSK=" << hsk << ", HSV=" << hsv << ", mmq=" << mmq << ", total_size=" << total_size << ", supported=" << supported);
@@ -9271,24 +9288,58 @@ static vk_conv_shapes ggml_vk_conv_select_shape(ggml_backend_vk_context * ctx, u
}
}
static vk_conv_shapes ggml_vk_conv3d_select_shape(ggml_backend_vk_context * ctx, uint32_t K, uint32_t NPQ) {
const auto & block = vk_conv_block_sizes[CONV_SHAPE_256x128];
static vk_conv_shapes ggml_vk_conv3d_select_shape(ggml_backend_vk_context * ctx, const ggml_tensor * kernel, const ggml_tensor * dst, uint32_t & split_k) {
split_k = 1;
const uint32_t K = ggml_get_op_params_i32(dst, 11);
const uint32_t NPQ = ggml_get_op_params_i32(dst, 10) * dst->ne[0] * dst->ne[1] * dst->ne[2];
const vk_conv_shapes fallback = ggml_vk_conv_select_shape(ctx, K, NPQ);
const uint32_t shader_core_count = ctx->device->shader_core_count;
// Slowdowns were observed on DGX Spark with drivers older than R615.
const bool driver_supported = ctx->device->driver_id != vk::DriverId::eNvidiaProprietary ||
(ctx->device->properties.driverVersion >> 22) >= 615;
if (ctx->device->coopmat2 && shader_core_count > 0 && K % block.K == 0 && driver_supported) {
const uint64_t n_tiles = (uint64_t)(K / block.K) * CEIL_DIV((uint64_t)NPQ, block.NPQ);
// Use the largest cm2 padding and include the dummy Csh[8].
const uint32_t pad = 8, csh_elems = 8;
const uint32_t shmem = (block.K * (block.CRS + pad) + block.CRS * (block.NPQ + pad) + csh_elems) * sizeof(uint16_t);
// Reuse each loaded input across 256 output channels instead of 128.
// Require full channel tiles and at least as many workgroups as GPU cores.
if (n_tiles >= shader_core_count && shmem <= ctx->device->properties.limits.maxComputeSharedMemorySize) {
return CONV_SHAPE_256x128;
}
if (!ctx->device->coopmat2 || shader_core_count == 0 || K == 0 || NPQ == 0 || !driver_supported) {
return fallback;
}
return ggml_vk_conv_select_shape(ctx, K, NPQ);
const auto & block = vk_conv_block_sizes[CONV_SHAPE_256x128];
const auto & limits = ctx->device->properties.limits;
// Use the largest cm2 padding and include the dummy Csh[8].
const uint32_t pad = 8, csh_elems = 8;
const uint32_t shmem = (block.K * (block.CRS + pad) + block.CRS * (block.NPQ + pad) + csh_elems) * sizeof(uint16_t);
if (shmem > limits.maxComputeSharedMemorySize) {
return fallback;
}
const uint32_t channel_tiles = CEIL_DIV(K, block.K);
const uint64_t n_tiles = (uint64_t)channel_tiles * CEIL_DIV(NPQ, block.NPQ);
// Keep full channel tiles when the grid already fills the GPU.
if (n_tiles >= shader_core_count) {
return K % block.K == 0 ? CONV_SHAPE_256x128 : fallback;
}
static const bool disable_split_k = getenv("GGML_VK_DISABLE_CONV3D_SPLIT_K") != nullptr;
// Do not use a tile with more than half its output channels padded.
if (disable_split_k || K < block.K / 2 || !ggml_is_contiguous(dst) || get_misalign_bytes(ctx, dst) != 0) {
return fallback;
}
// Use power-of-two splits to give each core at least one workgroup.
uint64_t splits = 1;
while (n_tiles * splits < shader_core_count) {
splits *= 2;
}
const uint64_t crs = (uint64_t)ggml_get_op_params_i32(dst, 9) * kernel->ne[0] * kernel->ne[1] * kernel->ne[2];
const uint64_t operand_bytes = block.K * ggml_type_size(kernel->type) + block.NPQ * sizeof(float);
const uint64_t scratch_bytes = 2 * sizeof(float) * block.K * block.NPQ;
// Do not add more scratch bytes than the estimated operand load bytes.
if (crs * operand_bytes < splits * scratch_bytes) {
return fallback;
}
const uint64_t ne = ggml_nelements(dst);
const uint64_t bytes = ggml_nbytes(dst);
if (ne == 0 || CEIL_DIV(ne, ctx->device->pipeline_matmul_split_k_reduce->wg_denoms[0]) > limits.maxComputeWorkGroupCount[0] ||
splits > limits.maxStorageBufferRange / bytes ||
splits > limits.maxComputeWorkGroupCount[0] / channel_tiles) {
return fallback;
}
split_k = (uint32_t)splits;
return CONV_SHAPE_256x128;
}
static vk_pipeline ggml_vk_rms_norm_pipeline(const std::array<vk_pipeline, 3> & pipelines, int64_t ncols) {
@@ -9997,7 +10048,8 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const
const uint32_t IC = (uint32_t)ggml_get_op_params_i32(dst, 9);
const uint32_t N = (uint32_t)ggml_get_op_params_i32(dst, 10);
const uint32_t NPQ = N * dst->ne[2] * dst->ne[1] * dst->ne[0];
const vk_conv_shapes shape = ggml_vk_conv3d_select_shape(ctx, OC, NPQ);
uint32_t split_k;
const vk_conv_shapes shape = ggml_vk_conv3d_select_shape(ctx, src0, dst, split_k);
const uint32_t KW = (uint32_t)src0->ne[0];
const uint32_t KH = (uint32_t)src0->ne[1];
@@ -10020,7 +10072,7 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const
(CRS % BS_CRS == 0) &&
(NPQ % BS_NPQ == 0)) ? 1u : 0u;
vk_conv3d_pipeline_state conv3d_pipeline_state(s0, s1, s2, p0, p1, p2, d0, d1, d2, KW, KH, KD, aligned);
vk_conv3d_pipeline_state conv3d_pipeline_state(s0, s1, s2, p0, p1, p2, d0, d1, d2, KW, KH, KD, aligned, split_k);
std::map<vk_conv3d_pipeline_state, vk_pipeline> *pipelines = nullptr;
if (src0->type == GGML_TYPE_F32) {
@@ -10126,6 +10178,7 @@ static void ggml_vk_op_f32(ggml_backend_vk_context * ctx, vk_context& subctx, co
init_pushconst_tensor_offsets(ctx, pc, src0, src1, src2, src3, dst);
std::array<uint32_t, 3> elements;
uint32_t split_k = 1;
switch (op) {
case GGML_OP_NORM:
@@ -10301,10 +10354,12 @@ static void ggml_vk_op_f32(ggml_backend_vk_context * ctx, vk_context& subctx, co
case GGML_OP_CONV_3D:
if constexpr (std::is_same_v<PC, vk_op_conv3d_push_constants>) {
const uint32_t NPQ = pc.N * pc.OD * pc.OH * pc.OW;
const vk_conv_shapes shape = ggml_vk_conv3d_select_shape(ctx, pc.OC, NPQ);
const uint32_t NPQ_blocks = CEIL_DIV(NPQ, vk_conv_block_sizes[shape].NPQ);
const vk_conv_shapes shape = ggml_vk_conv3d_select_shape(ctx, src0, dst, split_k);
const auto & block = vk_conv_block_sizes[shape];
const uint32_t NPQ_blocks = CEIL_DIV(NPQ, block.NPQ);
const uint32_t channel_groups = split_k > 1 ? CEIL_DIV(pc.OC, block.K) * block.K * split_k : pc.OC;
elements = { pc.OC, NPQ_blocks, 1 };
elements = { channel_groups, NPQ_blocks, 1 };
if (elements[1] > 512) {
elements[2] = CEIL_DIV(elements[1], 512);
elements[1] = 512;
@@ -10435,6 +10490,29 @@ static void ggml_vk_op_f32(ggml_backend_vk_context * ctx, vk_context& subctx, co
break;
}
if constexpr (std::is_same_v<PC, vk_op_conv3d_push_constants>) {
if (split_k > 1) {
const uint32_t ne = (uint32_t)ggml_nelements(dst);
const size_t scratch_size = ggml_nbytes(dst) * split_k;
if (ctx->prealloc_size_split_k < scratch_size) {
ctx->prealloc_size_split_k = scratch_size;
ggml_vk_preallocate_buffers(ctx, subctx);
}
if (ctx->prealloc_split_k_need_sync) {
ggml_vk_sync_buffers(ctx, subctx);
}
vk_subbuffer scratch = { ctx->prealloc_split_k, 0, scratch_size };
pc.dst_offset = 0;
ggml_pipeline_request_descriptor_sets(ctx, ctx->device->pipeline_matmul_split_k_reduce, 1);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { src0_buf, src1_buf, scratch }, pc, elements);
ggml_vk_sync_buffers(ctx, subctx);
const std::array<uint32_t, 2> reduce_pc = { ne, split_k };
ggml_vk_dispatch_pipeline(ctx, subctx, ctx->device->pipeline_matmul_split_k_reduce, { scratch, dst_buf }, reduce_pc, { ne, 1, 1 });
ctx->prealloc_split_k_need_sync = true;
return;
}
}
if (op == GGML_OP_ADD || op == GGML_OP_RMS_NORM) {
vk_subbuffer a_buf = src0_buf;
if (ctx->do_add_rms_partials) {
@@ -17228,7 +17306,7 @@ void vk_queue_handle_unsynchronized::submit(vk::ArrayProxy<const vk::SubmitInfo>
vk_device_struct::~vk_device_struct() {
// the build failed before the device was created, nothing to destroy
if (device == VK_NULL_HANDLE) {
if (!device) {
return;
}
@@ -112,6 +112,8 @@ const uint warps_M = BS_K / WM;
const uint warps_N = BS_NPQ / WN;
#endif
layout(constant_id = 22) const uint split_k = 1;
// without padding, ID_idx/IH_idx/IW_idx are in bounds by construction
const bool dhw_in_bounds = (p0 == 0) && (p1 == 0) && (p2 == 0);
@@ -170,8 +172,11 @@ D,R,S=KD,KH,KW
Z,P,Q=OD,OH,OW
*/
uint32_t B_idx_K = gl_WorkGroupID.x;
// Packing splits into the low bits would avoid runtime division, but adjacent channel tiles give better input cache reuse.
uint32_t B_idx_K = split_k > 1 ? gl_WorkGroupID.x % splitWork(K, BS_K) : gl_WorkGroupID.x;
uint32_t B_idx_split = split_k > 1 ? gl_WorkGroupID.x / splitWork(K, BS_K) : 0;
uint32_t B_idx_NPQ = gl_WorkGroupID.y + gl_WorkGroupID.z * 512;
uint32_t split_offset = B_idx_split * K * NPQ;
uint32_t T_y = tid / NT_NPQ;
uint32_t T_x = tid % NT_NPQ;
@@ -219,7 +224,7 @@ ACC_TYPE perElemOpStore(const in uint32_t r, const in uint32_t c, const in ACC_T
split_npq(NPQ_idx, N_idx, OD_idx, OH_idx, OW_idx);
uint32_t dst_idx = OW_idx + OH_idx * p.nb1 + OD_idx * p.nb2 + (N_idx * p.OC + K_idx) * p.nb3;
if (aligned != 0 || (K_idx < K && NPQ_idx < NPQ)) {
dst_data[dst_idx + p.dst_offset] = D_TYPE(elem);
dst_data[dst_idx + p.dst_offset + split_offset] = D_TYPE(elem);
}
return elem;
}
@@ -249,7 +254,11 @@ void main() {
}
#endif
/* Advance block in CRS dim */
[[dont_unroll]] for (uint32_t B_idx_CRS = 0; B_idx_CRS < NB_CRS; B_idx_CRS++) {
const uint32_t crs_count = NB_CRS / split_k;
const uint32_t crs_extra = NB_CRS % split_k;
const uint32_t crs_begin = B_idx_split * crs_count + min(B_idx_split, crs_extra);
const uint32_t crs_end = crs_begin + crs_count + uint32_t(B_idx_split < crs_extra);
[[dont_unroll]] for (uint32_t B_idx_CRS = crs_begin; B_idx_CRS < crs_end; B_idx_CRS++) {
uint32_t CRS_idx_a = B_idx_CRS * BS_CRS + Ac;
uint32_t IC_idx_a;
uint32_t KD_idx_a;
@@ -398,7 +407,7 @@ void main() {
split_npq(NPQ_idx, N_idx, OD_idx, OH_idx, OW_idx);
uint32_t dst_idx = OW_idx + OH_idx * p.nb1 + OD_idx * p.nb2 + (N_idx * p.OC + K_idx) * p.nb3;
if (aligned != 0 || (K_idx < K && NPQ_idx < NPQ)) {
dst_data[dst_idx + p.dst_offset] = D_TYPE(Csh[k_local * Csh_stride + npq_thread]);
dst_data[dst_idx + p.dst_offset + split_offset] = D_TYPE(Csh[k_local * Csh_stride + npq_thread]);
}
}
}
@@ -420,7 +429,7 @@ void main() {
split_npq(NPQ_idx, N_idx, OD_idx, OH_idx, OW_idx);
uint32_t dst_idx = OW_idx + OH_idx * p.nb1 + OD_idx * p.nb2 + (N_idx * p.OC + K_idx) * p.nb3;
if (aligned != 0 || (K_idx < K && NPQ_idx < NPQ)) {
dst_data[dst_idx + p.dst_offset] = D_TYPE(regC[T_ly][T_lx]);
dst_data[dst_idx + p.dst_offset + split_offset] = D_TYPE(regC[T_ly][T_lx]);
}
}
}
@@ -2,11 +2,20 @@
#extension GL_EXT_control_flow_attributes : enable
#extension GL_EXT_shader_16bit_storage : require
#if USE_SUBGROUPS
#extension GL_KHR_shader_subgroup_basic : enable
#extension GL_KHR_shader_subgroup_arithmetic : enable
#endif
#include "types.glsl"
layout(constant_id = 0) const int BLOCK_SIZE = 1024;
layout(constant_id = 1) const int QSA = 0; // 1: fuse the qwen4 QSA indexer gather + f16 mask
#if USE_SUBGROUPS
layout(constant_id = 2) const int SUBGROUP_SIZE = 32;
const uint NUM_SUBGROUPS = BLOCK_SIZE / SUBGROUP_SIZE;
#endif
layout(local_size_x_id = 0, local_size_y = 1, local_size_z = 1) in;
@@ -26,12 +35,19 @@ layout (push_constant) uniform parameter {
} p;
#define RADIX_BITS 8
#define UNROLL 4
#define RADIX_SIZE (1 << RADIX_BITS)
shared uint histo[RADIX_SIZE];
shared uint sh_bucket;
shared uint sh_above;
shared uint sh_tied;
shared uint out_count;
#if USE_SUBGROUPS
shared uint sg_tied[NUM_SUBGROUPS];
#else
shared uint tie_scan[2 * BLOCK_SIZE];
#endif
// order-preserving float -> uint mapping
uint f2ui(float x) {
@@ -87,52 +103,155 @@ void topk(const uint row) {
const bool first = (shift == 32 - RADIX_BITS);
const uint hi_mask = (shift + RADIX_BITS >= 32) ? 0u : (0xFFFFFFFFu << uint(shift + RADIX_BITS));
const uint prefix_hi = prefix & hi_mask;
for (uint i = tid; i < ncols; i += BLOCK_SIZE) {
const uint key = f2ui(load(row, i, first));
if ((key & hi_mask) == prefix_hi) {
atomicAdd(histo[(key >> uint(shift)) & (RADIX_SIZE - 1)], 1u);
uint run_b = RADIX_SIZE;
uint run_n = 0;
for (uint i0 = tid; i0 < ncols; i0 += UNROLL * BLOCK_SIZE) {
uint keys[UNROLL];
[[unroll]] for (uint u = 0; u < UNROLL; ++u) {
const uint i = i0 + u * BLOCK_SIZE;
keys[u] = f2ui(load(row, min(i, ncols - 1), first));
}
[[unroll]] for (uint u = 0; u < UNROLL; ++u) {
const uint key = keys[u];
if (i0 + u * BLOCK_SIZE < ncols && (key & hi_mask) == prefix_hi) {
const uint b = (key >> uint(shift)) & (RADIX_SIZE - 1);
if (b != run_b) {
if (run_n != 0) {
atomicAdd(histo[run_b], run_n);
}
run_b = b;
run_n = 0;
}
run_n++;
}
}
}
if (run_n != 0) {
atomicAdd(histo[run_b], run_n);
}
barrier();
// top-down scan for the bucket holding the K-th value
if (tid == 0) {
uint acc = 0;
uint b = 0;
for (int bb = RADIX_SIZE - 1; bb >= 0; --bb) {
const uint c = histo[bb];
if (acc + c >= desired) { b = uint(bb); break; }
acc += c;
#if USE_SUBGROUPS
uint c = 0, incl = 0;
if (tid < RADIX_SIZE) {
c = histo[RADIX_SIZE - 1 - tid];
incl = subgroupInclusiveAdd(c);
if (gl_SubgroupInvocationID == gl_SubgroupSize - 1) {
sg_tied[gl_SubgroupID] = incl;
}
sh_bucket = b;
sh_above = acc;
}
barrier();
if (tid < RADIX_SIZE) {
for (uint s = 0; s < gl_SubgroupID; ++s) {
incl += sg_tied[s];
}
if (incl - c < desired && incl >= desired) {
sh_bucket = RADIX_SIZE - 1 - tid;
sh_above = incl - c;
sh_tied = c;
}
}
barrier();
#else
uint c = 0;
if (tid < RADIX_SIZE) {
c = histo[RADIX_SIZE - 1 - tid];
tie_scan[tid] = c;
}
barrier();
uint src = 0;
[[unroll]] for (uint off = 1; off < RADIX_SIZE; off <<= 1) {
if (tid < RADIX_SIZE) {
const uint v = tie_scan[src + tid];
tie_scan[(RADIX_SIZE - src) + tid] = tid >= off ? v + tie_scan[src + tid - off] : v;
}
src = RADIX_SIZE - src;
barrier();
}
if (tid < RADIX_SIZE) {
const uint incl = tie_scan[src + tid];
if (incl - c < desired && incl >= desired) {
sh_bucket = RADIX_SIZE - 1 - tid;
sh_above = incl - c;
sh_tied = c;
}
}
barrier();
#endif
prefix |= sh_bucket << uint(shift);
desired -= sh_above;
barrier();
}
const uint threshold = prefix;
const bool all_ties = sh_tied == desired;
if (tid == 0) {
out_count = 0;
}
barrier();
// emit everything above the threshold, then fill the rest from ties
const uint threshold = prefix;
if (all_ties) {
for (uint i = tid; i < ncols; i += BLOCK_SIZE) {
if (f2ui(load(row, i, false)) >= threshold) {
data_d[row_out + atomicAdd(out_count, 1u)] = int(i);
}
}
return;
}
// more ties than slots: rank them in invocation order, which does not depend on the scheduling
uint ct = 0;
for (uint i = tid; i < ncols; i += BLOCK_SIZE) {
if (f2ui(load(row, i, false)) > threshold) {
const uint key = f2ui(load(row, i, false));
if (key > threshold) {
data_d[row_out + atomicAdd(out_count, 1u)] = int(i);
}
ct += key == threshold ? 1u : 0u;
}
#if USE_SUBGROUPS
const uint sg = gl_SubgroupID;
const uint st = subgroupAdd(ct);
if (subgroupElect()) {
sg_tied[sg] = st;
}
barrier();
if (sg == 0) {
const uint lane = gl_SubgroupInvocationID;
uint acc = 0;
for (uint c0 = 0; c0 < NUM_SUBGROUPS; c0 += SUBGROUP_SIZE) {
const uint c = c0 + lane;
const uint t = c < NUM_SUBGROUPS ? sg_tied[c] : 0u;
const uint e = subgroupExclusiveAdd(t);
if (c < NUM_SUBGROUPS) {
sg_tied[c] = acc + e;
}
acc += subgroupAdd(t);
}
}
barrier();
for (uint i = tid; i < ncols; i += BLOCK_SIZE) {
uint pos = p.k - desired + sg_tied[sg] + subgroupExclusiveAdd(ct);
#else
// inclusive Hillis-Steele scan, ping-ponging between the two halves
uint src = 0;
tie_scan[tid] = ct;
barrier();
for (uint off = 1; off < BLOCK_SIZE; off <<= 1) {
const uint v = tie_scan[src + tid];
tie_scan[(BLOCK_SIZE - src) + tid] = tid >= off ? v + tie_scan[src + tid - off] : v;
src = BLOCK_SIZE - src;
barrier();
}
uint pos = p.k - desired + tie_scan[src + tid] - ct;
#endif
const uint pos_end = min(p.k, pos + ct);
for (uint i = tid; i < ncols && pos < pos_end; i += BLOCK_SIZE) {
if (f2ui(load(row, i, false)) == threshold) {
const uint pos = atomicAdd(out_count, 1u);
if (pos < p.k) {
data_d[row_out + pos] = int(i);
}
data_d[row_out + pos++] = int(i);
}
}
}
@@ -1115,6 +1115,7 @@ void process_shaders() {
string_to_spv("topk_argsort_f32", "topk_argsort.comp", {{"A_TYPE", "float"}});
string_to_spv("topk_nary_search_f32", "topk_nary_search.comp", {{"A_TYPE", "float"}});
string_to_spv("topk_radix_select_f32", "topk_radix_select.comp", {{"A_TYPE", "float"}});
string_to_spv("topk_radix_select_f32_subgroup", "topk_radix_select.comp", {{"A_TYPE", "float"}, {"USE_SUBGROUPS", "1"}});
string_to_spv("argmax_f32", "argmax.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "int"}}));
string_to_spv("sum_rows_f32", "sum_rows.comp", merge_maps(base_dict, {{"A_TYPE", "float"}, {"D_TYPE", "float"}}));
+33 -15
View File
@@ -962,13 +962,16 @@ inline uint32_t ggml_webgpu_flash_attn_get_vec_kv_tile(size_t wg_mem_limit_byt
uint32_t head_dim_qk,
uint32_t head_dim_v,
bool has_mask,
bool kv_direct) {
const uint32_t max_kv_tile =
ggml_webgpu_flash_attn_max_kv_tile(wg_mem_limit_bytes, 1u, 1u, head_dim_qk, head_dim_v, has_mask, kv_direct);
bool k_direct,
bool v_direct) {
// kv_shmem is dropped only when both K and V are read directly
const uint32_t max_kv_tile = ggml_webgpu_flash_attn_max_kv_tile(wg_mem_limit_bytes, 1u, 1u, head_dim_qk, head_dim_v,
has_mask, k_direct && v_direct);
GGML_ASSERT(max_kv_tile > 0);
uint32_t kv_tile = std::min(GGML_WEBGPU_FLASH_ATTN_VEC_MAX_KV_TILE, max_kv_tile);
if (kv_direct) {
// a direct read has no bounds check, so the tile must divide the KV padding
if (k_direct || v_direct) {
kv_tile = std::min(kv_tile, GGML_WEBGPU_KV_SEQ_PAD);
while (GGML_WEBGPU_KV_SEQ_PAD % kv_tile != 0) {
kv_tile -= 1u;
@@ -1261,15 +1264,16 @@ class ggml_webgpu_shader_lib {
wgpu::Device device;
pre_wgsl::Preprocessor preprocessor;
std::unordered_map<int, webgpu_pipeline> sum_rows_pipelines; // key is whether to compute the mean
std::unordered_map<int, webgpu_pipeline> argmax_pipelines; // key is vec4
std::unordered_map<int, webgpu_pipeline> argsort_pipelines; // key is order
std::unordered_map<int, webgpu_pipeline> argsort_merge_pipelines; // key is order
std::unordered_map<int, webgpu_pipeline> cumsum_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> arange_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> count_equal_pipelines; // key is src_overlap
std::unordered_map<int, webgpu_pipeline> sum_rows_pipelines; // key is whether to compute the mean
std::unordered_map<int, webgpu_pipeline> argmax_pipelines; // key is vec4
std::unordered_map<int, webgpu_pipeline> argsort_pipelines; // key is order
std::unordered_map<int, webgpu_pipeline> argsort_merge_pipelines; // key is order
std::unordered_map<int, webgpu_pipeline> cumsum_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> arange_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> count_equal_pipelines; // key is src_overlap
std::unordered_map<int, webgpu_pipeline> timestep_embedding_pipelines; // key is fixed, no variants yet
std::unordered_map<ggml_webgpu_row_norm_pipeline_key, webgpu_pipeline, ggml_webgpu_row_norm_pipeline_key_hash>
row_norm_pipelines; // op/inplace
row_norm_pipelines; // op/inplace
std::unordered_map<ggml_webgpu_get_rows_pipeline_key, webgpu_pipeline, ggml_webgpu_get_rows_pipeline_key_hash>
get_rows_pipelines; // src_type, vectorized
@@ -1630,6 +1634,20 @@ class ggml_webgpu_shader_lib {
return cumsum_pipelines[1];
}
webgpu_pipeline get_timestep_embedding_pipeline(const ggml_webgpu_shader_lib_context & context) {
auto it = timestep_embedding_pipelines.find(1);
if (it != timestep_embedding_pipelines.end()) {
return it->second;
}
std::vector<std::string> defines;
defines.push_back(std::string("WG_SIZE=") + std::to_string(context.max_wg_size));
auto processed = preprocessor.preprocess(wgsl_timestep_embedding, defines);
timestep_embedding_pipelines[1] = ggml_webgpu_create_pipeline(device, processed, "timestep_embedding");
return timestep_embedding_pipelines[1];
}
webgpu_pipeline get_argsort_pipeline(const ggml_webgpu_shader_lib_context & context) {
bool is_top_k = context.dst->op == GGML_OP_TOP_K;
// ascending order is 0, descending order is 1
@@ -3050,7 +3068,7 @@ class ggml_webgpu_shader_lib {
uint32_t max_kv_tile = ggml_webgpu_flash_attn_max_kv_tile(
context.wg_mem_limit_bytes, decisions.q_tile, decisions.use_sg_matrix ? context.sg_mat_n : 1u,
key.common.head_dim_qk, key.common.head_dim_v, key.common.has_mask,
key.common.k_direct || key.common.v_direct);
key.common.k_direct && key.common.v_direct);
// WorkGroup storage size isn't enough for some params with subgroup matrices path (ref. https://github.com/ggml-org/llama.cpp/pull/26566)
if (max_kv_tile == 0) {
@@ -3063,7 +3081,7 @@ class ggml_webgpu_shader_lib {
key.use_sg_matrix = false;
max_kv_tile = ggml_webgpu_flash_attn_max_kv_tile(
context.wg_mem_limit_bytes, decisions.q_tile, 1u, key.common.head_dim_qk, key.common.head_dim_v,
key.common.has_mask, key.common.k_direct || key.common.v_direct);
key.common.has_mask, key.common.k_direct && key.common.v_direct);
GGML_ASSERT(max_kv_tile > 0);
}
@@ -3125,7 +3143,7 @@ class ggml_webgpu_shader_lib {
ggml_webgpu_flash_attn_vec_decisions decisions = {};
decisions.kv_tile = ggml_webgpu_flash_attn_get_vec_kv_tile(context.wg_mem_limit_bytes, key.common.head_dim_qk,
key.common.head_dim_v, key.common.has_mask,
key.common.k_direct || key.common.v_direct);
key.common.k_direct, key.common.v_direct);
decisions.wg_size = context.max_subgroup_size;
std::string variant = "flash_attn_vec";
+76 -19
View File
@@ -90,7 +90,7 @@ static inline uint32_t ggml_webgpu_u32_from_f32(float value) {
#define WEBGPU_NUM_PARAM_SLOT_SAFETY_MARGIN 10u
#define WEBGPU_RUNTIME_WAIT_TIMEOUT_MS 30000u
#define WEBGPU_RUNTIME_WAIT_TIMEOUT_NS (WEBGPU_RUNTIME_WAIT_TIMEOUT_MS * 1e6)
#define WEBGPU_PARAMS_BUF_SIZE_BYTES 128 // enough for 32 parameters
#define WEBGPU_PARAMS_BUF_SIZE_BYTES 256 // enough for 64 parameters
#define WEBGPU_SET_ROWS_ERROR_BUF_SIZE_BYTES 4
#define WEBGPU_STORAGE_BUF_BINDING_MULT 4 // a storage buffer binding size must be a multiple of 4
@@ -2104,8 +2104,10 @@ static ggml_webgpu_flash_attn_op ggml_webgpu_flash_attn_prepare(webgpu_context &
(uint32_t) (V->nb[2] / ggml_type_size(V->type)), // stride (elements/blocks) of V in dimension 2
(uint32_t) (V->nb[3] / ggml_type_size(V->type)), // stride (elements/blocks) of V in dimension 3
op.has_mask ? (uint32_t) (mask->nb[3] / ggml_type_size(mask->type)) : 0, // stride of mask dim 3
(uint32_t) (Q->ne[2] / K->ne[2]), // repeat factor for K/V in dim 2 (MHA/MQA/GQA)
ggml_webgpu_u32_from_f32(scale), // scale (possibly adjusted for logit softcap)
op.has_mask ? (uint32_t) (mask->nb[2] / ggml_type_size(mask->type)) : 0, // stride of mask dim 2
op.has_mask ? (uint32_t) mask->ne[2] : 1u, // mask heads, broadcast over the Q heads
(uint32_t) (Q->ne[2] / K->ne[2]), // repeat factor for K/V in dim 2 (MHA/MQA/GQA)
ggml_webgpu_u32_from_f32(scale), // scale (possibly adjusted for logit softcap)
ggml_webgpu_u32_from_f32(max_bias),
ggml_webgpu_u32_from_f32(logit_softcap),
ggml_webgpu_u32_from_f32(n_head_log2),
@@ -2211,7 +2213,9 @@ static webgpu_encoded_op ggml_webgpu_flash_attn_vec(webgpu_context & ct
blk_nblk1 = (uint32_t) Q->ne[1];
blk_buf = ggml_webgpu_tensor_buf(dst);
const uint32_t stride_mask3 = (uint32_t) (mask->nb[3] / ggml_type_size(mask->type));
blk_batch_count = stride_mask3 > 0 ? (uint32_t) Q->ne[3] : 1u;
const uint32_t stride_mask2 = (uint32_t) (mask->nb[2] / ggml_type_size(mask->type));
const uint32_t mask_ne2 = (uint32_t) mask->ne[2];
blk_batch_count = (stride_mask3 > 0 ? (uint32_t) Q->ne[3] : 1u) * mask_ne2;
const uint64_t blk_elems = (uint64_t) blk_nblk0 * blk_nblk1 * blk_batch_count;
blk_size_bytes = ROUNDUP_POW2(blk_elems * sizeof(uint32_t), WEBGPU_STORAGE_BUF_BINDING_MULT);
const ggml_webgpu_shader_lib_context blk_shader_ctx = op.shader_lib_ctx;
@@ -2222,6 +2226,8 @@ static webgpu_encoded_op ggml_webgpu_flash_attn_vec(webgpu_context & ct
(uint32_t) Q->ne[1], // seq_len_q
(uint32_t) K->ne[1], // seq_len_kv
stride_mask3, // stride_mask3
stride_mask2, // stride_mask2
mask_ne2, // mask_ne2
blk_nblk0, // nblk0
blk_nblk1, // nblk1
};
@@ -3357,6 +3363,44 @@ static webgpu_encoded_op ggml_webgpu_cumsum(webgpu_context & ctx, ggml_tensor *
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_timestep_embedding(webgpu_context & ctx, ggml_tensor * src, ggml_tensor * dst) {
const int dim = ggml_get_op_params_i32(dst, 0);
const int max_period = ggml_get_op_params_i32(dst, 1);
GGML_ASSERT(dim >= 1);
GGML_ASSERT(max_period > 0);
std::vector<uint32_t> params = {
(uint32_t) dim,
ggml_webgpu_u32_from_f32((float) max_period),
(uint32_t) src->ne[0],
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, src) / ggml_type_size(src->type)),
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, dst) / ggml_type_size(dst->type)),
};
std::vector<wgpu::BindGroupEntry> entries = {
ggml_webgpu_make_tensor_bind_group_entry(ctx, 0, src),
ggml_webgpu_make_tensor_bind_group_entry(ctx, 1, dst),
};
ggml_webgpu_shader_lib_context shader_lib_ctx = {};
shader_lib_ctx.src0 = src;
shader_lib_ctx.dst = dst;
shader_lib_ctx.max_wg_size = ctx->global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
webgpu_pipeline pipeline = ctx->shader_lib->get_timestep_embedding_pipeline(shader_lib_ctx);
// Each thread calculates both sin and cos for a given frequency,
// so divide params.dim by 2.
const uint32_t invocation_count = (uint32_t) src->ne[0] * (dim == 1 ? 1 : dim / 2);
const uint32_t workgroups = CEIL_DIV(invocation_count, shader_lib_ctx.max_wg_size);
uint32_t wg_x, wg_y;
compute_2d_workgroups(workgroups, ctx->global_ctx->capabilities.limits.maxComputeWorkgroupsPerDimension, wg_x,
wg_y);
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_sum_rows(webgpu_context & ctx, ggml_tensor * src, ggml_tensor * dst) {
bool total_sum = dst->op == GGML_OP_SUM;
std::vector<uint32_t> params = { (uint32_t) (ggml_webgpu_tensor_misalignment(ctx, src) / ggml_type_size(src->type)),
@@ -3650,6 +3694,8 @@ static std::optional<webgpu_encoded_op> ggml_webgpu_encode(webgpu_context ctx,
return ggml_webgpu_im2col(ctx, src0, src1, node);
case GGML_OP_UPSCALE:
return ggml_webgpu_upscale(ctx, src0, node);
case GGML_OP_TIMESTEP_EMBEDDING:
return ggml_webgpu_timestep_embedding(ctx, src0, node);
default:
return std::nullopt;
}
@@ -4381,12 +4427,11 @@ static size_t ggml_backend_webgpu_buffer_type_get_alloc_size(ggml_backend_buffer
// longest vec batch and the size stays monotonic in the batch length
const int64_t n_q = std::min<int64_t>(Q->ne[1], GGML_WEBGPU_FLASH_ATTN_VEC_MAX_SEQ_LEN - 1);
if (ggml_webgpu_flash_attn_use_vec_path(ctx->webgpu_global_ctx, Q, K, V, n_q)) {
const bool kv_direct =
ggml_webgpu_flash_attn_k_direct(Q, K, GGML_WEBGPU_FLASH_ATTN_TILE_KV_VEC_WIDTH) ||
ggml_webgpu_flash_attn_v_direct(Q, V, GGML_WEBGPU_FLASH_ATTN_TILE_KV_VEC_WIDTH);
const uint32_t kv_tile = ggml_webgpu_flash_attn_get_vec_kv_tile(
capabilities.limits.maxComputeWorkgroupStorageSize, (uint32_t) Q->ne[0], (uint32_t) V->ne[0],
mask != nullptr, kv_direct);
mask != nullptr,
ggml_webgpu_flash_attn_k_direct(Q, K, GGML_WEBGPU_FLASH_ATTN_TILE_KV_VEC_WIDTH),
ggml_webgpu_flash_attn_v_direct(Q, V, GGML_WEBGPU_FLASH_ATTN_TILE_KV_VEC_WIDTH));
const uint32_t vec_nwg_cap = capabilities.min_subgroup_size;
uint32_t nwg = ggml_webgpu_flash_attn_vec_nwg(vec_nwg_cap, kv_tile, (uint32_t) K->ne[1]);
@@ -4403,11 +4448,12 @@ static size_t ggml_backend_webgpu_buffer_type_get_alloc_size(ggml_backend_buffer
res += WEBGPU_STORAGE_BUF_BINDING_MULT + align;
}
if (mask != nullptr) {
const uint32_t blk_nblk0 = CEIL_DIV((uint32_t) K->ne[1], kv_tile);
const uint32_t blk_nblk1 = (uint32_t) n_q;
const uint32_t stride_mask3 = (uint32_t) (mask->nb[3] / ggml_type_size(mask->type));
const uint32_t blk_batch_count = stride_mask3 > 0 ? (uint32_t) Q->ne[3] : 1u;
const uint64_t blk_elems = (uint64_t) blk_nblk0 * blk_nblk1 * blk_batch_count;
const uint32_t blk_nblk0 = CEIL_DIV((uint32_t) K->ne[1], kv_tile);
const uint32_t blk_nblk1 = (uint32_t) n_q;
const uint32_t stride_mask3 = (uint32_t) (mask->nb[3] / ggml_type_size(mask->type));
const uint32_t blk_batch_count =
(stride_mask3 > 0 ? (uint32_t) Q->ne[3] : 1u) * (uint32_t) mask->ne[2];
const uint64_t blk_elems = (uint64_t) blk_nblk0 * blk_nblk1 * blk_batch_count;
const size_t blk_size_bytes =
ROUNDUP_POW2(blk_elems * sizeof(uint32_t), WEBGPU_STORAGE_BUF_BINDING_MULT);
res += blk_size_bytes + align;
@@ -5014,14 +5060,22 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
const uint32_t q_tile =
use_subgroup_matrix ? capabilities.sg_mat_m : GGML_WEBGPU_FLASH_ATTN_TILE_Q_TILE;
const uint32_t kv_granularity = use_subgroup_matrix ? capabilities.sg_mat_n : 1u;
const bool kv_direct = use_subgroup_matrix ?
ggml_webgpu_flash_attn_k_direct(src0, src1, capabilities.sg_mat_k) ||
ggml_webgpu_flash_attn_v_direct(src0, src2, capabilities.sg_mat_k) :
false;
const uint32_t max_kv_tile = ggml_webgpu_flash_attn_max_kv_tile(
// same direct read rule as get_flash_attn_pipeline: F16 only, on the subgroup matrix path
const bool kv_direct = use_subgroup_matrix && src1->type == GGML_TYPE_F16 &&
src2->type == GGML_TYPE_F16 &&
ggml_webgpu_flash_attn_k_direct(src0, src1, capabilities.sg_mat_k) &&
ggml_webgpu_flash_attn_v_direct(src0, src2, capabilities.sg_mat_k);
const uint32_t max_kv_tile = ggml_webgpu_flash_attn_max_kv_tile(
capabilities.limits.maxComputeWorkgroupStorageSize, q_tile, kv_granularity, (uint32_t) src0->ne[0],
(uint32_t) src2->ne[0], op->src[3] != nullptr, kv_direct);
supports_op = max_kv_tile > 0;
// get_flash_attn_pipeline falls back to the register tile path when the subgroup matrix tile does not fit
const uint32_t fallback_kv_tile =
use_subgroup_matrix ?
ggml_webgpu_flash_attn_max_kv_tile(
capabilities.limits.maxComputeWorkgroupStorageSize, GGML_WEBGPU_FLASH_ATTN_TILE_Q_TILE, 1u,
(uint32_t) src0->ne[0], (uint32_t) src2->ne[0], op->src[3] != nullptr, false) :
0u;
supports_op = max_kv_tile > 0 || fallback_kv_tile > 0;
break;
}
case GGML_OP_GROUP_NORM:
@@ -5184,6 +5238,9 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
case GGML_OP_CUMSUM:
supports_op = op->type == GGML_TYPE_F32 && src0->type == op->type;
break;
case GGML_OP_TIMESTEP_EMBEDDING:
supports_op = src0->type == GGML_TYPE_F32 && ggml_is_contiguous(src0);
break;
case GGML_OP_SUM:
supports_op = op->type == GGML_TYPE_F32 && src0->type == op->type && ggml_is_contiguously_allocated(src0);
break;
@@ -116,7 +116,7 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
#ifdef MASK
// mask offset
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + q_row_start * params.seq_len_kv;
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + (head_idx % params.mask_ne2) * params.stride_mask2 + q_row_start * params.seq_len_kv;
#endif
// note that the output is permuted, the layout is [head_dim_v, n_heads, seq_len_q, batch_size]
@@ -65,6 +65,8 @@ struct Params {
stride_v2: u32,
stride_v3: u32,
stride_mask3: u32,
stride_mask2: u32,
mask_ne2: u32,
// repeat factors for K/V, e.g., MHA vs. MQA vs. GQA
q_per_kv: u32,
@@ -49,7 +49,7 @@ fn load_k_tile_block(local_x: u32, kv_count: u32, kv_tile: u32, k_head_offset: u
let row_offset = k_row * HEAD_DIM_QK;
let global_block_idx = k_head_offset + global_k_row * params.stride_k1 + block_k;
let block_byte_base = global_block_idx * K_BLOCK_SIZE_BYTES;
let d = f16_from_u16(load_k_u16_at(block_byte_base));
let d = unpack2x16float(load_k_u16_at(block_byte_base)).x;
let thread_byte_offset = block_offset * K_BYTES_PER_THREAD;
let shmem_idx = row_offset + block_k * BLOCK_SIZE + thread_byte_offset;
for (var j = 0u; j < K_BYTES_PER_THREAD / K_BYTES_PER_INNER_LOOP; j += 1u) {
@@ -98,7 +98,7 @@ fn load_v_tile_block(local_x: u32, kv_count: u32, kv_tile: u32, v_head_offset: u
let row_offset = v_row * HEAD_DIM_V;
let global_block_idx = v_head_offset + global_v_row * params.stride_v1 + block_k;
let block_byte_base = global_block_idx * V_BLOCK_SIZE_BYTES;
let d = f16_from_u16(load_v_u16_at(block_byte_base));
let d = unpack2x16float(load_v_u16_at(block_byte_base)).x;
let thread_byte_offset = block_offset * V_BYTES_PER_THREAD;
let shmem_idx = row_offset + block_k * BLOCK_SIZE + thread_byte_offset;
for (var j = 0u; j < V_BYTES_PER_THREAD / V_BYTES_PER_INNER_LOOP; j += 1u) {
@@ -65,7 +65,7 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
let row_active = subgroup_id < Q_TILE && global_q_row < params.seq_len_q;
#ifdef MASK
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + q_row_start * params.seq_len_kv;
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + (head_idx % params.mask_ne2) * params.stride_mask2 + q_row_start * params.seq_len_kv;
#endif
let dst_global_offset = dst_batch_offset + q_row_start * dst2_stride + head_idx * HEAD_DIM_V;
@@ -9,6 +9,8 @@ struct Params {
seq_len_q: u32,
seq_len_kv: u32,
stride_mask3: u32,
stride_mask2: u32,
mask_ne2: u32,
// Number of KV blocks and Q blocks per batch.
// nblk0 = ceil(seq_len_kv / KV_TILE), nblk1 = seq_len_q.
nblk0: u32,
@@ -30,11 +32,12 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
@builtin(local_invocation_id) local_id: vec3<u32>) {
// Dispatch mapping:
// - x indexes KV blocks
// - y flattens (batch_idx, q_blk) as y = batch_idx * nblk1 + q_blk
// - y flattens (batch_idx, mask_head, q_blk) as y = (batch_idx * mask_ne2 + mask_head) * nblk1 + q_blk
let kv_blk = wg_id.x;
let y = wg_id.y;
let q_blk = y % params.nblk1;
let batch_idx = y / params.nblk1;
let mask_head = (y / params.nblk1) % params.mask_ne2;
let batch_idx = y / (params.nblk1 * params.mask_ne2);
if (kv_blk >= params.nblk0) {
return;
}
@@ -43,7 +46,7 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
let k_start = kv_blk * KV_TILE;
let mask_batch = select(0u, batch_idx, params.stride_mask3 > 0u);
let mask_batch_base = params.offset_mask + mask_batch * params.stride_mask3;
let mask_batch_base = params.offset_mask + mask_batch * params.stride_mask3 + mask_head * params.stride_mask2;
// We keep min/max to classify:
// - fully masked (max <= MASK_MIN)
@@ -95,7 +98,7 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
}
}
let blk_idx = (batch_idx * params.nblk1 + q_blk) * params.nblk0 + kv_blk;
let blk_idx = ((batch_idx * params.mask_ne2 + mask_head) * params.nblk1 + q_blk) * params.nblk0 + kv_blk;
blk[blk_idx] = state;
}
}
@@ -107,7 +107,7 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
#ifdef MASK
// mask offset
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + q_row_start * params.seq_len_kv;
let mask_global_offset = params.offset_mask + batch_idx * params.stride_mask3 + (head_idx % params.mask_ne2) * params.stride_mask2 + q_row_start * params.seq_len_kv;
#endif
let head = f32(head_idx);
@@ -129,7 +129,8 @@ fn main(@builtin(workgroup_id) wg_id: vec3<u32>,
let q_blk = q_row_start;
let kv_blk = kv_tile / KV_TILE;
let blk_batch = select(0u, batch_idx, params.stride_mask3 > 0u);
let blk_idx = params.blk_base + (blk_batch * params.blk_nblk1 + q_blk) * params.blk_nblk0 + kv_blk;
let blk_head = head_idx % params.mask_ne2;
let blk_idx = params.blk_base + ((blk_batch * params.mask_ne2 + blk_head) * params.blk_nblk1 + q_blk) * params.blk_nblk0 + kv_blk;
let blk_state_local = blk[blk_idx];
#else
let blk_state_local = 1u;
@@ -98,13 +98,13 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let byte_in_block = (global_k_start % BLOCK_SIZE) / 8u;
let src0_idx = batch_offset + global_m * params.stride_01 + block_k;
let block_byte_base = src0_idx * BLOCK_SIZE_BYTES;
let d = load_f16_at_src0(block_byte_base);
let d = load_f16_as_f32_at_src0(block_byte_base);
let q_byte = load_u32_at_src0(block_byte_base + 2u + byte_in_block) & 0xFFu;
for (var bit = 0u; bit < NQ; bit++) {
let global_k = global_k_start + bit;
if (global_k < params.k) {
shmem[i + bit] = select(-d, d, ((q_byte >> bit) & 1u) != 0u);
shmem[i + bit] = f16(select(-d, d, ((q_byte >> bit) & 1u) != 0u));
}
}
}
@@ -123,11 +123,11 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
if (global_m < params.m && global_k < params.k) {
let block_idx = batch_offset + global_m * params.stride_01 + global_k / BLOCK_SIZE;
let block_byte_base = block_idx * BLOCK_SIZE_BYTES;
let d = load_f16_at_src0(block_byte_base);
let d = load_f16_as_f32_at_src0(block_byte_base);
let q_packed = load_u32_at_src0(block_byte_base + 2u + (global_k % BLOCK_SIZE) / 4u);
for (var k = 0u; k < NQ; k++) {
let q = (q_packed >> (2u * k)) & 3u;
shmem[i + k] = (f16(q) - 1.0) * d;
shmem[i + k] = f16((f32(q) - 1.0) * d);
}
}
}
@@ -163,7 +163,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
#if defined(INIT_SRC0_SHMEM_Q4_0)
let block_byte_base = src0_idx * 18u; // BLOCK_SIZE_BYTES = 18u;
let d = load_f16_at_src0(block_byte_base);
let d = load_f16_as_f32_at_src0(block_byte_base);
// load NQ(16) weights
for (var j = 0u; j < BYTES_PER_THREAD / BYTES_PER_INNER_LOOP; j += 1) {
@@ -176,8 +176,8 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
#if defined(INIT_SRC0_SHMEM_Q4_1)
let block_byte_base = src0_idx * 20u; // BLOCK_SIZE_BYTES = 20u;
let dm = unpack2x16float(load_u32_at_src0_aligned(block_byte_base));
let d = f16(dm[0]);
let m = f16(dm[1]);
let d = dm[0];
let m = dm[1];
// load NQ(16) weights
for (var j = 0u; j < BYTES_PER_THREAD / BYTES_PER_INNER_LOOP; j += 1) {
@@ -186,10 +186,10 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
for (var k = 0u; k < BYTES_PER_INNER_LOOP; k++) {
let q_byte = get_byte(q_packed, k);
let q_lo = f16(q_byte & 0xF) * d + m;
let q_hi = f16((q_byte >> 4) & 0xF) * d + m;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = q_lo;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = q_hi;
let q_lo = f32(q_byte & 0xF) * d + m;
let q_hi = f32((q_byte >> 4) & 0xF) * d + m;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = f16(q_lo);
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = f16(q_hi);
}
}
#endif // INIT_SRC0_SHMEM_Q4_1
@@ -197,7 +197,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
#if defined(INIT_SRC0_SHMEM_Q5_0)
let block_byte_base = src0_idx * 22u; // BLOCK_SIZE_BYTES = 22u;
let d = load_f16_at_src0(block_byte_base);
let d = load_f16_as_f32_at_src0(block_byte_base);
let qh_packed = load_u32_at_src0(block_byte_base + 2u);
// load NQ(16) weights
@@ -210,11 +210,11 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let byte_idx = block_offset * BYTES_PER_THREAD + j * BYTES_PER_INNER_LOOP + k;
let qh_hi = (qh_packed >> (byte_idx + 12u)) & 0x10;
let q_hi = (f16(((q_byte >> 4) & 0xF) | qh_hi) - 16.0) * d;
let q_hi = (f32(((q_byte >> 4) & 0xF) | qh_hi) - 16.0) * d;
let qh_lo = ((qh_packed >> byte_idx) << 4) & 0x10;
let q_lo = (f16((q_byte & 0xF) | qh_lo) - 16.0) * d;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = q_lo;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = q_hi;
let q_lo = (f32((q_byte & 0xF) | qh_lo) - 16.0) * d;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = f16(q_lo);
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = f16(q_hi);
}
}
#endif // INIT_SRC0_SHMEM_Q5_0
@@ -223,8 +223,8 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let block_byte_base = src0_idx * 24u; // BLOCK_SIZE_BYTES = 24u;
let dm = unpack2x16float(load_u32_at_src0_aligned(block_byte_base));
let d = f16(dm[0]);
let m = f16(dm[1]);
let d = dm[0];
let m = dm[1];
let qh_packed = load_u32_at_src0_aligned(block_byte_base + 4u);
// load NQ(16) weights
@@ -237,18 +237,18 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let byte_idx = block_offset * BYTES_PER_THREAD + j * BYTES_PER_INNER_LOOP + k;
let qh_hi = (qh_packed >> (byte_idx + 12u)) & 0x10;
let q_hi = f16(((q_byte >> 4) & 0xF) | qh_hi) * d + m;
let q_hi = f32(((q_byte >> 4) & 0xF) | qh_hi) * d + m;
let qh_lo = ((qh_packed >> byte_idx) << 4) & 0x10;
let q_lo = f16((q_byte & 0xF) | qh_lo) * d + m;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = q_lo;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = q_hi;
let q_lo = f32((q_byte & 0xF) | qh_lo) * d + m;
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k] = f16(q_lo);
shmem[shmem_idx + j * BYTES_PER_INNER_LOOP + k + 16u] = f16(q_hi);
}
}
#endif // INIT_SRC0_SHMEM_Q5_1
#if defined(INIT_SRC0_SHMEM_Q8_0)
let block_byte_base = src0_idx * 34u; // BLOCK_SIZE_BYTES = 34u;
let d = load_f16_at_src0(block_byte_base);
let d = load_f16_as_f32_at_src0(block_byte_base);
// load NQ(16) weights
for (var j = 0u; j < BYTES_PER_THREAD / BYTES_PER_INNER_LOOP; j += 1) {
@@ -381,8 +381,8 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let dm_byte_base = block_byte_base + 80u;
let d_packed = unpack2x16float(load_u32_at_src0_aligned(dm_byte_base));
let d = f16(d_packed[0]);
let dmin = f16(d_packed[1]);
let d = d_packed[0];
let dmin = d_packed[1];
let chunk = k_in_block / 128u;
let pos_in_chunk = k_in_block % 32u;
@@ -391,19 +391,19 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
// whole 2 bits (4 elems)
let qs_word = load_u32_at_src0_aligned(qs_byte_base + 32u * chunk + 1u * pos_in_chunk);
let qs_vec4 = vec4<f16>(
f16((qs_word >> (2u * shift_phase + 0u)) & 0x3u),
f16((qs_word >> (2u * shift_phase + 8u)) & 0x3u),
f16((qs_word >> (2u * shift_phase + 16u)) & 0x3u),
f16((qs_word >> (2u * shift_phase + 24u)) & 0x3u),
let qs_vec4 = vec4<f32>(
f32((qs_word >> (2u * shift_phase + 0u)) & 0x3u),
f32((qs_word >> (2u * shift_phase + 8u)) & 0x3u),
f32((qs_word >> (2u * shift_phase + 16u)) & 0x3u),
f32((qs_word >> (2u * shift_phase + 24u)) & 0x3u),
);
let scale = load_byte_at_src0_aligned(scales_byte_base + sub_block);
let dl = d * f16(scale & 0xFu);
let ml = dmin * f16(scale >> 4u);
let dl = d * f32(scale & 0xFu);
let ml = dmin * f32(scale >> 4u);
store_shmem_kquants(qs_vec4 * dl - ml, elem_idx);
store_shmem_kquants(vec4<f16>(qs_vec4 * dl - ml), elem_idx);
#endif // INIT_SRC0_SHMEM_Q2_K
#if defined(INIT_SRC0_SHMEM_Q3_K)
@@ -412,7 +412,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let qs_byte_base = block_byte_base + 32u;
let scales_byte_base = block_byte_base + 96u;
let d_all = load_f16_at_src0(block_byte_base + 108u);
let d_all = load_f16_as_f32_at_src0(block_byte_base + 108u);
let chunk = k_in_block / 128u;
let pos_in_chunk = k_in_block % 32u;
@@ -424,29 +424,29 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
// low 2 bits (4 elems)
let q_lo2_word = load_u32_at_src0(qs_byte_base + 32u * chunk + 1u * hmask_block);
let q_lo2_vec4 = vec4<f16>(
f16((q_lo2_word >> (2u * shift_phase + 0u)) & 3u),
f16((q_lo2_word >> (2u * shift_phase + 8u)) & 3u),
f16((q_lo2_word >> (2u * shift_phase + 16u)) & 3u),
f16((q_lo2_word >> (2u * shift_phase + 24u)) & 3u)
let q_lo2_vec4 = vec4<f32>(
f32((q_lo2_word >> (2u * shift_phase + 0u)) & 3u),
f32((q_lo2_word >> (2u * shift_phase + 8u)) & 3u),
f32((q_lo2_word >> (2u * shift_phase + 16u)) & 3u),
f32((q_lo2_word >> (2u * shift_phase + 24u)) & 3u)
);
// high 1 bit (4 elems)
let q_hi1_word = load_u32_at_src0(hmask_byte_base + pos_in_chunk);
let q_hi1_vec4 = vec4<f16>(
f16(select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 0u)) & 1u) == 1u)),
f16(select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 8u)) & 1u) == 1u)),
f16(select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 16u)) & 1u) == 1u)),
f16(select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 24u)) & 1u) == 1u))
let q_hi1_vec4 = vec4<f32>(
select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 0u)) & 1u) == 1u),
select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 8u)) & 1u) == 1u),
select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 16u)) & 1u) == 1u),
select(4.0, 0.0, ((q_hi1_word >> (1u * hmask_shift_phase + 24u)) & 1u) == 1u)
);
let q_vec4 = q_lo2_vec4 - q_hi1_vec4;
let scale_low4 = (load_byte_at_src0_aligned(scales_byte_base + (sub_block % 8u)) >> (4u * (sub_block / 8u))) & 0xFu;
let scale_hi2 = (load_byte_at_src0_aligned(scales_byte_base + 8u + (sub_block % 4u)) >> (2u * (sub_block / 4u))) & 3u;
let dl = d_all * (f16((scale_hi2 << 4u) | scale_low4) - 32.0);
let dl = d_all * (f32((scale_hi2 << 4u) | scale_low4) - 32.0);
store_shmem_kquants(dl * q_vec4, elem_idx);
store_shmem_kquants(vec4<f16>(dl * q_vec4), elem_idx);
#endif // INIT_SRC0_SHMEM_Q3_K
#if defined(INIT_SRC0_SHMEM_Q4_K)
@@ -456,8 +456,8 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let qs_byte_base = block_byte_base + 16u;
let dm = unpack2x16float(load_u32_at_src0_aligned(dm_byte_base));
let d = f16(dm[0]);
let dmin = f16(dm[1]);
let d = dm[0];
let dmin = dm[1];
let chunk = k_in_block / 64u;
let pos_in_chunk = (k_in_block % 64u) % 32u;
@@ -466,11 +466,11 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
// whole 4 bits (4 elems)
let qs_word = load_u32_at_src0_aligned(qs_byte_base + 32u * chunk + 1u * pos_in_chunk);
let qs_vec4 = vec4<f16>(
f16((qs_word >> (4u * shift_phase + 0u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 8u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 16u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 24u)) & 0xFu)
let qs_vec4 = vec4<f32>(
f32((qs_word >> (4u * shift_phase + 0u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 8u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 16u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 24u)) & 0xFu)
);
var sc: u32;
@@ -489,10 +489,10 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
mn = (sc_min_lo >> 4u) | ((min_hi >> 6u) << 4u);
}
let dl = d * f16(sc);
let ml = dmin * f16(mn);
let dl = d * f32(sc);
let ml = dmin * f32(mn);
store_shmem_kquants(dl * qs_vec4 - vec4(ml, ml, ml, ml), elem_idx);
store_shmem_kquants(vec4<f16>(dl * qs_vec4 - vec4(ml, ml, ml, ml)), elem_idx);
#endif // INIT_SRC0_SHMEM_Q4_K
#if defined(INIT_SRC0_SHMEM_Q5_K)
@@ -503,8 +503,8 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let qs_byte_base = block_byte_base + 48u;
let dm = unpack2x16float(load_u32_at_src0_aligned(dm_byte_base));
let d = f16(dm[0]);
let dmin = f16(dm[1]);
let d = dm[0];
let dmin = dm[1];
let chunk = k_in_block / 64u;
let pos_in_chunk = (k_in_block % 64u) % 32u;
@@ -516,20 +516,20 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
// low 4 bits (4 elems)
let qs_word = load_u32_at_src0_aligned(qs_byte_base + 32u * chunk + 1u * pos_in_chunk);
let qs_lo4_vec4 = vec4<f16>(
f16((qs_word >> (4u * shift_phase + 0u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 8u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 16u)) & 0xFu),
f16((qs_word >> (4u * shift_phase + 24u)) & 0xFu)
let qs_lo4_vec4 = vec4<f32>(
f32((qs_word >> (4u * shift_phase + 0u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 8u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 16u)) & 0xFu),
f32((qs_word >> (4u * shift_phase + 24u)) & 0xFu)
);
// high 1 bit (4 elems)
let qh_word = load_u32_at_src0_aligned(qh_byte_base + qh_block);
let qh_vec4 = vec4<f16>(
f16(select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 0u)) & 1u) == 1u)),
f16(select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 8u)) & 1u) == 1u)),
f16(select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 16u)) & 1u) == 1u)),
f16(select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 24u)) & 1u) == 1u))
let qh_vec4 = vec4<f32>(
select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 0u)) & 1u) == 1u),
select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 8u)) & 1u) == 1u),
select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 16u)) & 1u) == 1u),
select(0.0, 16.0, ((qh_word >> (1u * qh_shift_phase + 24u)) & 1u) == 1u)
);
var sc: u32;
@@ -548,10 +548,10 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
mn = (sc_min_lo >> 4u) | ((min_hi >> 6u) << 4u);
}
let dl = d * f16(sc);
let ml = dmin * f16(mn);
let dl = d * f32(sc);
let ml = dmin * f32(mn);
store_shmem_kquants((qh_vec4 + qs_lo4_vec4) * dl - vec4<f16>(ml, ml, ml, ml), elem_idx);
store_shmem_kquants(vec4<f16>((qh_vec4 + qs_lo4_vec4) * dl - vec4<f32>(ml, ml, ml, ml)), elem_idx);
#endif // INIT_SRC0_SHMEM_Q5_K
#if defined(INIT_SRC0_SHMEM_Q6_K)
@@ -561,7 +561,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let scales_byte_base = block_byte_base + 192u;
let d_byte_base = block_byte_base + 208u;
let d = load_f16_at_src0(d_byte_base);
let d = load_f16_as_f32_at_src0(d_byte_base);
let chunk = k_in_block / 128u;
let ql_pos_in_chunk = (k_in_block % 128u) % 64u;
@@ -588,13 +588,13 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
((qh_word >> (2u * qh_shift_phase + 24u)) & 0x3u) << 4u,
);
let q_vec4 = vec4<f16>(qh_hi2_vec4 | ql_lo4_vec4) - vec4<f16>(32.0, 32.0, 32.0, 32.0);
let q_vec4 = vec4<f32>(qh_hi2_vec4 | ql_lo4_vec4) - vec4<f32>(32.0, 32.0, 32.0, 32.0);
let scale_byte = scales_byte_base + 1u * sub_block;
let scale_word = load_u32_at_src0_aligned(scale_byte);
let scale = get_byte_i32(scale_word, scale_byte & 3u);
store_shmem_kquants(d * q_vec4 * f16(scale), elem_idx);
store_shmem_kquants(vec4<f16>(d * q_vec4 * f32(scale)), elem_idx);
#endif // INIT_SRC0_SHMEM_Q6_K
}
}
@@ -626,17 +626,17 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let d_byte_base = block_byte_base + 0u;
let qs_byte_base = block_byte_base + 2u;
let d = load_f16_at_src0(d_byte_base);
let d = load_f16_as_f32_at_src0(d_byte_base);
let id_qtr = (k_in_block % 16u) / 4u;
let shift_phase = k_in_block / 16u;
let qs_u32 = load_u32_at_src0(qs_byte_base + 4u * id_qtr);
shmem[elem_idx + 0u] = d * f16(kvalues_iq4nl[(qs_u32 >> ( 0u + 4u * shift_phase)) & 0xFu]);
shmem[elem_idx + 1u] = d * f16(kvalues_iq4nl[(qs_u32 >> ( 8u + 4u * shift_phase)) & 0xFu]);
shmem[elem_idx + 2u] = d * f16(kvalues_iq4nl[(qs_u32 >> (16u + 4u * shift_phase)) & 0xFu]);
shmem[elem_idx + 3u] = d * f16(kvalues_iq4nl[(qs_u32 >> (24u + 4u * shift_phase)) & 0xFu]);
shmem[elem_idx + 0u] = f16(d * f32(kvalues_iq4nl[(qs_u32 >> ( 0u + 4u * shift_phase)) & 0xFu]));
shmem[elem_idx + 1u] = f16(d * f32(kvalues_iq4nl[(qs_u32 >> ( 8u + 4u * shift_phase)) & 0xFu]));
shmem[elem_idx + 2u] = f16(d * f32(kvalues_iq4nl[(qs_u32 >> (16u + 4u * shift_phase)) & 0xFu]));
shmem[elem_idx + 3u] = f16(d * f32(kvalues_iq4nl[(qs_u32 >> (24u + 4u * shift_phase)) & 0xFu]));
}
}
#endif // INIT_SRC0_SHMEM_IQ4_NL
@@ -670,12 +670,12 @@ fn create_iq_gw4(dl: f32, gw: u32, shift_base: u32, delta: f32) -> vec4<f16> {
#endif
#if defined(INIT_SRC0_SHMEM_IQ4_XS)
fn create_iq_gw4(dl: f16, qs_u32: u32, shift_phase: u32) -> vec4<f16> {
fn create_iq_gw4(dl: f32, qs_u32: u32, shift_phase: u32) -> vec4<f16> {
return vec4<f16>(
dl * f16(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 0u)) & 0xFu]),
dl * f16(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 8u)) & 0xFu]),
dl * f16(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 16u)) & 0xFu]),
dl * f16(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 24u)) & 0xFu]),
f16(dl * f32(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 0u)) & 0xFu])),
f16(dl * f32(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 8u)) & 0xFu])),
f16(dl * f32(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 16u)) & 0xFu])),
f16(dl * f32(kvalues_iq4nl[(qs_u32 >> (4 * shift_phase + 24u)) & 0xFu])),
);
}
#endif
@@ -775,7 +775,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let qs_byte_base = block_byte_base + 8u;
let d_scales_h = load_u32_at_src0_aligned(d_byte_base);
let d = bitcast<vec2<f16>>(d_scales_h).x;
let d = unpack2x16float(d_scales_h).x;
let scales_h = d_scales_h >> 16u;
let sub_block = k_in_block / 32u;
@@ -784,7 +784,7 @@ fn init_shmem_src0(thread_id: u32, batch_offset: u32, offset_m: u32, k_outer: u3
let scales_l_u32 = load_u32_at_src0_aligned(scales_l_byte_base);
let ls_lo = (get_byte(scales_l_u32, sub_block / 2u) >> (4u * (sub_block % 2u))) & 0xFu;
let ls_hi = ((scales_h >> (2u * sub_block)) & 3u) << 4u;
let dl = d * f16(i32(ls_lo | ls_hi) - 32);
let dl = d * f32(i32(ls_lo | ls_hi) - 32);
let qs_0_3_u32 = load_u32_at_src0_aligned(qs_byte_base + 16u * sub_block + 0u);
let qs_4_7_u32 = load_u32_at_src0_aligned(qs_byte_base + 16u * sub_block + 4u);
@@ -1,21 +1,19 @@
#ifdef U32_DEQUANT_HELPERS
fn dequant_q4_0_packed_to_shmem(q_packed: u32, d: f16, dst_idx: u32) {
let scale = QUANT_OUT_TYPE(d);
fn dequant_q4_0_packed_to_shmem(q_packed: u32, d: f32, dst_idx: u32) {
for (var k = 0u; k < 4u; k++) {
let q_byte = get_byte(q_packed, k);
let q_hi = (QUANT_OUT_TYPE((q_byte >> 4) & 0xFu) - QUANT_OUT_TYPE(8.0)) * scale;
let q_lo = (QUANT_OUT_TYPE(q_byte & 0xFu) - QUANT_OUT_TYPE(8.0)) * scale;
QUANT_SHMEM[dst_idx + k] = q_lo;
QUANT_SHMEM[dst_idx + k + 16u] = q_hi;
let q_hi = (f32((q_byte >> 4) & 0xFu) - 8.0) * d;
let q_lo = (f32(q_byte & 0xFu) - 8.0) * d;
QUANT_SHMEM[dst_idx + k] = QUANT_OUT_TYPE(q_lo);
QUANT_SHMEM[dst_idx + k + 16u] = QUANT_OUT_TYPE(q_hi);
}
}
fn dequant_q8_0_packed_to_shmem(q_packed: u32, d: f16, dst_idx: u32) {
let scale = QUANT_OUT_TYPE(d);
fn dequant_q8_0_packed_to_shmem(q_packed: u32, d: f32, dst_idx: u32) {
for (var k = 0u; k < 4u; k++) {
let q_byte = get_byte_i32(q_packed, k);
let q_val = QUANT_OUT_TYPE(q_byte) * scale;
QUANT_SHMEM[dst_idx + k] = q_val;
let q_val = f32(q_byte) * d;
QUANT_SHMEM[dst_idx + k] = QUANT_OUT_TYPE(q_val);
}
}
#endif

Some files were not shown because too many files have changed in this diff Show More