Compare commits

...
16 Commits
Author SHA1 Message Date
Pascal e8fbcad855 tests: print the exception message in test-llama-context (#32428)
A case that throws only reported "unhandled exception", which hides
the cause of the intermittent failures on the Vulkan Apple runner.
2026-11-16 21:53:32 +01:00
Xuan-Son Nguyen 95e837aaf1 skills: explicitly specify paths to README-dev.md (#32431) 2026-11-16 21:52:12 +01:00
Bartowski d998c676ff convert: Fix inkling 2 mmproj conversion (#32430) 2026-11-16 20:39:32 +01:00
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00
shaofeiqi ad5cbfa911 opencl: add A7 Q4_K non-MoE binary kernel (#30948) 2026-11-16 10:58:29 -08:00
Hongqiang Wang d261e49e7e opencl: fix three defects of the bin flash-attention prefill path (#32110)
* opencl: release the WMM flash-attention mask fallback image

The fallback image is created on every call of the bin flash-attention
kernel over a sub-buffer of the K cache and was never released, so each
call kept a K-cache sub-buffer alive. Every call with more than one query
row takes this path, including each speculative-decoding verify step, so
the objects pile up and the server slows with every request.

* opencl: keep q8_0 K/V at head size 512 off the WMM flash-attention kernel

With q8_0 K/V at DK=512 the bin flash-attention kernel gives wrong results
whenever V holds different values from K (test-backend-ops FLASH_ATTN_EXT
hsk=512 q8_0, max error 2.04 on X2-90); f16 K/V at DK=512 is correct. The
backend's own flash-attention kernels already decline q8_0 at DK=512, so the
shape is now handled the same way with or without the kernel library.

* opencl: decline WMM flash-attention prefill beyond the device's 3D image limits

The WMM path repacks Q/K/V into 3D images; V is n_kv wide, so past the device's
maximum 3D image width (about 16k tokens of context on X2-90) clCreateImage
failed and the call asserted. Such shapes now take the regular flash-attention
path.
2026-11-16 10:57:48 -08:00
Aldehir Rojas 4ddf2b9470 chat : implement inkling parser (#32381) 2026-11-16 19:48:24 +01:00
Xuan-Son Nguyen 4303cd94ae contributing: mark agents.md and code-review as required (#32416) 2026-11-16 19:26:10 +01:00
Beinsezii c7518dbec5 MMVQ: Tune ne11 Gate for RDNA3_5 (#32308) 2026-11-17 00:56:20 +08:00
Xuan-Son Nguyen a68c6136bb mtmd: support fish audio s2 (#32330)
* add fisk-audio-s2 support

* add voice cloning

* cont merge master

* nits

* nits

* clean up

* nits

* use new batch api
2026-11-16 17:32:27 +01:00
Aleksander Grygier 3a7660371d ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat

The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's package @ggml-org/llama-chat

It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the built-in web UI Llama Chat in docs, help text and comments

The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.

Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name tools/ui as a former path of Llama Chat in the code-review skill

Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's path in the security policy's exclusions

Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-16 17:23:36 +01:00
Shubham RoyandDeveloper e8698ac163 fix:- Misc. bug: Settings/Tools is lying about tool picks inside a ch… (#32223)
* fix:- Misc. bug: Settings/Tools is lying about tool picks inside a chat #32214

* checks run lint error

---------

Co-authored-by: Developer <dev@example.com>
2026-11-16 17:15:16 +01:00
Rudransh Shekhar 7804be0634 arg : allow up to llama_max_devices() configs for --tensor-split and --fit-target (#13293) (#31435) 2026-11-16 17:07:45 +01:00
Xuan-Son Nguyen ef6bc6f367 jinja: impl scope-chain key lookup (#32396)
* jinja: impl scope-chain key lookup

* add test for call_statement lambda
2026-11-16 15:48:46 +01:00
Niklas Wenzel a06044b934 llama : deprecate old batch API (#32306) 2026-11-16 15:16:13 +01:00
Niklas Wenzel 62a6d3a302 llama : migrate to llama_batch_ext (#32356)
* llama : migrate to llama_batch_ext

* mtmd : migrate zonos3-mini code

Added in https://github.com/ggml-org/llama.cpp/pull/32020
2026-11-16 15:16:12 +01:00
1098 changed files with 3955 additions and 534 deletions
+4 -4
View File
@@ -19,12 +19,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${CANN_BASE_IMAGE} AS build
@@ -40,7 +40,7 @@ WORKDIR /app
# -- Copy project files --
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# -- Set CANN environment variables (required for compilation) --
# Using ENV instead of `source` allows environment variables to persist across the entire image layer
+4 -4
View File
@@ -9,12 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -28,7 +28,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "arm64" ]; then \
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON; \
+4 -4
View File
@@ -17,12 +17,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_CUDA_DEV_CONTAINER} AS build
@@ -40,7 +40,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${CUDA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=${CUDA_DOCKER_ARCH}"; \
+4 -4
View File
@@ -11,12 +11,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/intel/deep-learning-essentials:$ONEAPI_VERSION AS build
@@ -41,7 +41,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${GGML_SYCL_F16}" = "ON" ]; then \
echo "GGML_SYCL_F16 is set" \
+4 -4
View File
@@ -15,12 +15,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM ${BASE_MUSA_DEV_CONTAINER} AS build
@@ -45,7 +45,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN if [ "${MUSA_DOCKER_ARCH}" != "default" ]; then \
export CMAKE_ARGS="-DMUSA_ARCHITECTURES=${MUSA_DOCKER_ARCH}"; \
+5 -5
View File
@@ -133,11 +133,11 @@ effectiveStdenv.mkDerivation (finalAttrs: {
src = lib.cleanSource ../../.;
};
# Builds the webui locally, taking care not to require updating any sha256 hash.
# Builds Llama Chat locally, taking care not to require updating any sha256 hash.
webui = stdenvNoCC.mkDerivation {
pname = "webui";
version = llamaVersion;
src = lib.cleanSource ../../tools/ui;
src = lib.cleanSource ../../ui/apps/chat;
nativeBuildInputs = [
nodejs
@@ -146,7 +146,7 @@ effectiveStdenv.mkDerivation (finalAttrs: {
# no sha256 required when using buildNodeModules
npmDeps = importNpmLock.buildNodeModules {
npmRoot = ../../tools/ui;
npmRoot = ../../ui/apps/chat;
inherit nodejs;
};
@@ -156,8 +156,8 @@ effectiveStdenv.mkDerivation (finalAttrs: {
};
postPatch = lib.optionalString useWebUi ''
cp -r ${finalAttrs.webui} tools/ui/dist
chmod -R u+w tools/ui/dist
cp -r ${finalAttrs.webui} ui/apps/chat/dist
chmod -R u+w ui/apps/chat/dist
'';
# With PR#6015 https://github.com/ggml-org/llama.cpp/pull/6015,
+4 -4
View File
@@ -28,12 +28,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
## Build Image
@@ -83,7 +83,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
# Build Stage
RUN bash -c "source ${OpenVINO_DIR}/setupvars.sh && \
+4 -4
View File
@@ -17,12 +17,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
### Build image
@@ -52,7 +52,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
+4 -4
View File
@@ -9,12 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -31,7 +31,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -B build -DGGML_NATIVE=OFF -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON && \
cmake --build build --config Release -j$(nproc)
+4 -4
View File
@@ -9,12 +9,12 @@ FROM docker.io/node:$NODE_VERSION AS web
ARG APP_VERSION
WORKDIR /app/tools/ui
WORKDIR /app/ui/apps/chat
COPY tools/ui/package.json tools/ui/package-lock.json ./
COPY ui/apps/chat/package.json ui/apps/chat/package-lock.json ./
RUN npm ci
COPY tools/ui/ ./
COPY ui/apps/chat/ ./
RUN LLAMA_BUILD_NUMBER="$APP_VERSION" npm run build
FROM docker.io/ubuntu:$UBUNTU_VERSION AS build
@@ -28,7 +28,7 @@ WORKDIR /app
COPY . .
COPY --from=web /app/tools/ui/dist tools/ui/dist
COPY --from=web /app/ui/apps/chat/dist ui/apps/chat/dist
RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_ZENDNN=ON && \
cmake --build build -j $(nproc)
+1 -1
View File
@@ -10,7 +10,7 @@
build*/
tools/ui/node_modules/
ui/apps/chat/node_modules/
models/*
+1 -1
View File
@@ -48,7 +48,7 @@ insert_final_newline = unset
[vendor/**.patch]
trim_trailing_whitespace = unset
[tools/ui/**]
[ui/**]
indent_style = unset
indent_size = unset
end_of_line = unset
+1 -2
View File
@@ -48,7 +48,6 @@ examples:
- all-globs-to-all-files:
- '!tools/server/**'
- '!tools/mtmd/**'
- '!tools/ui/**'
testing:
- changed-files:
- any-glob-to-any-file:
@@ -72,7 +71,7 @@ android:
server/ui:
- changed-files:
- any-glob-to-any-file:
- tools/ui/**
- ui/**
server:
- changed-files:
- any-glob-to-any-file:
+1 -1
View File
@@ -182,7 +182,7 @@ jobs:
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
# - name: Set up QEMU
# if: ${{ contains(matrix.config.platforms, 'linux/amd64') }}
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
BODY: ${{ github.event.pull_request.body }}
HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'video demo') }}
run: |
# sections written by demo:pr (see tools/ui/demo/cli.ts): the managed comment
# sections written by demo:pr (see ui/apps/chat/demo/cli.ts): the managed comment
# markers, or a preview linking to a demo video uploaded for this PR
has_demo=false
if { grep -qF '<!-- demo:start -->' <<< "$BODY" && grep -qF '<!-- demo:end -->' <<< "$BODY"; } ||
+12 -12
View File
@@ -130,7 +130,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -196,7 +196,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -274,7 +274,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -381,7 +381,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -479,7 +479,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Set up JDK
uses: actions/setup-java@v5
@@ -571,7 +571,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -624,7 +624,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Build
id: cmake_build
@@ -672,7 +672,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: ccache
uses: ggml-org/ccache-action@v1.2.24
@@ -894,7 +894,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Install Ninja
run: |
@@ -1460,7 +1460,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
@@ -1532,7 +1532,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Free up disk space
uses: ggml-org/free-disk-space@v1.3.1
@@ -1647,7 +1647,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist
path: ui/apps/chat/dist
- name: Dependencies
id: depends
+4 -4
View File
@@ -23,20 +23,20 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+5 -5
View File
@@ -37,26 +37,26 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build application
env:
LLAMA_BUILD_NUMBER: ${{ steps.version.outputs.ui_version }}
run: npm run build
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run PWA unit tests (versioned build output)
run: npx vitest --project=unit --run tests/unit/pwa.spec.ts
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Upload built UI
uses: actions/upload-artifact@v7
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
retention-days: 1
+5 -5
View File
@@ -33,15 +33,15 @@ jobs:
uses: actions/download-artifact@v8
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
run-id: ${{ inputs.run_id }}
github-token: ${{ github.token }}
- name: Create distribution archive
run: |
tar -czf dist.tar.gz -C tools/ui/dist .
tar -czf dist.tar.gz -C ui/apps/chat/dist .
sha256sum dist.tar.gz > dist.tar.gz.sha256
mv dist.tar.gz dist.tar.gz.sha256 tools/ui/dist/
mv dist.tar.gz dist.tar.gz.sha256 ui/apps/chat/dist/
- name: Install Hugging Face Hub CLI
run: pip install -U huggingface_hub
@@ -52,12 +52,12 @@ jobs:
- name: Sync built files to Hugging Face bucket (version tag)
run: |
# Upload the built files to the Hugging Face bucket under the release version
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/${{ inputs.version_tag }} --delete --quiet
- name: Sync built files to Hugging Face bucket (latest)
run: |
# Also upload to the 'latest' directory for fallback downloads
hf buckets sync tools/ui/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
hf buckets sync ui/apps/chat/dist hf://buckets/ggml-org/${{ env.HF_BUCKET_NAME }}/latest --delete --quiet
- name: Verify upload
run: |
+13 -13
View File
@@ -17,7 +17,7 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -25,7 +25,7 @@ on:
paths: [
'.github/workflows/ui-self-hosted.yml',
'.github/workflows/ui-build-self-hosted.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'tools/server/tests/**.*'
]
@@ -67,33 +67,33 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
e2e-tests:
name: E2E Tests
@@ -109,25 +109,25 @@ jobs:
- name: Install dependencies
id: setup
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Build Storybook
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+17 -17
View File
@@ -13,7 +13,7 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'tools/server/tests/**.*'
]
pull_request:
@@ -21,7 +21,7 @@ on:
paths: [
'.github/workflows/ui.yml',
'.github/workflows/ui-build.yml',
'tools/ui/**.*',
'ui/apps/chat/**.*',
'tools/server/tests/**.*'
]
@@ -66,46 +66,46 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Download built UI artifacts
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run type checking
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run check
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run linting
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npm run lint
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Client tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:client
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run Unit tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:unit
working-directory: tools/ui
working-directory: ui/apps/chat
e2e-tests:
name: E2E Tests
@@ -124,38 +124,38 @@ jobs:
with:
node-version: "24"
# cache: "npm"
# cache-dependency-path: "tools/ui/package-lock.json"
# cache-dependency-path: "ui/apps/chat/package-lock.json"
package-manager-cache: false
- name: Install dependencies
id: setup
if: ${{ steps.node.conclusion == 'success' }}
run: npm ci
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Download built UI artifacts (reuses ui-build)
uses: actions/download-artifact@v6
with:
name: llama-ui.zip
path: tools/ui/dist/
path: ui/apps/chat/dist/
- name: Install Playwright browsers
id: playwright
if: ${{ always() && steps.setup.conclusion == 'success' }}
run: npx playwright install --with-deps
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Build Storybook
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run build-storybook
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run UI tests
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:ui -- --testTimeout=60000
working-directory: tools/ui
working-directory: ui/apps/chat
- name: Run E2E tests (uses pre-built dist/ from ui-build)
if: ${{ always() && steps.playwright.conclusion == 'success' }}
run: npm run test:e2e
working-directory: tools/ui
working-directory: ui/apps/chat
+1 -1
View File
@@ -134,7 +134,7 @@ option(LLAMA_BUILD_TOOLS "llama: build tools"
option(LLAMA_BUILD_EXAMPLES "llama: build examples" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_SERVER "llama: build server example" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_APP "llama: build the unified binary" ${LLAMA_STANDALONE})
option(LLAMA_BUILD_UI "llama: build the embedded Web UI for server" OFF)
option(LLAMA_BUILD_UI "llama: build Llama Chat, the embedded chat app, for server" OFF)
option(LLAMA_USE_PREBUILT_UI "llama: use prebuilt UI from HF Bucket when available" ON)
option(LLAMA_CONNECT "llama: include the llama-connect tool (downloads a prebuilt binary)" OFF)
+1 -1
View File
@@ -102,7 +102,7 @@
/tools/rpc/ @ggml-org/ggml-rpc
/tools/server/* @ggml-org/llama-server # no subdir
/tools/server/tests/ @ggml-org/llama-server
/tools/ui/ @ggml-org/llama-ui
/ui/ @ggml-org/llama-ui
/tools/tokenize/ @ggerganov
/tools/tts/ @ggerganov
/vendor/ @ggerganov
+12
View File
@@ -16,6 +16,8 @@ The project differentiates between 3 levels of contributors:
>
> Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
### Requirements
If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
1. Explicitly disclose the manner in which AI was employed.
@@ -24,6 +26,16 @@ If AI is used to generate any portion of the code, contributors must adhere to t
4. Be prepared to explain every line of code you submit when asked about it by a maintainer.
5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
### Checklist
> [!IMPORTANT]
>
> **If you use AI, do all of the following:**
>
> 1. **Make sure [AGENTS.md](AGENTS.md) is loaded into your agent's context.** Skipping this step is one of the most common reasons PRs get rejected.
> 2. **Search for related discussions, PRs and issues** to get the broader context before writing any code.
> 3. **Run a local review pass before pushing anything**, using the [`code-review`](skills/code-review/SKILL.md) skill, once the change is implemented. Fix what it finds.
For more info, please refer to the [AGENTS.md](AGENTS.md) file.
# Pull requests (for contributors & collaborators)
+2 -2
View File
@@ -57,8 +57,8 @@ llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
<i>VLM session with <b>llama cli</b></i>
</td>
<td align="center">
<img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Built-in web UI against <b>llama serve</b></i>
<img width="1392" height="958" alt="Llama Chat, the built-in chat app, against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" />
<i>Llama Chat, the built-in chat app, against <b>llama serve</b></i>
</td>
</tr>
<table>
+1 -1
View File
@@ -46,7 +46,7 @@ Only vulnerabilities that fall within these parts of the project are considered
- `ggml/**/*`
- `gguf-py/**/*`
- `tools/server/*`, **excluding** the following topics:
- Web UI
- Llama Chat, the built-in chat app (`ui/apps/chat`)
- Features marked as experimental
- Features not recommended for use in untrusted environments (e.g., router, MCP)
- Bugs that can lead to Denial-of-Service attack
+12 -4
View File
@@ -2829,7 +2829,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -2895,7 +2895,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
const std::regex regex{ R"([,/]+)" };
std::sregex_token_iterator it{ arg_next.begin(), arg_next.end(), regex, -1 };
std::vector<std::string> split_arg{ it, {} };
if (split_arg.size() >= llama_max_devices()) {
if (split_arg.size() > llama_max_devices()) {
throw std::invalid_argument(
string_format("got %zu input configs, but system only has %zu devices", split_arg.size(), llama_max_devices())
);
@@ -3416,7 +3416,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui-mcp-proxy", "--webui-mcp-proxy"},
{"--no-ui-mcp-proxy", "--no-webui-mcp-proxy"},
"experimental: whether to enable MCP CORS proxy, for the Web UI to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)\n"
"experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)\n"
"note: for security reasons, this will limit --cors-origins to localhost by default",
[](common_params & params, bool value) {
params.ui_mcp_proxy = value;
@@ -3485,7 +3485,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
add_opt(common_arg(
{"--ui", "--webui"},
{"--no-ui", "--no-webui"},
string_format("whether to enable the Web UI (default: %s)", params.ui ? "enabled" : "disabled"),
string_format("whether to enable Llama Chat, the built-in chat app (default: %s)", params.ui ? "enabled" : "disabled"),
[](common_params & params, bool value) {
params.ui = value;
}
@@ -4522,6 +4522,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.tts_speaker_file = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
add_opt(common_arg(
{"--tts-speaker-text"}, "TEXT",
"transcript of --tts-speaker-file, for models that need it to clone the voice\n"
"see docs/multimodal/tts.md for per-model usage notes",
[](common_params & params, const std::string & value) {
params.tts_speaker_text = value;
}
).set_examples({LLAMA_EXAMPLE_TTS}));
//
// diffusion params
+6 -3
View File
@@ -1444,6 +1444,12 @@ static common_chat_params_init_fn common_chat_template_detect_params_init(const
return common_chat_params_init_qwen3_coder;
}
// Inkling 1 and 2
if (is_inkling_template(src)) {
LOG_DBG("Using specialized template: Inkling\n");
return common_chat_params_init_inkling;
}
return nullptr;
}
@@ -1626,9 +1632,6 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
auto_params.thinking_end_tags = {std::move(end_tag)};
}
}
if (is_inkling2_template(src)) {
common_chat_inkling2_split_media(auto_params.prompt);
}
const auto & arena = auto_params.parser;
LOG_DBG("%s: generated parser:\n%s\n\nparser generation prompt: %s\n", __func__, arena.dump(arena.root()).c_str(), auto_params.generation_prompt.c_str());
return auto_params;
+1
View File
@@ -796,6 +796,7 @@ struct common_params {
std::string tts_lang = "";
std::string tts_voice = "";
std::string tts_speaker_file = "";
std::string tts_speaker_text = "";
bool is_gen_docs = false; // whether we are running inside llama-gen-docs
};
+4 -3
View File
@@ -811,13 +811,14 @@ value call_statement::execute_impl(context & ctx) const {
}
auto * callee_func = cast_val<value_func>(callee_val);
context caller_ctx(ctx); // new scope for caller execution
// caller() can be saved and called after this scope ended, it owns a snapshot of the variables
auto caller_ctx = std::make_shared<context>(ctx.snapshot());
const func_handler func = [this, caller_ctx = std::move(caller_ctx)](const func_args & args) -> value {
const func_handler func = [this, caller_ctx](const func_args & args) -> value {
if (args.ctx.macro_depth >= JINJA_RECURSIVE_LIMIT) {
throw std::runtime_error("macro calls are nested more than " + std::to_string(JINJA_RECURSIVE_LIMIT) + " levels deep");
}
context block_ctx(caller_ctx); // new scope for block execution
context block_ctx(*caller_ctx); // new scope for block execution
block_ctx.macro_depth = args.ctx.macro_depth + 1;
bind_parameters("caller", this->caller_args, args, block_ctx);
+30 -3
View File
@@ -81,12 +81,38 @@ struct context {
current_time(parent.current_time),
is_get_stats(parent.is_get_stats),
macro_depth(parent.macro_depth),
// inherit variables (for example, when entering a new scope)
env(mk_val<value_object>(*parent.env)) {}
// child scope: own variables only, lookups fall back to the parent (which must outlive this scope)
parent(&parent),
env(mk_val<value_object>()) {
env->has_builtins = false;
}
context(context &&) = default;
// copy with all visible variables flattened in, for a scope that can outlive its parents (caller() of a call block)
context snapshot() const {
std::vector<const context *> chain;
for (const context * c = this; c; c = c->parent) {
chain.push_back(c);
}
context res(*this);
res.parent = nullptr;
for (auto it = chain.rbegin(); it != chain.rend(); ++it) {
for (const auto & kv : (*it)->env->as_ordered_object()) {
res.env->insert(kv.first, kv.second);
}
}
return res;
}
// returns nullptr if not found
value find_val(const value & key) {
return env->find(key);
for (const context * c = this; c; c = c->parent) {
if (value v = c->env->find(key)) {
return v;
}
}
return nullptr;
}
value get_val(const std::string & name) {
@@ -107,6 +133,7 @@ struct context {
}
private:
const context * parent = nullptr;
value_object env;
};
+165
View File
@@ -0,0 +1,165 @@
#include "parsers.h"
bool is_inkling_template(const std::string & src) {
return src.find("<|message_user|>") != std::string::npos &&
src.find("<|content_image|>") != std::string::npos;
}
// the template makes one message block per content part, but a media marker comes inside the text of a message
// <header><|content_text|>{media}text<|end_message|> --> <header>{media}<|end_message|><header><|content_text|>text<|end_message|>
static void inkling_split_media(common_chat_prompt & prompt) {
const std::string TEXT = "<|content_text|>";
const std::string END = "<|end_message|>";
const std::string MEDIA_BEG = "<__media_";
const std::string MEDIA_END = "__>";
std::vector<common_chat_prompt_part> out;
for (size_t i = 0; i < prompt.parts.size(); i++) {
const auto & part = prompt.parts[i];
const bool in_text_block = part.is_input && part.text.find(MEDIA_BEG) != std::string::npos &&
!out.empty() && !out.back().is_input && string_ends_with(out.back().text, TEXT) &&
i + 1 < prompt.parts.size() && !prompt.parts[i + 1].is_input && string_starts_with(prompt.parts[i + 1].text, END);
const size_t header_pos = in_text_block ? out.back().text.rfind("<|message_") : std::string::npos;
if (header_pos == std::string::npos) {
out.push_back(part);
continue;
}
const std::string header = out.back().text.substr(header_pos, out.back().text.size() - TEXT.size() - header_pos);
std::string rest = part.text;
for (bool first = true; !rest.empty(); first = false) {
const size_t beg = rest.find(MEDIA_BEG);
const size_t end = beg == std::string::npos ? beg : rest.find(MEDIA_END, beg);
const bool is_media = beg == 0 && end != std::string::npos;
const size_t len = is_media ? end + MEDIA_END.size() : (end == std::string::npos ? rest.size() : beg);
if (!first) {
out.push_back({ END + header + (is_media ? "" : TEXT), false });
} else if (is_media) {
// mtmd opens the media block
out.back().text.resize(out.back().text.size() - TEXT.size());
}
out.push_back({ rest.substr(0, len), true });
rest.erase(0, len);
}
}
prompt.parts = std::move(out);
}
// Inkling 1 and 2 - the assistant turn is a list of blocks, all closed by <|end_message|>:
// <|message_model|><|content_thinking|>...<|end_message|>
// <|message_model|><|content_text|>...<|end_message|>
// <|message_model|>NAME<|content_invoke_tool_json|>{"name":"NAME","args":{...}}<|end_message|>
common_chat_params common_chat_params_init_inkling(const common_chat_template & tmpl,
const autoparser::generation_params & inputs) {
common_chat_params data;
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs);
data.generation_prompt = common_chat_template_generation_prompt_impl(tmpl, inputs);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.supports_thinking = true;
inkling_split_media(data.prompt);
const std::string HEADER = "<|message_model|>";
const std::string THINK = "<|content_thinking|>";
const std::string TEXT = "<|content_text|>";
const std::string TOOL_JSON = "<|content_invoke_tool_json|>";
const std::string END = "<|end_message|>";
data.preserved_tokens = { HEADER, THINK, TEXT, TOOL_JSON, END };
data.thinking_start_tag = THINK;
data.thinking_end_tags = { END };
data.message_delimiters = {
{ COMMON_CHAT_ROLE_ASSISTANT, HEADER },
{ COMMON_CHAT_ROLE_USER, "<|message_user|>" },
{ COMMON_CHAT_ROLE_SYSTEM, "<|message_system|>" },
{ COMMON_CHAT_ROLE_TOOL, "<|message_tool|>" },
};
if (inputs.has_continuation()) {
const auto & msg = inputs.continue_msg;
data.generation_prompt.clear();
if (!msg.reasoning_content.empty()) {
data.generation_prompt = HEADER + THINK + msg.reasoning_content;
}
if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
if (!data.generation_prompt.empty()) {
data.generation_prompt += END;
}
data.generation_prompt += HEADER + TEXT + msg.render_content();
}
data.prompt.append(data.generation_prompt);
}
auto has_tools = inputs.tools.is_array() && !inputs.tools.empty();
auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
auto use_tools = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto header = p.literal(HEADER);
auto end = p.literal(END);
auto think = p.literal(THINK);
auto thinking = p.rule("thinking", extract_reasoning ? think + p.reasoning(p.until(END)) + end :
p.atomic(p.content(think)) + p.content(p.until(END) + end));
auto text = p.rule("text", p.literal(TEXT) + p.content(p.until(END)) + end);
if (has_response_format) {
auto response_format = p.rule("response-format", p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)));
return p.zero_or_more(header + thinking) + header + p.literal(TEXT) + response_format + end;
}
auto blocks = p.zero_or_more(header + (thinking | text));
if (!use_tools) {
return blocks;
}
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
const auto & function = tool.at("function");
std::string name = function.at("name");
auto open = p.tool_name(p.literal(name)) + p.literal(TOOL_JSON) + p.literal("{\"name\":" + json(name).dump() + ",\"args\":");
auto args = p.tool_args(p.json());
tool_choice |= p.rule("tool-" + std::to_string(tool_index), p.tool(p.tool_open(open) + args + p.tool_close(p.literal("}")) + end));
});
// tool calls are the last blocks of the turn
auto calls = inputs.parallel_tool_calls ? tool_choice + p.zero_or_more(header + tool_choice) : tool_choice;
auto tool_calls = header + p.trigger_rule("tool-calls", calls);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return blocks + tool_calls;
}
return blocks + p.optional(tool_calls);
});
if (has_response_format || use_tools) {
data.grammar_lazy = !has_response_format && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_REQUIRED;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
});
if (data.grammar_lazy) {
// a header followed by text and not by a content token is a tool call, the grammar starts at the name
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|message_model\\|>([^<])" },
};
// Inkling 1: the first header is in the generation prompt, and a lazy grammar does not get the prompt.
// an output that starts with no content token is a tool call. not for a continuation, it starts with text
if (!inputs.has_continuation()) {
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN_FULL, "[^<]+" });
}
}
}
return data;
}
-46
View File
@@ -1,46 +0,0 @@
#include "parsers.h"
bool is_inkling2_template(const std::string & src) {
return src.find("<|message_user|>") != std::string::npos &&
src.find("<|content_image|>") != std::string::npos;
}
// the template makes one message block per content part, but a media marker comes inside the text of a message
// <header><|content_text|>{media}text<|end_message|> --> <header>{media}<|end_message|><header><|content_text|>text<|end_message|>
void common_chat_inkling2_split_media(common_chat_prompt & prompt) {
const std::string TEXT = "<|content_text|>";
const std::string END = "<|end_message|>";
const std::string MEDIA_BEG = "<__media_";
const std::string MEDIA_END = "__>";
std::vector<common_chat_prompt_part> out;
for (size_t i = 0; i < prompt.parts.size(); i++) {
const auto & part = prompt.parts[i];
const bool in_text_block = part.is_input && part.text.find(MEDIA_BEG) != std::string::npos &&
!out.empty() && !out.back().is_input && string_ends_with(out.back().text, TEXT) &&
i + 1 < prompt.parts.size() && !prompt.parts[i + 1].is_input && string_starts_with(prompt.parts[i + 1].text, END);
const size_t header_pos = in_text_block ? out.back().text.rfind("<|message_") : std::string::npos;
if (header_pos == std::string::npos) {
out.push_back(part);
continue;
}
const std::string header = out.back().text.substr(header_pos, out.back().text.size() - TEXT.size() - header_pos);
std::string rest = part.text;
for (bool first = true; !rest.empty(); first = false) {
const size_t beg = rest.find(MEDIA_BEG);
const size_t end = beg == std::string::npos ? beg : rest.find(MEDIA_END, beg);
const bool is_media = beg == 0 && end != std::string::npos;
const size_t len = is_media ? end + MEDIA_END.size() : (end == std::string::npos ? rest.size() : beg);
if (!first) {
out.push_back({ END + header + (is_media ? "" : TEXT), false });
} else if (is_media) {
// mtmd opens the media block
out.back().text.resize(out.back().text.size() - TEXT.size());
}
out.push_back({ rest.substr(0, len), true });
rest.erase(0, len);
}
}
prompt.parts = std::move(out);
}
+3 -4
View File
@@ -41,10 +41,7 @@ std::string common_chat_template_generation_prompt_impl(
bool is_lfm2_template(const std::string & src);
bool is_inkling2_template(const std::string & src);
// give each media marker of a text block its own message block
void common_chat_inkling2_split_media(common_chat_prompt & prompt);
bool is_inkling_template(const std::string & src);
common_chat_params common_chat_params_init_cohere2moe(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
@@ -60,6 +57,8 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
common_chat_params common_chat_params_init_harmony_xml(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_inkling(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_k2_horizon(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
common_chat_params common_chat_params_init_kimi_k2(const common_chat_template & tmpl, const autoparser::generation_params & inputs);
+1 -1
View File
@@ -10,7 +10,7 @@ set(LLAMA_CHAT_PARSERS_SOURCES
${CMAKE_CURRENT_LIST_DIR}/gigachat-v3.cpp
${CMAKE_CURRENT_LIST_DIR}/gpt-oss.cpp
${CMAKE_CURRENT_LIST_DIR}/harmony-xml.cpp
${CMAKE_CURRENT_LIST_DIR}/inkling2.cpp
${CMAKE_CURRENT_LIST_DIR}/inkling.cpp
${CMAKE_CURRENT_LIST_DIR}/k2-horizon.cpp
${CMAKE_CURRENT_LIST_DIR}/kimi-k2.cpp
${CMAKE_CURRENT_LIST_DIR}/kimi-k3.cpp
+4
View File
@@ -284,6 +284,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"Qwen3OmniMoeForConditionalGeneration": "qwen3vl",
"PocketTTSModel": "pockettts",
"Zonos3Model": "zonos3",
"YuE2ForCausalLM": "yue2",
"Qwen3TTSForConditionalGeneration": "qwen3tts",
"Qwen3VLForConditionalGeneration": "qwen3vl",
"Qwen3VLMoeForConditionalGeneration": "qwen3vl",
@@ -336,6 +337,7 @@ TEXT_MODEL_MAP: dict[str, str] = {
"XverseForCausalLM": "xverse",
"YoutuForCausalLM": "deepseek",
"YoutuVLForConditionalGeneration": "deepseek",
"fish_qwen3_omni": "fish_s2",
"modeling_grove_moe.GroveMoeForCausalLM": "grovemoe",
"modeling_sarvam_moe.SarvamMoEForCausalLM": "bailingmoe",
}
@@ -360,6 +362,7 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"Exaone4_5_ForConditionalGeneration": "exaone",
"FishOmniModel": "drama3",
"Zonos3Model": "zonos3",
"YuE2ForCausalLM": "yue2",
"Gemma3ForConditionalGeneration": "gemma",
"Gemma3nForConditionalGeneration": "gemma",
"Gemma4ForConditionalGeneration": "gemma",
@@ -438,6 +441,7 @@ MMPROJ_MODEL_MAP: dict[str, str] = {
"VoxtralForConditionalGeneration": "ultravox",
"VoxtralRealtimeForConditionalGeneration": "ultravox",
"YoutuVLForConditionalGeneration": "youtuvl",
"fish_qwen3_omni": "fish_s2",
}
+3
View File
@@ -3488,6 +3488,9 @@ def get_model_architecture(hparams: dict[str, Any], model_type: ModelType) -> st
# some models (llava_onevision2) only declare the architecture inside text_config
if arch is None and text_config.get("architectures") is not None:
arch = text_config["architectures"][0]
# some configs only have a model_type (fishaudio/s2-pro)
if arch is None:
arch = hparams.get("model_type")
if arch is None:
raise ValueError("Failed to detect model architecture")
return arch
+32 -19
View File
@@ -18,6 +18,30 @@ _TOWER_SHARED = "audio_decoder_mtp_shared.layers."
_TOWER_RESIDUAL = "audio_decoder_mtp.layers."
def load_codec(dir_model) -> dict[str, Tensor]:
sd = torch.load(dir_model / "codec.pth", map_location="cpu", mmap=True, weights_only=True)
return sd.get("state_dict", sd)
def fold_conv(sd: dict[str, Tensor], prefix: str) -> Tensor:
# fold weight norm
if prefix + ".conv.weight" in sd:
return sd[prefix + ".conv.weight"].float()
g = sd[prefix + ".conv.parametrizations.weight.original0"].float()
v = sd[prefix + ".conv.parametrizations.weight.original1"].float()
return g * v / v.norm(dim=(1, 2), keepdim=True)
def fold_wn(g: Tensor, v: Tensor) -> Tensor:
# fold old-style weight norm of a 1x1 conv, returns [out, in]
return (g.float() * v.float() / v.float().norm(dim=(1, 2), keepdim=True)).squeeze(-1)
def fold_snake(alpha: Tensor) -> tuple[Tensor, Tensor]:
alpha = alpha.float().flatten()
return alpha, 1.0 / (alpha + 1e-9)
@ModelBase.register("FishOmniModel")
@ModelBase.example("fishaudio/drama-3")
class Drama3TextModel(Qwen3_5MoeTextModel):
@@ -36,6 +60,7 @@ class Drama3TextModel(Qwen3_5MoeTextModel):
class Drama3MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = False
gen_projector_type = gguf.VisionProjectorType.DRAMA3_GEN
_LAYER_MAP = {
"enorm.weight": (gguf.MODEL_TENSOR.A_GEN_CODE_ENORM, True),
@@ -69,7 +94,7 @@ class Drama3MmprojModel(MmprojModel):
cfg = self.hparams
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.DRAMA3_GEN)
self.gguf_writer.add_clip_gen_audio_projector_type(self.gen_projector_type)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(cfg["dim"])
self.gguf_writer.add_gen_audio_feed_forward_length(cfg["intermediate_size"])
@@ -153,32 +178,20 @@ class Drama3MmprojModel(MmprojModel):
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
# only the DAC decoder, the encoder is for voice cloning (not supported yet)
sd = torch.load(self.dir_model / "codec.pth", map_location="cpu", mmap=True, weights_only=True)
sd = sd.get("state_dict", sd)
sd = load_codec(self.dir_model)
T = gguf.MODEL_TENSOR
def get(name: str) -> Tensor:
return sd[name].float()
def conv(prefix: str) -> Tensor:
# fold weight norm
if prefix + ".conv.weight" in sd:
return get(prefix + ".conv.weight")
g = get(prefix + ".conv.parametrizations.weight.original0")
v = get(prefix + ".conv.parametrizations.weight.original1")
return g * v / v.norm(dim=(1, 2), keepdim=True)
return fold_conv(sd, prefix)
def quant_table(prefix: str) -> Tensor:
# fold out_proj into the codebook, so decoding is a row lookup
g = get(prefix + ".out_proj.weight_g")
v = get(prefix + ".out_proj.weight_v")
w = (g * v / v.norm(dim=(1, 2), keepdim=True)).squeeze(-1)
w = fold_wn(sd[prefix + ".out_proj.weight_g"], sd[prefix + ".out_proj.weight_v"])
return get(prefix + ".codebook.weight") @ w.T + get(prefix + ".out_proj.bias")
def snake(alpha: Tensor) -> tuple[Tensor, Tensor]:
alpha = alpha.flatten()
return alpha, 1.0 / (alpha + 1e-9)
q = "quantizer."
yield (self.format_tensor_name(T.A_GEN_WAV_QUANT_FIRST_CB), quant_table(q + "semantic_quantizer.quantizers.0"))
n_rest = sum(1 for k in sd if k.startswith(q + "quantizer.quantizers.") and k.endswith(".codebook.weight"))
@@ -228,7 +241,7 @@ class Drama3MmprojModel(MmprojModel):
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_ENTRY, suffix=".bias"), get(d + "0.conv.bias"))
for bid in range(4):
p = f"{d}{bid + 1}.block."
a, b = snake(get(p + "0.alpha"))
a, b = fold_snake(get(p + "0.alpha"))
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_SNAKE, bid, suffix=".alpha"), a)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_SNAKE, bid, suffix=".beta"), b)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_UP_CONV, bid, suffix=".weight"), conv(p + "1"))
@@ -236,7 +249,7 @@ class Drama3MmprojModel(MmprojModel):
for xid in range(3):
r = f"{p}{xid + 2}.block."
for act, tensor in (("0", T.A_GEN_WAV_DAC_RES_ACT1), ("2", T.A_GEN_WAV_DAC_RES_ACT2)):
a, b = snake(get(r + act + ".alpha"))
a, b = fold_snake(get(r + act + ".alpha"))
name = gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
yield (name + ".alpha", a)
yield (name + ".beta", b)
@@ -244,7 +257,7 @@ class Drama3MmprojModel(MmprojModel):
name = gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
yield (name + ".weight", conv(r + cv))
yield (name + ".bias", get(r + cv + ".conv.bias"))
a, b = snake(get(d + "5.alpha"))
a, b = fold_snake(get(d + "5.alpha"))
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_SNAKE, suffix=".alpha"), a)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_SNAKE, suffix=".beta"), b)
yield (self.format_tensor_name(T.A_GEN_WAV_DAC_POST_CONV, suffix=".weight"), conv(d + "6"))
+276
View File
@@ -0,0 +1,276 @@
from __future__ import annotations
from pathlib import Path
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, gguf
from .drama3 import Drama3MmprojModel, fold_conv, fold_snake, fold_wn, load_codec
from .qwen import Qwen3Model
# Fish Audio S2 Pro (dual-AR)
# text model: Qwen3 backbone (slow AR), samples the semantic code of each frame
# mmproj: fast AR code predictor (other codebooks) and Fish DAC decoder (codec.pth, same as drama3)
# residual codebook size of codec.pth, the fast AR head is cut to it
_N_RESIDUAL_CODES = 1024
# codec encoder strides (encoder_rates of the codec config), the quantizer then downsamples 2x2
_CODEC_ENC_STRIDES = (2, 4, 8, 8)
# codec encoder transformers, both 1024 wide with 16 heads of 64
# generic audio layers a.blk.0..3: last encoder block, a.blk.4..11: quantizer pre_module
_N_ENC_TFM_LAYERS = 4
_N_PRE_TFM_LAYERS = 8
_CODEC_HEAD_DIM = 64
def _load_hparams(dir_model: Path) -> dict[str, Any]:
hparams = ModelBase.load_hparams(dir_model, is_mistral_format=False)
cfg = hparams["text_config"]
hparams["text_config"] = {
"hidden_size": cfg["dim"],
"num_hidden_layers": cfg["n_layer"],
"num_attention_heads": cfg["n_head"],
"num_key_value_heads": cfg["n_local_heads"],
"head_dim": cfg["head_dim"],
"intermediate_size": cfg["intermediate_size"],
"rms_norm_eps": cfg["norm_eps"],
"rope_theta": cfg["rope_base"],
"max_position_embeddings": cfg["max_seq_len"],
"vocab_size": cfg["vocab_size"],
"tie_word_embeddings": cfg["tie_word_embeddings"],
}
return hparams
def _neox_rows(w: Tensor, head_dim: int) -> Tensor:
# checkpoint uses interleaved RoPE, reorder the rows of each head so NEOX RoPE gives the same result
order = torch.cat([torch.arange(0, head_dim, 2), torch.arange(1, head_dim, 2)])
return w.reshape(-1, head_dim, *w.shape[1:])[:, order].reshape(w.shape)
@ModelBase.register("fish_qwen3_omni")
@ModelBase.example("fishaudio/s2-pro")
class FishS2TextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
super().__init__(dir_model, *args, hparams=_load_hparams(dir_model), **kwargs)
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# the rest is mapped by tensor_mapping.py
name = name.removeprefix("text_model.")
T = gguf.MODEL_TENSOR
head_dim = self.hparams["head_dim"]
if name.endswith(".attention.wqkv.weight"):
n_q = self.hparams["num_attention_heads"] * head_dim
n_kv = self.hparams["num_key_value_heads"] * head_dim
q, k, v = data_torch.split([n_q, n_kv, n_kv], dim=0)
yield (self.format_tensor_name(T.ATTN_Q, bid), _neox_rows(q, head_dim))
yield (self.format_tensor_name(T.ATTN_K, bid), _neox_rows(k, head_dim))
yield (self.format_tensor_name(T.ATTN_V, bid), v)
return
if name.endswith(".attention.q_norm.weight"):
yield (self.format_tensor_name(T.ATTN_Q_NORM, bid), _neox_rows(data_torch, head_dim))
return
if name.endswith(".attention.k_norm.weight"):
yield (self.format_tensor_name(T.ATTN_K_NORM, bid), _neox_rows(data_torch, head_dim))
return
yield from super().modify_tensors(data_torch, name, bid)
@ModelBase.register("fish_qwen3_omni")
@ModelBase.example("fishaudio/s2-pro")
class FishS2MmprojModel(Drama3MmprojModel):
has_audio_encoder = True # codec encoder, for voice cloning
gen_projector_type = gguf.VisionProjectorType.FISH_S2_GEN
_FAST_AR_MAP = {
"attention_norm.weight": gguf.MODEL_TENSOR.A_GEN_CODE_ATTN_NORM,
"attention.wo.weight": gguf.MODEL_TENSOR.A_GEN_CODE_ATTN_OUT,
"ffn_norm.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_NORM,
"feed_forward.w1.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_GATE,
"feed_forward.w3.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_UP,
"feed_forward.w2.weight": gguf.MODEL_TENSOR.A_GEN_CODE_FFN_DOWN,
}
def __init__(self, dir_model: Path, *args, **kwargs):
kwargs.pop("hparams", None)
super().__init__(dir_model, *args, hparams=_load_hparams(dir_model), **kwargs)
def get_audio_config(self) -> dict[str, Any] | None:
cfg = dict(self.global_config["audio_decoder_config"])
cfg["num_hidden_layers"] = cfg["n_layer"]
return cfg
def set_gguf_parameters(self):
super().set_gguf_parameters()
# codec encoder: raw 44.1 kHz waveform -> backbone input rows, other hparams are hardcoded in tools/mtmd
self.gguf_writer.add_clip_has_audio_encoder(True)
self.gguf_writer.add_clip_audio_projector_type(gguf.VisionProjectorType.FISH_S2_SPKENC)
self.gguf_writer.add_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_audio_num_mel_bins(1)
self.gguf_writer.add_audio_block_count(_N_ENC_TFM_LAYERS + _N_PRE_TFM_LAYERS)
self.gguf_writer.add_audio_embedding_length(1024)
self.gguf_writer.add_audio_feed_forward_length(3072)
self.gguf_writer.add_audio_head_count(1024 // _CODEC_HEAD_DIM)
self.gguf_writer.add_audio_attention_layernorm_eps(1e-5)
def tensor_force_quant(self, name, new_name, bid, n_dims):
# the VQ picks the nearest codebook entry, small errors in the encoder change the codes
if new_name.startswith("a.codec."):
return gguf.GGMLQuantizationType.F32
# codec transformers: F16 gives codes closer to the reference than BF16
if new_name.startswith("a.blk.") and n_dims == 2:
return gguf.GGMLQuantizationType.F16
return super().tensor_force_quant(name, new_name, bid, n_dims)
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name = item[0]
if name.startswith("audio_decoder.") or name == "text_model.model.embeddings.weight":
return item
return None
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
T = gguf.MODEL_TENSOR
cfg = self.hparams
# already named by generate_extra_tensors()
if name.startswith("a."):
yield (name, data_torch)
return
if name == "text_model.model.embeddings.weight":
# semantic band of the backbone embeddings
begin = self.global_config["semantic_start_token_id"]
yield (self.format_tensor_name(T.A_GEN_CODE_OUT_EMBD), data_torch[begin:begin + cfg["vocab_size"]])
return
if name == "audio_decoder.codebook_embeddings.weight":
# backbone input of a speech row, one table per codebook
yield (self.format_tensor_name(T.A_GEN_CODE_EMBD), data_torch.reshape(cfg["num_codebooks"], -1, data_torch.shape[-1]))
return
if name == "audio_decoder.embeddings.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_TOK_EMBD), data_torch)
return
if name == "audio_decoder.output.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_HEAD), data_torch[:_N_RESIDUAL_CODES])
return
if name == "audio_decoder.norm.weight":
yield (self.format_tensor_name(T.A_GEN_CODE_OUTPUT_NORM), data_torch)
return
assert bid is not None and name.startswith(f"audio_decoder.layers.{bid}.")
key = name.removeprefix(f"audio_decoder.layers.{bid}.")
if key == "attention.wqkv.weight":
head_dim = cfg["head_dim"]
n_q = cfg["n_head"] * head_dim
n_kv = cfg["n_local_heads"] * head_dim
q, k, v = data_torch.split([n_q, n_kv, n_kv], dim=0)
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_Q, bid), _neox_rows(q, head_dim))
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_K, bid), _neox_rows(k, head_dim))
yield (self.format_tensor_name(T.A_GEN_CODE_ATTN_V, bid), v)
return
yield (self.format_tensor_name(self._FAST_AR_MAP[key], bid), data_torch)
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
yield from super().generate_extra_tensors()
sd = load_codec(self.dir_model)
T = gguf.MODEL_TENSOR
def get(name: str) -> Tensor:
return sd[name].float()
def tn(tensor: gguf.MODEL_TENSOR, bid: int | None = None, xid: int | None = None) -> str:
return gguf.TENSOR_NAMES[tensor].format(bid=bid, xid=xid)
def conv_wb(tensor: gguf.MODEL_TENSOR, prefix: str, bid: int | None = None, xid: int | None = None, stride: int = 1):
# taps-major [K / stride, OC, stride * IC], see conv() in tools/mtmd/models/fish-s2-spkenc.cpp
w = fold_conv(sd, prefix)
n_out, n_in, k = w.shape
w = w.reshape(n_out, n_in, k // stride, stride).permute(2, 0, 3, 1).reshape(k // stride, n_out, stride * n_in)
yield (tn(tensor, bid, xid) + ".weight", w)
yield (tn(tensor, bid, xid) + ".bias", get(prefix + ".conv.bias"))
def snake_ab(tensor: gguf.MODEL_TENSOR, name: str, bid: int | None = None, xid: int | None = None):
a, b = fold_snake(get(name))
yield (tn(tensor, bid, xid) + ".alpha", a)
yield (tn(tensor, bid, xid) + ".beta", b)
def tfm_layers(prefix: str, n_layer: int, offset: int):
for il in range(n_layer):
p = f"{prefix}layers.{il}."
bid = offset + il
wq, wk, wv = get(p + "attention.wqkv.weight").chunk(3, dim=0)
yield (self.format_tensor_name(T.A_ENC_ATTN_Q, bid), _neox_rows(wq, _CODEC_HEAD_DIM))
yield (self.format_tensor_name(T.A_ENC_ATTN_K, bid), _neox_rows(wk, _CODEC_HEAD_DIM))
yield (self.format_tensor_name(T.A_ENC_ATTN_V, bid), wv)
yield (self.format_tensor_name(T.A_ENC_OUTPUT, bid), get(p + "attention.wo.weight"))
yield (self.format_tensor_name(T.A_ENC_INPUT_NORM, bid), get(p + "attention_norm.weight"))
yield (self.format_tensor_name(T.A_ENC_ATTN_SCALE, bid), get(p + "attention_layer_scale.gamma"))
yield (self.format_tensor_name(T.A_ENC_OUTPUT_NORM, bid), get(p + "ffn_norm.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_GATE, bid), get(p + "feed_forward.w1.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_UP, bid), get(p + "feed_forward.w3.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_DOWN, bid), get(p + "feed_forward.w2.weight"))
yield (self.format_tensor_name(T.A_ENC_FFN_SCALE_LS, bid), get(p + "ffn_layer_scale.gamma"))
# encoder: entry conv, 4x (3 residual units, Snake, strided conv), Snake, post conv
e = "encoder.block."
yield from conv_wb(T.A_CODEC_ENC_ENTRY, e + "0")
for bid, stride in enumerate(_CODEC_ENC_STRIDES):
p = f"{e}{bid + 1}.block."
for xid in range(3):
r = f"{p}{xid}.block."
yield from snake_ab(T.A_CODEC_ENC_RES_ACT1, r + "0.alpha", bid, xid)
yield from conv_wb(T.A_CODEC_ENC_RES_CONV1, r + "1", bid, xid)
yield from snake_ab(T.A_CODEC_ENC_RES_ACT2, r + "2.alpha", bid, xid)
yield from conv_wb(T.A_CODEC_ENC_RES_CONV2, r + "3", bid, xid)
yield from snake_ab(T.A_CODEC_ENC_SNAKE, p + "3.alpha", bid)
yield from conv_wb(T.A_CODEC_ENC_CONV, p + "4", bid, stride=stride)
# the last block ends with a windowed transformer
yield from tfm_layers(f"{e}4.block.5.", _N_ENC_TFM_LAYERS, 0)
yield (self.format_tensor_name(T.A_POST_NORM), get(f"{e}4.block.5.norm.weight"))
yield from snake_ab(T.A_CODEC_ENC_POST_SNAKE, e + "5.alpha")
yield from conv_wb(T.A_CODEC_ENC_POST_CONV, e + "6")
# quantizer: 2x (strided conv + ConvNeXt), windowed transformer, then semantic + residual VQ
q = "quantizer."
for bid in range(2):
p = f"{q}downsample.{bid}."
yield from conv_wb(T.A_CODEC_DOWN_CONV, p + "0", bid, stride=2)
# depthwise: one [C] row per tap
yield (tn(T.A_CODEC_DOWN_DWCONV, bid) + ".weight", fold_conv(sd, p + "1.dwconv").squeeze(1).T.contiguous())
yield (tn(T.A_CODEC_DOWN_DWCONV, bid) + ".bias", get(p + "1.dwconv.conv.bias"))
for key, tensor in (("norm", T.A_CODEC_DOWN_NORM), ("pwconv1", T.A_CODEC_DOWN_PW1), ("pwconv2", T.A_CODEC_DOWN_PW2)):
yield (tn(tensor, bid) + ".weight", get(f"{p}1.{key}.weight"))
yield (tn(tensor, bid) + ".bias", get(f"{p}1.{key}.bias"))
yield (tn(T.A_CODEC_DOWN_GAMMA, bid), get(p + "1.gamma"))
yield from tfm_layers(q + "pre_module.", _N_PRE_TFM_LAYERS, _N_ENC_TFM_LAYERS)
yield (self.format_tensor_name(T.A_CODEC_PRE_NORM), get(q + "pre_module.norm.weight"))
# nearest codebook entry by cosine similarity, see VectorQuantize.decode_latents()
def vq(prefix: str) -> tuple[Tensor, Tensor, Tensor]:
w = fold_wn(sd[prefix + ".in_proj.weight_g"], sd[prefix + ".in_proj.weight_v"])
cb = torch.nn.functional.normalize(get(prefix + ".codebook.weight"), dim=-1)
return w, get(prefix + ".in_proj.bias"), cb
w, b, cb = vq(q + "semantic_quantizer.quantizers.0")
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_IN, suffix=".weight"), w)
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_IN, suffix=".bias"), b)
yield (self.format_tensor_name(T.A_CODEC_VQ_FIRST_CB), cb)
n_rest = self.hparams["num_codebooks"] - 1
rest = [vq(f"{q}quantizer.quantizers.{i}") for i in range(n_rest)]
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_IN, suffix=".weight"), torch.stack([r[0] for r in rest]))
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_IN, suffix=".bias"), torch.stack([r[1] for r in rest]))
yield (self.format_tensor_name(T.A_CODEC_VQ_REST_CB), torch.stack([r[2] for r in rest]))
+4
View File
@@ -302,6 +302,10 @@ class Inkling2VisionModel(MmprojModel):
w.add_vision_attention_layernorm_eps(config["layer_norm_eps"])
w.add_vision_spatial_merge_size(config["spatial_merge_size"])
def dequant_model(self):
# vision tensors are BF16, only the text experts are quantized
return
@classmethod
def filter_tensors(cls, item):
name, gen = item
+234
View File
@@ -0,0 +1,234 @@
from __future__ import annotations
import base64
from typing import Any, Callable, Iterable, TYPE_CHECKING
import torch
if TYPE_CHECKING:
from torch import Tensor
from .base import ModelBase, MmprojModel, gguf, logger
from .qwen import QwenModel, Qwen3Model
# token ids of the yue2_infer protocol (protocol.py)
EOD = 151643
ABC_START, ABC_END = 151847, 151848
MUSIC_START, MUSIC_END = 151851, 151852
CODEC_OFFSET, CODEC_SIZE = 151853, 32768
LATENT_START = 184621
# the VAE is a separate repo (m-a-p/YuE2-Vae), downloaded into this subdir of the model dir
VAE_DIR = "vae"
INSTRUCTIONS = {
"off": "Generate music with codec tokens from the given conditions.",
"melody": "Generate a melody-only ABC transcription without chord symbols, then generate music with codec tokens from the given conditions.",
"full": "Generate a chord-annotated ABC transcription, then generate music with codec tokens from the given conditions.",
}
# the text is "[Tags]\n<style>\n[Lyrics]\n<lyrics>", optionally followed by "\n[ABC]\n<score>" (cover mode)
# voice picks the planning mode: full (default), melody or off
TTS_TEMPLATE = (
"{%- set text = messages[0]['content'] -%}"
"{%- set mode = (voice | lower) if voice is defined and voice and (voice | lower) in ['full', 'melody', 'off'] else 'full' -%}"
"{%- set parts = text.split('\\n[ABC]\\n') -%}"
"{%- set instructions = {"
+ ", ".join(f"'{k}': '{v}'" for k, v in INSTRUCTIONS.items())
+ "} -%}"
"{{- '<|endoftext|>' + instructions[mode] + '\\n' + parts[0].rstrip() + '\\n' -}}"
"{%- if mode == 'off' -%}"
"{{- '<abc></abc><|music_start|>' -}}"
"{%- elif parts | length > 1 -%}"
"{{- '<abc>' + parts[1] + '</abc><|music_start|>' -}}"
"{%- else -%}"
"{{- '<abc>' -}}"
"{%- endif -%}"
)
def _is_nar_tensor(name: str) -> bool:
return ".nar_" in name or name.startswith(("llm2vae.", "vae2llm.", "time_embedder.", "latent_pos_embed."))
@ModelBase.register("YuE2ForCausalLM")
class YuE2TextModel(Qwen3Model):
model_arch = gguf.MODEL_ARCH.QWEN3
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
# the NAR path goes to the mmproj
if _is_nar_tensor(item[0]):
return None
return super().filter_tensors(item)
def set_gguf_parameters(self):
super().set_gguf_parameters()
# the codes are sampled with classifier-free guidance, the uncond prompt runs as a second sequence
self.gguf_writer.add_classifier_free_guidance(True)
def set_vocab(self):
# checkpoint-native tiktoken file, see tokenization_yue2.py
ranks: dict[bytes, int] = {}
with open(self.dir_model / "qwen.tiktoken", "rb") as f:
for line in f.read().splitlines():
if line:
tok, rank = line.split()
ranks[base64.b64decode(tok)] = int(rank)
if len(ranks) != EOD:
raise ValueError(f"YuE2: expected {EOD} BPE tokens, got {len(ranks)}")
vocab_size = self.hparams["vocab_size"]
tokens: list[str] = [""] * vocab_size
toktypes: list[int] = [gguf.TokenType.UNUSED] * vocab_size
merges: list[str] = []
for tok, rank in ranks.items():
tokens[rank] = QwenModel.token_bytes_to_string(tok)
toktypes[rank] = gguf.TokenType.NORMAL
if len(tok) > 1:
merged = QwenModel.bpe(ranks, tok, max_rank=rank)
assert len(merged) == 2
merges.append(" ".join(map(QwenModel.token_bytes_to_string, merged)))
specials = ["<|endoftext|>", "<|im_start|>", "<|im_end|>", "<R>", "<S>", "<X>", "<mask>", "<sep>"]
specials += [f"<extra_{i}>" for i in range(200)]
specials[204:206] = ["<abc>", "</abc>"]
named = {EOD + i: s for i, s in enumerate(specials)}
named[MUSIC_START] = "<|music_start|>"
named[MUSIC_END] = "<|music_end|>"
for i in range(CODEC_SIZE):
named[CODEC_OFFSET + i] = f"<|codec_{i}|>"
for i, s in enumerate(["<|latent_start|>", "<|latent_end|>", "<|latent_pad|>"]):
named[LATENT_START + i] = s
for i in range(EOD, vocab_size):
tokens[i] = named.get(i, f"[PAD{i}]")
toktypes[i] = gguf.TokenType.CONTROL if i in named else gguf.TokenType.UNUSED
self.gguf_writer.add_tokenizer_model("gpt2")
self.gguf_writer.add_tokenizer_pre("qwen2")
self.gguf_writer.add_token_list(tokens)
self.gguf_writer.add_token_types(toktypes)
self.gguf_writer.add_token_merges(merges)
self.gguf_writer.add_bos_token_id(EOD)
self.gguf_writer.add_eos_token_id(MUSIC_END)
self.gguf_writer.add_pad_token_id(EOD)
self.gguf_writer.add_add_bos_token(False)
self.gguf_writer.add_chat_template([{"name": "tts", "template": TTS_TEMPLATE}])
@ModelBase.register("YuE2ForCausalLM")
class YuE2MmprojModel(MmprojModel):
has_vision_encoder = False
has_audio_encoder = False
def get_audio_config(self) -> dict[str, Any] | None:
# the NAR path: same shapes as the backbone layers
return {k: self.global_config[k] for k in ("num_hidden_layers", "hidden_size", "intermediate_size",
"num_attention_heads", "num_key_value_heads", "rms_norm_eps")}
@classmethod
def filter_tensors(cls, item: tuple[str, Callable[[], Tensor]]) -> tuple[str, Callable[[], Tensor]] | None:
name = item[0]
if name == "latent_pos_embed.pe":
return None # sinusoidal, computed at runtime
if _is_nar_tensor(name) or name == "model.norm.weight":
return item
return None
def set_gguf_parameters(self):
hp = self.hparams
self.gguf_writer.add_file_type(self.ftype)
self.gguf_writer.add_clip_has_gen_audio_encoder(True)
self.gguf_writer.add_clip_gen_audio_projector_type(gguf.VisionProjectorType.YUE2_GEN)
self.gguf_writer.add_gen_audio_projection_dim(self.n_embd_text)
self.gguf_writer.add_gen_audio_embedding_length(hp["hidden_size"])
self.gguf_writer.add_gen_audio_feed_forward_length(hp["intermediate_size"])
self.gguf_writer.add_gen_audio_block_count(hp["num_hidden_layers"])
self.gguf_writer.add_gen_audio_head_count(hp["num_attention_heads"])
self.gguf_writer.add_gen_audio_head_count_kv(hp["num_key_value_heads"])
self.gguf_writer.add_gen_audio_attention_layernorm_eps(hp["rms_norm_eps"])
# rope theta, the ODE solver and the VAE hparams are hard-coded in tools/mtmd
def tensor_force_quant(self, name, new_name, bid, n_dims):
# read on the host, or small convs and projections of the flow / VAE graphs
if not new_name.startswith("a.gen.nar.blk.") or n_dims < 2:
return gguf.GGMLQuantizationType.F32
return super().tensor_force_quant(name, new_name, bid, n_dims)
@staticmethod
def _wn(sd: dict[str, Tensor], prefix: str) -> Tensor:
# torch weight norm, over every dim but 0
g, v = sd[prefix + ".weight_g"].float(), sd[prefix + ".weight_v"].float()
return g * v / v.flatten(1).norm(dim=1).view(-1, *[1] * (v.dim() - 1))
def generate_extra_tensors(self) -> Iterable[tuple[str, Tensor]]:
from safetensors.torch import load_file
path = self.dir_model / VAE_DIR / "model.safetensors"
if not path.is_file():
raise FileNotFoundError(f"YuE2: VAE not found at {path}, download m-a-p/YuE2-Vae into it")
logger.info(f"gguf: loading VAE decoder from '{path}'")
sd = {k[len("decoder.layers."):]: v for k, v in load_file(str(path)).items() if k.startswith("decoder.layers.")}
p = "a.gen.vae"
# SnakeBeta with log-scale params: x + 1/(exp(beta) + 1e-9) * sin(exp(alpha) * x)^2
def snake(dst: str, src: str) -> Iterable[tuple[str, Tensor]]:
yield (f"{dst}.alpha", torch.exp(sd[src + ".alpha"].float()))
yield (f"{dst}.beta", 1.0 / (torch.exp(sd[src + ".beta"].float()) + 1e-9))
yield (f"{p}.conv_in.weight", self._wn(sd, "0"))
yield (f"{p}.conv_in.bias", sd["0.bias"])
n_blk = sum(1 for k in sd if k.endswith(".layers.1.weight_g") and k.count(".") == 3)
for b in range(n_blk):
s = f"{b + 1}.layers"
yield from snake(f"{p}.blk.{b}.snake", f"{s}.0")
yield (f"{p}.blk.{b}.up.weight", self._wn(sd, f"{s}.1"))
yield (f"{p}.blk.{b}.up.bias", sd[f"{s}.1.bias"])
for r in range(3):
rs = f"{s}.{r + 2}.layers"
yield from snake(f"{p}.blk.{b}.res.{r}.snake1", f"{rs}.0")
yield (f"{p}.blk.{b}.res.{r}.conv1.weight", self._wn(sd, f"{rs}.1"))
yield (f"{p}.blk.{b}.res.{r}.conv1.bias", sd[f"{rs}.1.bias"])
yield from snake(f"{p}.blk.{b}.res.{r}.snake2", f"{rs}.2")
yield (f"{p}.blk.{b}.res.{r}.conv2.weight", self._wn(sd, f"{rs}.3"))
yield (f"{p}.blk.{b}.res.{r}.conv2.bias", sd[f"{rs}.3.bias"])
yield from snake(f"{p}.post_snake", f"{n_blk + 1}")
yield (f"{p}.conv_out.weight", self._wn(sd, f"{n_blk + 2}"))
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
if name.startswith("a.gen.vae."):
yield (name, data_torch)
return
p = "a.gen.nar"
aux = {
"model.norm.weight": f"{p}.output_norm.weight",
"vae2llm.weight": f"{p}.inp.weight",
"vae2llm.bias": f"{p}.inp.bias",
"llm2vae.weight": f"{p}.out.weight",
"llm2vae.bias": f"{p}.out.bias",
"time_embedder.mlp.0.weight": f"{p}.time.0.weight",
"time_embedder.mlp.0.bias": f"{p}.time.0.bias",
"time_embedder.mlp.2.weight": f"{p}.time.2.weight",
"time_embedder.mlp.2.bias": f"{p}.time.2.bias",
}
if name in aux:
yield (aux[name], data_torch)
return
# the NAR layers keep the libllama names, mtmd loads them as a text model
assert bid is not None, name
layer = {
"nar_input_layernorm.weight": "attn_norm.weight",
"nar_self_attn.q_proj.weight": "attn_q.weight",
"nar_self_attn.k_proj.weight": "attn_k.weight",
"nar_self_attn.v_proj.weight": "attn_v.weight",
"nar_self_attn.o_proj.weight": "attn_output.weight",
"nar_self_attn.q_norm.weight": "attn_q_norm.weight",
"nar_self_attn.k_norm.weight": "attn_k_norm.weight",
"nar_pre_mlp_layernorm.weight": "ffn_norm.weight",
"nar_mlp.gate_proj.weight": "ffn_gate.weight",
"nar_mlp.up_proj.weight": "ffn_up.weight",
"nar_mlp.down_proj.weight": "ffn_down.weight",
}
key = name.split(".", 3)[3] # strip "model.layers.{bid}."
yield (f"{p}.blk.{bid}.{layer[key]}", data_torch)
+1 -1
View File
@@ -692,7 +692,7 @@ llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_
# Or Using llama-server executable
./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024
# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.
# Option 1: Open your browser to http://localhost:8080 to open Llama Chat, the built-in chat app of the llama.cpp server.
# Option 2: In a NEW terminal, test the server with curl
# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
+91
View File
@@ -12,6 +12,8 @@ Supported models:
- [Chatterbox](#chatterbox)
- [Drama 3](#drama-3)
- [ZONOS3](#zonos3)
- [Fish Audio S2 Pro](#fish-audio-s2-pro)
- [YuE2](#yue2) (songs)
The sections below list the params supported by each model, with `llama-tts` examples. In `llama-server`, the same options are set with `--tts-lang` and `--tts-voice`, or per request (see the [`/tts` endpoint](../../tools/server/README.md#post-tts-generate-speech-audio-from-text)).
@@ -215,3 +217,92 @@ Convert the root of the release repository, not the `lm/` subdirectory. The mmpr
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf
```
## Fish Audio S2 Pro
[Fish Audio S2 Pro](https://huggingface.co/fishaudio/s2-pro): a Qwen3 backbone (4B) samples each frame's semantic token, a 4-layer fast AR predicts the 9 other codes, and the Fish DAC (same as Drama 3) decodes them to 44.1 kHz audio.
Available params:
- Speaker tags `<|speaker:0|>`, `<|speaker:1|>`, ... in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>`
- Inline instructions in brackets control the delivery, for example `[laughing]`, `[whisper]` or `[pause]`
- The language is the one of the text, `--tts-lang` is not used
- Voice cloning: `--tts-speaker-file` with the reference clip, optionally `--tts-speaker-text` with its transcript (it is put in the prompt as the reference does). The clip is encoded by the codec encoder half of `codec.pth`, it is truncated to the first 30 s
- Note: `--temp`, `--top-k` and `--top-p` are not used (only `--seed`), frames are sampled with the settings of the reference (semantic token: repetition penalty 1.1 over the last 16 frames, top-k 30, top-p 0.8, temperature 0.8, or top-p 0.9 and temperature 1.0 when the last 4 frames repeat a token; other codes: greedy)
- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
Example usage:
```sh
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
-p "<|speaker:0|>Did you remember to buy the bread? <|speaker:1|>Of course I did [laughing] it is on the kitchen table." \
-n 1000 \
--output out.wav
```
Voice cloning (`--tts-speaker-text` can be left out):
```sh
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
--tts-speaker-file reference.wav \
--tts-speaker-text "The words spoken in reference.wav." \
-p "Hello, this is a cloned voice." \
--output out.wav
```
**Note for GGUF conversion:**
The mmproj holds the fast AR and `codec.pth` (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16. The codec encoder is always stored in F16/F32, small errors in it change the codes picked by its VQ:
```sh
python convert_hf_to_gguf.py path/to/s2-pro --outtype q8_0 --outfile s2-pro.gguf
python convert_hf_to_gguf.py path/to/s2-pro --mmproj --outtype bf16 --outfile mmproj-s2-pro.gguf
```
## YuE2
[m-a-p/YuE2-3B](https://huggingface.co/m-a-p/YuE2-3B) generates songs (vocals and accompaniment) from a style prompt and lyrics, as 48 kHz stereo. The backbone (a Qwen3 text model) writes an ABC score, then one semantic code per 40 ms frame; the NAR half of the checkpoint turns the codes into VAE latents by flow matching, attending to the keys and values of the backbone, and the [YuE2-Vae](https://huggingface.co/m-a-p/YuE2-Vae) decoder makes the audio.
The prompt is the text of the reference request:
```
[Tags]
City Pop, upbeat, danceable, groovy bass, electric guitar, synth
[Lyrics]
[Verse]
...
```
Available params:
- `--tts-voice` picks the planning mode of the reference (`cot`): `full` (default, a chord-annotated score), `melody` (a melody-only score) or `off` (no score)
- Cover mode: append the score to the prompt as `\n[ABC]\n<score>`, with `--tts-voice melody` or `full`; the model then goes straight to the codes
- The generation stops at the limits of the reference (4096 score tokens, then 9000 codes, 6 minutes); `-n` sets a lower limit on the score tokens and the codes together
- The context must keep the default size (24576 positions, the one of the reference): a smaller `-c` is rejected when the model is loaded
- Note: `--top-k` and `--top-p` are not used, both phases are sampled with the settings of the reference (`yue2_generation_config.json`); `--temp` changes the temperature of the codes
- Note: the backbone context must use an F16 KV cache (default): the NAR pass loads its keys and values
- The codes are sampled with classifier-free guidance like the reference: scale 1.01 with `--tts-voice off`, off (1.0) otherwise. `llama-server` takes `cfg_scale` to change it, the reference suggests 1.2 for a stronger text guidance. The uncond prompt runs as a second sequence, so the context must also hold it (the instruction, the score and the codes once more)
- The NAR pass solves the flow-matching ODE in 32 midpoint steps like the reference; `llama-server` takes `flow_steps` to change it. The NAR time is about proportional to it: 16 steps are about 35% faster for the NAR and the VAE together, with a small deviation from the reference (about 0.2 dB of spectral difference); it grows with fewer steps (about 0.4 dB at 8, 0.8 dB at 4)
- The NAR pass runs on chunks of codes like the reference: one chunk for a song that fits in 24576 positions, so the audio arrives after the last code. When streaming, each 30 s of codes is a chunk, processed during the generation, so the audio arrives in 30 s steps; the chunks are independent (each sees the prompt, the score and its own codes), which may be audible at the boundaries
Example usage:
```sh
llama-tts -m yue2-3b.gguf \
-mm mmproj-yue2-3b.gguf \
-f song.txt \
--output song.wav
```
**Note for GGUF conversion:**
The VAE is a separate repository, download it into the `vae` subdirectory of the model:
```sh
hf download m-a-p/YuE2-3B --local-dir YuE2-3B
hf download m-a-p/YuE2-Vae --local-dir YuE2-3B/vae
python convert_hf_to_gguf.py YuE2-3B --outfile yue2-3b.gguf
python convert_hf_to_gguf.py YuE2-3B --mmproj --outfile mmproj-yue2-3b.gguf
```
The mmproj holds the NAR transformer layers (loaded with libllama as a Qwen3 text model, in the `--outtype` type), and the latent projections and the VAE decoder, always in F32.
+29 -31
View File
@@ -87,36 +87,32 @@ for id: llama_token in tokens {
print("\n")
var batch = llama_batch_init(max(Int32(tokens.count), Int32(n_parallel)), 0, 1)
let batch = llama_batch_ext_init(context)
defer {
llama_batch_free(batch)
llama_batch_ext_free(batch)
}
// evaluate the initial prompt
batch.n_tokens = Int32(tokens.count)
for (i, token) in tokens.enumerated() {
batch.token[i] = token
batch.pos[i] = Int32(i)
batch.n_seq_id[i] = 1
// batch.seq_id[i][0] = 0
// TODO: is this the proper way to do this?
if let seq_id = batch.seq_id[i] {
seq_id[0] = 0
let idx = llama_batch_ext_add_token(batch, 0, token)
if idx < 0 {
print("llama_batch_ext_add_token() failed: \(idx)")
exit(1)
}
batch.logits[i] = 0
var pos = llama_pos(i)
llama_batch_ext_set_pos(batch, idx, &pos)
}
// llama_decode will output logits only for the last token of the prompt
batch.logits[Int(batch.n_tokens) - 1] = 1
// llama_process will output logits only for the last token of the prompt
llama_batch_ext_set_output_logits(batch, Int32(tokens.count) - 1, true)
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
exit(1)
}
for i in 1 ..< n_parallel {
llama_memory_seq_cp(llama_get_memory(context), 0, Int32(i), 0, batch.n_tokens)
llama_memory_seq_cp(llama_get_memory(context), 0, Int32(i), 0, Int32(tokens.count))
}
if n_parallel > 1 {
@@ -125,16 +121,17 @@ if n_parallel > 1 {
var streams: [String] = .init(repeating: "", count: n_parallel)
var streamBuffers: [[CChar]] = .init(repeating: [], count: n_parallel)
var i_batch = [Int32](repeating: batch.n_tokens - 1, count: n_parallel)
var i_batch = [Int32](repeating: Int32(tokens.count) - 1, count: n_parallel)
var n_cur = batch.n_tokens
var n_cur = Int32(tokens.count)
var n_decode = 0
let t_main_start = ggml_time_us()
while n_cur <= n_len {
// prepare the next batch
batch.n_tokens = 0
llama_batch_ext_clear(batch)
var n_tokens: Int32 = 0
// sample the next token for each parallel sequence / stream
for i in 0 ..< n_parallel {
@@ -165,31 +162,32 @@ while n_cur <= n_len {
streams[i] += nextStringPiece
// push this new token for next evaluation
batch.token[Int(batch.n_tokens)] = new_token_id
batch.pos[Int(batch.n_tokens)] = n_cur
batch.n_seq_id[Int(batch.n_tokens)] = 1
if let seq_id = batch.seq_id[Int(batch.n_tokens)] {
seq_id[0] = Int32(i)
let idx = llama_batch_ext_add_token(batch, Int32(i), new_token_id)
if idx < 0 {
print("llama_batch_ext_add_token() failed: \(idx)")
exit(1)
}
batch.logits[Int(batch.n_tokens)] = 1
var pos = n_cur
llama_batch_ext_set_pos(batch, idx, &pos)
llama_batch_ext_set_output_logits(batch, idx, true)
i_batch[i] = batch.n_tokens
i_batch[i] = idx
batch.n_tokens += 1
n_tokens += 1
n_decode += 1
}
// all streams are finished
if batch.n_tokens == 0 {
if n_tokens == 0 {
break
}
n_cur += 1
// evaluate the current batch with the transformer model
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
exit(1)
}
}
@@ -5,20 +5,16 @@ enum LlamaError: Error {
case couldNotInitializeContext
}
func llama_batch_clear(_ batch: inout llama_batch) {
batch.n_tokens = 0
}
func llama_batch_add(_ batch: inout llama_batch, _ id: llama_token, _ pos: llama_pos, _ seq_ids: [llama_seq_id], _ logits: Bool) {
batch.token [Int(batch.n_tokens)] = id
batch.pos [Int(batch.n_tokens)] = pos
batch.n_seq_id[Int(batch.n_tokens)] = Int32(seq_ids.count)
for i in 0..<seq_ids.count {
batch.seq_id[Int(batch.n_tokens)]![Int(i)] = seq_ids[i]
@discardableResult
func llama_batch_add(_ batch: OpaquePointer, _ id: llama_token, _ pos: llama_pos, _ seq_id: llama_seq_id, _ logits: Bool) -> Int32 {
let idx = llama_batch_ext_add_token(batch, seq_id, id)
if idx < 0 {
return idx
}
batch.logits [Int(batch.n_tokens)] = logits ? 1 : 0
batch.n_tokens += 1
var pos = pos
llama_batch_ext_set_pos(batch, idx, &pos)
llama_batch_ext_set_output_logits(batch, idx, logits)
return idx
}
actor LlamaContext {
@@ -26,7 +22,7 @@ actor LlamaContext {
private var context: OpaquePointer
private var vocab: OpaquePointer
private var sampling: UnsafeMutablePointer<llama_sampler>
private var batch: llama_batch
private var batch: OpaquePointer
private var tokens_list: [llama_token]
var is_done: Bool = false
@@ -42,7 +38,7 @@ actor LlamaContext {
self.model = model
self.context = context
self.tokens_list = []
self.batch = llama_batch_init(512, 0, 1)
self.batch = llama_batch_ext_init(context)
self.temporary_invalid_cchars = []
let sparams = llama_sampler_chain_default_params()
self.sampling = llama_sampler_chain_init(sparams)
@@ -53,7 +49,7 @@ actor LlamaContext {
deinit {
llama_sampler_free(sampling)
llama_batch_free(batch)
llama_batch_ext_free(batch)
llama_model_free(model)
llama_free(context)
llama_backend_free()
@@ -110,10 +106,6 @@ actor LlamaContext {
return SwiftString
}
func get_n_tokens() -> Int32 {
return batch.n_tokens;
}
func completion_init(text: String) {
print("attempting to complete \"\(text)\"")
@@ -133,25 +125,27 @@ actor LlamaContext {
print(String(cString: token_to_piece(token: id) + [0]))
}
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
for i1 in 0..<tokens_list.count {
let i = Int(i1)
llama_batch_add(&batch, tokens_list[i], Int32(i), [0], false)
}
batch.logits[Int(batch.n_tokens) - 1] = 1 // true
if llama_decode(context, batch) != 0 {
print("llama_decode() failed")
let idx = llama_batch_add(batch, tokens_list[i], Int32(i), 0, i == tokens_list.count - 1)
if idx < 0 {
print("llama_batch_add() failed: \(idx)")
}
}
n_cur = batch.n_tokens
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed")
}
n_cur = Int32(tokens_list.count)
}
func completion_loop() -> String {
var new_token_id: llama_token = 0
new_token_id = llama_sampler_sample(sampling, context, batch.n_tokens - 1)
new_token_id = llama_sampler_sample(sampling, context, -1)
if llama_vocab_is_eog(vocab, new_token_id) || n_cur == n_len {
print("\n")
@@ -178,13 +172,13 @@ actor LlamaContext {
print(new_token_str)
// tokens_list.append(new_token_id)
llama_batch_clear(&batch)
llama_batch_add(&batch, new_token_id, n_cur, [0], true)
llama_batch_ext_clear(batch)
llama_batch_add(batch, new_token_id, n_cur, 0, true)
n_decode += 1
n_cur += 1
if llama_decode(context, batch) != 0 {
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("failed to evaluate llama!")
}
@@ -201,21 +195,20 @@ actor LlamaContext {
for _ in 0..<nr {
// bench prompt processing
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
let n_tokens = pp
for i in 0..<n_tokens {
llama_batch_add(&batch, 0, Int32(i), [0], false)
llama_batch_add(batch, 0, Int32(i), 0, i == n_tokens - 1)
}
batch.logits[Int(batch.n_tokens) - 1] = 1 // true
llama_memory_clear(llama_get_memory(context), false)
let t_pp_start = DispatchTime.now().uptimeNanoseconds / 1000;
if llama_decode(context, batch) != 0 {
print("llama_decode() failed during prompt")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed during prompt")
}
llama_synchronize(context)
@@ -228,14 +221,14 @@ actor LlamaContext {
let t_tg_start = DispatchTime.now().uptimeNanoseconds / 1000;
for i in 0..<tg {
llama_batch_clear(&batch)
llama_batch_ext_clear(batch)
for j in 0..<pl {
llama_batch_add(&batch, 0, Int32(i), [Int32(j)], true)
llama_batch_add(batch, 0, Int32(i), Int32(j), true)
}
if llama_decode(context, batch) != 0 {
print("llama_decode() failed during text generation")
if llama_process(context, LLAMA_PROCESS_TYPE_DECODE, batch) != 0 {
print("llama_process() failed during text generation")
}
llama_synchronize(context)
}
+8
View File
@@ -378,6 +378,14 @@ bool ggml_cuda_should_use_mmvq(enum ggml_type type, int cc, int64_t ne11) {
return ne11 <= MMVQ_MAX_BATCH_SIZE;
}
}
if (GGML_CUDA_CC_IS_RDNA3_5(cc)) {
switch (type) { // tuned on Radeon 8060S (gfx1151)
case GGML_TYPE_Q8_0:
return ne11 <= 6;
default:
return ne11 <= MMVQ_MAX_BATCH_SIZE;
}
}
if (GGML_CUDA_CC_IS_CDNA(cc)) {
if (GGML_CUDA_CC_IS_CDNA1(cc)) {
switch (type) {
+112 -1
View File
@@ -712,6 +712,9 @@ struct ggml_backend_opencl_context {
size_t image_max_buffer_size;
size_t image2d_max_width;
size_t image2d_max_height;
size_t image3d_max_width;
size_t image3d_max_height;
size_t image3d_max_depth;
cl_device_svm_capabilities svm_caps;
@@ -1431,6 +1434,7 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_gemv_noshuffle_q4_k_f32_mc3; // multi-column (N=3) verify GEMV
cl_kernel kernel_gemm_noshuffle_q4_k_f32;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin;
cl_kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin;
cl_kernel kernel_gemv_noshuffle_q4_k_f32_32b_trans;
cl_kernel kernel_gemm_noshuffle_q4_k_q8_1_dp4a = nullptr; // dp4a (int8) dense prefill GEMM
@@ -6701,6 +6705,7 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin = nullptr;
backend_ctx->kernel_gemm_noshuffle_q4_k_q8_1_dp4a_ila_a8_bin = nullptr;
{
if (backend_ctx->has_vector_subgroup_broadcast) {
@@ -6745,6 +6750,17 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
CL_CHECK(clReleaseProgram(bin_prog));
GGML_LOG_CONT(".");
}
kernel_bin = (const char *)backend_ctx->get_adreno_bin_kernel("gemm_noshuffle_q4_k_f32_32b_trans_ila_a7", &bin_size);
if (kernel_bin && bin_size > 0) {
cl_program bin_prog =
build_program_from_binary(backend_ctx->context, backend_ctx->device, kernel_bin, "", bin_size);
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin =
clCreateKernel(bin_prog, "kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7", &err), err));
CL_CHECK(clReleaseProgram(bin_prog));
GGML_LOG_CONT(".");
}
}
}
@@ -9258,6 +9274,9 @@ static ggml_backend_opencl_context * ggml_cl_init(ggml_backend_dev_t dev) {
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE_MAX_BUFFER_SIZE, sizeof(size_t), &backend_ctx->image_max_buffer_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE2D_MAX_WIDTH, sizeof(size_t), &backend_ctx->image2d_max_width, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE2D_MAX_HEIGHT, sizeof(size_t), &backend_ctx->image2d_max_height, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_WIDTH, sizeof(size_t), &backend_ctx->image3d_max_width, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_HEIGHT, sizeof(size_t), &backend_ctx->image3d_max_height, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_IMAGE3D_MAX_DEPTH, sizeof(size_t), &backend_ctx->image3d_max_depth, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_MAX_WORK_GROUP_SIZE, sizeof(size_t), &backend_ctx->max_workgroup_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_LOCAL_MEM_SIZE, sizeof(cl_ulong), &backend_ctx->local_mem_size, NULL));
CL_CHECK(clGetDeviceInfo(device, CL_DEVICE_SVM_CAPABILITIES, sizeof(cl_device_svm_capabilities), &backend_ctx->svm_caps, 0));
@@ -11660,6 +11679,26 @@ static bool use_fa_bin_kernels_prefill(const ggml_backend_opencl_context * backe
constexpr bool prefill_only = true;
// The repacked Q/K/V are 3D images: Q {n_q, n_batch*n_head, dk/4}, K {dk, n_kv/4, n_batch*n_head_kv},
// V {n_kv, dv/4, n_batch*n_head_kv}. Past the device's 3D image limits clCreateImage fails and the
// call asserts (X2-90: V is n_kv wide, so any prefill past ~16k tokens of context), so decline those shapes.
const size_t n_kv = (size_t) k->ne[1];
const size_t n_bh = (size_t) q->ne[3] * (size_t) q->ne[2];
const size_t n_bhkv = (size_t) q->ne[3] * (size_t) k->ne[2];
if ((size_t) n_q > backend_ctx->image3d_max_width || n_bh > backend_ctx->image3d_max_height ||
(size_t) dk / 4 > backend_ctx->image3d_max_depth ||
(size_t) dk > backend_ctx->image3d_max_width || (n_kv + 3) / 4 > backend_ctx->image3d_max_height ||
n_kv > backend_ctx->image3d_max_width || (size_t) dv / 4 > backend_ctx->image3d_max_height ||
n_bhkv > backend_ctx->image3d_max_depth) {
return false;
}
// q8_0 K/V at DK=512 gives wrong results here when V differs from K (f16 K/V is correct).
// Decline it, as this backend does without the kernel library.
if (is_q8_0 && dk == 512) {
return false;
}
return (backend_ctx->gpu_family == GPU_FAMILY::ADRENO &&
(is_mixed || is_q8_0) && (dk == dv)
&& (dk == 64 || dk == 128 || dk == 256 || dk == 512)
@@ -11775,7 +11814,8 @@ inline bool use_q6_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
inline bool use_q4_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx, const ggml_tensor *tensor) {
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans ||
!backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin) {
!(backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -22794,6 +22834,7 @@ static void ggml_cl_flash_attn_prefill_bin(ggml_backend_t backend, const ggml_te
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
CL_CHECK(clReleaseMemObject(mem_tex_matrixO_1dbuf));
CL_CHECK(clReleaseMemObject(mem_tex_mask_fallback_1dbuf));
CL_CHECK(clReleaseMemObject(img_q_wmm));
CL_CHECK(clReleaseMemObject(img_k_wmm));
CL_CHECK(clReleaseMemObject(img_v_wmm));
@@ -27675,6 +27716,8 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno_ila(ggml_backend_t backend, const gg
const int gemm_tile_n = 64;
int N_pad = CEIL_DIV(N, gemm_tile_n) * gemm_tile_n;
const bool have_a8 = backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin != nullptr;
static const char * q4_k_bin_dp4a_env = getenv("GGML_OPENCL_Q4_K_BIN_DP4A");
bool q4_k_bin_dp4a_on = q4_k_bin_dp4a_env
? (atoi(q4_k_bin_dp4a_env) != 0)
@@ -27744,6 +27787,74 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno_ila(ggml_backend_t backend, const gg
return;
}
if (!have_a8) {
cl_mem b_src_sub = nullptr;
cl_mem b_pad_sub = nullptr;
cl_mem b_buf = nullptr;
region.origin = offset1;
region.size = (size_t)K * N * sizeof(float);
CL_CHECK((b_src_sub = clCreateSubBuffer(extra1->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
if (N_pad == N) {
b_buf = b_src_sub;
} else {
size_t padded_bytes = (size_t)K * N_pad * sizeof(float);
backend_ctx->prealloc_act_trans.allocate(context, padded_bytes);
region.origin = 0;
region.size = padded_bytes;
CL_CHECK((b_pad_sub = clCreateSubBuffer(backend_ctx->prealloc_act_trans.buffer, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err), err));
const float b_pad_zero = 0.0f;
CL_CHECK(clEnqueueFillBuffer(backend_ctx->queue, b_pad_sub, &b_pad_zero, sizeof(float),
0, padded_bytes, 0, NULL, NULL));
CL_CHECK(clEnqueueCopyBuffer(backend_ctx->queue, b_src_sub, b_pad_sub, 0, 0, (size_t)K * N * sizeof(float), 0, NULL, NULL));
b_buf = b_pad_sub;
}
img_fmt = { CL_R, CL_FLOAT };
memset(&img_desc, 0, sizeof(img_desc));
img_desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
img_desc.image_width = (size_t)K * N_pad;
img_desc.buffer = b_buf;
cl_mem b_img_a7;
CL_CHECK((b_img_a7 = clCreateImage(context, CL_MEM_READ_ONLY, &img_fmt, &img_desc, NULL, &err), err));
const int c_offset_for_kernel = (int)offsetd;
int line_stride_B = K * (int)sizeof(float);
int line_stride_C = M * (int)sizeof(float);
kernel = backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin;
cl_uint k_arg = 0;
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->q_img));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->d));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->dm));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extra0_q4_k->s));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &b_img_a7));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &offset1));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &c_offset_for_kernel));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne00));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(cl_uint), &ne01));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_B));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &line_stride_C));
CL_CHECK(clSetKernelArg(kernel, k_arg++, sizeof(int), &N));
size_t local_work_size[3] = { 64, 2, 4 };
size_t m_tiles = (size_t)CEIL_DIV(M, 64);
size_t global_y = ((m_tiles + 1) / 2) * 2;
size_t global_work_size[3] = { 64, global_y, (size_t)CEIL_DIV(N_pad, 32) };
backend_ctx->enqueue_ndrange_kernel(kernel, 3, global_work_size, local_work_size, dst);
CL_CHECK(clReleaseMemObject(b_img_a7));
if (b_src_sub) {
CL_CHECK(clReleaseMemObject(b_src_sub));
}
if (b_pad_sub) {
CL_CHECK(clReleaseMemObject(b_pad_sub));
}
return;
}
cl_mem b_sub_buf = nullptr;
cl_mem b_padded = nullptr;
cl_mem b_buf = nullptr;
+67
View File
@@ -1335,6 +1335,28 @@ class MODEL_TENSOR(IntEnum):
A_GEN_CODE_EH_PROJ = auto() # drama3: per-layer [code embd; hidden] -> hidden projection
A_GEN_CODE_BLK_EMBD = auto() # drama3: per-layer code embedding table
A_GEN_CODE_BLK_OUT_NORM = auto() # drama3: per-layer output norm
A_GEN_CODE_TOK_EMBD = auto() # fish_s2: code predictor input embedding, shared by all codebooks
# fish_s2 codec encoder (voice cloning): reference clip -> codes
A_CODEC_ENC_ENTRY = auto() # entry conv
A_CODEC_ENC_SNAKE = auto() # encoder block, Snake before the strided conv
A_CODEC_ENC_CONV = auto() # encoder block, strided conv
A_CODEC_ENC_RES_ACT1 = auto()
A_CODEC_ENC_RES_CONV1 = auto()
A_CODEC_ENC_RES_ACT2 = auto()
A_CODEC_ENC_RES_CONV2 = auto()
A_CODEC_ENC_POST_SNAKE = auto()
A_CODEC_ENC_POST_CONV = auto()
A_CODEC_DOWN_CONV = auto() # quantizer downsample, strided conv + ConvNeXt
A_CODEC_DOWN_DWCONV = auto()
A_CODEC_DOWN_NORM = auto()
A_CODEC_DOWN_PW1 = auto()
A_CODEC_DOWN_PW2 = auto()
A_CODEC_DOWN_GAMMA = auto()
A_CODEC_PRE_NORM = auto() # output norm of the quantizer pre_module
A_CODEC_VQ_FIRST_IN = auto() # semantic VQ input projection
A_CODEC_VQ_FIRST_CB = auto() # semantic VQ codebook, l2-normalized
A_CODEC_VQ_REST_IN = auto() # residual VQ input projections, merged 3D
A_CODEC_VQ_REST_CB = auto() # residual VQ codebooks, l2-normalized, merged 3D
# qwen3tts code2wav: RVQ codes -> raw PCM
A_GEN_WAV_QUANT_FIRST_IN = auto() # semantic RVQ, in_proj (1x1 conv, loaded as 2D)
A_GEN_WAV_QUANT_FIRST_OUT = auto() # semantic RVQ, out_proj
@@ -2277,6 +2299,27 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
MODEL_TENSOR.A_GEN_CODE_EH_PROJ: "a.gen.code.blk.{bid}.eh_proj",
MODEL_TENSOR.A_GEN_CODE_BLK_EMBD: "a.gen.code.blk.{bid}.embd",
MODEL_TENSOR.A_GEN_CODE_BLK_OUT_NORM: "a.gen.code.blk.{bid}.out_norm",
MODEL_TENSOR.A_GEN_CODE_TOK_EMBD: "a.gen.code.tok_embd",
MODEL_TENSOR.A_CODEC_ENC_ENTRY: "a.codec.enc.entry",
MODEL_TENSOR.A_CODEC_ENC_SNAKE: "a.codec.enc.blk.{bid}.snake",
MODEL_TENSOR.A_CODEC_ENC_CONV: "a.codec.enc.blk.{bid}.conv",
MODEL_TENSOR.A_CODEC_ENC_RES_ACT1: "a.codec.enc.blk.{bid}.res.{xid}.act1",
MODEL_TENSOR.A_CODEC_ENC_RES_CONV1: "a.codec.enc.blk.{bid}.res.{xid}.conv1",
MODEL_TENSOR.A_CODEC_ENC_RES_ACT2: "a.codec.enc.blk.{bid}.res.{xid}.act2",
MODEL_TENSOR.A_CODEC_ENC_RES_CONV2: "a.codec.enc.blk.{bid}.res.{xid}.conv2",
MODEL_TENSOR.A_CODEC_ENC_POST_SNAKE: "a.codec.enc.post_snake",
MODEL_TENSOR.A_CODEC_ENC_POST_CONV: "a.codec.enc.post_conv",
MODEL_TENSOR.A_CODEC_DOWN_CONV: "a.codec.down.blk.{bid}.conv",
MODEL_TENSOR.A_CODEC_DOWN_DWCONV: "a.codec.down.blk.{bid}.dwconv",
MODEL_TENSOR.A_CODEC_DOWN_NORM: "a.codec.down.blk.{bid}.norm",
MODEL_TENSOR.A_CODEC_DOWN_PW1: "a.codec.down.blk.{bid}.pw1",
MODEL_TENSOR.A_CODEC_DOWN_PW2: "a.codec.down.blk.{bid}.pw2",
MODEL_TENSOR.A_CODEC_DOWN_GAMMA: "a.codec.down.blk.{bid}.gamma",
MODEL_TENSOR.A_CODEC_PRE_NORM: "a.codec.pre_norm",
MODEL_TENSOR.A_CODEC_VQ_FIRST_IN: "a.codec.vq.first.in_proj",
MODEL_TENSOR.A_CODEC_VQ_FIRST_CB: "a.codec.vq.first.codebook",
MODEL_TENSOR.A_CODEC_VQ_REST_IN: "a.codec.vq.rest.in_proj",
MODEL_TENSOR.A_CODEC_VQ_REST_CB: "a.codec.vq.rest.codebook",
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_IN: "a.gen.wav.quant.first.in_proj",
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_OUT: "a.gen.wav.quant.first.out_proj",
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_CB: "a.gen.wav.quant.first.codebook",
@@ -2711,6 +2754,27 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
MODEL_TENSOR.A_GEN_CODE_EH_PROJ,
MODEL_TENSOR.A_GEN_CODE_BLK_EMBD,
MODEL_TENSOR.A_GEN_CODE_BLK_OUT_NORM,
MODEL_TENSOR.A_GEN_CODE_TOK_EMBD,
MODEL_TENSOR.A_CODEC_ENC_ENTRY,
MODEL_TENSOR.A_CODEC_ENC_SNAKE,
MODEL_TENSOR.A_CODEC_ENC_CONV,
MODEL_TENSOR.A_CODEC_ENC_RES_ACT1,
MODEL_TENSOR.A_CODEC_ENC_RES_CONV1,
MODEL_TENSOR.A_CODEC_ENC_RES_ACT2,
MODEL_TENSOR.A_CODEC_ENC_RES_CONV2,
MODEL_TENSOR.A_CODEC_ENC_POST_SNAKE,
MODEL_TENSOR.A_CODEC_ENC_POST_CONV,
MODEL_TENSOR.A_CODEC_DOWN_CONV,
MODEL_TENSOR.A_CODEC_DOWN_DWCONV,
MODEL_TENSOR.A_CODEC_DOWN_NORM,
MODEL_TENSOR.A_CODEC_DOWN_PW1,
MODEL_TENSOR.A_CODEC_DOWN_PW2,
MODEL_TENSOR.A_CODEC_DOWN_GAMMA,
MODEL_TENSOR.A_CODEC_PRE_NORM,
MODEL_TENSOR.A_CODEC_VQ_FIRST_IN,
MODEL_TENSOR.A_CODEC_VQ_FIRST_CB,
MODEL_TENSOR.A_CODEC_VQ_REST_IN,
MODEL_TENSOR.A_CODEC_VQ_REST_CB,
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_IN,
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_OUT,
MODEL_TENSOR.A_GEN_WAV_QUANT_FIRST_CB,
@@ -7223,6 +7287,9 @@ class VisionProjectorType:
DRAMA3_GEN = "drama3_gen" # audio generation: codebook tower + Fish DAC decoder
ZONOS3_GEN = "zonos3_gen" # audio generation: 10 codebook heads + ZAC decoder
ZONOS3_SPKENC = "zonos3_spkenc" # audio: MECT-B2 speaker encoder
YUE2_GEN = "yue2_gen" # audio generation: flow-matching NAR + Oobleck VAE decoder
FISH_S2_GEN = "fish_s2_gen" # audio generation: fast AR code predictor + Fish DAC decoder
FISH_S2_SPKENC = "fish_s2_spkenc" # audio: Fish DAC encoder + RVQ, reference clip -> prompt rows
HUNYUANVL = "hunyuanvl"
PARAKEET = "parakeet" # audio
MINIMAXM3 = "minimax_m3"
+15 -9
View File
@@ -271,6 +271,7 @@ extern "C" {
// - if not: only the last token is output
// )
//
// [DEPRECATED] use llama_batch_ext instead
typedef struct llama_batch {
int32_t n_tokens;
@@ -945,9 +946,10 @@ extern "C" {
//
// NOTE: this is a helper function to facilitate transition to the new batch API - avoid using it
//
LLAMA_API struct llama_batch llama_batch_get_one(
DEPRECATED(LLAMA_API struct llama_batch llama_batch_get_one(
llama_token * tokens,
int32_t n_tokens);
int32_t n_tokens),
"use llama_batch_ext_init and llama_batch_ext_add_token instead");
// Allocates a batch of tokens on the heap that can hold a maximum of n_tokens
// Each token can be assigned up to n_seq_max sequence ids
@@ -956,13 +958,15 @@ extern "C" {
// Otherwise, llama_batch.token will be allocated to store n_tokens llama_token
// The rest of the llama_batch members are allocated with size n_tokens
// All members are left uninitialized
LLAMA_API struct llama_batch llama_batch_init(
DEPRECATED(LLAMA_API struct llama_batch llama_batch_init(
int32_t n_tokens,
int32_t embd,
int32_t n_seq_max);
int32_t n_seq_max),
"use llama_batch_ext_init instead");
// Frees a batch of tokens allocated with llama_batch_init()
LLAMA_API void llama_batch_free(struct llama_batch batch);
DEPRECATED(LLAMA_API void llama_batch_free(struct llama_batch batch),
"use llama_batch_ext_free instead");
// Process a batch of tokens.
// In contrast to llama_decode() - this call does not use KV cache.
@@ -970,9 +974,10 @@ extern "C" {
// Can store the encoder output internally for later use by the decoder's cross-attention layers.
// 0 - success
// < 0 - error. the memory state is restored to the state before this call
LLAMA_API int32_t llama_encode(
DEPRECATED(LLAMA_API int32_t llama_encode(
struct llama_context * ctx,
struct llama_batch batch);
struct llama_batch batch),
"use llama_process with LLAMA_PROCESS_TYPE_ENCODE instead");
// Process a batch of tokens.
// Requires the context to have a memory.
@@ -986,9 +991,10 @@ extern "C" {
// 2 - aborted (processed ubatches will remain in the context's memory)
// -1 - invalid input batch
// < -1 - fatal error (processed ubatches will remain in the context's memory)
LLAMA_API int32_t llama_decode(
DEPRECATED(LLAMA_API int32_t llama_decode(
struct llama_context * ctx,
struct llama_batch batch);
struct llama_batch batch),
"use llama_process with LLAMA_PROCESS_TYPE_DECODE instead");
//
// Extended batch API
@@ -0,0 +1,130 @@
{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "xhigh": 0.99, "max": 0.99} -%}
{%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
{%- macro emit_thinking_effort() -%}
{%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%}
{%- if eff is string -%}
{%- set key = eff | trim -%}
{%- if key not in effort_map -%}
{{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
{%- endif -%}
{%- set num = effort_map[key] -%}
{%- else -%}
{%- set num = eff | float -%}
{%- endif -%}
{%- if num < 0.0 or num > 0.99 -%}
{{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
{%- endif -%}
{{- "<|message_system|><|content_text|>Thinking effort level: " -}}
{%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
{{- "<|end_message|>" -}}
{%- endmacro -%}
{%- if tools -%}
{%- set tool_state = namespace(specs=[]) -%}
{%- for tool in tools -%}
{%- set fn = tool.function if tool.function is defined else tool -%}
{%- set spec = {
"description": (fn.description if fn.description is defined and fn.description else ""),
"name": fn.name,
"parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}),
"type": (tool.type if tool.type is defined and tool.type else "function"),
} -%}
{%- set tool_state.specs = tool_state.specs + [spec] -%}
{%- endfor -%}
{{- "<|message_system|>tool_declare<|content_xml|>" -}}
{{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}}
{{- "<|end_message|>" -}}
{%- endif -%}
{%- set state = namespace(effort_emitted=false) -%}
{%- for message in messages -%}
{%- if message.role not in role_token -%}
{{- raise_exception("Unknown message role: " ~ message.role) -}}
{%- endif -%}
{%- if not state.effort_emitted and message.role != "system" -%}
{{- emit_thinking_effort() -}}
{%- set state.effort_emitted = true -%}
{%- endif -%}
{%- set header = namespace(text=role_token[message.role]) -%}
{%- if message.role == "tool" -%}
{%- if message.name is defined and message.name -%}
{%- set header.text = header.text ~ message.name -%}
{%- elif message.tool_call_id is defined and message.tool_call_id -%}
{%- for prev in messages -%}
{%- if prev.role == "assistant" and prev.tool_calls -%}
{%- for tc in prev.tool_calls -%}
{%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%}
{%- set header.text = header.text ~ tc.function.name -%}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- elif message.role == "assistant" and message.name is defined and message.name -%}
{%- set header.text = header.text ~ message.name -%}
{%- endif -%}
{%- set rtok = header.text -%}
{%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%}
{{- rtok ~ "<|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}}
{%- endif -%}
{%- if message.content is string and (message.content or not (message.role == "assistant" and (message.tool_calls or message.reasoning_content))) -%}
{{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}}
{%- elif message.content -%}
{%- for part in message.content -%}
{%- if part is string -%}
{{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}}
{%- elif part.type is not defined or part.type in ("text", "input_text") -%}
{%- set text_part = (part.text if part.text is defined and part.text is string else "") -%}
{{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}}
{%- elif part.type in ("image", "input_image", "image_url") -%}
{%- if message.role not in ("user", "tool") -%}
{{- raise_exception("Image content is only supported for user and tool messages") -}}
{%- endif -%}
{{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}}
{%- elif part.type in ("audio", "input_audio", "audio_url") -%}
{{- raise_exception("Inkling 2 does not accept audio input") -}}
{%- else -%}
{{- raise_exception("Unsupported content part type: " ~ part.type) -}}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- if message.role == "assistant" and message.tool_calls -%}
{%- for tc in message.tool_calls -%}
{%- set fn = tc.function -%}
{%- if fn.name is not defined or fn.name is not string -%}
{{- raise_exception("tool call function name must be a string") -}}
{%- endif -%}
{%- if message.name is defined and message.name and message.name != fn.name -%}
{{- raise_exception("assistant name " ~ message.name ~ " does not match its tool call " ~ fn.name) -}}
{%- endif -%}
{%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%}
{%- if args is string -%}
{{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}}
{%- endif -%}
{%- if args is not mapping -%}
{{- raise_exception("tool call arguments must be an object") -}}
{%- endif -%}
{{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}}
{{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}}
{{- "{" -}}
{%- for key, value in args | dictsort(case_sensitive=true) -%}
{{- (key | tojson) ~ ":" ~ (value | tojson(separators=(",", ":"))) -}}
{%- if not loop.last -%},{%- endif -%}
{%- endfor -%}
{{- "}}<|end_message|>" -}}
{%- endfor -%}
{%- endif -%}
{%- if message.role == "assistant" -%}
{{- "<|content_model_end_sampling|>" -}}
{%- endif -%}
{%- endfor -%}
{%- if not state.effort_emitted -%}
{{- emit_thinking_effort() -}}
{%- endif -%}
@@ -0,0 +1,129 @@
{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "max": 0.99} -%}
{%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
{%- macro emit_thinking_effort() -%}
{%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%}
{%- if eff is string -%}
{%- set key = eff | trim -%}
{%- if key not in effort_map -%}
{{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
{%- endif -%}
{%- set num = effort_map[key] -%}
{%- else -%}
{%- set num = eff | float -%}
{%- endif -%}
{%- if num < 0.0 or num > 0.99 -%}
{{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
{%- endif -%}
{{- "<|message_system|><|content_text|>Thinking effort level: " -}}
{%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
{{- "<|end_message|>" -}}
{%- endmacro -%}
{%- if tools -%}
{%- set tool_state = namespace(specs=[]) -%}
{%- for tool in tools -%}
{%- set fn = tool.function if tool.function is defined else tool -%}
{%- set spec = {
"description": (fn.description if fn.description is defined and fn.description else ""),
"name": fn.name,
"parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}),
"type": (tool.type if tool.type is defined and tool.type else "function"),
} -%}
{%- set tool_state.specs = tool_state.specs + [spec] -%}
{%- endfor -%}
{{- "<|message_system|>tool_declare<|content_xml|>" -}}
{{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}}
{{- "<|end_message|>" -}}
{%- endif -%}
{%- set state = namespace(effort_emitted=false) -%}
{%- for message in messages -%}
{%- if message.role not in role_token -%}
{{- raise_exception("Unknown message role: " ~ message.role) -}}
{%- endif -%}
{%- if not state.effort_emitted and message.role != "system" -%}
{{- emit_thinking_effort() -}}
{%- set state.effort_emitted = true -%}
{%- endif -%}
{%- set rtok = role_token[message.role] -%}
{%- if message.role == "tool" -%}
{%- set tool_name_state = namespace(name="") -%}
{%- if message.name is defined and message.name -%}
{%- set tool_name_state.name = message.name -%}
{%- elif message.tool_call_id is defined and message.tool_call_id -%}
{%- for prev in messages -%}
{%- if prev.role == "assistant" and prev.tool_calls -%}
{%- for tc in prev.tool_calls -%}
{%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%}
{%- set tool_name_state.name = tc.function.name -%}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{{- rtok -}}
{%- if tool_name_state.name -%}{{- tool_name_state.name -}}{%- endif -%}
{{- "<|content_text|>" -}}
{%- if message.content is string -%}{{- message.content -}}{%- endif -%}
{{- "<|end_message|>" -}}
{%- else -%}
{%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%}
{{- "<|message_model|><|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}}
{%- endif -%}
{%- if message.content is string -%}
{{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}}
{%- elif message.content -%}
{%- for part in message.content -%}
{%- if part is string -%}
{{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}}
{%- elif part.type is not defined or part.type in ("text", "input_text") -%}
{%- set text_part = (part.text if part.text is defined and part.text is string else "") -%}
{{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}}
{%- elif part.type in ("image", "input_image", "image_url") -%}
{{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}}
{%- elif part.type in ("audio", "input_audio", "audio_url") -%}
{{- rtok ~ "<|content_audio_input|><|unused_200053|><|audio_end|><|end_message|>" -}}
{%- else -%}
{{- raise_exception("Unsupported content part type: " ~ part.type) -}}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- if message.role == "assistant" and message.tool_calls -%}
{%- for tc in message.tool_calls -%}
{%- set fn = tc.function -%}
{%- if fn.name is not defined or fn.name is not string -%}
{{- raise_exception("tool call function name must be a string") -}}
{%- endif -%}
{%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%}
{%- if args is string -%}
{{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}}
{%- endif -%}
{%- if args is not mapping -%}
{{- raise_exception("tool call arguments must be an object") -}}
{%- endif -%}
{{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}}
{{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}}
{{- (args | tojson(sort_keys=true, separators=(",", ":"))) -}}
{{- "}<|end_message|>" -}}
{%- endfor -%}
{%- endif -%}
{%- if message.role == "assistant" -%}
{{- "<|content_model_end_sampling|>" -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if not state.effort_emitted -%}
{{- emit_thinking_effort() -}}
{%- endif -%}
{%- if add_generation_prompt -%}
{{- "<|message_model|>" -}}
{%- endif -%}
+1 -1
View File
@@ -33,7 +33,7 @@ Write a summary of the change log in a few sections:
### UI changes (if applicable)
[summarize the changes to `/tools/ui/`]
[summarize the changes to `/ui/apps/chat/`]
### ggml changes (if applicable)
+3 -3
View File
@@ -144,7 +144,7 @@ endfunction()
# Generate ui.cpp/ui.h embedding every file of ${dist_dir} (empty table when
# it has no index.html), gzip-compressed when LLAMA_UI_GZIP is enabled.
function(emit_files dist_dir)
set(UI_TEMPLATE_DIR "${LLAMA_SOURCE_DIR}/tools/ui")
set(UI_TEMPLATE_DIR "${LLAMA_SOURCE_DIR}/ui/apps/chat")
# Collect the asset list once and reuse it for the fingerprint,
# validation, compression and embedding.
@@ -518,7 +518,7 @@ function(hf_download version out_var out_resolved)
endfunction()
# ---------------------------------------------------------------------------
# 1. Priority 1: pre-built assets supplied in tools/ui/dist
# 1. Priority 1: pre-built assets supplied in ui/apps/chat/dist
# ---------------------------------------------------------------------------
if(EXISTS "${SRC_DIST_DIR}/index.html")
message(STATUS "UI: using pre-built assets from ${SRC_DIST_DIR}")
@@ -585,7 +585,7 @@ if(NOT provisioned)
"In a disconnected environment, download the pre-built UI "
"from a llama.cpp release at "
"https://github.com/ggml-org/llama.cpp/releases and "
"extract to tools/ui/dist.")
"extract to ui/apps/chat/dist.")
endif()
endif()
+18 -5
View File
@@ -22,11 +22,24 @@ Identify what actually changed and which area checklists below apply. Run `git d
- `ggml/` (any backend, op, or `ggml.h`) -> **ggml / backend**
- `include/llama.h` and other public headers -> **Public API**
- `tools/server/` -> **Server**
- `tools/ui/` (or the legacy `tools/server/webui/`) -> **WebUI**
- `ui/apps/chat/` (or the legacy `tools/ui/`, `tools/server/webui/`) -> **Llama Chat**
- `tests/`, `tools/server/tests/`, or any other added/changed test -> **Tests**
- anything else, plus all of the above -> **General** (always runs)
Always run the **Scope and quick-reject gate**, the **Security review**, and the **General** checklist. Run each area checklist whose paths were touched. Additionally, if the diff introduces a new component, subsystem, or piece of infrastructure (a new file/class/module, a new abstraction, or hand-rolled machinery), run the **Approach and design** review. Tell the user which checklists you're running and why.
Some areas have a developer README that documents internals and conventions the review must enforce. If the diff touches one of these areas, read the matching file into context before reviewing (required, not optional):
- `tools/server/` -> [tools/server/README-dev.md](../../tools/server/README-dev.md)
- `tools/mtmd/` -> [tools/mtmd/README-dev.md](../../tools/mtmd/README-dev.md)
Always run the **Prior review context** step, the **Scope and quick-reject gate**, the **Security review**, and the **General** checklist. Run each area checklist whose paths were touched. Additionally, if the diff introduces a new component, subsystem, or piece of infrastructure (a new file/class/module, a new abstraction, or hand-rolled machinery), run the **Approach and design** review. Tell the user which checklists you're running and why.
## Prior review context (always)
Maintainers tend to repeat the same comments on the same parts of the code. Before reviewing, find out what they said last time so this diff doesn't repeat a mistake that was already flagged.
- For each touched file (or the hot spots in a large diff), find recent PRs that changed the same code: `git log --oneline -n 20 -- <file>` (merged PR numbers appear as `(#NNNNN)` in the subject), and `git blame -L <start>,<end> master -- <file>` for the exact lines being changed. For in-flight work, `gh pr list --state all --search "<file or symbol>"`.
- Read the reviewer feedback on the most relevant few: `gh pr view <n> --comments` for the conversation and `gh api repos/ggml-org/llama.cpp/pulls/<n>/comments` for inline review comments.
- Note recurring reviewer requests and frequently-occurring mistakes for this area (e.g. a helper that should be reused, a naming or layering rule, a backend that keeps getting broken, a test that is always asked for), then check the current diff against each one. Report a match as a finding and cite the earlier PR so the user can read the original context.
## Scope and quick-reject gate (always)
@@ -119,9 +132,9 @@ Public API changes carry a higher bar than internal ones (`CONTRIBUTING.md`). Re
- Security: don't trust client-supplied headers (e.g. `X-Forwarded-For`) or add footguns; things like IP allowlisting belong at a reverse proxy unless there's a trusted-proxy design.
- Wire new behavior into the existing request/response and checkpoint paths correctly; watch for resource leaks across requests.
## WebUI (`tools/ui/`)
## Llama Chat (`ui/apps/chat/`)
These apply only to changes under `tools/ui/` (formerly `tools/server/webui/`) - the SvelteKit + Svelte 5 frontend. They are frontend conventions, not repo-wide ones, so do not apply them to C++ or other tool code.
These apply only to changes under `ui/apps/chat/` (formerly `tools/ui/`, `tools/server/webui/`) - the SvelteKit + Svelte 5 frontend. They are frontend conventions, not repo-wide ones, so do not apply them to C++ or other tool code.
The dominant review theme in this area is placement: most accepted nits are about putting a value, function, or piece of logic in the layer where it belongs.
@@ -192,7 +205,7 @@ Adjacent data-shape rule: empty string is not a valid "unset" initial state - us
- Sanitize anything rendered from model or remote data: the rehype/`{@html}` pipeline, SVG content, and URLs (use the `$lib/utils` URL sanitizer rather than ad-hoc checks or a third-party lib for trivial cases).
- Proxy headers and agent/tool filesystem access are security boundaries - scope enforcement (e.g. browse-root sandboxing) must be validated end-to-end, not just in the UI; loop in server reviewers for the C++ side.
### PR hygiene specific to `tools/ui`
### PR hygiene specific to `ui/apps/chat`
- Build artifacts are not committed - the UI is built by CI (`.github/workflows/ui-build.yml`) and downloaded/bundled at release time. `npm run build` must still succeed locally; a red UI build job is blocking.
- `npm run lint && npm run check` (and `npm run format`) must pass; a failing Storybook test is blocking.
+13 -16
View File
@@ -4254,7 +4254,7 @@ void llama_context::opt_epoch_iter(
ggml_opt_result_t result,
const std::vector<llama_token> & tokens,
const std::vector<llama_token> & labels_sparse,
llama_batch & batch,
llama_batch_ext & batch,
ggml_opt_epoch_callback callback,
bool train,
int64_t idata_in_loop,
@@ -4268,22 +4268,21 @@ void llama_context::opt_epoch_iter(
memory->clear(true);
for (uint32_t pos_ctx = 0; pos_ctx < n_ctx; pos_ctx += n_batch) {
batch.n_tokens = n_batch;
batch.clear();
for (uint32_t pos_batch = 0; pos_batch < n_batch; ++pos_batch) {
batch.token [pos_batch] = tokens[pos_ctx + pos_batch];
batch.pos [pos_batch] = pos_ctx + pos_batch;
batch.n_seq_id[pos_batch] = 1;
batch.seq_id [pos_batch][0] = 0;
batch.logits [pos_batch] = true;
}
// TODO: use llama_batch_ext here
{
llama_batch_compat compat(this, batch);
if (!balloc->init(*compat.batch_ext, model.vocab, true)) {
const int32_t idx = batch.add_token(0);
const llama_pos pos = pos_ctx + pos_batch;
if (idx < 0 || !batch.set_token_id(idx, tokens[pos_ctx + pos_batch])) {
LLAMA_LOG_ERROR("%s: failed to initialize batch\n", __func__);
return;
}
batch.set_token_pos(idx, &pos);
batch.set_output(idx, true);
}
if (!balloc->init(batch, model.vocab, true)) {
LLAMA_LOG_ERROR("%s: failed to initialize batch\n", __func__);
return;
}
const uint32_t n_tokens_all = balloc->get_n_tokens();
@@ -4382,7 +4381,7 @@ void llama_context::opt_epoch(
const uint32_t ubatch_per_ctx = n_ctx / n_ubatch;
struct llama_batch batch = llama_batch_init(n_batch, 0, 1);
llama_batch_ext batch(this);
std::vector<llama_token> tokens(n_ctx);
std::vector<llama_token> labels_sparse(n_ctx);
@@ -4409,8 +4408,6 @@ void llama_context::opt_epoch(
opt_epoch_iter(dataset, result_eval, tokens, labels_sparse, batch,
callback_eval, train, idata_in_loop, ndata_in_loop, t_loop_start);
}
llama_batch_free(batch);
}
//
+1 -1
View File
@@ -233,7 +233,7 @@ struct llama_context {
ggml_opt_result_t result,
const std::vector<llama_token> & tokens,
const std::vector<llama_token> & labels_sparse,
llama_batch & batch,
llama_batch_ext & batch,
ggml_opt_epoch_callback callback,
bool train,
int64_t idata_in_loop,
+2
View File
@@ -403,6 +403,8 @@ size_t llama_grammar_trigger_pattern::find(const std::string & input) const {
if (std::regex_match(input, match, regex)) {
return find_start_pos(match);
}
// some regex implementations match ^/$ at line breaks
return std::string::npos;
}
// search anywhere
+2 -1
View File
@@ -557,7 +557,8 @@ struct case_state_rotation : ctx_test {
}
llama_token token = 0;
if (llama_decode(src.get(), llama_batch_get_one(&token, 1))) {
common_batch batch = common_batch_get_one(src.get(), &token, 1);
if (llama_process(src.get(), LLAMA_PROCESS_TYPE_DECODE, batch.get())) {
TLOG_ERR("%s: failed to decode a token\n", __func__);
success = false;
break;
+3
View File
@@ -89,6 +89,9 @@ std::vector<case_run> run_cases(model_run & mr, const std::vector<const ctx_test
}
}
}
} catch (const std::exception & e) {
run.status = test_status::FAIL;
run.reason = std::string("exception: ") + e.what();
} catch (...) {
run.status = test_status::FAIL;
run.reason = "unhandled exception";
+27
View File
@@ -333,6 +333,33 @@ static void test(void) {
assert(params.lora_adapters[2].path == "file3\"3\".gguf");
assert(params.lora_adapters[3].path == "file4\".gguf");
{
std::string ts_exact;
for (size_t i = 0; i < llama_max_devices(); ++i) {
ts_exact += (i > 0 ? "," : "") + std::to_string(i + 1);
}
common_params p_ts;
argv = {"binary_name", "-m", "model.gguf", "-ts", ts_exact};
assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), p_ts, LLAMA_EXAMPLE_COMMON));
for (size_t i = 0; i < llama_max_devices(); ++i) {
assert(p_ts.tensor_split[i] == static_cast<float>(i + 1));
}
common_params p_fitt;
argv = {"binary_name", "-m", "model.gguf", "-fitt", ts_exact};
assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), p_fitt, LLAMA_EXAMPLE_COMMON));
for (size_t i = 0; i < llama_max_devices(); ++i) {
assert(p_fitt.fit_params_target[i] == static_cast<size_t>(i + 1) * 1024 * 1024);
}
std::string ts_too_many = ts_exact + ",99";
argv = {"binary_name", "-m", "model.gguf", "-ts", ts_too_many};
assert(false == common_params_parse(argv.size(), list_str_to_char(argv).data(), p_ts, LLAMA_EXAMPLE_COMMON));
argv = {"binary_name", "-m", "model.gguf", "-fitt", ts_too_many};
assert(false == common_params_parse(argv.size(), list_str_to_char(argv).data(), p_fitt, LLAMA_EXAMPLE_COMMON));
}
// skip this part on windows, because setenv is not supported
#ifdef _WIN32
printf("test-arg-parser: skip on windows build\n");
+6 -2
View File
@@ -978,7 +978,9 @@ static void test_compat(testing & t) {
mem.ranges[0] = {0, 9};
llama_token token[3] = { 5, 6, 7 };
llama_batch lb = llama_batch_get_one(token, 3);
llama_batch lb = {};
lb.n_tokens = 3;
lb.token = token;
batch_builder bb(2, &mem, 4, 1, /*n_vocab*/ 100);
llama_batch_compat::init(bb.b, lb);
@@ -999,7 +1001,9 @@ static void test_compat(testing & t) {
t.test("auto_pos_starts_at_zero_without_memory", [&](testing & t) {
llama_token token[2] = { 5, 6 };
llama_batch lb = llama_batch_get_one(token, 2);
llama_batch lb = {};
lb.n_tokens = 2;
lb.token = token;
batch_builder bb(2, nullptr, 4, 1, /*n_vocab*/ 100);
llama_batch_compat::init(bb.b, lb);
+96
View File
@@ -5516,6 +5516,102 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
.run();
}
// Inkling 1 and 2
for (const int version : { 1, 2 }) {
auto tst = peg_tester(version == 1 ? "models/templates/thinkingmachines-Inkling.jinja" :
"models/templates/thinkingmachines-Inkling-2-Small.jinja", detailed_debug);
// Inkling 1 has the first header in the generation prompt, Inkling 2 writes it
const std::string H = "<|message_model|>";
const std::string P = version == 1 ? "" : H;
const std::string E = "<|end_message|>";
const std::string CALL = "special_function<|content_invoke_tool_json|>{\"name\":\"special_function\",\"args\":{\"arg1\":1}}" + E;
// Content only
tst.test(P + "<|content_text|>Hello, world!\nWhat's up?" + E)
.expect(message_assist)
.run();
// Thinking block, then content block
tst.test(P + "<|content_thinking|>I'm thinking" + E + H + "<|content_text|>Hello, world!\nWhat's up?" + E)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect(simple_assist_msg("Hello, world!\nWhat's up?", "I'm thinking"))
.run();
// Without reasoning extraction, the thinking block stays in the content
tst.test(P + "<|content_thinking|>I'm thinking" + E + H + "<|content_text|>Hello, world!\nWhat's up?" + E)
.reasoning_format(COMMON_REASONING_FORMAT_NONE)
.expect_content("<|content_thinking|>I'm thinking" + E + "Hello, world!\nWhat's up?")
.run();
// Cut by the token limit, in thinking and in content
tst.test(P + "<|content_thinking|>I was still thinking")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect_reasoning("I was still thinking")
.run();
tst.test(P + "<|content_thinking|>I'm thinking" + E + H + "<|content_text|>Hello, wor")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect(simple_assist_msg("Hello, wor", "I'm thinking"))
.run();
// Single tool call
tst.test(P + CALL)
.tools({ special_function_tool })
.expect_tool_calls({
{ "special_function", R"({"arg1":1})", "" },
})
.expect_rules({ "tool-0" })
.run();
// Thinking, content, then a tool call
tst.test(P + "<|content_thinking|>I should call it" + E + H + "<|content_text|>On it." + E + H + CALL)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool })
.expect(simple_assist_msg("On it.", "I should call it", "special_function", R"({"arg1":1})", ""))
.run();
// Tools given, but the model answers with content
tst.test(P + "<|content_thinking|>No tool needed" + E + H + "<|content_text|>Hello, world!\nWhat's up?" + E)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool })
.expect(simple_assist_msg("Hello, world!\nWhat's up?", "No tool needed"))
.run();
// Parallel tool calls, one block each
tst.test(P + CALL + H +
"special_function_with_opt<|content_invoke_tool_json|>{\"name\":\"special_function_with_opt\",\"args\":{\"arg1\":1,\"arg2\":2}}" + E)
.parallel_tool_calls(true)
.tools({ special_function_tool, special_function_tool_with_optional_param })
.expect_tool_calls({
{ "special_function", R"({"arg1":1})", "" },
{ "special_function_with_opt", R"({"arg1":1,"arg2":2})", "" },
})
.run();
// The block closer inside a string argument does not end the block
tst.test(P + "python<|content_invoke_tool_json|>{\"name\":\"python\",\"args\":{\"code\":\"print('<|end_message|>')\"}}" + E)
.tools({ python_tool })
.expect_tool_calls({
{ "python", R"JSON({"code":"print('<|end_message|>')"})JSON", "" },
})
.run();
// Forced tool call
tst.test(P + "<|content_thinking|>I should call it" + E + H + CALL)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
.tools({ special_function_tool })
.expect(simple_assist_msg("", "I should call it", "special_function", R"({"arg1":1})", ""))
.run();
// Response format
tst.test(P + "<|content_thinking|>I'm thinking" + E + H + "<|content_text|>{\"amount\": 123.45}" + E)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.json_schema(invoice_schema)
.expect(simple_assist_msg("{\"amount\": 123.45}", "I'm thinking"))
.run();
}
// Kimi-K2-Thinking tests - custom parser
// Unique feature: tool call ID embeds function name as functions.<name>:<counter>
{
+9
View File
@@ -1475,6 +1475,15 @@ static void test_macros(testing & t) {
"a+b"
);
test_template(t, "caller saved and called after its scope ended",
"{% set ns = namespace(f=none) %}"
"{% macro keep() %}{% set ns.f = caller %}{% endmacro %}"
"{% macro outer(p) %}{% call keep() %}[{{ p }}]{% endcall %}{% endmacro %}"
"{{ outer('a') }}{{ outer('b') }}{{ ns.f() }}{{ ns.f() }}",
json::object(),
"[b][b]"
);
test_template(t, "macro with caller",
"\
{%- macro nest_dict(o, i, ff='') %}\n\
+2 -1
View File
@@ -22,7 +22,8 @@ else()
add_subdirectory(perplexity)
add_subdirectory(quantize)
if (LLAMA_BUILD_SERVER)
add_subdirectory(ui)
# the chat app lives in ui/apps/chat, outside tools/
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../ui/apps/chat ${CMAKE_BINARY_DIR}/ui/apps/chat)
add_subdirectory(cli)
add_subdirectory(server)
endif()
+3
View File
@@ -84,7 +84,10 @@ add_library(mtmd
models/qwen3tts-spkenc.cpp
models/qwen3tts-gen.cpp
models/drama3-gen.cpp
models/fish-s2-gen.cpp
models/fish-s2-spkenc.cpp
models/zonos3-gen.cpp
models/yue2-gen.cpp
models/zonos3-spkenc.cpp
models/pockettts-seanet.cpp
models/pockettts-spkenc.cpp
+16 -1
View File
@@ -39,6 +39,7 @@ Rules of thumb:
- Models and backends must outlive the subcontexts, `mtmd_context` declares them first for that reason
- Models having both audio and vision encoders must have different arch names (projector types), postfixed by `a` and `v`. See Gemma 4 as an example (`gemma4a` / `gemma4v`)
- Locking happens only at the public API boundary (`mtmd.cpp`): a new public function that runs a graph must take `compute_mutex`. The internals (subcontexts, preprocessors, `mtmd_audio_log_mel()`) assume it is held and never lock
- No magic numbers for a model's fixed shapes (layer counts, codebook sizes, window lengths...): define them once as `static constexpr` in the graph struct in `models/models.h`, and refer to them from the loader or the subcontext (e.g. `mtmd_graph_zonos3_gen::n_codebooks`, `mtmd_graph_fish_s2_spkenc::n_tfm_layer`)
## Pipeline
@@ -83,6 +84,18 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i
- Core API `mtmd.h`: handles main inference. Important: `mtmd_gen_state` holds only the per-sequence data that the mmproj itself writes and reads back (RNG, decoder KV / conv state, output buffers). Data that goes to or comes from the backbone (hidden states, codes, PCM) must stay explicit in `mtmd_gen_inp` / `mtmd_gen_out`; the caller drives the backbone and handles audio frame accumulation.
- Helper API `mtmd-helper.h`: provides a model-agnostic stateful API. Usage example can be found in the `tools/tts` directory.
### Running a transformer stage on libllama (mtmd-owned `llama_context`)
Some stages are a full LLM-sized transformer that attends to the keys/values of the backbone, for example the flow-matching NAR of YuE2 (same layer shapes as the backbone, its own weights, bidirectional over its rows, and it reads the backbone KV). Do not re-implement it as an mtmd graph. Instead, mtmd can own a second `llama_model` + `llama_context` (see `mtmd_yue2_nar_load()` and `mtmd_gen_flow_yue2()` in `mtmd.cpp`):
- Conversion: store the layers in the mmproj under `a.gen.<stage>.` with the libllama tensor names of an existing arch (e.g. `a.gen.nar.blk.0.attn_q.weight`)
- Loading: build a synthetic `gguf_context` with the arch hparams (`general.architecture`, `tokenizer.ggml.model = none`, `vocab_size = 1`, ...) and one tensor info per mmproj tensor (only the type is used, the shapes come from the hparams), then call `llama_model_init_from_user()`. The `set_tensor_data` callback reads the data from the mmproj file, so no copy is kept in the mtmd model
- libllama also creates every optional tensor of the arch (biases, `*.scale`, `*.input_scale`, `dense_2`, ...) for a model built from metadata. The callback must fill them with values that leave the graph unchanged: 0 for biases, 1 for scales, identity for `dense_2`. Zeros everywhere silently make every matmul output 0
- Backbone keys/values: the helper passes the backbone sequence state (`llama_state_seq_get_data()`) through `mtmd_gen_inp`, mtmd loads it into its context with `llama_state_seq_set_data()`. This only checks the layer count and the K/V types and row sizes, so it works across models with the same KV layout. The two contexts must agree on the KV type and on flash attention (`v_trans`)
- Inputs and outputs are embedding rows (`llama_batch::embd`, `llama_get_embeddings()` with `LLAMA_POOLING_TYPE_NONE`), the projections around them stay small mtmd graphs
- Non-causal rows: `LLAMA_ATTENTION_TYPE_NON_CAUSAL`, all rows in one ubatch (`n_ubatch` >= rows), they see every cached position of the sequence. Remove them with `llama_memory_seq_rm()` after each pass
- Size the context per call (`n_ctx`, `n_ubatch`) instead of keeping one for the whole session
### Checklist for porting new audio generation models to mtmd
1. Make sure to consult merged PRs about adding new TTS models, especially reviewer comments
@@ -109,7 +122,9 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i
- 60% changes inside `mtmd-helper-gen.cpp`
- 10% changes inside `libmtmd` core (`mtmd-model`, `mtmd-subcontext`, `models/`)
- The rest downstream code (CLI, server) should have no changes at all
5. Update usage documentation in `docs/multimodal/tts.md`
5. Validate with several concurrent requests on `llama-server` (e.g. `-np 2`, one plain and one streaming request at the same time), not only with `llama-tts`
- The server interleaves the slots on one `llama_context`, so a pipeline must copy what it needs (hidden state, logits) right after its own decode, not read it at the next step
6. Update usage documentation in `docs/multimodal/tts.md`
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, **open an issue first for discussion**.
+74
View File
@@ -0,0 +1,74 @@
#include "models.h"
#include <cmath>
#include <vector>
// one frame: backbone hidden + semantic code -> the codes of this frame + next backbone input
// fast AR row 0 is the backbone hidden, row p > 0 is the embedding of code p - 1 and picks code p (greedy, as the reference)
ggml_cgraph * mtmd_graph_fish_s2_gen::build_fast_ar() {
const int64_t n_cb = model.gen_code_embd_w->ne[2];
const int64_t n_embd_cb = model.gen_code_embd_w->ne[0];
const int64_t n_embd = model.gen_code_tok_embd_w->ne[0];
const int64_t n_kv = model.layers[0].k_w->ne[1];
const int n_layer = (int) model.layers.size();
ggml_tensor * h = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, n_mmproj_embd);
ggml_set_name(h, "inp_raw");
ggml_set_input(h);
ggml_tensor * inp_code0 = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, 1);
ggml_set_name(inp_code0, "inp_code0");
ggml_set_input(inp_code0);
mtmd_graph_qwen3tts_gen::code_gen cg(*this, top_k, top_p);
std::vector<ggml_tensor *> k_cache(n_layer), v_cache(n_layer);
for (int il = 0; il < n_layer; il++) {
k_cache[il] = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_kv, n_cb), 0.0f);
v_cache[il] = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_kv, n_cb), 0.0f);
}
ggml_tensor * out_codes = cg.cache_set(ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, 1, n_cb), 0, inp_code0);
for (int p = 0; p < n_cb; p++) {
ggml_tensor * cur = h;
if (p > 0) {
ggml_tensor * code = ggml_view_1d(ctx0, out_codes, 1, (size_t) (p - 1) * out_codes->nb[1]);
cur = ggml_reshape_1d(ctx0, ggml_get_rows(ctx0, model.gen_code_tok_embd_w, code), n_embd);
}
ggml_tensor * kq_mask = cg.causal_mask_row(n_cb, p);
ggml_tensor * inp_pos = cg.const_i32(k_cache[0], (float) p);
for (int il = 0; il < n_layer; il++) {
cur = cg.layer_forward(cur, model.layers[il], inp_pos, kq_mask, k_cache[il], v_cache[il], n_cb, p, il);
}
if (p == 0) {
continue; // only fills the KV
}
cur = ggml_mul(ctx0, ggml_rms_norm(ctx0, cur, hparams.eps), model.gen_code_norm_w);
ggml_tensor * logits = ggml_mul_mat(ctx0, model.gen_code_head_w, cur);
out_codes = cg.cache_set(out_codes, p, ggml_argmax(ctx0, logits));
}
out_codes = ggml_cont(ctx0, out_codes);
ggml_set_name(out_codes, "out_codes");
ggml_set_output(out_codes);
ggml_build_forward_expand(gf, out_codes);
// backbone input of the next row: (E_lm[semantic] + sum of E_cb[g][code g]) / sqrt(n_cb + 1)
ggml_tensor * out_embd = ggml_get_rows(ctx0, model.gen_code_out_embd_w, inp_code0);
for (int64_t g = 0; g < n_cb; g++) {
ggml_tensor * table = ggml_view_2d(ctx0, model.gen_code_embd_w, n_embd_cb, model.gen_code_embd_w->ne[1],
model.gen_code_embd_w->nb[1], (size_t) g * model.gen_code_embd_w->nb[2]);
ggml_tensor * code = ggml_view_1d(ctx0, out_codes, 1, (size_t) g * out_codes->nb[1]);
out_embd = ggml_add(ctx0, out_embd, ggml_get_rows(ctx0, table, code));
}
out_embd = ggml_scale(ctx0, out_embd, 1.0f / sqrtf((float) (n_cb + 1)));
cb(out_embd, "gen_audio_out", -1);
ggml_build_forward_expand(gf, out_embd);
return gf;
}
ggml_cgraph * mtmd_graph_fish_s2_gen::build() {
return gen_process == MTMD_SUBCTX_GEN_PROCESS_GEN_WAV ? build_codec() : build_fast_ar();
}
+135
View File
@@ -0,0 +1,135 @@
#include "models.h"
#include <cmath>
#include <string>
// reference clip -> codec frames -> backbone input rows, see DAC.encode() and DownsampleResidualVectorQuantize.encode()
// activations are [C, T]; a conv is a sum of matmuls over tap-shifted views: exact in F32, no im2col buffer
// causal conv, w is taps-major [stride * IC, OC, K / stride], see conversion/fish_s2.py
ggml_tensor * mtmd_graph_fish_s2_spkenc::conv(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int stride, int dilation) const {
const int64_t n_tap = w->ne[2];
const int pad = (int) (n_tap - 1) * stride * dilation;
if (pad > 0) {
x = ggml_pad_ext(ctx0, x, 0, 0, pad, 0, 0, 0, 0, 0);
}
if (stride > 1) {
// stride frames per column, then each tap group is one matmul
x = ggml_reshape_2d(ctx0, x, x->ne[0] * stride, x->ne[1] / stride);
}
const int64_t n_out = x->ne[1] - (n_tap - 1) * dilation;
ggml_tensor * y = nullptr;
for (int64_t k = 0; k < n_tap; k++) {
ggml_tensor * wk = ggml_view_2d(ctx0, w, w->ne[0], w->ne[1], w->nb[1], (size_t) k * w->nb[2]);
ggml_tensor * xk = ggml_view_2d(ctx0, x, x->ne[0], n_out, x->nb[1], (size_t) k * dilation * x->nb[1]);
ggml_tensor * yk = ggml_mul_mat(ctx0, wk, xk);
y = y ? ggml_add(ctx0, y, yk) : yk;
}
return ggml_add(ctx0, y, b);
}
// x + sin(alpha * x)^2 / alpha, beta = 1 / alpha is folded at conversion
// TODO @ngxson: on Metal, residual codes match the reference less often than on CPU (89-98% vs 99%), maybe the fast-math sin
ggml_tensor * mtmd_graph_fish_s2_spkenc::snake(ggml_tensor * x, ggml_tensor * alpha, ggml_tensor * beta) const {
ggml_tensor * s = ggml_sqr(ctx0, ggml_sin(ctx0, ggml_mul(ctx0, x, alpha)));
return ggml_add(ctx0, x, ggml_mul(ctx0, s, beta));
}
// causal depthwise conv (w [C, K]) -> LayerNorm -> pwconv1 -> GELU -> pwconv2 -> layer scale -> residual
ggml_tensor * mtmd_graph_fish_s2_spkenc::convnext(ggml_tensor * x, const mtmd_code2wav::upsample_block & blk) const {
const int64_t n_tap = blk.dwconv_w->ne[1];
const int64_t n_pos = x->ne[1];
ggml_tensor * xp = ggml_pad_ext(ctx0, x, 0, 0, (int) n_tap - 1, 0, 0, 0, 0, 0);
ggml_tensor * h = nullptr;
for (int64_t k = 0; k < n_tap; k++) {
ggml_tensor * wk = ggml_view_1d(ctx0, blk.dwconv_w, blk.dwconv_w->ne[0], (size_t) k * blk.dwconv_w->nb[1]);
ggml_tensor * xk = ggml_view_2d(ctx0, xp, xp->ne[0], n_pos, xp->nb[1], (size_t) k * xp->nb[1]);
ggml_tensor * hk = ggml_mul(ctx0, xk, wk);
h = h ? ggml_add(ctx0, h, hk) : hk;
}
h = ggml_add(ctx0, h, blk.dwconv_b);
h = ggml_add(ctx0, ggml_mul(ctx0, ggml_norm(ctx0, h, 1e-6f), blk.norm_w), blk.norm_b);
h = ggml_gelu_erf(ctx0, ggml_add(ctx0, ggml_mul_mat(ctx0, blk.pw1_w, h), blk.pw1_b));
h = ggml_add(ctx0, ggml_mul_mat(ctx0, blk.pw2_w, h), blk.pw2_b);
return ggml_add(ctx0, x, ggml_mul(ctx0, h, blk.gamma));
}
ggml_cgraph * mtmd_graph_fish_s2_spkenc::build() {
const auto & c2w = model.c2w;
// the preprocessor hands over the waveform as a single-row "mel", [n_samples, 1] is also [1, n_samples]
ggml_tensor * inp_raw = build_inp_raw(1);
ggml_tensor * x = ggml_reshape_2d(ctx0, inp_raw, 1, inp_raw->ne[0]);
// windowed causal transformer on a cold start: zero state, positions from 0
mtmd_graph_qwen3tts_gen::code2wav dec(*this);
auto zero_state = [&](const std::string & name, int64_t n0, int64_t n1) {
dec.state_in[name] = ggml_fill(ctx0, ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n0, n1), 0.0f);
};
auto transformer = [&](ggml_tensor * cur, int il0, int il1, int64_t window, ggml_tensor * norm_w) {
const int64_t n_kv = model.layers[il0].k_w->ne[1];
zero_state("tfm_pos", 1, 1);
for (int il = il0; il < il1; il++) {
zero_state("tfm_k_" + std::to_string(il), n_kv, window - 1);
zero_state("tfm_v_" + std::to_string(il), n_kv, window - 1);
cur = dec.tfm_layer_forward(cur, model.layers[il], il, window);
}
return ggml_mul(ctx0, ggml_rms_norm(ctx0, cur, hparams.wav_tfm_eps), norm_w);
};
// encoder: entry conv, 4x (3 residual units, Snake, strided conv), transformer, Snake, post conv
x = conv(x, c2w.dac_entry_w, c2w.dac_entry_b, 1, 1);
static constexpr int DAC_DILATIONS[3] = { 1, 3, 9 };
for (const auto & blk : c2w.dac) {
for (size_t ir = 0; ir < blk.res.size(); ir++) {
const auto & res = blk.res[ir];
ggml_tensor * h = snake(x, res.act1_alpha, res.act1_beta);
h = conv(h, res.conv1_w, res.conv1_b, 1, DAC_DILATIONS[ir]);
h = snake(h, res.act2_alpha, res.act2_beta);
h = conv(h, res.conv2_w, res.conv2_b, 1, 1);
x = ggml_add(ctx0, x, h);
}
x = snake(x, blk.snake_alpha, blk.snake_beta);
x = conv(x, blk.conv_w, blk.conv_b, (int) (blk.conv_w->ne[0] / x->ne[0]), 1);
}
x = transformer(x, 0, n_enc_tfm_layer, 512, model.post_ln_w);
x = snake(x, c2w.dac_post_snake_alpha, c2w.dac_post_snake_beta);
x = conv(x, c2w.dac_post_conv_w, c2w.dac_post_conv_b, 1, 1);
cb(x, "codec_enc", -1);
// quantizer: 2x (strided conv + ConvNeXt), transformer
for (const auto & down : c2w.upsample) {
x = conv(x, down.conv_w, down.conv_b, (int) (down.conv_w->ne[0] / x->ne[0]), 1);
x = convnext(x, down);
}
ggml_tensor * z = transformer(x, n_enc_tfm_layer, (int) model.layers.size(), 128, c2w.tfm_output_norm_w);
cb(z, "codec_latent", -1);
// nearest entry by cosine similarity, then the entry through out_proj is removed from the residual
auto vq = [&](ggml_tensor * in_w, ggml_tensor * in_b, ggml_tensor * cb_norm, ggml_tensor * table) {
ggml_tensor * e = ggml_add(ctx0, ggml_mul_mat(ctx0, in_w, z), in_b);
e = ggml_l2_norm(ctx0, e, 1e-12f);
ggml_tensor * idx = ggml_argmax(ctx0, ggml_mul_mat(ctx0, cb_norm, e));
z = ggml_sub(ctx0, z, ggml_get_rows(ctx0, table, idx));
return idx;
};
auto page = [&](ggml_tensor * t, int64_t g) {
return ggml_view_2d(ctx0, t, t->ne[0], t->ne[1], t->nb[1], (size_t) g * t->nb[2]);
};
// backbone row: (E_lm[semantic] + sum of E_cb[g][code g]) / sqrt(n_cb + 1), same as fish_s2_gen
const int64_t n_cb = model.gen_code_embd_w->ne[2];
ggml_tensor * idx = vq(c2w.quant_first_in_w, c2w.quant_first_in_b, c2w.quant_first_cb_norm_w, c2w.quant_first_cb_w);
ggml_tensor * out = ggml_add(ctx0, ggml_get_rows(ctx0, model.gen_code_out_embd_w, idx),
ggml_get_rows(ctx0, page(model.gen_code_embd_w, 0), idx));
for (int64_t g = 1; g < n_cb; g++) {
ggml_tensor * in_b = ggml_view_1d(ctx0, c2w.quant_rest_in_b, c2w.quant_rest_in_b->ne[0], (size_t) (g - 1) * c2w.quant_rest_in_b->nb[1]);
idx = vq(page(c2w.quant_rest_in_w, g - 1), in_b, page(c2w.quant_rest_cb_norm_w, g - 1), page(c2w.quant_rest_cb_w, g - 1));
out = ggml_add(ctx0, out, ggml_get_rows(ctx0, page(model.gen_code_embd_w, g), idx));
}
out = ggml_scale(ctx0, out, 1.0f / sqrtf((float) (n_cb + 1)));
cb(out, "spk_rows", -1);
ggml_build_forward_expand(gf, out);
return gf;
}
+55 -1
View File
@@ -462,7 +462,8 @@ struct mtmd_graph_qwen3tts_gen : mtmd_graph {
ggml_tensor * snake(ggml_tensor * x, ggml_tensor * alpha, ggml_tensor * beta) const;
ggml_tensor * quant_decode(ggml_tensor * inp_codes) const;
ggml_tensor * tfm_layer_forward(ggml_tensor * cur, const mtmd_layer & layer, int il) const;
// window <= 0 uses hparams.wav_tfm_swa
ggml_tensor * tfm_layer_forward(ggml_tensor * cur, const mtmd_layer & layer, int il, int64_t window = 0) const;
ggml_tensor * convnext_block(ggml_tensor * x, const mtmd_code2wav::upsample_block & blk, const std::string & state_prefix) const;
ggml_tensor * dac_res_unit(ggml_tensor * x, const mtmd_code2wav::dac_res & res, int dilation, const std::string & state_name) const;
@@ -569,6 +570,32 @@ struct mtmd_graph_drama3_gen : mtmd_graph {
ggml_tensor * inp_pos, ggml_tensor * k_slot, ggml_tensor * v_slot, int pos);
};
//
// fish_s2 generation:
// GEN_CODE = fast AR, backbone hidden + semantic code -> other codes of the frame + next backbone input
// GEN_WAV = Fish DAC decoder, same as drama3
//
struct mtmd_graph_fish_s2_gen : mtmd_graph_drama3_gen {
using mtmd_graph_drama3_gen::mtmd_graph_drama3_gen;
ggml_cgraph * build() override;
ggml_cgraph * build_fast_ar();
};
// fish_s2 voice cloning: raw 44.1 kHz clip -> codec encoder + RVQ -> one backbone input row per frame
struct mtmd_graph_fish_s2_spkenc : mtmd_graph {
mtmd_graph_fish_s2_spkenc(mtmd_subcontext * ctx, const mtmd_image_f32 & img) : mtmd_graph(ctx, img) {}
ggml_cgraph * build() override;
// codec encoder transformer layers, the rest of model.layers is the quantizer pre_module
static constexpr int n_enc_tfm_layer = 4;
static constexpr int n_tfm_layer = 12;
ggml_tensor * conv(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int stride, int dilation) const;
ggml_tensor * snake(ggml_tensor * x, ggml_tensor * alpha, ggml_tensor * beta) const;
ggml_tensor * convnext(ggml_tensor * x, const mtmd_code2wav::upsample_block & blk) const;
};
//
// zonos3 generation:
// GEN_CODE = 10 codebook heads, or the backbone row of a sampled frame
@@ -610,6 +637,33 @@ struct mtmd_graph_zonos3_gen : mtmd_graph {
std::vector<c2w_state_slot> list_zonos3_gen_state_slots(const mtmd_hparams & hparams, const mtmd_model & model);
//
// YuE2 generation, the NAR transformer itself runs on a llama_context, see mtmd_gen_flow_yue2():
// LATENT_IN = latent rows -> backbone embd rows
// LATENT_OUT = backbone hidden rows -> velocity rows
// GEN_WAV = Oobleck VAE decoder, latent rows -> interleaved stereo PCM
//
struct mtmd_graph_yue2_gen : mtmd_graph {
mtmd_graph_yue2_gen(mtmd_subcontext * ctx, const mtmd_image_f32 & img, mtmd_subctx_gen_process gen_process, int n_rows)
: mtmd_graph(ctx, img), gen_process(gen_process), n_rows(n_rows) {}
ggml_cgraph * build() override;
static constexpr int n_latent = 64;
static constexpr int n_freq = 256; // time embedder input
static constexpr int n_channels = 2;
static constexpr int vae_ratio = 1920; // output samples per latent frame
static constexpr int vae_trim = 64; // a decode of n frames has vae_ratio * n - vae_trim samples
static constexpr float rope_theta = 1000000.0f;
static constexpr int n_ctx_max = 24576; // the AR prefix and the NAR rows of one chunk
static constexpr int ode_steps = 32; // midpoint solver
mtmd_subctx_gen_process gen_process;
int n_rows;
ggml_tensor * snake(ggml_tensor * x, const mtmd_yue2::snake & s) const;
ggml_tensor * conv1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation) const;
};
struct mtmd_graph_soprano : mtmd_graph {
mtmd_graph_soprano(mtmd_subcontext * ctx, const mtmd_image_f32 & img, int n_frames) : mtmd_graph(ctx, img), n_frames(n_frames) {}
ggml_cgraph * build() override;
+9 -6
View File
@@ -156,10 +156,13 @@ ggml_tensor * mtmd_graph_qwen3tts_gen::code_gen::layer_forward(
q = ggml_reshape_3d(ctx0, q, d_head, n_head, 1);
k = ggml_reshape_3d(ctx0, k, d_head, n_head_kv, 1);
q = ggml_rms_norm(ctx0, q, hparams.eps);
q = ggml_mul(ctx0, q, layer.q_norm);
k = ggml_rms_norm(ctx0, k, hparams.eps);
k = ggml_mul(ctx0, k, layer.k_norm);
// fish_s2 fast AR has no qk norm
if (layer.q_norm) {
q = ggml_rms_norm(ctx0, q, hparams.eps);
q = ggml_mul(ctx0, q, layer.q_norm);
k = ggml_rms_norm(ctx0, k, hparams.eps);
k = ggml_mul(ctx0, k, layer.k_norm);
}
q = ggml_rope_ext(ctx0, q, inp_pos, nullptr, (int) d_head, GGML_ROPE_TYPE_NEOX, 0,
hparams.rope_theta, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
@@ -418,12 +421,12 @@ ggml_tensor * mtmd_graph_qwen3tts_gen::code2wav::quant_decode(ggml_tensor * inp_
// one pre_transformer layer over a batch of N = sliding_window new frames
// attention runs over [(W-1)-frame prefix from the last batch] + [N new frames]
// RoPE positions come from a persisted counter, so phases line up across batches
ggml_tensor * mtmd_graph_qwen3tts_gen::code2wav::tfm_layer_forward(ggml_tensor * cur, const mtmd_layer & layer, int il) const {
ggml_tensor * mtmd_graph_qwen3tts_gen::code2wav::tfm_layer_forward(ggml_tensor * cur, const mtmd_layer & layer, int il, int64_t window) const {
const int n_head = hparams.wav_tfm_n_head;
const int n_head_kv = hparams.wav_tfm_n_head_kv;
const int64_t d_head = layer.q_w->ne[1] / n_head;
const float kq_scale = 1.0f / sqrtf((float) d_head);
const int64_t W = hparams.wav_tfm_swa; // == N, frames per batch
const int64_t W = window > 0 ? window : hparams.wav_tfm_swa; // code2wav: == N, frames per batch
const int64_t N = cur->ne[1];
const int64_t prefix = W - 1;
const int64_t total_kv = prefix + N;
+89
View File
@@ -0,0 +1,89 @@
#include "models.h"
// SnakeBeta, alpha and beta folded at conversion: x + beta * sin(alpha * x)^2
// x: [T, C], alpha/beta: [C]
ggml_tensor * mtmd_graph_yue2_gen::snake(ggml_tensor * x, const mtmd_yue2::snake & s) const {
ggml_tensor * a = ggml_reshape_2d(ctx0, s.alpha, 1, s.alpha->ne[0]);
ggml_tensor * b = ggml_reshape_2d(ctx0, s.beta, 1, s.beta->ne[0]);
// expand reshapes first so mul/sin/sqr/mul/add lands as consecutive nodes, letting backends fuse them
ggml_build_forward_expand(gf, a);
ggml_build_forward_expand(gf, b);
ggml_tensor * y = ggml_sqr(ctx0, ggml_sin(ctx0, ggml_mul(ctx0, x, a)));
return ggml_add(ctx0, x, ggml_mul(ctx0, y, b));
}
// same-length conv, stride 1; x: [T, IC], w: [K, IC, OC], returns [T, OC]
// im2col in F32: the reference decoder runs in FP32
ggml_tensor * mtmd_graph_yue2_gen::conv1d(ggml_tensor * x, ggml_tensor * w, ggml_tensor * b, int dilation) const {
const int pad = (int) (w->ne[0] - 1) / 2 * dilation;
ggml_tensor * col = ggml_im2col(ctx0, w, x, 1, 0, pad, 0, dilation, 0, false, GGML_TYPE_F32); // [1, T, IC * K]
ggml_tensor * y = ggml_mul_mat(ctx0,
ggml_reshape_2d(ctx0, col, col->ne[0], col->ne[1] * col->ne[2]),
ggml_reshape_2d(ctx0, w, w->ne[0] * w->ne[1], w->ne[2]));
if (b) {
y = ggml_add(ctx0, y, ggml_reshape_2d(ctx0, b, 1, b->ne[0]));
}
return y;
}
ggml_cgraph * mtmd_graph_yue2_gen::build() {
const auto & y = model.yue2;
if (gen_process == MTMD_SUBCTX_GEN_PROCESS_LATENT_IN || gen_process == MTMD_SUBCTX_GEN_PROCESS_LATENT_OUT) {
const bool is_in = gen_process == MTMD_SUBCTX_GEN_PROCESS_LATENT_IN;
ggml_tensor * w = is_in ? y.inp_w : y.out_w;
ggml_tensor * b = is_in ? y.inp_b : y.out_b;
ggml_tensor * cur = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, w->ne[0], n_rows);
ggml_set_name(cur, "inp_feats");
ggml_set_input(cur);
cur = ggml_add(ctx0, build_mm(w, cur), b);
ggml_set_name(cur, "out_feats");
ggml_set_output(cur);
ggml_build_forward_expand(gf, cur);
return gf;
}
GGML_ASSERT(gen_process == MTMD_SUBCTX_GEN_PROCESS_GEN_WAV);
static constexpr int dilations[3] = { 1, 3, 9 };
ggml_tensor * inp = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_latent, n_rows);
ggml_set_name(inp, "inp_feats");
ggml_set_input(inp);
ggml_tensor * x = ggml_cont(ctx0, ggml_transpose(ctx0, inp)); // [T, n_latent]
x = conv1d(x, y.vae_in_w, y.vae_in_b, 1);
for (size_t ib = 0; ib < y.vae.size(); ib++) {
const auto & blk = y.vae[ib];
const int stride = (int) (blk.up_w->ne[0] / 2);
const int pad = (stride + 1) / 2; // ceil(stride / 2) of the reference
x = snake(x, blk.act);
{
// ConvTranspose1d as GEMM + col2im, same as qwen3tts; up_w: [K, OC, IC]
const int64_t K = blk.up_w->ne[0];
const int64_t OC = blk.up_w->ne[1];
ggml_tensor * w2 = ggml_cont(ctx0, ggml_transpose(ctx0, ggml_reshape_2d(ctx0, blk.up_w, K * OC, blk.up_w->ne[2])));
ggml_tensor * col = ggml_mul_mat(ctx0, w2, ggml_cont(ctx0, ggml_transpose(ctx0, x))); // [K * OC, T]
x = ggml_col2im_1d(ctx0, col, stride, (int) OC, pad);
}
x = ggml_add(ctx0, x, ggml_reshape_2d(ctx0, blk.up_b, 1, blk.up_b->ne[0]));
for (size_t ir = 0; ir < blk.res.size(); ir++) {
const auto & ru = blk.res[ir];
ggml_tensor * h = snake(x, ru.act1);
h = conv1d(h, ru.conv1_w, ru.conv1_b, dilations[ir]);
h = snake(h, ru.act2);
h = conv1d(h, ru.conv2_w, ru.conv2_b, 1);
x = ggml_add(ctx0, x, h);
}
cb(x, "vae_block", (int) ib);
}
x = snake(x, y.vae_post);
x = conv1d(x, y.vae_out_w, nullptr, 1); // [T, n_channels]
x = ggml_cont(ctx0, ggml_transpose(ctx0, x)); // interleaved
ggml_set_name(x, "out_audio");
ggml_set_output(x);
ggml_build_forward_expand(gf, x);
return gf;
}
+10
View File
@@ -2242,6 +2242,16 @@ bool mtmd_audio_preprocessor_pockettts::process(const float * sa
return true;
}
bool mtmd_audio_preprocessor_fish_s2::process(const float * samples, size_t n_samples, std::vector<mtmd_audio_mel> & output, bool shape_only) const {
const size_t max_samples = (size_t) mtmd_hparams::fish_s2_max_spk_seconds * hparams.audio_sample_rate;
if (n_samples > max_samples) {
LOG_WRN("%s: speaker reference is %.1f s, truncating to the first %d s\n", __func__,
(double) n_samples / hparams.audio_sample_rate, mtmd_hparams::fish_s2_max_spk_seconds);
n_samples = max_samples;
}
return mtmd_audio_preprocessor_muse_spark_audio::process(samples, n_samples, output, shape_only);
}
//
// mtmd_audio_preprocessor_vibevoice_asr
//
+6
View File
@@ -129,6 +129,12 @@ struct mtmd_audio_preprocessor_muse_spark_audio : mtmd_audio_preprocessor {
bool process(const float * samples, size_t n_samples, std::vector<mtmd_audio_mel> & output, bool shape_only) const override;
};
// fish_s2 speaker reference: same as muse spark (raw waveform, whole frames), truncated to fish_s2_max_spk_seconds
struct mtmd_audio_preprocessor_fish_s2 : mtmd_audio_preprocessor_muse_spark_audio {
using mtmd_audio_preprocessor_muse_spark_audio::mtmd_audio_preprocessor_muse_spark_audio;
bool process(const float * samples, size_t n_samples, std::vector<mtmd_audio_mel> & output, bool shape_only) const override;
};
// vibevoice: gain to -25 dBFS RMS, without clipping the peak
// the RMS is floored at -50 dBFS, so that silence or noise alone is not boosted to full scale
double mtmd_audio_vibevoice_gain(double sum_sq, int64_t n_samples, double peak);
+24 -3
View File
@@ -973,6 +973,10 @@ std::unique_ptr<mtmd_graph> mtmd_get_graph_builder(mtmd_subcontext * ctx, const
{
builder = std::make_unique<mtmd_graph_pockettts_spkenc>(ctx, img);
} break;
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
builder = std::make_unique<mtmd_graph_fish_s2_spkenc>(ctx, img);
} break;
case PROJECTOR_TYPE_POCKETTTS_GEN:
{
const auto gen_process = params ? params->gen_process : MTMD_SUBCTX_GEN_PROCESS_GEN_CODE;
@@ -1034,14 +1038,31 @@ std::unique_ptr<mtmd_graph> mtmd_get_graph_builder(mtmd_subcontext * ctx, const
? (int) (params->codes->size() / n_cb) : 1;
builder = std::make_unique<mtmd_graph_zonos3_gen>(ctx, img, gen_process, all_known, n_frames);
} break;
case PROJECTOR_TYPE_YUE2_GEN:
{
const auto gen_process = params ? params->gen_process : MTMD_SUBCTX_GEN_PROCESS_LATENT_IN;
int n_rows = 1;
if (params && params->feats) {
const size_t n_col = gen_process == MTMD_SUBCTX_GEN_PROCESS_LATENT_OUT ? ctx->model.yue2.out_w->ne[0] : mtmd_graph_yue2_gen::n_latent;
GGML_ASSERT(!params->feats->empty() && params->feats->size() % n_col == 0);
n_rows = (int) (params->feats->size() / n_col);
}
builder = std::make_unique<mtmd_graph_yue2_gen>(ctx, img, gen_process, n_rows);
} break;
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_FISH_S2_GEN:
{
const auto gen_process = params ? params->gen_process : MTMD_SUBCTX_GEN_PROCESS_GEN_CODE;
const int64_t n_cb = ctx->model.gen_code_embd_w->ne[2];
const int n_frames = params && params->codes ? (int) (params->codes->size() / n_cb) : 1;
builder = std::make_unique<mtmd_graph_drama3_gen>(ctx, img, gen_process, params ? params->gen_state : nullptr, n_frames,
params ? params->top_k : 30, params ? params->top_p : 0.9f,
params ? params->temp : 1.0f);
const int top_k = params ? params->top_k : 30;
const float top_p = params ? params->top_p : 0.9f;
const float temp = params ? params->temp : 1.0f;
if (ctx->model.proj_type == PROJECTOR_TYPE_FISH_S2_GEN) {
builder = std::make_unique<mtmd_graph_fish_s2_gen>(ctx, img, gen_process, params ? params->gen_state : nullptr, n_frames, top_k, top_p, temp);
} else {
builder = std::make_unique<mtmd_graph_drama3_gen>(ctx, img, gen_process, params ? params->gen_state : nullptr, n_frames, top_k, top_p, temp);
}
} break;
case PROJECTOR_TYPE_QWEN3TTS_GEN:
{
File diff suppressed because it is too large Load Diff
+25 -4
View File
@@ -287,8 +287,8 @@ struct mtmd_helper_gen_audio;
typedef struct mtmd_helper_gen_audio mtmd_helper_gen_audio;
enum mtmd_helper_gen_audio_outtype {
MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM, // raw PCM
MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV, // WAV PCM 16-bit LE, mono
MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM, // raw F32 PCM, channels interleaved
MTMD_HELPER_GEN_AUDIO_OUTTYPE_WAV, // WAV PCM 16-bit LE
};
// what a response that continues a sequence (n_past > 0) outputs
enum mtmd_helper_gen_audio_out_mode {
@@ -310,6 +310,7 @@ struct mtmd_helper_gen_audio_inp {
enum mtmd_helper_gen_audio_out_mode out_mode; // n_past > 0 only (default: auto)
mtmd_bitmap * speaker_ref; // optional, can be NULL: audio to clone, or a voice bitmap
const char * speaker_ref_text; // optional, can be NULL: transcript of speaker_ref, for models that put it in the prompt (e.g. Fish S2)
const char * lang; // optional, can be NULL
int32_t top_k;
@@ -321,11 +322,20 @@ struct mtmd_helper_gen_audio_inp {
float speed; // speaking speed relative to a typical pace, > 0 (default: 1.0 = the model's own pace)
float loudness; // target loudness in LUFS, < 0 (default: 0.0 = the model's own loudness)
// classifier-free guidance scale of the audio codes, > 0, 1.0 = no guidance (only for models that support it)
// default: 0.0 = the model's own; other models reject non-default values
float cfg_scale;
// steps of the flow-matching ODE solver, fewer is faster with less accurate audio (only for models that support it)
// default: 0 = the model's own; other models reject non-default values
int32_t flow_steps;
enum mtmd_helper_gen_audio_outtype out_type;
};
MTMD_API struct mtmd_helper_gen_audio_inp mtmd_helper_gen_audio_inp_default(void);
// returns nullptr if the context cannot run the pipeline of the model (e.g. a context too small)
MTMD_API mtmd_helper_gen_audio * mtmd_helper_gen_audio_init(
struct llama_context * lctx,
struct mtmd_context * mctx);
@@ -334,12 +344,15 @@ MTMD_API void mtmd_helper_gen_audio_free(mtmd_helper_gen_audio * ctx);
MTMD_API void mtmd_helper_gen_audio_reset(mtmd_helper_gen_audio * ctx);
// frame limit for a caller that has none of its own (e.g. no -n); 0 = step_gen() stops by itself
MTMD_API int32_t mtmd_helper_gen_audio_max_frames(const mtmd_helper_gen_audio * ctx);
// returns 0 on success, 2 if the input is invalid for the model (e.g. unsupported speed or loudness), 1 on other errors
MTMD_API int32_t mtmd_helper_gen_audio_set_input(
mtmd_helper_gen_audio * ctx,
const struct mtmd_helper_gen_audio_inp * inp);
// processes at most n_batch prompt tokens per call
// processes at most n_batch prompt tokens per call, n_batch must not exceed llama_n_batch() of the context
// returns: >0 = number of prompt tokens remaining, 0 = done, <0 = error
MTMD_API int32_t mtmd_helper_gen_audio_step_prompt(
mtmd_helper_gen_audio * ctx,
@@ -369,7 +382,7 @@ MTMD_API int32_t mtmd_helper_audio_stream_get_logit_bias(const mtmd_context * mc
llama_logit_bias * out, int32_t n_max);
// out_data valid until next get_output() or reset() call
// out_n_samples (optional, can be NULL) receives the number of generated PCM samples
// out_n_samples (optional, can be NULL) receives the number of generated PCM samples per channel (see mtmd_gen_audio_info::n_channels)
// if inp->stream is true: returns only audio produced since the previous call, and
// *out_data_len == 0 whenever a full window_frames batch hasn't accumulated yet
MTMD_API int32_t mtmd_helper_gen_audio_get_output(
@@ -423,6 +436,7 @@ struct gen_audio {
mtmd_helper_gen_audio_inp data = mtmd_helper_gen_audio_inp_default();
std::string prompt_str;
std::string lang_str;
std::string speaker_ref_text_str;
mtmd::bitmap_ptr speaker_ref_ptr;
inp() = default;
@@ -434,6 +448,7 @@ struct gen_audio {
void set_prompt (std::string p) { prompt_str = std::move(p); }
void set_lang (std::string l) { lang_str = std::move(l); }
void set_speaker_ref(mtmd::bitmap_ptr bmp) { speaker_ref_ptr = std::move(bmp); }
void set_speaker_ref_text(std::string t) { speaker_ref_text_str = std::move(t); }
// pointers are only valid as long as *this is alive
const mtmd_helper_gen_audio_inp * get() {
@@ -441,6 +456,7 @@ struct gen_audio {
data.prompt_len = prompt_str.size();
data.lang = lang_str.empty() ? nullptr : lang_str.c_str();
data.speaker_ref = speaker_ref_ptr.get();
data.speaker_ref_text = speaker_ref_text_str.empty() ? nullptr : speaker_ref_text_str.c_str();
return &data;
}
};
@@ -452,6 +468,11 @@ struct gen_audio {
bool valid() const {
return ctx.get() != nullptr;
}
// caller's limit if > 0, else the limit of the model
int32_t max_frames(int32_t n_predict) const {
const int32_t n_model = mtmd_helper_gen_audio_max_frames(ctx.get());
return n_predict > 0 ? n_predict : n_model > 0 ? n_model : INT32_MAX;
}
void reset() {
mtmd_helper_gen_audio_reset(ctx.get());
}
+31
View File
@@ -271,6 +271,31 @@
#define TN_A_GEN_CODE_BLK_EMBD "a.gen.code.blk.%d.embd.%s"
#define TN_A_GEN_CODE_BLK_OUT_NORM "a.gen.code.blk.%d.out_norm.%s"
// fish_s2: fast AR input embedding, shared by all codebooks
#define TN_A_GEN_CODE_TOK_EMBD "a.gen.code.tok_embd.%s"
// fish_s2 codec encoder, the transformer layers use the generic a.blk.* names
#define TN_A_CODEC_ENC_ENTRY "a.codec.enc.entry.%s"
#define TN_A_CODEC_ENC_SNAKE "a.codec.enc.blk.%d.snake.%s"
#define TN_A_CODEC_ENC_CONV "a.codec.enc.blk.%d.conv.%s"
#define TN_A_CODEC_ENC_RES_ACT1 "a.codec.enc.blk.%d.res.%d.act1.%s"
#define TN_A_CODEC_ENC_RES_CONV1 "a.codec.enc.blk.%d.res.%d.conv1.%s"
#define TN_A_CODEC_ENC_RES_ACT2 "a.codec.enc.blk.%d.res.%d.act2.%s"
#define TN_A_CODEC_ENC_RES_CONV2 "a.codec.enc.blk.%d.res.%d.conv2.%s"
#define TN_A_CODEC_ENC_POST_SNAKE "a.codec.enc.post_snake.%s"
#define TN_A_CODEC_ENC_POST_CONV "a.codec.enc.post_conv.%s"
#define TN_A_CODEC_DOWN_CONV "a.codec.down.blk.%d.conv.%s"
#define TN_A_CODEC_DOWN_DWCONV "a.codec.down.blk.%d.dwconv.%s"
#define TN_A_CODEC_DOWN_NORM "a.codec.down.blk.%d.norm.%s"
#define TN_A_CODEC_DOWN_PW1 "a.codec.down.blk.%d.pw1.%s"
#define TN_A_CODEC_DOWN_PW2 "a.codec.down.blk.%d.pw2.%s"
#define TN_A_CODEC_DOWN_GAMMA "a.codec.down.blk.%d.gamma"
#define TN_A_CODEC_PRE_NORM "a.codec.pre_norm.%s"
#define TN_A_CODEC_VQ_FIRST_IN "a.codec.vq.first.in_proj.%s"
#define TN_A_CODEC_VQ_FIRST_CB "a.codec.vq.first.codebook.%s"
#define TN_A_CODEC_VQ_REST_IN "a.codec.vq.rest.in_proj.%s"
#define TN_A_CODEC_VQ_REST_CB "a.codec.vq.rest.codebook.%s"
// qwen3tts code2wav (RVQ codes -> raw PCM)
// pre_transformer layers use the generic TN_ATTN_*/TN_FFN_*/TN_LN_*/TN_LS_* macros, prefix "a.gen.wav.tfm"
#define TN_A_GEN_WAV_QUANT_FIRST_IN "a.gen.wav.quant.first.in_proj.%s"
@@ -588,6 +613,9 @@ enum projector_type {
PROJECTOR_TYPE_DRAMA3_GEN,
PROJECTOR_TYPE_ZONOS3_GEN,
PROJECTOR_TYPE_ZONOS3_SPKENC,
PROJECTOR_TYPE_YUE2_GEN,
PROJECTOR_TYPE_FISH_S2_GEN,
PROJECTOR_TYPE_FISH_S2_SPKENC,
PROJECTOR_TYPE_MUSE_GLIMMER,
PROJECTOR_TYPE_SENSENOVA_U1,
PROJECTOR_TYPE_COHERE2V,
@@ -680,6 +708,9 @@ static std::map<projector_type, std::string> PROJECTOR_TYPE_NAMES = {
{ PROJECTOR_TYPE_DRAMA3_GEN, "drama3_gen"},
{ PROJECTOR_TYPE_ZONOS3_GEN, "zonos3_gen"},
{ PROJECTOR_TYPE_ZONOS3_SPKENC, "zonos3_spkenc"},
{ PROJECTOR_TYPE_YUE2_GEN, "yue2_gen"},
{ PROJECTOR_TYPE_FISH_S2_GEN, "fish_s2_gen"},
{ PROJECTOR_TYPE_FISH_S2_SPKENC, "fish_s2_spkenc"},
{ PROJECTOR_TYPE_MUSE_GLIMMER, "muse-glimmer"},
{ PROJECTOR_TYPE_SENSENOVA_U1, "sensenova_u1"},
{ PROJECTOR_TYPE_COHERE2V, "cohere2v"},
+183 -3
View File
@@ -1093,8 +1093,16 @@ struct mtmd_model_loader {
{
hparams.audio_sample_rate = 44100;
} break;
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_YUE2_GEN:
{
hparams.audio_sample_rate = 48000;
} break;
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_FISH_S2_GEN:
{
// fish_s2 fast AR, drama3 hardcodes its tower RoPE in the graph
hparams.rope_theta = 1000000.0f;
// Fish DAC decoder, codec.pth has no config
hparams.audio_sample_rate = 44100;
hparams.wav_tfm_n_layer = 8;
@@ -1125,6 +1133,19 @@ struct mtmd_model_loader {
hparams.flow_n_step = 1;
hparams.gen_eos_threshold = -4.0f;
} break;
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
// Fish DAC encoder, codec.pth has no config
// one row per 2048 samples: encoder strides, then the quantizer downsample
hparams.audio_sample_rate = 44100;
hparams.audio_patch_sizes = { 2, 4, 8, 8, 2, 2 };
// whole frames, about 10 s like a typical reference clip
hparams.warmup_audio_size = hparams.audio_samples_per_token() * 215;
hparams.wav_tfm_n_head = 16;
hparams.wav_tfm_n_head_kv = 16;
hparams.wav_tfm_eps = 1e-5f;
hparams.wav_tfm_rope_theta = 10000.0f;
} break;
case PROJECTOR_TYPE_CHATTERBOX:
{
// the classic cfm solver unrolls up to 10 steps x 2 cfg estimator evaluations in one graph
@@ -1372,7 +1393,8 @@ struct mtmd_model_loader {
const bool fft_based = model.proj_type != PROJECTOR_TYPE_GEMMA4UA &&
model.proj_type != PROJECTOR_TYPE_MUSE_SPARK_AUDIO &&
model.proj_type != PROJECTOR_TYPE_VIBEVOICE_ASR &&
model.proj_type != PROJECTOR_TYPE_POCKETTTS_SPKENC;
model.proj_type != PROJECTOR_TYPE_POCKETTTS_SPKENC &&
model.proj_type != PROJECTOR_TYPE_FISH_S2_SPKENC;
// Validate audio hparams loaded from GGUF metadata
if (hparams.n_mel_bins <= 0 || (fft_based && hparams.n_mel_bins > 256)) {
@@ -1547,7 +1569,8 @@ struct mtmd_model_loader {
model.proj_type != PROJECTOR_TYPE_MUSE_SPARK_AUDIO &&
model.proj_type != PROJECTOR_TYPE_VIBEVOICE_ASR &&
model.proj_type != PROJECTOR_TYPE_CHATTERBOX &&
model.proj_type != PROJECTOR_TYPE_CHATTERBOX_SPKENC);
model.proj_type != PROJECTOR_TYPE_CHATTERBOX_SPKENC &&
model.proj_type != PROJECTOR_TYPE_YUE2_GEN);
// layers
const int n_layers_to_load = has_standard_layers ? hparams.n_layer : 0;
@@ -2526,6 +2549,71 @@ struct mtmd_model_loader {
model.mm_fc_w = get_tensor(mtmd_format(TN_MM_AUDIO_FC, "weight"));
model.mm_fc_b = get_tensor(mtmd_format(TN_MM_AUDIO_FC, "bias"));
} break;
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
// codec encoder in the code2wav structs: dac = encoder blocks, upsample = quantizer downsample
// generic a.blk.* layers: 0..3 last encoder block, 4..11 quantizer pre_module
auto & c2w = model.c2w;
c2w.dac_entry_w = get_tensor(mtmd_format(TN_A_CODEC_ENC_ENTRY, "weight"));
c2w.dac_entry_b = get_tensor(mtmd_format(TN_A_CODEC_ENC_ENTRY, "bias"));
c2w.dac.resize(4);
for (int il = 0; il < (int) c2w.dac.size(); il++) {
auto & blk = c2w.dac[il];
blk.snake_alpha = get_tensor(mtmd_format(TN_A_CODEC_ENC_SNAKE, il, "alpha"));
blk.snake_beta = get_tensor(mtmd_format(TN_A_CODEC_ENC_SNAKE, il, "beta"));
blk.conv_w = get_tensor(mtmd_format(TN_A_CODEC_ENC_CONV, il, "weight"));
blk.conv_b = get_tensor(mtmd_format(TN_A_CODEC_ENC_CONV, il, "bias"));
blk.res.resize(3);
for (int ir = 0; ir < (int) blk.res.size(); ir++) {
auto & res = blk.res[ir];
res.act1_alpha = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_ACT1, il, ir, "alpha"));
res.act1_beta = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_ACT1, il, ir, "beta"));
res.conv1_w = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_CONV1, il, ir, "weight"));
res.conv1_b = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_CONV1, il, ir, "bias"));
res.act2_alpha = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_ACT2, il, ir, "alpha"));
res.act2_beta = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_ACT2, il, ir, "beta"));
res.conv2_w = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_CONV2, il, ir, "weight"));
res.conv2_b = get_tensor(mtmd_format(TN_A_CODEC_ENC_RES_CONV2, il, ir, "bias"));
}
}
c2w.dac_post_snake_alpha = get_tensor(mtmd_format(TN_A_CODEC_ENC_POST_SNAKE, "alpha"));
c2w.dac_post_snake_beta = get_tensor(mtmd_format(TN_A_CODEC_ENC_POST_SNAKE, "beta"));
c2w.dac_post_conv_w = get_tensor(mtmd_format(TN_A_CODEC_ENC_POST_CONV, "weight"));
c2w.dac_post_conv_b = get_tensor(mtmd_format(TN_A_CODEC_ENC_POST_CONV, "bias"));
c2w.upsample.resize(2);
for (int il = 0; il < (int) c2w.upsample.size(); il++) {
auto & down = c2w.upsample[il];
down.conv_w = get_tensor(mtmd_format(TN_A_CODEC_DOWN_CONV, il, "weight"));
down.conv_b = get_tensor(mtmd_format(TN_A_CODEC_DOWN_CONV, il, "bias"));
down.dwconv_w = get_tensor(mtmd_format(TN_A_CODEC_DOWN_DWCONV, il, "weight"));
down.dwconv_b = get_tensor(mtmd_format(TN_A_CODEC_DOWN_DWCONV, il, "bias"));
down.norm_w = get_tensor(mtmd_format(TN_A_CODEC_DOWN_NORM, il, "weight"));
down.norm_b = get_tensor(mtmd_format(TN_A_CODEC_DOWN_NORM, il, "bias"));
down.pw1_w = get_tensor(mtmd_format(TN_A_CODEC_DOWN_PW1, il, "weight"));
down.pw1_b = get_tensor(mtmd_format(TN_A_CODEC_DOWN_PW1, il, "bias"));
down.pw2_w = get_tensor(mtmd_format(TN_A_CODEC_DOWN_PW2, il, "weight"));
down.pw2_b = get_tensor(mtmd_format(TN_A_CODEC_DOWN_PW2, il, "bias"));
down.gamma = get_tensor(mtmd_format(TN_A_CODEC_DOWN_GAMMA, il));
}
c2w.tfm_output_norm_w = get_tensor(mtmd_format(TN_A_CODEC_PRE_NORM, "weight"));
if (hparams.n_layer != mtmd_graph_fish_s2_spkenc::n_tfm_layer) {
throw std::runtime_error(mtmd_format("%s: fish_s2_spkenc expects %d transformer layers\n", __func__, mtmd_graph_fish_s2_spkenc::n_tfm_layer));
}
// VQ: search with the normalized codebooks, subtract the codec decoder's out_proj tables
c2w.quant_first_in_w = get_tensor(mtmd_format(TN_A_CODEC_VQ_FIRST_IN, "weight"));
c2w.quant_first_in_b = get_tensor(mtmd_format(TN_A_CODEC_VQ_FIRST_IN, "bias"));
c2w.quant_first_cb_norm_w = get_tensor(mtmd_format(TN_A_CODEC_VQ_FIRST_CB, "weight"));
c2w.quant_rest_in_w = get_tensor(mtmd_format(TN_A_CODEC_VQ_REST_IN, "weight"));
c2w.quant_rest_in_b = get_tensor(mtmd_format(TN_A_CODEC_VQ_REST_IN, "bias"));
c2w.quant_rest_cb_norm_w = get_tensor(mtmd_format(TN_A_CODEC_VQ_REST_CB, "weight"));
c2w.quant_first_cb_w = get_tensor(mtmd_format(TN_A_GEN_WAV_QUANT_FIRST_CB, "weight"));
c2w.quant_rest_cb_w = get_tensor(mtmd_format(TN_A_GEN_WAV_QUANT_REST_CB, "weight"));
// codes -> backbone rows, same tables as fish_s2_gen
model.gen_code_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_EMBD, "weight"));
model.gen_code_out_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_OUT_EMBD, "weight"));
} break;
case PROJECTOR_TYPE_POCKETTTS_SPKENC:
{
load_seanet(model.seanet, false);
@@ -2809,6 +2897,73 @@ struct mtmd_model_loader {
z.rs_new = rs->ne[1];
z.rs_kernel_len = rs->ne[0];
} break;
case PROJECTOR_TYPE_YUE2_GEN:
{
auto & y = model.yue2;
y.inp_w = get_tensor("a.gen.nar.inp.weight");
y.inp_b = get_tensor("a.gen.nar.inp.bias");
y.out_w = get_tensor("a.gen.nar.out.weight");
y.out_b = get_tensor("a.gen.nar.out.bias");
if (y.inp_w->ne[0] != mtmd_graph_yue2_gen::n_latent || y.out_w->ne[1] != mtmd_graph_yue2_gen::n_latent ||
y.inp_w->ne[1] != hparams.n_embd || y.out_w->ne[0] != hparams.n_embd) {
throw std::runtime_error(mtmd_format("%s: yue2: unexpected latent projection shape\n", __func__));
}
auto read_host = [&](const std::string & name, std::vector<float> & dst, int64_t ne0, int64_t ne1) {
const ggml_tensor * t = ggml_get_tensor(ctx_meta.get(), name.c_str());
if (!t || t->type != GGML_TYPE_F32 || t->ne[0] != ne0 || t->ne[1] != ne1 || ggml_n_dims(t) > 2) {
throw std::runtime_error(mtmd_format("%s: invalid or missing tensor %s\n", __func__, name.c_str()));
}
dst.resize(ggml_nelements(t));
fin.seekg(tensor_offset.at(name), std::ios::beg);
if (!fin.read(reinterpret_cast<char *>(dst.data()), ggml_nbytes(t))) {
throw std::runtime_error(mtmd_format("%s: failed to read %s\n", __func__, name.c_str()));
}
};
const int64_t n_embd = hparams.n_embd;
read_host("a.gen.nar.time.0.weight", y.time0_w, mtmd_graph_yue2_gen::n_freq, n_embd);
read_host("a.gen.nar.time.0.bias", y.time0_b, n_embd, 1);
read_host("a.gen.nar.time.2.weight", y.time2_w, n_embd, n_embd);
read_host("a.gen.nar.time.2.bias", y.time2_b, n_embd, 1);
const std::string p = "a.gen.vae";
auto load_snake = [&](const std::string & q) {
return mtmd_yue2::snake{ get_tensor(q + ".alpha"), get_tensor(q + ".beta") };
};
y.vae_in_w = get_tensor(p + ".conv_in.weight");
y.vae_in_b = get_tensor(p + ".conv_in.bias");
for (int b = 0; gguf_find_tensor(ctx_gguf.get(), mtmd_format("%s.blk.%d.up.weight", p.c_str(), b).c_str()) >= 0; b++) {
const std::string q = p + ".blk." + std::to_string(b);
mtmd_yue2::block blk;
blk.act = load_snake(q + ".snake");
blk.up_w = get_tensor(q + ".up.weight");
blk.up_b = get_tensor(q + ".up.bias");
for (int r = 0; r < 3; r++) {
const std::string rq = q + ".res." + std::to_string(r);
mtmd_yue2::res_unit ru;
ru.act1 = load_snake(rq + ".snake1");
ru.conv1_w = get_tensor(rq + ".conv1.weight");
ru.conv1_b = get_tensor(rq + ".conv1.bias");
ru.act2 = load_snake(rq + ".snake2");
ru.conv2_w = get_tensor(rq + ".conv2.weight");
ru.conv2_b = get_tensor(rq + ".conv2.bias");
blk.res.push_back(ru);
}
y.vae.push_back(std::move(blk));
}
y.vae_post = load_snake(p + ".post_snake");
y.vae_out_w = get_tensor(p + ".conv_out.weight");
// the tile cropping in mtmd_gen_wav_yue2() depends on the total stride
int64_t ratio = 1;
for (const auto & blk : y.vae) {
ratio *= blk.up_w->ne[0] / 2;
}
if (ratio != mtmd_graph_yue2_gen::vae_ratio || y.vae_in_w->ne[1] != mtmd_graph_yue2_gen::n_latent ||
y.vae_out_w->ne[2] != mtmd_graph_yue2_gen::n_channels) {
throw std::runtime_error(mtmd_format("%s: yue2: unexpected VAE decoder layout\n", __func__));
}
} break;
case PROJECTOR_TYPE_DRAMA3_GEN:
{
// generic a.gen.code.blk.* tensors are loaded by the loop above
@@ -2827,6 +2982,20 @@ struct mtmd_model_loader {
model.gen_code_out_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_OUT_EMBD, "weight")); // backbone rows of the semantic band
load_code2wav(false);
} break;
case PROJECTOR_TYPE_FISH_S2_GEN:
{
// generic a.gen.code.blk.* tensors are loaded by the loop above
model.gen_code_tok_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_TOK_EMBD, "weight")); // fast AR input
model.gen_code_head_w = get_tensor(mtmd_format(TN_A_GEN_CODE_HEAD, "weight")); // fast AR head, residual codebooks
model.gen_code_norm_w = get_tensor(mtmd_format(TN_A_GEN_CODE_NORM, "weight"));
model.gen_code_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_EMBD, "weight")); // merged 3D, backbone input tables
model.gen_code_out_embd_w = get_tensor(mtmd_format(TN_A_GEN_CODE_OUT_EMBD, "weight")); // backbone rows of the semantic band
load_code2wav(false);
// the graph reads model.layers[0]
if (hparams.n_layer < 1) {
throw std::runtime_error(mtmd_format("%s: fish_s2_gen has no fast AR layer\n", __func__));
}
} break;
case PROJECTOR_TYPE_CHATTERBOX:
{
auto & c = model.cbx;
@@ -4440,10 +4609,17 @@ int mtmd_model::n_output_tokens(const mtmd_image_f32 * img) const {
case PROJECTOR_TYPE_LFM3A_GEN:
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_ZONOS3_GEN:
case PROJECTOR_TYPE_YUE2_GEN:
case PROJECTOR_TYPE_FISH_S2_GEN:
{
// one hidden-state vector fed back to the talker per call
n_patches = 1;
} break;
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
// one backbone row per codec frame, the preprocessor pads to whole frames
n_patches = (int) (img->nx() / hparams.audio_samples_per_token());
} break;
case PROJECTOR_TYPE_POCKETTTS_SPKENC:
{
// one conditioning row per 12.5Hz frame
@@ -4630,7 +4806,11 @@ int mtmd_model::n_mmproj_embd() const {
case PROJECTOR_TYPE_LFM3A_GEN:
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_ZONOS3_GEN:
case PROJECTOR_TYPE_FISH_S2_GEN:
case PROJECTOR_TYPE_FISH_S2_SPKENC:
return gen_code_out_embd_w->ne[0];
case PROJECTOR_TYPE_YUE2_GEN:
return yue2.inp_w->ne[1];
case PROJECTOR_TYPE_POCKETTTS_SPKENC:
return spk_proj_w->ne[1];
case PROJECTOR_TYPE_POCKETTTS_GEN:
+50
View File
@@ -213,6 +213,8 @@ struct mtmd_hparams {
// pocket-tts
static constexpr int32_t pockettts_max_spk_seconds = 30;
// fish_s2: the codec encoder runs at 44.1 kHz, memory grows with the clip
static constexpr int32_t fish_s2_max_spk_seconds = 30;
int32_t seanet_n_stage = 0;
std::vector<int32_t> seanet_ratios; // encoder order (reversed compared to the config)
int32_t mimi_downsample = 0; // encoder frame rate / model frame rate
@@ -622,6 +624,12 @@ struct mtmd_code2wav {
ggml_tensor * quant_rest_out_w = nullptr;
ggml_tensor * quant_rest_cb_w = nullptr; // codebooks, merged 3D [15, vocab, dim]
// fish_s2 codec encoder: in_proj bias and l2-normalized codebooks for the nearest-entry search
ggml_tensor * quant_first_in_b = nullptr;
ggml_tensor * quant_rest_in_b = nullptr;
ggml_tensor * quant_first_cb_norm_w = nullptr;
ggml_tensor * quant_rest_cb_norm_w = nullptr;
ggml_tensor * pre_conv_w = nullptr;
ggml_tensor * pre_conv_b = nullptr;
@@ -935,6 +943,46 @@ struct mtmd_zonos3_wav {
int64_t rs_kernel_len = 0;
};
// yue2: projections of the flow-matching NAR (a.gen.nar.*) and the Oobleck VAE decoder (a.gen.vae.*)
// the NAR transformer layers are loaded by libllama, see mtmd_yue2_nar_load()
struct mtmd_yue2 {
struct snake {
ggml_tensor * alpha = nullptr; // exp(alpha)
ggml_tensor * beta = nullptr; // 1 / (exp(beta) + eps)
};
struct res_unit {
snake act1;
ggml_tensor * conv1_w = nullptr; // k = 7, dilated
ggml_tensor * conv1_b = nullptr;
snake act2;
ggml_tensor * conv2_w = nullptr; // k = 1
ggml_tensor * conv2_b = nullptr;
};
struct block {
snake act;
ggml_tensor * up_w = nullptr; // ConvTranspose1d, k = 2 * stride
ggml_tensor * up_b = nullptr;
std::vector<res_unit> res;
};
ggml_tensor * inp_w = nullptr; // latent -> backbone embd
ggml_tensor * inp_b = nullptr;
ggml_tensor * out_w = nullptr; // backbone hidden -> velocity
ggml_tensor * out_b = nullptr;
// time embedder MLP, read on the host
std::vector<float> time0_w; // [256, n_embd]
std::vector<float> time0_b;
std::vector<float> time2_w; // [n_embd, n_embd]
std::vector<float> time2_b;
ggml_tensor * vae_in_w = nullptr;
ggml_tensor * vae_in_b = nullptr;
std::vector<block> vae;
snake vae_post;
ggml_tensor * vae_out_w = nullptr;
};
struct mtmd_vocos {
ggml_tensor * input_w = nullptr;
ggml_tensor * input_b = nullptr;
@@ -1208,6 +1256,7 @@ struct mtmd_model {
ggml_tensor * gen_code_head_w = nullptr; // per-codebook output head, merged 3D
ggml_tensor * gen_code_out_embd_w = nullptr; // codebook-0 embedding, fed back into the talker
ggml_tensor * gen_code_norm_w = nullptr; // final norm
ggml_tensor * gen_code_tok_embd_w = nullptr; // fish_s2: input embedding shared by all codebooks
// lfm3a depthformer (MaskGIT)
ggml_tensor * gen_code_head_norm_w = nullptr; // per-codebook norm before the tied head, merged 2D
@@ -1217,6 +1266,7 @@ struct mtmd_model {
mtmd_code2wav c2w;
std::vector<mtmd_drama3_layer> drama3_layers;
mtmd_zonos3_wav zonos3;
mtmd_yue2 yue2;
// pocket-tts: SEANet stack, shared by the encoder (speaker path) and the decoder (gen path)
mtmd_seanet seanet;
+13 -2
View File
@@ -389,6 +389,7 @@ bool mtmd_subcontext::encode_batch(int n_threads, const mtmd_image_f32_batch * i
static std::vector<c2w_state_slot> list_gen_state_slots(const mtmd_hparams & hparams, const mtmd_model & model) {
switch (model.proj_type) {
case PROJECTOR_TYPE_QWEN3TTS_GEN: return list_c2w_state_slots(hparams, model);
case PROJECTOR_TYPE_FISH_S2_GEN: return list_c2w_state_slots(hparams, model);
case PROJECTOR_TYPE_DRAMA3_GEN: return list_drama3_state_slots(hparams, model);
case PROJECTOR_TYPE_POCKETTTS_GEN: return list_pockettts_state_slots(hparams, model);
case PROJECTOR_TYPE_LFM3A_GEN: return list_lfm3a_gen_state_slots(hparams, model);
@@ -1505,6 +1506,7 @@ bool mtmd_subcontext::encode(mtmd_subctx_encode_params * params) {
}
} break;
case PROJECTOR_TYPE_SOPRANO:
case PROJECTOR_TYPE_YUE2_GEN:
{
set_input_f32("inp_feats", *params->feats);
} break;
@@ -1680,6 +1682,7 @@ bool mtmd_subcontext::encode(mtmd_subctx_encode_params * params) {
case PROJECTOR_TYPE_MUSE_SPARK_AUDIO:
case PROJECTOR_TYPE_QWEN3TTS_SPKENC:
case PROJECTOR_TYPE_ZONOS3_SPKENC:
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
// do nothing
} break;
@@ -1789,6 +1792,7 @@ bool mtmd_subcontext::encode(mtmd_subctx_encode_params * params) {
}
} break;
case PROJECTOR_TYPE_DRAMA3_GEN:
case PROJECTOR_TYPE_FISH_S2_GEN:
{
const int64_t n_cb = model.gen_code_embd_w->ne[2];
const int64_t n_vocab = model.gen_code_embd_w->ne[1];
@@ -1807,6 +1811,13 @@ bool mtmd_subcontext::encode(mtmd_subctx_encode_params * params) {
}
set_input_i32("inp_codes", codes);
set_gen_state_in();
} else if (model.proj_type == PROJECTOR_TYPE_FISH_S2_GEN) {
if (params->code0 < 0 || params->code0 >= n_vocab) {
LOG_ERR("%s: code0 out of range (%d, vocab %lld)\n", __func__, params->code0, (long long) n_vocab);
return false;
}
std::vector<int32_t> code0 = { params->code0 };
set_input_i32("inp_code0", code0);
} else {
// code0 < 0: next row is text (prompt), the tower only fills its KV
// codes: input frame codes for a speech row, empty for a text row
@@ -2622,8 +2633,8 @@ bool mtmd_subcontext::encode(mtmd_subctx_encode_params * params) {
mtmd_graph_soprano::decode_spectrum(model, spectrum, out_audio);
}
// drop the tail audio that comes from the code-0 rear padding, drama3 has no padding
if (model.proj_type != PROJECTOR_TYPE_DRAMA3_GEN) {
// drop the tail audio that comes from the code-0 rear padding, drama3, fish_s2 and yue2 have no padding
if (model.proj_type != PROJECTOR_TYPE_DRAMA3_GEN && model.proj_type != PROJECTOR_TYPE_FISH_S2_GEN && model.proj_type != PROJECTOR_TYPE_YUE2_GEN) {
const int64_t n_codes = params->codes ? model.gen_code_head_w->ne[2] + 1 : 0;
const int64_t n_frames_w = hparams.wav_tfm_swa;
const int64_t n_frames = params->codes ? (int64_t) params->codes->size() / n_codes : n_frames_w;
+2
View File
@@ -69,6 +69,8 @@ enum mtmd_subctx_gen_process {
MTMD_SUBCTX_GEN_PROCESS_TTS, // full utterance of codes to raw PCM audio
MTMD_SUBCTX_GEN_PROCESS_TTS_VOCODE, // internal: mel + source stft to istft input
MTMD_SUBCTX_GEN_PROCESS_TOKENIZE, // raw PCM audio to semantic speech tokens
MTMD_SUBCTX_GEN_PROCESS_LATENT_IN, // internal: flow latent rows to backbone embd rows
MTMD_SUBCTX_GEN_PROCESS_LATENT_OUT, // internal: backbone hidden rows to flow velocity rows
};
// persistent state of a causal audio encoder, for streaming input (one per stream)
+338
View File
@@ -8,6 +8,7 @@
#include "debug/mtmd-debug.h"
#include "llama.h"
#include "llama-cpp.h"
#include "../src/llama-ext.h"
// fix problem with std::min and std::max
@@ -613,6 +614,103 @@ static bool cbx_is_mtl(const mtmd_model & model) {
return model.cbx_read_tensor("a.cenc.perceiver.pre_attention_query", nullptr, 0) > 0;
}
// yue2: the NAR layers are a qwen3 text model, libllama loads them from a synthetic GGUF
// the tensor data is read from the mmproj file, mtmd keeps no copy
struct yue2_nar_source {
std::ifstream f;
std::map<std::string, std::pair<size_t, size_t>> tensors; // name in the synthetic model -> offset and size in the mmproj
};
static void yue2_nar_set_tensor_data(ggml_tensor * t, void * user_data) {
auto * src = (yue2_nar_source *) user_data;
std::vector<uint8_t> buf(ggml_nbytes(t), 0);
const std::string name = t->name;
auto it = src->tensors.find(name);
if (it != src->tensors.end()) {
if (it->second.second != buf.size()) {
throw std::runtime_error(mtmd_format("yue2: unexpected size of %s", t->name));
}
src->f.seekg((std::streamoff) it->second.first, std::ios::beg);
if (!src->f.read((char *) buf.data(), (std::streamsize) buf.size())) {
throw std::runtime_error(mtmd_format("yue2: failed to read %s", t->name));
}
} else {
// optional tensors that libllama creates by itself, give them neutral values (scale = 1, dense_2 = identity, others = 0)
GGML_ASSERT(t->type == GGML_TYPE_F32);
float * w = (float *) buf.data();
if (name.size() >= 5 && name.compare(name.size() - 5, 5, "scale") == 0) {
std::fill(w, w + ggml_nelements(t), 1.0f);
} else if (name == "dense_2.weight") {
GGML_ASSERT(t->ne[0] == t->ne[1]);
for (int64_t i = 0; i < t->ne[0]; i++) {
w[i * t->ne[0] + i] = 1.0f;
}
}
}
ggml_backend_tensor_set(t, buf.data(), 0, buf.size());
}
static llama_model * mtmd_yue2_nar_load(const char * fname, ggml_backend_dev_t device, const mtmd_hparams & hparams) {
ggml_context * meta = nullptr;
gguf_init_params gparams = { /*no_alloc =*/ true, /*ctx =*/ &meta };
gguf_context_ptr gsrc(gguf_init_from_file(fname, gparams));
ggml_context_ptr meta_ptr(meta);
if (!gsrc) {
throw std::runtime_error(mtmd_format("yue2: failed to read %s", fname));
}
const llm_arch arch = LLM_ARCH_QWEN3;
gguf_context_ptr gdst(gguf_init_empty());
auto key = [&](llm_kv kv) { return llama_arch_kv_name(arch, kv); };
gguf_set_val_str(gdst.get(), key(LLM_KV_GENERAL_ARCHITECTURE).c_str(), llama_arch_name(arch));
gguf_set_val_str(gdst.get(), key(LLM_KV_TOKENIZER_MODEL).c_str(), "none");
gguf_set_val_u32(gdst.get(), key(LLM_KV_VOCAB_SIZE).c_str(), 1);
gguf_set_val_u32(gdst.get(), key(LLM_KV_CONTEXT_LENGTH).c_str(), mtmd_graph_yue2_gen::n_ctx_max);
gguf_set_val_u32(gdst.get(), key(LLM_KV_EMBEDDING_LENGTH).c_str(), hparams.n_embd);
gguf_set_val_u32(gdst.get(), key(LLM_KV_FEED_FORWARD_LENGTH).c_str(), hparams.n_ff);
gguf_set_val_u32(gdst.get(), key(LLM_KV_BLOCK_COUNT).c_str(), hparams.n_layer);
gguf_set_val_u32(gdst.get(), key(LLM_KV_ATTENTION_HEAD_COUNT).c_str(), hparams.n_head);
gguf_set_val_u32(gdst.get(), key(LLM_KV_ATTENTION_HEAD_COUNT_KV).c_str(), hparams.n_head_kv);
gguf_set_val_f32(gdst.get(), key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS).c_str(), hparams.eps);
gguf_set_val_f32(gdst.get(), key(LLM_KV_ROPE_FREQ_BASE).c_str(), mtmd_graph_yue2_gen::rope_theta);
yue2_nar_source src;
const std::string prefix = "a.gen.nar.";
for (int64_t i = 0; i < gguf_get_n_tensors(gsrc.get()); i++) {
const std::string name = gguf_get_tensor_name(gsrc.get(), i);
if (name.rfind(prefix + "blk.", 0) != 0 && name != prefix + "output_norm.weight") {
continue;
}
// the synthetic model only takes the type from here, the shapes come from the hparams
ggml_tensor t = *ggml_get_tensor(meta, name.c_str());
ggml_set_name(&t, name.substr(prefix.size()).c_str());
gguf_add_tensor(gdst.get(), &t);
src.tensors[t.name] = { gguf_get_data_offset(gsrc.get()) + gguf_get_tensor_offset(gsrc.get(), i), ggml_nbytes(&t) };
}
if (src.tensors.size() != (size_t) hparams.n_layer * 11 + 1) {
throw std::runtime_error("yue2: unexpected NAR tensors in the mmproj");
}
src.f = open_ifstream_binary(fname);
if (!src.f) {
throw std::runtime_error(mtmd_format("yue2: failed to open %s", fname));
}
llama_model_params mparams = llama_model_default_params();
std::vector<ggml_backend_dev_t> devs;
if (device && ggml_backend_dev_type(device) != GGML_BACKEND_DEVICE_TYPE_CPU) {
devs = { device, nullptr };
mparams.devices = devs.data();
mparams.n_gpu_layers = -1;
} else {
mparams.n_gpu_layers = 0;
}
llama_model * model = llama_model_init_from_user(gdst.get(), yue2_nar_set_tensor_data, &src, mparams);
if (!model) {
throw std::runtime_error("yue2: failed to load the NAR layers");
}
return model;
}
struct mtmd_context {
// must outlive the subcontexts below
mtmd_models models;
@@ -628,6 +726,10 @@ struct mtmd_context {
// generation context
std::unique_ptr<mtmd_subcontext> ctx_gen_a; // audio
// yue2: the NAR layers, run on a llama_context of their own, see mtmd_gen_flow_yue2()
llama_model_ptr yue2_nar;
llama_flash_attn_type yue2_flash_attn = LLAMA_FLASH_ATTN_TYPE_AUTO;
// gen-side reference data of the last encoded chunk (chatterbox), see mtmd_gen_state_set_ref()
mtmd_gen_ref gen_ref;
@@ -761,6 +863,10 @@ struct mtmd_context {
if (models.gen_a) {
ctx_gen_a = std::make_unique<mtmd_subcontext>(*models.gen_a, *backends, sctx_params);
}
if (models.gen_a && models.gen_a->proj_type == PROJECTOR_TYPE_YUE2_GEN && !no_alloc) {
yue2_nar.reset(mtmd_yue2_nar_load(mmproj_fname, device, models.gen_a->hparams));
yue2_flash_attn = ctx_params.flash_attn_type;
}
for (size_t i = 0; i < backends->ptrs.size(); ++i) {
const size_t size = ggml_backend_sched_get_buffer_size(backends->sched.get(), backends->ptrs[i]);
if (size > 1) {
@@ -1314,6 +1420,10 @@ struct mtmd_context {
{
audio_preproc = std::make_unique<mtmd_audio_preprocessor_pockettts>(models.a.get());
} break;
case PROJECTOR_TYPE_FISH_S2_SPKENC:
{
audio_preproc = std::make_unique<mtmd_audio_preprocessor_fish_s2>(models.a.get());
} break;
case PROJECTOR_TYPE_CHATTERBOX_SPKENC:
{
audio_preproc = std::make_unique<mtmd_audio_preprocessor_chatterbox_ref>(models.a.get(), cbx_is_mtl(*models.a));
@@ -2455,6 +2565,7 @@ float * mtmd_get_output_embd(mtmd_context * ctx) {
mtmd_gen_audio_info mtmd_gen_audio_get_info(const mtmd_context * ctx) {
mtmd_gen_audio_info info{};
info.model_variant = "";
info.n_channels = 1;
if (!ctx->ctx_gen_a) {
info.type = MTMD_GEN_AUDIO_TYPE_NONE;
return info;
@@ -2490,6 +2601,15 @@ mtmd_gen_audio_info mtmd_gen_audio_get_info(const mtmd_context * ctx) {
info.type = MTMD_GEN_AUDIO_TYPE_ZONOS3;
info.sample_rate = 44100;
break;
case PROJECTOR_TYPE_YUE2_GEN:
info.type = MTMD_GEN_AUDIO_TYPE_YUE2;
info.sample_rate = 48000;
info.n_channels = mtmd_graph_yue2_gen::n_channels;
break;
case PROJECTOR_TYPE_FISH_S2_GEN:
info.type = MTMD_GEN_AUDIO_TYPE_FISH_S2;
info.sample_rate = 44100;
break;
default:
info.type = MTMD_GEN_AUDIO_TYPE_NONE;
break;
@@ -3113,6 +3233,219 @@ static int32_t mtmd_gen_wav_lfm3a(mtmd_context * ctx, mtmd_gen_state * state, co
return 0;
}
// yue2: one pass of an mtmd graph, rows in -> rows out (PCM for GEN_WAV)
static bool yue2_encode(mtmd_context * ctx, mtmd_gen_state * state, mtmd_subctx_gen_process proc, const std::vector<float> & in, std::vector<float> & out) {
mtmd_image_f32 dummy;
dummy.set_size({1, 1}, false, true);
dummy.cpy_buf(std::vector<float>(1, 0.0f));
mtmd_image_f32_batch batch;
batch.is_audio = true;
batch.entries.push_back(std::move(dummy));
mtmd_subctx_encode_params params;
params.imgs = &batch;
params.n_threads = ctx->n_threads;
params.gen_process = proc;
params.feats = &in;
if (proc == MTMD_SUBCTX_GEN_PROCESS_GEN_WAV) {
params.out_audio = &out;
params.gen_state = state->gen_state;
} else {
params.out_feats = &out;
}
return ctx->ctx_gen_a->encode(&params);
}
// NAR context: the backbone keys/values, then the NAR rows of each pass
// the state only loads into the same KV layout, try both flash attention settings
static llama_context_ptr yue2_nar_context(mtmd_context * ctx, const mtmd_gen_inp * inp, int32_t n_rows) {
llama_context_params cparams = llama_context_default_params();
cparams.n_ctx = (uint32_t) (inp->backbone_n_past + n_rows);
cparams.n_batch = (uint32_t) n_rows;
cparams.n_ubatch = (uint32_t) n_rows; // non-causal, the rows of a pass must be in one ubatch
cparams.n_seq_max = 1;
cparams.n_threads = ctx->n_threads;
cparams.n_threads_batch = ctx->n_threads;
cparams.embeddings = true;
cparams.pooling_type = LLAMA_POOLING_TYPE_NONE;
cparams.attention_type = LLAMA_ATTENTION_TYPE_NON_CAUSAL;
cparams.no_perf = true;
std::vector<llama_flash_attn_type> fa_types = { ctx->yue2_flash_attn };
if (ctx->yue2_flash_attn == LLAMA_FLASH_ATTN_TYPE_AUTO) {
fa_types = { LLAMA_FLASH_ATTN_TYPE_ENABLED, LLAMA_FLASH_ATTN_TYPE_DISABLED };
}
for (llama_flash_attn_type fa : fa_types) {
cparams.flash_attn_type = fa;
llama_context_ptr lctx(llama_init_from_model(ctx->yue2_nar.get(), cparams));
if (!lctx) {
continue;
}
if (llama_state_seq_set_data(lctx.get(), inp->backbone_state, inp->backbone_state_size, 0) == 0) {
continue;
}
if (llama_memory_seq_pos_max(llama_get_memory(lctx.get()), 0) != inp->backbone_n_past - 1) {
LOG_ERR("%s: the backbone state does not hold %d positions\n", __func__, inp->backbone_n_past);
return nullptr;
}
return lctx;
}
LOG_ERR("%s: cannot load the backbone state, the backbone context must use an F16 KV cache\n", __func__);
return nullptr;
}
// yue2 GEN_CODE: flow matching from noise to the VAE latents of one chunk, see synthesize() in the reference (nar.py)
static int32_t mtmd_gen_flow_yue2(mtmd_context * ctx, mtmd_gen_state * state, const mtmd_gen_inp * inp, mtmd_gen_out * out) {
using g = mtmd_graph_yue2_gen;
const auto & y = ctx->ctx_gen_a->model.yue2;
const int64_t n_embd = ctx->ctx_gen_a->model.hparams.n_embd;
if (!ctx->yue2_nar || inp->n_codes == 0 || !inp->backbone_state || inp->backbone_n_past <= 0) {
LOG_ERR("%s: yue2 needs the frame count and the backbone state\n", __func__);
return 1;
}
const int64_t n_frames = (int64_t) inp->n_codes;
const int64_t n_rows = n_frames + 2; // latent start, frames, latent end
const int64_t n_past = inp->backbone_n_past;
if (n_past + n_rows > g::n_ctx_max) {
LOG_ERR("%s: song too long: %lld prompt positions + %lld frames, max %d positions\n", __func__,
(long long) n_past, (long long) n_frames, g::n_ctx_max);
return 1;
}
llama_context_ptr lctx = yue2_nar_context(ctx, inp, (int32_t) n_rows);
if (!lctx) {
return 1;
}
llama_memory_t mem = llama_get_memory(lctx.get());
// sinusoidal position of each NAR row (AudioPositionEmbedding), interleaved sin/cos
std::vector<float> pos_embd((size_t) (n_rows * n_embd));
for (int64_t i = 0; i < n_rows; i++) {
for (int64_t j = 0; j < n_embd / 2; j++) {
const double a = (double) i * std::exp((double) (2 * j) * -std::log(10000.0) / (double) n_embd);
pos_embd[i * n_embd + 2 * j] = (float) std::sin(a);
pos_embd[i * n_embd + 2 * j + 1] = (float) std::cos(a);
}
}
// TimestepEmbedder: [cos, sin] of t * freqs -> linear -> silu -> linear
auto time_embd = [&](double t) {
const double raw = std::clamp(std::log(t / (1.0 - t)), -20.0, 20.0);
const double ts = 1.0 / (1.0 + std::exp(-raw)); // timestep_shift = 1
const int64_t half = g::n_freq / 2;
std::vector<float> f(g::n_freq);
for (int64_t j = 0; j < half; j++) {
const double a = ts * std::exp(-std::log(10000.0) * (double) j / (double) half);
f[j] = (float) std::cos(a);
f[half + j] = (float) std::sin(a);
}
std::vector<float> h(y.time0_b), o(y.time2_b);
for (int64_t r = 0; r < n_embd; r++) {
for (int64_t c = 0; c < g::n_freq; c++) {
h[r] += y.time0_w[r * g::n_freq + c] * f[c];
}
h[r] = h[r] / (1.0f + std::exp(-h[r]));
}
for (int64_t r = 0; r < n_embd; r++) {
for (int64_t c = 0; c < n_embd; c++) {
o[r] += y.time2_w[r * n_embd + c] * h[c];
}
}
return o;
};
llama_batch_ext_ptr batch(llama_batch_ext_init(lctx.get()));
std::vector<float> x_rows((size_t) (n_rows * g::n_latent), 0.0f);
std::vector<float> embd, hidden, vel;
auto velocity = [&](const std::vector<float> & x, double t, std::vector<float> & v) -> bool {
std::copy(x.begin(), x.end(), x_rows.begin() + g::n_latent); // the start and end rows stay 0
if (!yue2_encode(ctx, state, MTMD_SUBCTX_GEN_PROCESS_LATENT_IN, x_rows, embd)) {
return false;
}
const std::vector<float> temb = time_embd(t);
llama_batch_ext_clear(batch.get());
for (int64_t i = 0; i < n_rows; i++) {
for (int64_t j = 0; j < n_embd; j++) {
embd[i * n_embd + j] = embd[i * n_embd + j] + temb[j] + pos_embd[i * n_embd + j];
}
const llama_embd row = { embd.data() + i * n_embd, 1, (size_t) n_embd };
const llama_pos p = (llama_pos) (n_past + i);
const int32_t idx = llama_batch_ext_add_embd(batch.get(), 0, row);
if (idx < 0) {
LOG_ERR("%s: failed to add a row to the batch (error %d)\n", __func__, idx);
return false;
}
llama_batch_ext_set_pos(batch.get(), idx, &p);
llama_batch_ext_set_output_embd(batch.get(), idx, true);
}
if (llama_process(lctx.get(), LLAMA_PROCESS_TYPE_DECODE, batch.get()) != 0) {
LOG_ERR("%s: NAR decode failed\n", __func__);
return false;
}
const float * h = llama_get_embeddings(lctx.get());
hidden.assign(h, h + n_rows * n_embd);
llama_memory_seq_rm(mem, 0, (llama_pos) n_past, -1);
if (!yue2_encode(ctx, state, MTMD_SUBCTX_GEN_PROCESS_LATENT_OUT, hidden, vel)) {
return false;
}
v.assign(vel.begin() + g::n_latent, vel.end() - g::n_latent);
return true;
};
std::normal_distribution<float> dist(0.0f, 1.0f);
std::vector<float> x((size_t) (n_frames * g::n_latent));
for (auto & v : x) {
v = dist(state->gen_state->rng);
}
// midpoint solver from t = 1 (noise) to t = 0
std::vector<float> v1, v2, mid(x.size());
const int n_steps = inp->n_steps > 0 ? inp->n_steps : g::ode_steps;
const double dt = 1.0 / n_steps;
bool ok = true;
for (int step = 0; step < n_steps && ok; step++) {
const double t = 1.0 - step * dt;
ok = velocity(x, t, v1);
for (size_t i = 0; ok && i < x.size(); i++) {
mid[i] = x[i] - v1[i] * (float) (dt / 2);
}
ok = ok && velocity(mid, t - dt / 2, v2);
for (size_t i = 0; ok && i < x.size(); i++) {
x[i] -= v2[i] * (float) dt;
}
}
if (!ok) {
return 1;
}
state->out_feats = std::move(x);
out->feats = state->out_feats.data();
out->n_feats = state->out_feats.size();
return 0;
}
// yue2 GEN_WAV: latents -> interleaved stereo PCM in one VAE pass, the caller splits a long song into tiles
static int32_t mtmd_gen_wav_yue2(mtmd_context * ctx, mtmd_gen_state * state, const mtmd_gen_inp * inp, mtmd_gen_out * out) {
using g = mtmd_graph_yue2_gen;
if (!inp->feats || inp->n_feats == 0 || inp->n_feats % g::n_latent != 0) {
LOG_ERR("%s: yue2 needs whole latent frames\n", __func__);
return 1;
}
const std::vector<float> feats(inp->feats, inp->feats + inp->n_feats);
auto & audio = state->out_audio;
if (!yue2_encode(ctx, state, MTMD_SUBCTX_GEN_PROCESS_GEN_WAV, feats, audio)) {
LOG_ERR("%s: encode failed (vae)\n", __func__);
return 1;
}
for (auto & v : audio) {
v = std::clamp(v, -1.0f, 1.0f);
}
out->audio = audio.data();
out->n_samples = audio.size();
return 0;
}
int32_t mtmd_gen_state_set_ref(mtmd_context * ctx, mtmd_gen_state * state) {
if (!state || state->sctx != ctx->ctx_gen_a.get()) {
LOG_ERR("%s: invalid gen state\n", __func__);
@@ -3139,6 +3472,11 @@ static int32_t mtmd_gen_audio_process_impl(mtmd_context * ctx, mtmd_gen_state *
*out = {};
if (sctx->model.proj_type == PROJECTOR_TYPE_YUE2_GEN) {
return inp->type == MTMD_GEN_PROCESS_TYPE_GEN_CODE ? mtmd_gen_flow_yue2(ctx, state, inp, out)
: mtmd_gen_wav_yue2(ctx, state, inp, out);
}
if (sctx->model.proj_type == PROJECTOR_TYPE_SOPRANO) {
const size_t n_embd = (size_t) sctx->model.n_mmproj_embd();
if (inp->type != MTMD_GEN_PROCESS_TYPE_GEN_WAV || !inp->feats || inp->n_feats < 2 * n_embd || inp->n_feats % n_embd != 0 || inp->codes) {
+14 -1
View File
@@ -473,11 +473,14 @@ enum mtmd_gen_audio_type {
MTMD_GEN_AUDIO_TYPE_CHATTERBOX,
MTMD_GEN_AUDIO_TYPE_DRAMA3,
MTMD_GEN_AUDIO_TYPE_ZONOS3,
MTMD_GEN_AUDIO_TYPE_YUE2,
MTMD_GEN_AUDIO_TYPE_FISH_S2,
};
struct mtmd_gen_audio_info {
enum mtmd_gen_audio_type type;
int32_t sample_rate; // in Hz, for example 24000 for qwen3tts
int32_t n_channels; // 1 = mono, 2 = stereo (mtmd_gen_out::audio is interleaved)
const char * model_variant; // name of the weight variant, can be nullptr if not applicable
int32_t n_voices; // predefined voices, see mtmd_gen_audio_voice_name()
};
@@ -500,6 +503,7 @@ enum mtmd_gen_process_type {
// for qwen3tts, this is code2wav
// for pocket-tts, this is mimi decoder
// for lfm3a, this is the codec decoder + ISTFT, n_codes == 0 flushes the tail
// for yue2, this is one VAE pass, the caller splits a long song into tiles
};
struct mtmd_gen_inp {
@@ -522,6 +526,15 @@ struct mtmd_gen_inp {
size_t n_codes;
const float * feats;
size_t n_feats;
// yue2 GEN_CODE: n_codes is the frame count, codes is not read (the backbone state holds them)
// backbone_state: llama_state_seq_get_data() of the backbone_n_past positions (prompt, codes, end token)
const uint8_t * backbone_state;
size_t backbone_state_size;
int32_t backbone_n_past;
// flow-matching GEN_CODE: steps of the ODE solver, <= 0 = model default (yue2: 32)
int32_t n_steps;
};
struct mtmd_gen_out {
@@ -538,7 +551,7 @@ struct mtmd_gen_out {
// for MTMD_GEN_PROCESS_TYPE_GEN_WAV
const float * audio;
size_t n_samples;
size_t n_samples; // total over all channels
};
struct mtmd_gen_state_params {
+8 -8
View File
@@ -15,7 +15,7 @@ In-scope types of feature:
- Multimodal input/output
- Memory management: save/load state, context checkpoints
- Model management
- Features that are required by the Web UI
- Features that are required by Llama Chat
- Frontend:
- Chat-oriented features, example: basic chat, image upload, edit messages
- Agentic features, example: MCP
@@ -182,7 +182,7 @@ For detailed instructions, see the [test documentation](./tests/README.md).
### API for tools
This endpoint is intended to be used internally by the Web UI and subject to change or to be removed in the future.
This endpoint is intended to be used internally by Llama Chat and subject to change or to be removed in the future.
**GET /tools**
@@ -279,7 +279,7 @@ For the direction from router to child:
Model management API was added via PR [#23976](https://github.com/ggml-org/llama.cpp/pull/23976)
The main goal of this API is to allow downloading models and/or removing models from the web UI. It relies on the model cache infrastructure under the hood to manage the list of models dynamically.
The main goal of this API is to allow downloading models and/or removing models from Llama Chat. It relies on the model cache infrastructure under the hood to manage the list of models dynamically.
Instead of building everything from the ground up (like what most AI agents will do when you ask them to implement a similar feature), we built on top of existing, already well-engineered components inside the codebase:
- Model cache infrastructure as mentioned above (`common/download.h`)
@@ -342,11 +342,11 @@ Endpoints created with `create_response(true)` (`/health`, `/props`, `/models`,
## Web UI
## Llama Chat
The project includes a web-based user interface for interacting with `llama-server`. It supports both single-model (`MODEL` mode) and multi-model (`ROUTER` mode) operation.
The project includes Llama Chat, a web-based chat app for interacting with `llama-server`. It supports both single-model (`MODEL` mode) and multi-model (`ROUTER` mode) operation.
The SvelteKit-based Web UI is introduced in this PR: https://github.com/ggml-org/llama.cpp/pull/14839
The SvelteKit-based app is introduced in this PR: https://github.com/ggml-org/llama.cpp/pull/14839
### Features
@@ -378,7 +378,7 @@ Routes → Components → Hooks → Stores → Services → Storage/API
- **Services** - stateless API/database communication (`ChatService`, `ModelsService`, `PropsService`, `DatabaseService`)
- **Hooks** - reusable logic (`useModelChangeValidation`, `useProcessingState`)
For detailed architecture diagrams, see [`tools/ui/docs/`](../ui/docs/):
For detailed architecture diagrams, see [`ui/apps/chat/docs/`](../../ui/apps/chat/docs/):
- `high-level-architecture.mmd` - full architecture with all modules
- `high-level-architecture-simplified.mmd` - simplified overview
@@ -390,7 +390,7 @@ For detailed architecture diagrams, see [`tools/ui/docs/`](../ui/docs/):
```sh
# make sure you have Node.js installed
cd tools/ui
cd ui/apps/chat
npm i
# run dev server (with hot reload)
+19 -13
View File
@@ -2,7 +2,7 @@
Fast, lightweight, pure C/C++ HTTP server based on [httplib](https://github.com/yhirose/cpp-httplib), [nlohmann::json](https://github.com/nlohmann/json) and **llama.cpp**.
Set of LLM REST APIs and a web UI to interact with llama.cpp.
Set of LLM REST APIs and Llama Chat, a built-in chat app, to interact with llama.cpp.
**Features:**
* LLM inference of F16 and quantized models on GPU and CPU
@@ -17,7 +17,7 @@ Set of LLM REST APIs and a web UI to interact with llama.cpp.
* Prefilling of assistant messages similar to the Claude API
* [Function calling](../../docs/function-calling.md) / tool use for ~any model
* Speculative decoding
* Easy-to-use web UI
* Llama Chat, an easy-to-use built-in chat app
For the full list of features, please refer to [server's changelog](https://github.com/ggml-org/llama.cpp/issues/9291)
@@ -202,14 +202,14 @@ For the full list of features, please refer to [server's changelog](https://gith
| `--api-prefix PREFIX` | prefix path the server serves from, without the trailing slash (default: )<br/>(env: LLAMA_ARG_API_PREFIX) |
| `--ui-config, --webui-config JSON` | JSON that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG) |
| `--ui-config-file, --webui-config-file PATH` | JSON file that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG_FILE) |
| `--ui-mcp-proxy, --webui-mcp-proxy, --no-ui-mcp-proxy, --no-webui-mcp-proxy` | experimental: whether to enable MCP CORS proxy, for the Web UI to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_UI_MCP_PROXY) |
| `--ui-mcp-proxy, --webui-mcp-proxy, --no-ui-mcp-proxy, --no-webui-mcp-proxy` | experimental: whether to enable MCP CORS proxy, for Llama Chat to reach public MCP servers without CORS headers; local and private addresses are refused - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_UI_MCP_PROXY) |
| `--tools TOOL1,TOOL2,...` | experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)<br/>specify "all" to enable all tools<br/>available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_TOOLS) |
| `--tools-runtime OPTION` | experimental: run tools in a separate runtime environment (default: none, use host environment)<br/>available options:<br/> 'docker:<image>', 'podman:<image>': spin up a new container and reuse it for all invocations, clean up on server exit<br/> 'docker-container:<id>', 'podman-container:<id>': use an existing container by ID, won't stop on server exit<br/> 'ssh:<target>': run tools on a remote POSIX host over SSH, key-based auth and a trusted host key are required<br/><br/>(env: LLAMA_ARG_TOOLS_RUNTIME) |
| `--allow-unauthenticated-tools` | allow server tools, MCP servers or MCP proxy on a non-loopback address without an API key - only for trusted networks (default: disabled)<br/>(env: LLAMA_ARG_ALLOW_UNAUTHENTICATED_TOOLS) |
| `--mcp-servers-config PATH` | experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_CONFIG) |
| `--mcp-servers-json JSON` | experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_JSON) |
| `-ag, --agent, -no-ag, --no-agent` | whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_AGENT) |
| `--ui, --webui, --no-ui, --no-webui` | whether to enable the Web UI (default: enabled)<br/>(env: LLAMA_ARG_UI) |
| `--ui, --webui, --no-ui, --no-webui` | whether to enable Llama Chat, the built-in chat app (default: enabled)<br/>(env: LLAMA_ARG_UI) |
| `--connect` | (experimental) open a peer-to-peer tunnel for server using llama-connect (default: disabled)<br/>(env: LLAMA_ARG_CONNECT) |
| `--connect-code CODE` | manually specify a 40-character code for --connect (default: generate a new one for each run)<br/>(env: LLAMA_ARG_CONNECT_CODE) |
| `--embedding, --embeddings` | restrict to only support embedding use case; use only with dedicated embedding models (default: disabled)<br/>(env: LLAMA_ARG_EMBEDDINGS) |
@@ -357,7 +357,7 @@ For more details, please refer to [multimodal documentation](../../docs/multimod
### Server tools support
The server includes a set of server tools that enable the LLM to access the local file system directly from the Web UI.
The server includes a set of server tools that enable the LLM to access the local file system directly from Llama Chat.
To use this feature, start the server with `--tools all`. You can also enable only specific tools by passing a comma-separated list: `--tools name1,name2,...`. Run `--help` for the full list of available tool names.
@@ -389,11 +389,11 @@ The same JSON can be passed inline with `--mcp-servers-json`. Each entry under `
| `cwd` | working directory of the child process |
| `timeout_ms` | per-tool-call timeout (default: 30000) |
Every server is spawned once at startup to list its tools, then stopped, and respawned on demand when one of its tools is called. Tools are exposed as `<server>_<tool>` alongside the built-in ones: they show up in the Web UI and in `GET /tools`, and the model calls them like any other tool. A name colliding with an already registered tool is skipped. This is independent of `--tools`, MCP servers can be the only tools available.
Every server is spawned once at startup to list its tools, then stopped, and respawned on demand when one of its tools is called. Tools are exposed as `<server>_<tool>` alongside the built-in ones: they show up in Llama Chat and in `GET /tools`, and the model calls them like any other tool. A name colliding with an already registered tool is skipped. This is independent of `--tools`, MCP servers can be the only tools available.
The child process runs with the same privileges as the server, so only declare commands you trust. As with `--tools`, `--cors-origins` then defaults to `localhost`.
Note: `--ui-mcp-proxy` is unrelated, it only lets the Web UI reach public MCP servers that do not send CORS headers. It refuses local and private addresses (loopback, LAN, link-local); connect to those from the Web UI directly.
Note: `--ui-mcp-proxy` is unrelated, it only lets Llama Chat reach public MCP servers that do not send CORS headers. It refuses local and private addresses (loopback, LAN, link-local); connect to those from Llama Chat directly.
Any server written against the [MCP specification](https://modelcontextprotocol.io) works as is, whether it uses an official SDK or not: the transport is one JSON-RPC message per line on stdio, so a script wrapping an existing program is a valid server too.
@@ -866,6 +866,8 @@ Returns raw audio bytes (`audio/wav` by default) rather than JSON. For the list
`speaker_ref_b64`: Base64-encoded reference audio to clone the speaker's voice. Optional. Alternatively, upload the reference audio as a `speaker_ref` file field via `multipart/form-data` (see example below) - if both are provided, the uploaded file takes precedence. Rejected when the model has no voice encoder (e.g. Pocket TTS without voice cloning).
`speaker_ref_text`: Transcript of the reference audio, put in the prompt by some models (e.g. Fish Audio S2 Pro). Optional.
`top_k`, `top_p`: Sampling params for both the backbone sampler and the acoustic code predictor. Optional, `0` or absent keeps the server `--top-k`/`--top-p` for the backbone and the model defaults for the code predictor.
`temperature`: Temperature for the backbone sampler and the audio frames, for models that support it. `0` is greedy sampling. Optional, absent keeps the server `--temp` for the backbone and the model defaults for the audio.
@@ -878,9 +880,13 @@ Returns raw audio bytes (`audio/wav` by default) rather than JSON. For the list
`loudness`: Target loudness in LUFS, `< 0` (default `0.0`, the model's own loudness). Only for models that support it, others reject a value other than `0.0` with a 400 error.
`n_predict`: Max number of audio frames to generate. Defaults to `512`; generation normally stops earlier once the model emits an end-of-speech token.
`cfg_scale`: Classifier-free guidance scale of the audio codes, `> 0`, `1.0` disables it (default `0.0`, the model's own). Only for models that support it (YuE2), others reject a value other than `0.0` with a 400 error.
`response_format`: `wav` (default), `pcm` (raw `float32` little-endian mono samples, no header; the response `Content-Type` carries the sample rate, e.g. `audio/pcm;rate=24000;encoding=float;bits=32`) or `pcm16` (same, as signed 16-bit samples, e.g. `audio/pcm;rate=24000;encoding=signed-int;bits=16`).
`flow_steps`: Steps of the flow-matching ODE solver, `1` to `1024` (default `0`, the model's own). Fewer steps are faster and the audio is less accurate. Only for models that support it (YuE2), others reject a value other than `0` with a 400 error.
`n_predict`: Max number of audio frames to generate. Defaults to the limit of the model (`512` for most models); generation normally stops earlier once the model emits an end-of-speech token.
`response_format`: `wav` (default), `pcm` (raw `float32` little-endian samples, no header, channels interleaved; the response `Content-Type` carries the sample rate and, for stereo models, the channel count, e.g. `audio/pcm;rate=24000;encoding=float;bits=32`) or `pcm16` (same, as signed 16-bit samples, e.g. `audio/pcm;rate=24000;encoding=signed-int;bits=16`).
`stream`: If `true`, the response is streamed as audio becomes available instead of waiting for the full generation to finish. WAV streaming writes an RFC-noncompliant header with an unknown (`0xFFFFFFFF`) size field, since the final length isn't known up front; most players and decoders (ffmpeg, VLC, ...) handle this by reading until EOF.
@@ -1672,7 +1678,7 @@ Generates speech from text, for models that can generate audio. The request is c
`stream_format`: `audio` streams the response as it becomes available, see `stream` in `/tts` for the WAV header it writes. `sse` is rejected. The response is not streamed when omitted.
`speed`: Speaking speed, see `speed` in `/tts`. Only models that support it accept a value other than `1.0`. `loudness` of `/tts` can be used as well.
`speed`: Speaking speed, see `speed` in `/tts`. Only models that support it accept a value other than `1.0`. `loudness`, `cfg_scale` and `flow_steps` of `/tts` can be used as well.
`instructions`: Ignored.
@@ -2044,7 +2050,7 @@ An invalid request returns the error `400`. A model that is not a decision model
## Server tools
The server exposes a REST API under `/tools` that allows the Web UI to call server tools. This endpoint is intended to be used internally by the Web UI and subject to change or to be removed in the future.
The server exposes a REST API under `/tools` that allows Llama Chat to call server tools. This endpoint is intended to be used internally by Llama Chat and subject to change or to be removed in the future.
**Please do NOT use this endpoint in a downstream application**
@@ -2643,9 +2649,9 @@ Apart from error types supported by OAI, we also have custom types that are spec
}
```
### Custom default Web UI preferences
### Custom default Llama Chat preferences
You can specify default preferences for the web UI using `--ui-config <JSON config>` or `--ui-config-file <path to JSON config>`. For example, you can disable pasting long text as attachments and enable rendering Markdown in user messages with this command:
You can specify default preferences for Llama Chat using `--ui-config <JSON config>` or `--ui-config-file <path to JSON config>`. For example, you can disable pasting long text as attachments and enable rendering Markdown in user messages with this command:
```bash
./llama-server -m model.gguf --ui-config '{"pasteLongTextToFileLen": 0, "renderUserContentAsMarkdown": true}'
+3 -3
View File
@@ -13,7 +13,7 @@
#include <thread>
#include <vector>
// share code = room code + pass code, must match llama-connect and the Web UI
// share code = room code + pass code, must match llama-connect and Llama Chat
// ref: https://github.com/ggml-org/llama-connect/blob/master/src/protocol.rs
static constexpr size_t CONNECT_ROOM_CODE_LEN = 8;
static constexpr size_t CONNECT_PASS_CODE_LEN = 32;
@@ -55,7 +55,7 @@ static std::string format_share_code(const std::string & code) {
return out;
}
// whitespace is tolerated, the Web UI shows the code in blocks and users paste it back
// whitespace is tolerated, Llama Chat shows the code in blocks and users paste it back
// returns an empty string if it is not a valid share code
static std::string normalize_share_code(const std::string & key) {
std::string code;
@@ -242,7 +242,7 @@ bool server_connect::start(const common_params & params, int port) {
SRV_INF("%s", "-----------------\n");
SRV_INF("%s", "remote access is enabled via llama-connect\n");
SRV_INF("share code (enter it in the Web UI under Settings -> Remote Access): %s\n",
SRV_INF("share code (enter it in Llama Chat under Settings -> Remote Access): %s\n",
format_share_code(code).c_str());
SRV_WRN("%s", "anyone with this code can use this server, do not share it publicly\n");
SRV_INF("%s", "-----------------\n");
+12 -6
View File
@@ -1540,6 +1540,10 @@ private:
if (has_cap_tts()) {
slot.tts.ctx.init(ctx_tgt, mctx);
if (!slot.tts.is_supported()) {
SRV_ERR("%s", "the context cannot run the audio generation of this model\n");
return false;
}
}
SLT_TRC(slot, "new slot, n_ctx = %d\n", slot.n_ctx);
@@ -1953,7 +1957,7 @@ private:
task.tts_inp.data.seq_id = slot.id;
const int32_t ret = slot.tts.ctx.set_input(task.tts_inp.get());
if (ret == 2) {
send_error(task, "invalid TTS input: the model may not support \"speed\" or \"loudness\", see the server log", ERROR_TYPE_INVALID_REQUEST);
send_error(task, "invalid TTS input: the model may not support \"speed\", \"loudness\", \"cfg_scale\" or \"flow_steps\", see the server log", ERROR_TYPE_INVALID_REQUEST);
return false;
}
if (ret != 0) {
@@ -3647,7 +3651,7 @@ private:
return;
}
const int32_t n_predict = slot.n_predict_max > 0 ? slot.n_predict_max : 512;
const int32_t n_predict = slot.tts.ctx.max_frames(slot.n_predict_max);
if (slot.tts.stop || slot.tts.n_decoded >= n_predict) {
int32_t sample_rate = 0;
const char * data = nullptr;
@@ -5650,12 +5654,12 @@ json server_routes::get_model_info() const {
// body of /tts and /v1/audio/speech; multipart form fields arrive as strings, the numeric and boolean params get their types back
static json parse_tts_body(const server_http_req & req) {
json body = json::parse(req.body);
for (const char * k : { "top_k", "n_predict", "max_tokens", "seed" }) {
for (const char * k : { "top_k", "n_predict", "max_tokens", "seed", "flow_steps" }) {
if (body.contains(k) && body[k].is_string()) {
body[k] = std::stoll(body[k].get<std::string>());
}
}
for (const char * k : { "top_p", "temperature", "repeat_penalty", "speed", "loudness" }) {
for (const char * k : { "top_p", "temperature", "repeat_penalty", "speed", "loudness", "cfg_scale" }) {
if (body.contains(k) && body[k].is_string()) {
body[k] = std::stof(body[k].get<std::string>());
}
@@ -6992,6 +6996,7 @@ std::unique_ptr<server_res_generator> server_routes::handle_tts_impl(const serve
}
task.tts_inp.set_prompt(tts_prompt.empty() ? prompt : tts_prompt);
task.tts_inp.set_lang(lang);
task.tts_inp.set_speaker_ref_text(json_value(body, "speaker_ref_text", std::string()));
task.tts_inp.data.stream = stream;
task.tts_inp.data.out_type = response_format == "pcm" || pcm16
? MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM
@@ -7062,8 +7067,9 @@ std::unique_ptr<server_res_generator> server_routes::handle_tts_impl(const serve
task.id = rd.get_new_id();
rd.post_task(std::move(task));
// raw LE mono samples; audio/L16 would imply 16-bit big-endian (RFC 2586)
const std::string pcm_type = "audio/pcm;rate=" + std::to_string(info.sample_rate);
// raw LE samples, channels interleaved; audio/L16 would imply 16-bit big-endian (RFC 2586)
const std::string pcm_type = "audio/pcm;rate=" + std::to_string(info.sample_rate) +
(info.n_channels > 1 ? ";channels=" + std::to_string(info.n_channels) : "");
const std::string content_type = pcm16 ? pcm_type + ";encoding=signed-int;bits=16"
: response_format == "pcm" ? pcm_type + ";encoding=float;bits=32"
: "audio/wav";
+1 -1
View File
@@ -481,7 +481,7 @@ bool server_http_context::init_listener(const common_params & params) {
};
//
// Web UI setup
// Llama Chat setup
//
// Use new `params.ui` field (backed by old `params.webui` for compat)
+1 -1
View File
@@ -2058,7 +2058,7 @@ void server_models_routes::init_routes() {
// main instance
auto res = std::make_unique<server_http_res>();
json props = {
// TODO: add support for this on web UI
// TODO: add support for this in Llama Chat
{"role", "router"},
{"max_instances", params.models_max},
{"models_autoload", params.models_autoload},
+13 -17
View File
@@ -1032,24 +1032,20 @@ bool server_realtime_slot::decode(const llama_token * tokens, const float * rows
if (n_past + n >= n_ctx) {
return fail("the conversation does not fit in the context", ERROR_TYPE_EXCEED_CONTEXT_SIZE);
}
for (int32_t i0 = 0; i0 < n; i0 += n_batch) {
const int32_t n_cur = std::min(n_batch, n - i0);
llama_batch b = llama_batch_init(n_cur, rows ? n_embd : 0, 1);
for (int32_t i = 0; i < n_cur; i++) {
if (rows) {
memcpy(b.embd + (size_t) i * n_embd, rows + (size_t) (i0 + i) * n_embd, n_embd * sizeof(float));
} else {
b.token[i] = tokens[i0 + i];
}
b.pos[i] = n_past + i0 + i;
b.n_seq_id[i] = 1;
b.seq_id[i][0] = seq_id;
b.logits[i] = logits_last && i0 + i == n - 1;
common_batch batch(ctx);
for (int32_t i = 0; i < n; i++) {
const llama_pos pos = n_past + i;
const bool output = logits_last && i == n - 1;
if (rows) {
const llama_embd e = { rows + (size_t) i * n_embd, 1, (size_t) n_embd };
batch.add_embd(e, &pos, seq_id, output);
} else {
batch.add(tokens[i], pos, seq_id, output);
}
b.n_tokens = n_cur;
const int ret = llama_decode(ctx, b);
llama_batch_free(b);
if (ret != 0) {
}
for (int32_t i = 0; i < n; i += n_batch) {
const int32_t n_cur = std::min(n_batch, n - i);
if (llama_process(ctx, LLAMA_PROCESS_TYPE_DECODE, batch.get_sub_batch(i, n_cur)) != 0) {
return fail("failed to decode the conversation");
}
}
+7
View File
@@ -561,6 +561,13 @@ std::vector<std::unique_ptr<field>> make_tts_schema(server_task & task) {
add((new field_num("loudness", task.tts_inp.data.loudness))
->set_desc("Target loudness in LUFS, for models that support it (0.0 = model default)"));
add((new field_num("cfg_scale", task.tts_inp.data.cfg_scale))
->set_desc("Classifier-free guidance scale of the audio codes, for models that support it (0.0 = model default, 1.0 = no guidance)"));
add((new field_num("flow_steps", task.tts_inp.data.flow_steps))
->set_hard_limits(0, 1024)
->set_desc("Steps of the flow-matching ODE solver, for models that support it (0 = model default); fewer is faster, less accurate"));
add((new field_num("n_predict", task.params.n_predict))
->set_hard_limits(-1, INT32_MAX)
->add_alias("max_tokens")
+1 -1
View File
@@ -50,7 +50,7 @@ void server_stream_session_manager_stop();
// decoded path would split before the param is captured
server_http_context::handler_t server_stream_make_get_handler();
// POST /v1/streams/lookup with body {"conversation_ids": [...]}: only answers for ids the
// caller already owns (the WebUI passes the convs visible in its sidebar), the server never
// caller already owns (Llama Chat passes the convs visible in its sidebar), the server never
// lists ids it has not been asked about, so a random caller cannot enumerate live sessions
server_http_context::handler_t server_stream_make_lookup_handler();
server_http_context::handler_t server_stream_make_delete_handler();
+2 -2
View File
@@ -1076,7 +1076,7 @@ struct server_tool_file_glob_search : server_tool {
}
// `base` is always absolute (resolve falls back to the server cwd), so
// API clients (e.g. the web UI picker) can join the relative entries
// API clients (e.g. the Llama Chat picker) can join the relative entries
// into absolute paths. `plain_text_response` is what the model sees;
// `entries` is the same data as structured JSON for the UI picker,
// which reads `entries`/`base` instead of re-parsing the text.
@@ -1953,7 +1953,7 @@ static std::vector<std::unique_ptr<server_tool>> build_tools() {
// IMPORTANT: for contributors, please keep this array of tools as minimal as possible
// we only accept minimal i/o and shell command tools here
// for example, do not add: web search, get date time, etc.
// high-level functionality should be added either via MCP or web UI
// high-level functionality should be added either via MCP or Llama Chat
std::vector<std::unique_ptr<server_tool>> tools;
tools.push_back(std::make_unique<server_tool_read_file>());
tools.push_back(std::make_unique<server_tool_file_glob_search>());
+1 -1
View File
@@ -279,7 +279,7 @@ static bool server_setup_features(
warn_names.push_back("peer-to-peer tunnel (llama-connect, experimental)");
}
// CORS proxy (EXPERIMENTAL, only used by the Web UI for MCP)
// CORS proxy (EXPERIMENTAL, only used by Llama Chat for MCP)
if (params.ui_mcp_proxy) {
ctx_http.get ("/cors-proxy", ex_wrapper(proxy_handler_get));
ctx_http.post("/cors-proxy", ex_wrapper(proxy_handler_post));
+1 -1
View File
@@ -25,7 +25,7 @@ def test_access_public_endpoint(endpoint: str):
def test_access_static_assets_without_api_key():
"""Static web UI assets should not require API key authentication (issue #21229)"""
"""Static Llama Chat assets should not require API key authentication (issue #21229)"""
global server
server.start()
for path in ["/", "/sw.js", "/manifest.webmanifest", "/_app/version.json"]:
+2 -1
View File
@@ -16,7 +16,8 @@ Note: it's highly recommended to always provide a speaker reference voice (via `
Common params:
- Sampling params such as `--top-k`, `--top-p`, `--temp`, etc. `--temp` also sets the temperature of the audio frames for models that support it, without it they use the model default
- `-n <number_of_frames>` limits the output length, e.g. `-n 500`. Note that how many milliseconds each frame represents varies by model
- `-n <number_of_frames>` limits the output length, e.g. `-n 500` (default: the limit of the model, 512 frames for most models). Note that how many milliseconds each frame represents varies by model
- `--tts-speaker-text` (optional) gives the transcript of `--tts-speaker-file`, some models put it in the prompt (e.g. Fish Audio S2 Pro)
- Core inference params such as `-ngl`, `-b`, `-ub`, etc.
## Supported models
+7 -2
View File
@@ -142,11 +142,16 @@ int main(int argc, char ** argv) {
mtmd_helper::gen_audio gen;
gen.init(lctx, mctx.get());
if (!gen.valid()) {
LOG_ERR("the context cannot run the audio generation of this model\n");
return 1;
}
mtmd_helper_gen_audio_inp inp = mtmd_helper_gen_audio_inp_default();
inp.seq_id = 0;
inp.prompt = prompt.c_str();
inp.prompt_len = prompt.size();
inp.speaker_ref = speaker_bitmap.get();
inp.speaker_ref_text = params.tts_speaker_text.empty() ? nullptr : params.tts_speaker_text.c_str();
inp.lang = params.tts_lang.c_str();
inp.top_k = params.sampling.top_k;
inp.top_p = params.sampling.top_p;
@@ -168,7 +173,7 @@ int main(int argc, char ** argv) {
const int64_t t_prompt_start_us = ggml_time_us();
for (;;) {
int32_t ret = gen.step_prompt(params.n_batch);
int32_t ret = gen.step_prompt(llama_n_batch(lctx));
if (ret < 0) {
LOG_ERR("prompt processing failed\n");
return 1;
@@ -185,7 +190,7 @@ int main(int argc, char ** argv) {
return t;
};
const int max_new = params.n_predict > 0 ? params.n_predict : 512;
const int max_new = gen.max_frames(params.n_predict);
int n_frames = 0;
llama_token sampled = sample_semantic_code();
const float * h_state = llama_get_embeddings_ith(lctx, -1);
@@ -13,7 +13,7 @@ VITE_PUBLIC_SERVER_ORIGIN='http://localhost:8033'
# DEMO_TRACES_DATASET='<owner>/<dataset>'
# DEMO_GITHUB_REPO='ggml-org/llama.cpp'
# DEMO_SERVER_ORIGIN='http://localhost:9931'
# DEMO_LLAMA_SERVER='../../build/bin/llama-server'
# DEMO_LLAMA_SERVER='../../../build/bin/llama-server'
# DEMO_COLOR_SCHEME='light'
# Device pixel ratio demos render and record at (1440x900 viewport)
# DEMO_SCALE='2'

Some files were not shown because too many files have changed in this diff Show More