From 86020c467a86f37e3cc452dc68aa33c8cf623c44 Mon Sep 17 00:00:00 2001 From: Xuan Son Nguyen Date: Sat, 1 Aug 2026 15:07:28 +0200 Subject: [PATCH] update dev docs --- tools/mtmd/README-dev.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/tools/mtmd/README-dev.md b/tools/mtmd/README-dev.md index 83256a4565..5b94e268c1 100644 --- a/tools/mtmd/README-dev.md +++ b/tools/mtmd/README-dev.md @@ -33,18 +33,17 @@ A typical pipeline of the core libmtmd is as follows: Audio generation is added to mtmd in PR [#26254](https://github.com/ggml-org/llama.cpp/pull/26254) Currently, we support the 3-stage pipeline below which should cover most TTS models: -1. (Optional) an audio encoder model that converts reference voice into codes or features -2. A backbone model that accepts text prompt and reference voice as input -3. A feature generator model that takes the hidden state from backbone and generate audio features (usually as audio codes or mel-spectrogram) -4. A model that converts audio features to the final PCM waveform +- Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input +- Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram) +- Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform For example, Qwen3-TTS: -1. Reference voice is encoded using ECAPA-TDNN speaker encoder (`speaker_encoder`) -2. Text prompt and reference voice are processed via a backbone (`talker.model`) -3. A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (`talker.code_predictor`) -4. 16 generated codes are converted into waveform (`code2wav`) +- Reference voice is encoded using ECAPA-TDNN speaker encoder (`speaker_encoder`) +- Text prompt and reference voice are processed via a backbone (`talker.model`) +- A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (`talker.code_predictor`) +- 16 generated codes are converted into waveform (`code2wav`) -### API design constraint +### API design constraints Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system is designed to be flexible and reusable by new models. @@ -55,13 +54,19 @@ Due to wide variety of audio generation pipelines, the `mtmd_gen_audio` system i ### Checklist for porting new audio generation models to mtmd 1. Establish a list of reusable and missing components from the current mtmd implementation. -2. Sidecar models (code2wav, bigvgan, etc) must live inside the same GGUF file (but can be in different `clip_context` if necessary) +2. For GGUF conversion: + - Backbone model should be converted to a normal text model (loadable via `libllama`) + - If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see `qwen3tts.py`) + - If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see `src/models/qwen3vl.cpp`) + - Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different `clip_context` if necessary) 3. Make sure most of the changes happen inside `mtmd-helper-gen.cpp`. A good PR looks like this: - 10-20% changes is to add new backbone (text) model and conversion - 60% changes inside `mtmd-helper-gen.cpp` - 10% changes inside `libmtmd` and `clip.cpp` systems - The rest downstream code (CLI, server) should have no changes at all +IMPORTANT: if the model requires any changes that doesn't fit into the existing infrastructure, **open an issue** first for discussion. Any new components must be verified to respect the API design constraints stated above + ## Helper We provide a set of helper functions via `mtmd_helper` to make using libmtmd easier. The helper provides: