Toki NasinandSigbjørn Skjæret db33d3cb89 vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734) ...

* vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3

The original tokenizer configs for PLaMo-2 and PLaMo-3 have
`add_bos_token: true` and `add_eos_token: false` , but
_set_vocab_plamo() did not write the BOS/EOS metadata. The
PLAMO2 tokenizer path also ignored add_bos/add_eos during
tokenization.

Write the settings from tokenizer_config.json and honor them in
the PLAMO2 tokenization path. GGUFs without these keys keep the
previous behavior.

* Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

2026-10-01 08:31:42 +03:00

..

__init__.py

add GLM-5.3-Flash (GLM5-Next) support (#27773)

2026-09-30 14:20:32 +08:00

afmoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

arctic.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

baichuan.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

bailingmoe3.py

model : add Ling 3.0 VL support (#29151)

2026-09-24 08:57:31 +02:00

bailingmoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

base.py

vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734)

2026-10-01 08:31:42 +03:00

bert.py

ci : bump ty to 0.0.84 (#29529)

2026-09-27 13:43:08 +02:00

bitnet.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

bloom.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

chameleon.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

chatglm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

codeshell.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

cogvlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

command_r.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

dbrx.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

deci.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

deepseek.py

convert : skip bias_vl tensor in DeepSeek-V4 DSpark conversion (#28294)

2026-09-03 10:37:23 +03:00

dots1.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

dots3.py

mtmd: support dots3-note vision+audio (#27524)

2026-08-22 10:35:50 +02:00

dotsocr.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

dream.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

ernie.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

exaone.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

falcon_h1.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

falcon.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gemma.py

model : support Gemma4 DSpark draft backbone (#29226)

2026-09-23 13:34:09 +03:00

glm.py

add GLM-5.3-Flash (GLM5-Next) support (#27773)

2026-09-30 14:20:32 +08:00

gpt2.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gpt_oss.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

gptneox.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

granite.py

model : GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM (#25505)

2026-08-19 16:53:31 +02:00

grok.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

grovemoe.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

hrm_text.py

model : add support for HrmTextForCausalLM (DFM Mimir 1B) (#27625)

2026-09-16 15:18:45 +02:00

hunyuan.py

spec : support DFlash for HunyuanOCR (#28890)

2026-09-22 15:04:52 +02:00

hy_v4.py

convert : refactor Hy4-preview conversion - move HC tensor mapping to the global map (#28451)

2026-09-07 15:20:58 +02:00

internlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

internvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

jais.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

jamba.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

januspro.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

kimi_k3.py

convert: add MiMo-V2.6 support (#29257)

2026-09-22 14:38:09 +02:00

kimi_linear.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

kimivl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

laguna.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

lfm2.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

lighton_ocr.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llada.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llama4.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llama.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

llava.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

maincoder.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

mamba.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

maple.py

llama: add Maple 20B-A1B ternary MoE architecture (CPU) (#27000)

2026-09-14 14:04:05 +03:00

mellum.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

mimo.py

llama : fix tensor split for fused qkv with uneven K/V head sizes (#29294)

2026-09-25 11:15:33 +08:00

minicpm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

minimax.py

ci : bump ty to 0.0.78 (#28548)

2026-09-07 21:10:06 +02:00

mistral3.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

mistral.py

convert : fix conversion for Mistral-Medium-3.5-128B (#24268)

2026-06-07 21:41:39 +02:00

mpt.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

muse_glimmer.py

ci : bump ty to 0.0.78 (#28548)

2026-09-07 21:10:06 +02:00

nanbeige.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

nemotron.py

gguf-py : TemplateProcessing has final word on add_special_token (#29417)

2026-09-25 11:55:38 +02:00

olmo.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

openelm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

orion.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pangu.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

phi.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pixtral.py

Refactor: convert_hf_to_gguf.py (#17114)

2026-05-15 15:18:12 +02:00

plamo.py

convert : export YaRN scaling parameters for PLaMo-3 (#29528)

2026-09-27 13:45:47 +02:00

plm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

pockettts.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

qwen3tts.py

mtmd: Fix Qwen3-tts-0.6b (#28231)

2026-09-02 12:46:16 +02:00

qwen3vl.py

add GLM-5.3-Flash (GLM5-Next) support (#27773)

2026-09-30 14:20:32 +08:00

qwen4exp.py

model: add Qwen3.8-Flash-Next (qwen4exp) (#27742)

2026-08-27 21:32:31 +02:00

qwen.py

convert : fix LoRA conversion crash for Qwen3.5 V-head reorder (#28324)

2026-09-30 19:30:21 +02:00

qwenvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

refact.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

rwkv.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

sarashina2.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

smallthinker.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

smolvlm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

spark2_5.py

[Model] Support for Spark2_5ForCausalLM implementation (#27868)

2026-09-06 17:43:58 +02:00

stablelm.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

starcoder.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

step3.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

t5.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

talkie.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

ultravox.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

wavtokenizer.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

xverse.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00

youtuvl.py

convert: add @ModelBase.example (#27208)

2026-08-17 10:15:11 +02:00