3 Commits
Author SHA1 Message Date
Baojiang LeeandGeorge Panchuk 40cca63d5f fix: make tokenizer metadata files optional (#693)
* fix: support optional tokenizer metadata files

* fix: pad id fallback chain and additional_special_tokens lists

* refactor: remove redundant tests and comments

---------

Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
2026-09-22 01:37:52 +07:00
GeorgeandMohammed Alshyakh a3a798f4f3 fix: preserve pad_to_multiple_of when normalizing padding (#717)
Co-authored-by: Mohammed Alshyakh <zzzzmmmm298@gmail.com>
2026-09-21 22:13:43 +07:00
George bdf6816da8 fix: normalize tokenizer padding to batch-longest (#716)
* fix: normalize tokenizer padding to batch-longest

* fix: don't use max position embeddings as max len
2026-09-21 21:05:57 +07:00