Saba Fallah
e5d8b4685c
mtmd: dsocr-tiles fixes ( #25481 )
...
* ds-ocr img-preproc fuse_row tile-drop fix for multi rows and columns images
* mtmd drop the duplicate redundant img_end
* deepseekocr graph simplify CLS broadcast cleanup
2026-07-09 13:45:21 +02:00
Xuan Son Nguyen
959ce58197
improve
2026-06-18 19:29:43 +02:00
Xuan Son Nguyen
39fffcda7b
Merge branch 'master' into xsn/mtmd_ds_ocr_tiles
2026-06-18 18:59:26 +02:00
Xuan-Son Nguyen
060ce1bf72
mtmd: refactor llava-uhd overview image handling (always use ov_img_first) ( #24769 )
...
* add dedicated "overview" for mtmd_image_preproc_out
* corrections
* correct (again)
* nits
* nits (2)
2026-06-18 18:53:49 +02:00
Xuan Son Nguyen
9158400c42
adapt to new preprocessor api
2026-06-18 16:32:08 +02:00
Xuan Son Nguyen
ea4d61c4f5
Merge branch 'master' into xsn/mtmd_ds_ocr_tiles
2026-06-18 16:20:15 +02:00
Xuan-Son Nguyen
24bba7b98e
mtmd: refactor preprocessor, add mtmd_image_preproc_out ( #24736 )
...
* add mtmd_image_preproc_out
* add dev docs
* remove unused clip API
* rm unused clip_image_f32_batch::grid
* change preprocess() call signature
2026-06-18 12:04:39 +02:00
Xuan-Son Nguyen
f3e1828164
mtmd: llava_uhd should no longer use batch dim ( #24732 )
2026-06-17 22:40:50 +02:00
Xuan Son Nguyen
6b482fd842
fuse row into a long image
2026-06-17 00:44:57 +02:00
Saba Fallah
d4bbef8083
mtmd: deepseek-ocr v1 multi-tile dynamic resolution + unified image-preprocessors for both versions (ds-ocr v1 and v2)
2026-06-16 22:36:39 +02:00
Xuan-Son Nguyen
31e82494c0
mtmd: support "frame merge" for qwen-vl-based models ( #21858 )
...
* feat: add video support for Qwen3.5
* various clean up
* revise the design
* fix llava-uhd case
* nits
* nits 2
---------
Co-authored-by: andrewmd5 <1297077+andrewmd5@users.noreply.github.com >
2026-06-06 21:17:25 +02:00
Xuan-Son Nguyen
f5c6ae1827
mtmd, server: add "placeholder bitmap" for counting tokens , add */input_tokens API ( #23913 )
...
* mtmd: add "placeholder bitmap" for counting tokens w/o preprocessing
* fast path skip preproc for placeholder
* fix build
* correct the api
* add server endpoint + tests
* add object name
* update docs
* add proxy handling
* fix build
* fix audio input path
* use is_placeholder in process_mtmd_prompt()
* nits
* nits (2)
* docs: clarify chat/completions/input_tokens is not official
* fix merge problem
2026-06-06 11:06:51 +02:00
Saba Fallah
da3f990a47
mtmd: Add DeepSeekOCR 2 Support ( #20975 )
...
* mtmd: DeepSeek-OCR 2 support, with multi-tile dynamic resolution
* introduced clip_image_f32::add_viewsep
* address PR review
- drop redundant ggml_cpy ops in both deepseekocr versions build
- drop no-op ggml_cont in build_sam
- assert num_image_tokens deepseekocr2
- view_seperator as (1, n_embd) at conversion (for both versions)
- drop redundant ggml_reshape_2d
* Update tools/mtmd/models/deepseekocr2.cpp
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com >
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com >
2026-05-29 16:13:51 +02:00
Saba Fallah
a8681a0ed2
mtmd : DeepSeek-OCR image processing fixes, img_tool::resize padding refactor ( #23345 )
...
* mtmd : deepseek-ocr fixes, improvements and refactoring
- image processing changes to achieve full parity with Pillow (reference impl)
- SAM mask casting only when flash-attn is on
- SAM refactor (build_sam() extracted so deepseek-ocr-2 can reuse it)
- llama-chat changes to fix server/WebUI issue (new media_markers_first())
- adapted test-chat-template and added test cases for deepseek-ocr
- changed regression test for deepseek-ocr to use CER+chrF scores for ground-truth comparison; removed embedding-model
- ty.toml ignore unresolved-import for tools/mtmd/tests/**
* image-text reordering fix removed
* refactor bool add_padding + pad_rounding enum into a single pad_style enum
2026-05-20 17:37:10 +02:00
tc-mb
2496f9c149
mtmd : support MiniCPM-V 4.6 ( #22529 )
...
* Support MiniCPM-V 4.6 in new branch
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix code bug
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix pre-commit
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix convert
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* rename clip_graph_minicpmv4_6
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* use new TYPE_MINICPMV4_6
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* use build_attn to allow flash attention support
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* no use legacy code, restored here.
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* use the existing tensors name
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* unused ctx->model.hparams.minicpmv_version
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* use n_merge for slice alignment
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* borrow wa_layer_indexes for vit_merger insertion point
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix code style
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* use filter_tensors and add model.vision_tower
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix chkhsh
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
* fix type check
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
---------
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-05-06 21:54:09 +02:00
Sergiu
82764d8f40
mtmd: fix crash when sending image under 2x2 pixels ( #21711 )
2026-04-12 23:59:21 +02:00
forforever73
09343c0198
model : support step3-vl-10b ( #21287 )
...
* feat: support step3-vl-10b
* use fused QKV && mapping tensor in tensor_mapping.py
* guard hardcoded params and drop crop metadata
* get understand_projector_stride from global config
* img_u8_resize_bilinear_to_f32 move in step3vl class
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* fix the \r\n mess
* add width and heads to MmprojModel.set_gguf_parameters
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-04-08 09:51:31 +02:00
Xuan-Son Nguyen
871f1a2d2f
mtmd: add more sanity checks ( #21047 )
2026-03-27 11:00:52 +01:00
Xuan-Son Nguyen
a73bbd5d92
mtmd: refactor image preprocessing ( #21031 )
...
* mtmd: refactor image pre-processing
* correct some places
* correct lfm2
* fix deepseek-ocr on server
* add comment to clarify about mtmd_image_preprocessor_dyn_size
2026-03-26 19:49:20 +01:00
