mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-10-02 10:57:33 -05:00
4dd127584b
* server: add read_image tool (#25875) Adds a server-tool that allows vision models to analyze server-side images. This tool is reading a single file for now: The image data is base64 encoded and passed to the UI, which decodes it, fills the <img> tag and removes the data URI before passing the tool result back to the model. * cleanup read_image tool: move magic strings to constants * Add dedicated constants file: tools/ui/src/lib/constants/read-image.ts with PREFIX_IMAGE, PREFIX_SIZE, PREFIX_MIME constants * Use ATTACHMENT_SAVED_REGEX from agentic.ts in ChatMessageToolCallBlockReadImage.svelte * Use NEWLINE constant from code.ts instead of hardcoded '\n' * Use PREFIX_SIZE in regex pattern for size parsing * Add SERVER_TOOL_READ_IMAGE_PREFIX_* constants in C++ server-tools.cpp to match the TypeScript PREFIX_* constants for consistency * server: rename read_image tool to read_media for images and audio * Rename server_tool_read_image to server_tool_read_media in C++ * Rename enum BuiltInTool.READ_IMAGE to READ_MEDIA * Rename UI constants, parser, and Svelte component files * Update display label from 'Read image' to 'Read media' * ui: consolidate audio data URI handling into shared utility * Extract getAudioInputFormat to a shared utility (was duplicated inline) * Store raw base64 in base64Data on the message object * Use base64Data to construct data URIs for audio rendering * Update agentic store to build INPUT_AUDIO parts from base64Data * server: read_media: restrict audio to wav/mp3 and minor fixes * Server get_mime_from_extension now only advertises audio/wav and audio/mpeg (the only formats the model's input_audio API accepts) * Case-insensitive extension matching (fixes .MP3, .Wav, etc.) * Unknown extensions return an error instead of a multi-MB data URI that inflates model context with garbage * Updated tool description to document supported formats * Frontend AUDIO_MIME_TO_EXTENSION trimmed to match server * fix a missing import in tools/ui/src/lib/stores/agentic.svelte.ts * server: read_media: add to --tools help text and README tool list * ui: fix indentation in ChatMessageToolCallBlockDefault.svelte * server: read_media tool: fix a cast to use the correct type * server: read_media: multiple fixes * server-tools.cpp import cctype, remove UTF-8 char, check mime before reading file * ui: add MimeTypePrefix.AUDIO and use it in agentic.svelte.ts * server: make read_media inherit from read_file and add uses_cwd * ui: fix formating issues * rm from server * move it to frontend-only tool * correct partial commit * rm unused * ui: address review from allozaur Replace the magic strings, regexes and number in the read_media parser and service with named constants. Path splitting reuses FILE_PATH_SEPARATOR_REGEX, the size header regex moves to READ_MEDIA_SIZE_REGEX derived from PREFIX_SIZE, and FILE_EXTENSION_SEPARATOR lands next to it in constants/code.ts. --------- Co-authored-by: ckrafft <ckrafft@epyc> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com>
28 lines
1.2 KiB
TypeScript
28 lines
1.2 KiB
TypeScript
export const NEWLINE = '\n';
|
|
export const TAB = '\t';
|
|
export const DEFAULT_LANGUAGE = 'text';
|
|
export const LANG_PATTERN = /^(\w*)\n?/;
|
|
export const AMPERSAND_REGEX = /&/g;
|
|
export const LT_REGEX = /</g;
|
|
export const GT_REGEX = />/g;
|
|
export const FENCE_PATTERN = /^```|\n```/g;
|
|
|
|
// Whitespace-only empty lines (between start of string and first non-empty line).
|
|
// Used by trimCodePadding to drop leading/trailing phantom blank rows from LLM
|
|
// payload wrappers without touching internal blank lines.
|
|
export const TRIM_LEADING_PADDING_REGEX = /^(?:[ \t]*\n)+/;
|
|
export const TRIM_TRAILING_PADDING_REGEX = /(?:\n[ \t]*)+$/;
|
|
|
|
// Matches either Unix or Windows path separators so `String.split(REGEX)` can
|
|
// recover the trailing file-name segment from either `/foo/bar.txt` or
|
|
// `C:\foo\bar.txt`. Used wherever a parameter accepts a user-supplied path.
|
|
export const FILE_PATH_SEPARATOR_REGEX = /[\\/]/;
|
|
|
|
// Separates a file name from its extension, e.g. the '.' in `cover.png`.
|
|
export const FILE_EXTENSION_SEPARATOR = '.';
|
|
|
|
// Matches the `text:` prefix that file-type identifiers use to denote a
|
|
// plain-text language (e.g. `text:typescript`). Used by tool-call renderers
|
|
// to recover the underlying highlight.js language.
|
|
export const TEXT_LANGUAGE_PREFIX_REGEX = /^text:/;
|