Load the selected model's details, file tree and README via
HuggingFaceService on selection change, and expose the download
progress type globally for the status feed. The download options
already read their state from the models status store.
Assisted-by: pi
Replace the device-memory tier badges with the simple memory estimate:
each quant tooltip shows the estimated runtime memory and the device/OS
chip is dropped, matching the estimateModelMemoryBytes model. Make the
download dialog callbacks optional so the component stays presentational
until wired to the live status store.
Assisted-by: pi
Port the discover UI from the scrapbook, adapted to the typed
sidecar API: searchable two-pane explorer (list, item, info, org
avatar with quant badge), model details (header, name badges,
download options grouped by bit depth with compatibility tiers,
terminal serve/cli commands per draft sidecar, README viewer, chat
template dialog), download confirmation dialog with progress, and
the full-screen dialog shell.
Presentational components take data and download state via props;
the loading container and store wiring land in the integration
branch. MarkdownContent gains a sanitized allowHtml option used by
the README viewer; ModelId gains context, size-range, params and
sidecar badges; the selector option passes thinking/tool flags
instead of the removed capabilities prop.
Basic Storybook stories cover each component with HF-shaped
fixtures.
Assisted-by: pi
Tag the pure-logic files and functions that llama.app (llama-pages)
can reuse as-is: model id parsing, HF name and quant conventions,
hardware compatibility estimation, chat-template capability
detectors, and the HF formatting and metadata helpers. App-specific
code is left unmarked.
The LLAMA-APP-REUSE prefix makes the reusable surface greppable and
distinguishable from regular comments: grep -rn LLAMA-APP-REUSE.
Assisted-by: pi
Port the download lifecycle into ModelsStatusManager: track per-entry
progress keyed by <repo>:<tag> from the /models/sse feed, record
failed downloads for the delete-and-retry path, and expose the
downloadModel / cancelDownload operations (POST/DELETE /models).
Add the ServerModelStatus.DOWNLOADED/DOWNLOADING cases and the
ModelDownloadProgress type.
Assisted-by: pi
Wire the model download flow: ModelsService.downloadModel (POST
/models) and cancelDownload (DELETE /models), the apiDelete helper,
ApiModelsDownloadRequest/Response types, the download_progress SSE
payload, and the download_finished/download_failed SSE event kinds
matching the server feed.
Add modelsHubStore owning the HuggingFace GGUF model list for the
discover dialog: curated catalog defaults on open, search replaces
the list across all of HuggingFace.
Assisted-by: pi
Replace the device-memory tier machinery with a plain file-size
estimate: required runtime memory is the model file size with
headroom for KV cache and allocator overhead (estimateModelMemoryBytes).
Callers present the requirement; there is no device detection and no
fit-versus-budget verdict.
Drops resolveDeviceMemoryGb, deviceMemoryBudgetMb,
computeFileCompatibilityTiers and the CompatibilityTier type, and the
barrel keeps only the new estimator.
Assisted-by: pi
Port the hardware-compatibility estimator from ggml-org/llama-macos:
map every GGUF file in a repo to a full/limited/none tier based on
the device memory budget (GPU working set approximated from RAM, less
fit slack and an OS floor) and the estimated weight + context memory.
Main quants are tiered individually; shards, mmproj and quant-matched
draft sidecars inherit their main quant's tier. Sidecar picking
mirrors the server's find_best_sibling ranking (deepest directory,
exact quant tag, closest bit depth).
Also port detectToolUseSupport (infers tool-calling support from a
chat template) and the browser get_info fallback helper.
Assisted-by: pi
Address review follow-up: the llama.app catalog endpoint belongs to
the models-discover feature, not the HF constants. Use Number() for
shard index parsing and name the UD-quant prefix segment lookup.
Assisted-by: pi
Address review on the HF data layer:
- replace the HfModelSort / SidecarForm / sibling entry type string
unions with enums (HfModelSort, SidecarForm, HfEntryType)
- move URLs, query params, regexes, limits, retry settings, shard
file conventions, tag tokens and formatting units into a dedicated
huggingface.constants.ts; reuse the existing PATH_SEPARATOR
- drop the task label / pipeline icon / library display maps: the
discover UI only presents GGUF models, so keep the task tags for
logic use only (parseTags)
- drop the hardcoded curated model list; the discover dialog gets its
default list from the llama.app /v1/catalog.json endpoint, which is
an acceptable online-only source since the feature requires internet
access anyway
Assisted-by: pi
Add HuggingFaceService for browsing and searching GGUF models on the
HF Hub: catalog/model search, model details, repo file tree, raw
README fetch, and the llama.app model catalog. Includes GGUF file
analysis helpers - extractQuantMeta (quant token plus sidecar type
and its form, prefix or suffix), shard collapsing, quant bit-depth
lookup, and download/size/likes formatting.
Add the HF API types and the curated model list shown in the
Discover Models sidebar.
Assisted-by: pi
Use lowercase values for the sidecar enums so the value doubles as
the filename token, derive the sidecar regexes from the enum values,
and rename the MODEL_ID regex keys to the _REGEX suffix used by the
rest of the constants files. Replace the tools capability magic
string with ModelCapability.TOOL_USE.
Assisted-by: pi
Add ModelDraftSidecar / ModelAuxSidecar enums with a ModelSidecar
union type; mmproj is the only auxiliary sidecar (single member,
covers vision and audio input). Add SIDECAR_PREFIX/SUFFIX_RE regex
matching the server's filename conventions, and type guards +
enum-file-token helpers in model-id.constants.ts.
Extend parseModelId to detect sidecar filename tokens (mtp-, mmproj-,
etc) and expose isDraftSidecar / isAuxSidecar / sidecarFromFileToken
helpers. Add ModelCapability.TOOL_USE with icon/label/flag mappings.
Assisted-by: pi
Drop the Router prefix from client-side API types; names now map
directly to the /models endpoint family (load/unload/download/list).
Merge ApiModelListResponse into ApiModelsListResponse (same endpoint
shape in both modes) and remove the duplicate ModelsService.listRouter().
Assisted-by: pi
* ui : fix MCP image attachments not displayed in tool block (#25789)
Fixes regression from #25450 where ChatMessageAgenticContent passed
message.extra instead of section.toolResultExtras to tool blocks,
leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX
which misclassified "[Attachment saved: ...]" as JSON.
Fixes#25789
Assisted-by: Muse Spark
* Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case
* ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* server : use pytest-xdist for server tests
This commit adds pytest-xdist to the server tests. This is pytest
plugin that distributes test execution across multiple CPU cores.
Assisted-by: pi:llama.cpp/qwen3.8-27B
Refs: https://github.com/ggml-org/llama.cpp/pull/26734#issuecomment-5220707042
* remove server_base_port and BASE_PORT
* use worksteal and pytest builting tmp_path
* mtmd : mark context as const in more methods
Mark `mtmd_context` as `const` in:
- mtmd_bitmap_init_lazy
- mtmd_tokenize
- mtmd_tokenize_from_parts
- mtmd_helper_support_video
- mtmd_helper_bitmap_init_from_file
- mtmd_helper_bitmap_init_from_buf
- mtmd_helper_video_init
- mtmd_helper_video_init_from_buf
- mtmd_helper_model_can_chat
The tokenization functions in particular are useful to have marked
`const`, as that allows more easily telling the compiler that we can
safely tokenize from multiple threads (`mtmd_tokenize` is already
documented as thread-safe, this just reifies that in the signature).
* mtmd : mark tokenization input pointer as const
Mark the `bitmaps` and `parts` pointers in `mtmd_tokenize` and
`mtmd_tokenize_from_parts` as `const`. This allows more easily calling
these with immutable arrays / vectors.
* mtmd : mark llama_context as const in mtmd_helper_model_can_chat
* server : accept data: URLs for input_video and input_audio
input_video and input_audio passed accept_base64_uri=false to
handle_media(), so data: URLs got treated as raw base64 strings and
failed later with a confusing media probe error (#27724).
pass true for these two content types the same way image_url already
does, and allow video/audio mime types in the data: url check instead
of image only. data URL validation now throws std::invalid_argument so
malformed input comes back as 400 instead of 500, matching the other
input validation in this file.
* server : simplify handle_media and drop unused accept_base64_uri flag
* server : update comment and add unit test for invalid data URI MIME
* common, server : enable preserve_reasoning kwarg by default, log its effective state
If the preserve_reasoning chat template kwarg is not specified explicitly
via --reasoning-preserve / --no-reasoning-preserve, it is enabled by
default after argument processing. The server logs the effective state of
the kwarg, warns that it is enabled by default when the template supports
it, and only warns "has no effect" when it was enabled explicitly on a
template that does not support it. Setting the kwarg via
--chat-template-kwargs is deprecated.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : update comment
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
---------
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* mtmd: load the qwen3-tts code predictor proj_in as optional
The talker and the code predictor share the hidden size on the 0.6B
checkpoints, so the reference builds no small_to_mtp_projection and
the conversion emits no tensor for it. The graph already falls back
to identity when the weight is missing, the loader now agrees.
* mtmd: keep the qwen3-tts code predictor ffn_down in F32
The code predictor carries a massive activation: its layer 2 FFN
intermediate peaks around 1.5e5, well past the 65504 ceiling of F16.
mul_mat casts its input to the weight type, so an F16 ffn_down turns
that peak into inf, the residual follows, and the next rms_norm yields
NaN. Reference forward in float32 gives 145109 against 145396 measured
in the graph.
* ui: copy the displayed text of grouped agentic responses
Agentic sessions render as a single entry anchored on the first
assistant turn, whose content is typically just the first tool call,
so the copy button wrote an empty string to the clipboard. Derive the
text sections of the whole session and copy them joined, matching the
visible response. Plain messages keep the previous behavior.
* const
* rpc: avoid serializing buffers from other servers
Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization.
Assisted-by: Codex
* cont : add ref
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the
internal lazy_mode parameter, and add a -lzm shorthand. Sync the READMEs.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* Add ctx-per-slot argument for unifid KV cache
* Swap out ctx fractions for ctx pool slots
* Formatting cleanup
* Remove ctx-pool-slots, make ctx-per-slot an int
* refactor it
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* ui : strip trailing container-format segments from parsed model names
* ui : show reasoning and modality icons on model options and search by modality
* ui : keep reasoning submenu visible regardless of model state
* ui : add show-org-name-in-trigger display setting
* ui : move model list into a submenu within the model selector
* ui : make model option hover and focus highlight override the active state
* ui : add raw model id tooltip to model selector options
* feat: Enable microphone input as default for audio models
* ui : fix eslint issues in chat form and model selector
* ui: show modality icons instead of file submenu in chat add menu
Assisted-by: pi
* chore: Format
* chore: Format
* ui: add ModelCapability enum and shared modality/capability icon constants
Assisted by: pi:GLM-5.3-Flash
* ui: derive modality badge icons and labels from shared constants
Assisted by: pi:GLM-5.3-Flash
* ui: split model option icons into capabilities and modalities
Replace the supportsThinking flag on ModelId with a capabilities object
keyed like ModelModalities, so future capabilities (tool calls, etc.)
slot in alongside reasoning. Icons and labels now come from the shared
CAPABILITY_ICONS/MODALITY_ICONS constants.
Assisted by: pi:GLM-5.3-Flash
* ui: replace per-conversation MCP overrides with per-conversation tool policy
MCP server enabled state is now global (server.enabled); per-conversation
control moves to disabled tool keys and categories seeded into each new
conversation. Aligns the add sheet with the dropdown options and flattens
MCP tool groups in the tools submenu.
Assisted-by: pi
* ui: keep tool policy migration running when defaults parse fails
A corrupt disabledToolKeys localStorage entry no longer aborts the
migration; it falls through with empty defaults so legacy MCP server
overrides still get converted.
Assisted-by: pi
* ui: fall back to global defaults when agentic flow has no tool policy
Passing empty disabled sets bypassed the global defaults and could
enable tools for callers that do not pass a policy yet.
Assisted-by: pi
* ui: align preferences section headers with their methods
The Reasoning Effort and Working Directory headers sat above tool
policy methods; move them above setCwd and setReasoningEffort. Also
clarify the disabled tools JSDoc: existing rows with an unset field
have an empty policy, defaults apply only when there is no active
conversation.
Assisted-by: pi
* ui: gate MCP server avatars on conversation tool policy
Servers whose tools are disabled for the current conversation (MCP
category or server-scoped key) no longer show as enabled for the chat.
Assisted-by: pi
* ui: drop unused MCP category toggle from tools panel hook
Per-conversation MCP control is server-granular; no component renders
a whole-category toggle, so remove the dead API.
Assisted-by: pi
* ui: skip MCP init when flow policy disables the MCP category
Resolve the effective tool policy before deciding whether to
initialize MCP so flows that will not send any MCP tools skip the
init work. Callers without a policy keep falling back to global
defaults.
Assisted-by: pi
* chore: format
* ui: restore reasoning section in mobile add sheet
The sheet rewrite dropped it; the desktop dropdown still has it.
MCP Prompts and Resources stay out of the sheet on purpose.
Assisted-by: pi
* ui: clear MCP server group key in enableAllToolsForServer
The group key disables every tool of the server regardless of
per-tool keys, so re-enabling a server from Settings did nothing
while it was set.
Assisted-by: pi
* ui: skip MCP init when no policy-enabled server remains
Extends the category-level check: the flow also skips MCP init when
every globally-enabled server has its server-scoped group key
disabled in the tool policy.
Assisted-by: pi
* ui: make Settings tools tab edit defaults with category toggles
Adds per-category checkboxes and a caption stating the tab applies
to new conversations; tool picks inside a chat only affect that
chat.
Assisted-by: pi
* ui: gate cwd picker and mention picker on effective tool policy
Both checked the global disabled set directly, so a conversation
that disabled file_search still showed search as available.
Assisted-by: pi
* ui: clean up tool key helpers and store docs
Documents getEnabledToolsForLLM properly, unstacks the JSDoc at
isEntryEnabled, makes setToolEnabled persist like setCategoryEnabled
(toggleTool now delegates to it), and routes the serverId-less MCP
branch of toolKey through getMcpServerToolsKey so both key formats
come from one place. Preferences banner comments become plain
comments so they no longer read as class member docs.
Assisted-by: pi
* ui: indeterminate group checkboxes and inert grayed rows
A category that is on with nothing enabled under it now shows the
mixed checkbox state instead of a checked box next to 0/N. Rows
grayed out by a disabled parent no longer stay clickable behind
opacity.
Assisted-by: pi
* ui: gate MCP prompt and resource capabilities on tool policy
hasPromptsCapability and hasResourcesCapability accept an optional
set of usable server ids; ChatFormActions resolves it from global
enablement minus the active conversation's policy. Restores the
per-chat gating the old mcpServerOverrides provided; callers without
arguments keep global behavior.
Assisted-by: pi
* ui: remove unmounted MCP submenu component
Never rendered anywhere; its entries are duplicates (prompts and
resources live in the attachment menu, servers in the add menu and
sheet) that would need capability wiring maintained for nothing.
Assisted-by: pi
* ui: fix model information dialog width on all screen sizes
The dialog sets container-type: inline-size, so auto width ignores
its contents and collapses to padding. Give it an explicit viewport
width on mobile and cap at 60rem on desktop.
Assisted-by: pi
* ui: scroll wide chat template in model information dialog
Long unbreakable Jinja tokens blew out the table and dialog width;
the block now scrolls horizontally instead of stretching.
Assisted-by: pi
* ui: use fixed table layout in model information dialog
Auto table layout sizes columns to content min-content, so the chat
template's long lines kept inflating the dialog despite the scroll
wrapper. Fixed layout pins the first column and gives the value
column a definite width the wrapper can scroll within. min-w-0 on
the grid item guards the same path on the grid side.
Assisted-by: pi
* ui: make model information dialog full-screen on mobile
Matches the settings dialog pattern: full viewport below md,
calc-sized and capped at 60rem on desktop.
Assisted-by: pi
* ui: stack chat template row in model information dialog
Label above the block in a single full-width cell, so the template
gets the whole table width and its horizontal scroll is usable on
narrow screens.
Assisted-by: pi
* ui: scroll model information header with the content
The base dialog header is sticky; this dialog overrides it to
relative so the title and description scroll away with the body.
relative keeps the header as the close button's containing block.
Assisted-by: pi
* ui: replace literal comment text in sheet group snippet
A // line inside the Svelte snippet rendered as visible text; use an
HTML comment.
Assisted-by: pi
* ui: let indeterminate state win over checked in group checkboxes
The checkbox indicator snippet renders the check icon whenever
checked, so the mixed state never showed. Pass the checked prop
as false while indeterminate.
Assisted-by: pi
* ui: initialize only policy-enabled MCP servers for a flow
ensureInitialized accepts an optional server id set; the agentic
flow passes the servers its tool policy leaves usable, so servers
disabled for the conversation no longer get connected. Callers
without arguments keep the global behavior.
Assisted-by: pi
* ui: derive group checkbox state in useToolsPanel
Moves the mixed-state derivation out of the submenu and sheet
snippets into one getGroupCheckState accessor; the snippets just
consume checked and indeterminate.
Assisted-by: pi
* ui: gate /prompt command on the conversation tool policy
The slash command's availability now follows the same rule as the
agentic flow instead of the global capability check, so it disables
itself when the conversation's policy leaves no usable MCP server.
Assisted-by: pi
* ui: remove dead MCP prompt menu trigger chain
The /prompt slash command is the surviving trigger; the menu-button
path (onMcpPromptClick, hasMcpPromptsSupport, showMcpPromptButton,
the MCP_PROMPT attachment item and its unrendered item arrays) has
no consumer left. Message display for inserted prompts is untouched.
Assisted-by: pi
* ui: render dash for mixed-state group checkboxes
The accessor refactor dropped the checked-and-not-indeterminate
guard, so the category-on flag won and the dash never showed. The
tooltip keeps using the raw parent flag since clicking a mixed
group still disables it.
Assisted-by: pi
* ui: fix group checkbox sticking checked after disable
Clicking a mixed-state group box let bits-ui optimistically flip
its internal checked flag; the derived checked prop did not change
across the transition (both mixed and off map to checked=false),
so Svelte never applied the settled value and the check icon stuck
while the count already read 0/7.
Pass the parent flag as checked and the mix as indeterminate, so
every group toggle changes checked; render the dash on top of a
checked box for the mixed state.
Assisted-by: pi
* fix: UI for Model Information dialog
* ui: keep MCP connections stable across policy switches
ensureInitialized folds the policy into its config signature, so
alternating two conversations with different policies tore down and
reconnected every server with health checks included. Tool collection
already filters by the flow policy, so initialize every
settings-enabled server instead and never pass a policy into the MCP
config. The duplicated policy-server check becomes one accessor on
ConversationPreferences.
Assisted-by: pi
* ui: remove dead MCP resources menu trigger chain
Same shape as the earlier prompt trigger cleanup: nothing renders the
MCP resources menu button, and the only live entry into resource
browsing is Settings > MCP Servers plus the attachment resource
picker. Drop onMcpResourcesClick, hasMcpResourcesSupport,
MCP_RESOURCES_CLICK, the AttachmentItemVisibleWhen enum and
hasResourcesCapability; the resources display, browser and picker
components are untouched.
Assisted-by: pi
* Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli
* Address review comments
* Address review comments
* Add some comments in the code
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops
* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers
* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
* ui : open MCP servers in a dialog from the chat form
Replace the MCP servers submenu with a single "MCP Servers" item that opens
a new DialogMcpServers dialog instead of navigating to the /mcp-servers route.
Assisted-by: pi
* ui : browse MCP resources from the server card
Make the Resources capability badge clickable so it opens the MCP resources
browser dialog, and drop the page-only chrome from SettingsMcpServers.
Assisted-by: pi
* ui : remove mcp-servers route and sidebar entry
MCP servers are now managed in a dialog, so drop the dedicated route and the
sidebar icon that navigated to it.
Assisted-by: pi
* ui : remove unused MCP servers submenu component
The submenu was replaced by the MCP servers dialog, so delete the component
and its export.
Assisted-by: pi
* feat(ui): add DialogSettingsChat dialog
* refactor(ui): switch SettingsChat to in-app section navigation
* feat(ui): open settings as dialog from sidebar
* refactor(ui): remove settings route and URL-based settings navigation
* fix(ui): adjust MCP dialogs for new base sizing
* chore: Formatting & linting
* feat(ui): make base dialog responsive and support sticky headers
* ui: move dialog close button to the sticky header
Assisted-by: pi
* chore: Formatting & linting
* rpc: support apple RDMA as an RPC transport
* remove set_tensor micro optimization, rpc socket pinning per CR
* remove transparent reconnect
* trigger apple builds on RPC changes
---------
Co-authored-by: Ryan Churaman <rschu@meta.com>
* server: fix tool calls getting silently stripped with --prefill-assistant
Last assistant carries tool_calls + --prefill-assistant is on → request
flips into continuation mode, add_generation_prompt forced off, tail
rebuilt from reasoning_content + content only. Tool calls just vanish.
- Auto-continuation now skips trailing assistant msgs that have tool calls
- continue_final_message on those throws a clear error instead of
silently corrupting the prompt
- Regression tests included, red before / green after
Fixes#27588
Developed with AI assistance, disclosed per the contribution policy.
* server : address review: fail on prefill-assistant + trailing tool_calls
Move validation into oaicompat_chat_params_parse (next to the existing
two-or-more-assistant check) and remove it from common_chat_templates_apply,
which has no precedent for validation. Drop the regression tests.
Per review: --prefill-assistant with a trailing assistant message
containing tool calls is not supported and should fail loudly.
Aleksander Grygier