* server: accept null sampling params
Extend the schema validation to treat a null value as absent, so
clients can send null on nullable params (temperature, top_p, ...)
to request the server default. This matches the OpenAI spec and the
json_value convention used elsewhere.
Add has_field() to skip null in the field eval guards.
* has_field -> has_value
* mtmd: deepseek-ocr v1 multi-tile dynamic resolution + unified image-preprocessors for both versions (ds-ocr v1 and v2)
* remove hacky API
* fuse row into a long image
* almost working
* adapt to new preprocessor api
* rm debugging printf
* improve
* mtmd: dsocr-tiles fixes (#25481)
* ds-ocr img-preproc fuse_row tile-drop fix for multi rows and columns images
* mtmd drop the duplicate redundant img_end
* deepseekocr graph simplify CLS broadcast cleanup
* test-deepseek-ocr: relax v1 single-view tolerance; drop trailing prompt space; make DRY opt-in and n_predict model-specific (#25486)
---------
Co-authored-by: Saba Fallah <10401143+sfallah@users.noreply.github.com>
Co-authored-by: Saba Fallah <sabafallah@gmail.com>
* llama-cli: fix crash on wrong server base url by catching exceptions and graceful exit
* review: leaner catch group: json error and standard exception
downloadConversation serialized activeMessages, the root -> currNode
path, so exporting a conversation with edited or regenerated messages
dropped every alternate version and kept only the selected one.
Fetch the whole message tree via getConversationMessages so the export
carries all message versions, matching the multi-conversation export
path which already did this. Keep the active conversation as the header
source to preserve an up-to-date currNode.
Forks are separate conversations, each with its own convId, and are
exported on their own.
A model whose chat template parses at init but fails parser generation
at apply time (e.g. uses {% call %}) throws std::invalid_argument from
common_chat_templates_support_enable_thinking(), which ran outside the
try/catch guarding common_chat_templates_init(). The throw was uncaught
and llama-cli aborted (SIGABRT) instead of failing to load. Moved the
probe inside that try/catch so an apply-time error fails load the same
way an init parse error does.
Signed-off-by: Jesse LaRose <jesse@taey.ai>
* cli: move to HTTP-based implementation
* wip
* working
* remote server ok
* cli support router mode
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
* case: router with only one model
* Apply suggestions from code review
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* remove outdated comment
* use destructor instead
* add ftype
* cli-view --> cli-ui
* pimpl
* no more json in header
* nits fixes
* also show model aliases
---------
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
* server-stream : pimpl
* server-stream: prefix free functions with server_stream_
address review from ggerganov: scope the public stream functions under the
server_stream_ prefix, matching server_stream_session_manager_start/stop.
* server-stream: guard session and manager state with the mutex
address review from ggerganov: make done, completed_ts and the GC running flag plain members under their
mutex and set the condvar predicates under the lock. keep cancelled atomic for
the lock-free should_stop poll.
* server-stream: trim comments to the non-obvious
address review from ggerganov: drop comments that restate the code, keep the
concurrency, lifetime and ordering rationale. de-stale a few comments left by the
pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone.
* server-stream: update dev docs for the pimpl and prefix
reflect server_stream_session_manager_start/stop and the server_stream_ prefix,
note the manager is now a file-static singleton hidden in the .cpp
* server-stream: move stream traces to debug level
keep the bring-up traces for diagnostics but off the default log: skip
drain, draining, drain ended, DELETE evict, attach_pipe, and the router
stream resume proxy.
* server-stream: align router stream resume proxy trace with upstream
the child-side bring-up traces are already SRV_TRC on master, move the
router stream resume proxy trace to the same level.
* server-stream: move stream_read_status enum to the cpp
it is only used by the hidden session and consumer types, so it belongs
with them behind the pimpl boundary, not on the public header surface.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* feat: WIP
* feat: Retire ChatScreenProcessingInfo component, context, and keepStatsVisible settings
* feat: Always-on gauge with active-model /props, conversation stats and live-reactive reading/output/avg
* feat: Add /tokenize endpoint, TokenizeService, FNV-1a and JSON Schema utilities
* feat: Surface enabled-tools token count in context hover card
* refactor(tools): make toolsStore the sole owner of the OpenAI wire format
Previously mcpStore.getToolDefinitionsForLLM() owned the MCP->OpenAI
shape conversion (plus normalizeSchemaProperties). That created two
sources of truth for what gets sent to the LLM, with the
duplication-prone risk of the deduplicated enabled list (which feeds
the token-count cache) drifting from the bytes actually shipped on
chat.
Now:
- mcpStore: pure protocol state + routing. Drop getToolDefinitionsForLLM
and the inline OpenAIToolDefinition conversion + normalizeSchemaProperties.
Doc comment adjusted to declare wire-format ownership as belonging
to toolsStore. Connection lifecycle, health checks, executeTool,
and the connections/toolsIndex remain.
- toolsStore: owns the wire shape (added earlier this series). mcpEntries()
inlines the MCP tool conversion; uses normalizeJsonSchema (the JSON
Schema util extracted in the prior commit) so missing 'type' fields
are inferred from defaults. mcpTools getter iterates mcpEntries() so
the Settings UI and the deduplicated enabled list see the same
definitions. getEnabledToolsForLLM iterates mcpEntries() instead of
calling mcpStore, so the JSON sent to the LLM is identical to what
toolsStore.refreshEnabledToolsTokenCount tokenizes.
- agentic: the chat-completion tools field's type was annotated as
ReturnType<typeof mcpStore.getToolDefinitionsForLLM>, claiming the
shape was owned by mcpStore. Switch to ReturnType<typeof
toolsStore.getEnabledToolsForLLM>, the actual source.
Assisted-by: Claude
* feat: UI WIP
* feat: UI WIP
* feat: UI WIP
* feat: Adjust reasoning submenu layout and spacing
* feat: Adjust context usage gauge thresholds and styling
* feat: Split context usage gauge stats into current and cumulative breakdowns
* chore: Format
* refactor: Cleanup
* refactor: Cleanup
* feat: improve token gauge accuracy and display
* refactor: remove MCP recommendation gating and simplify server visibility
* feat: add token audit logging to ChatStore for debugging
* refactor: Simplify context token reading to use server promptTokens directly
* feat: Replace last-known token tracking with live server-derived stats for accurate streaming gauges
* feat: UI Improvements
* feat: Move prompt processing stats to the preceding user message
* feat: Fix context token double-counting and refine gauge layout
* refactor: remove always-show-agentic-turns setting and simplify agentic turn display
* feat: track and display cache tokens in context gauge
* feat: add diagnostic logging for chat completion requests
* refactor: improve token audit console output with fresh/cached breakdown
* fix: invalidate enabled tools token count cache on tool changes
* test: add unit tests for tools store token count invalidation
* refactor: Remove tools token counting infrastructure
* refactor: Update ChatFormContextGauge to use simplified token tracking
* refactor: Update ChatStore to remove tools token counting
* chore: Formatting
* feat: Improve UI text
* feat: simplify context usage derivation and refine gauge labels
* refactor: cleanup logs
* cleaning
* fix: UI
* refactor: Enums
* refactor: Extract context gauge logic into hook and split UI into sub-components
* refactor: Cleanup comments
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* fix: draft model fit vs load inconsistency
* refactor(server): unify draft/mtp parameter initialization, model, and context load
- moves speculative init to speculative.cpp
- changes server_context_impl model_dft and ctx_dft to use raw pointers
- fix: don't throttle progress callback when loading draft model
- refactor: rename draft model/ctx load method
* fix: valign
Before this commit, --cache-ram was not a hard limit:
- The cache always kept at least one entry, even if that entry exceeded the
RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
one, which could cause the cache to temporarily exceed the RAM/token limits
even if individual entries were below the limit.
Now, ensure that the RAM limit is strict with these changes:
- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.
Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
* server: fix deadlock in load_models() when erasing a finished download
The download monitoring thread acquires the models mutex on its way out,
but load_models() joined it from the erase loop while holding that mutex.
Join it outside the lock via threads_to_join like the other monitoring
threads.
* server: add default timeout to test requests
A hung server now fails the test after 10 minutes instead of stalling
the CI job for hours. Explicit timeouts are unchanged.
* ui: Improve performance when streaming
* ui: build sibling info map in branching utils
Moves the node map and sibling map construction from the
.by block into buildSiblingInfoMap() in branching.ts.
The map is built once per structural change and only read
afterwards, so it does not need SvelteMap reactivity. Keeping
the construction in plain TypeScript fixes the
svelte/prefer-svelte-reactivity lint error and groups the
branching logic where it already lives.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ui: migrate legacy string-encoded booleans in persisted config
* ui: enable thinking by default
Fresh users and legacy conversations without a persisted thinking
preference now default to enabled. The per-conversation toggle and
the persisted localStorage choice keep taking precedence.
Picks up the enable_thinking default from #24876.
* server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections
* server + ui: sse_ping_interval becomes a per-request body field
Address review from ngxson: the global default returns to 30 so API
clients see no behavior change, and the WebUI sends sse_ping_interval: 1
in the request body since it owns the 3s visibility-kick contract and
declares the cadence it needs. Positive values keep the existing > 0
gate, -1 keeps its disabled semantics.
* server: move sse_ping_interval into the request schema
Address review from ngxson: the field is now a typed field_num with
hard limits (-1, INT32_MAX) bound to task_params, seeded from the CLI
default alongside the other inherited parameters. The raw json_value
read and its redundant comment are gone, and schema evaluation brings
type and range validation for free.
* feat: ui: Add predefined recommended MCP servers to settings
* feat: ui: Add MCP server recommendation dialog with custom server support
* feat: Auto-focus input fields on mount and dynamic addition
* feat: Add header validation to MCP server add and edit forms
* feat: Persist recommended MCP server opt-in selections
* test: Cover MCP configuration with tests
* chore: Format & cleanup
* feat: Centralize MCP server overrides to settings config and improve recommendation UI
* fix: Capture index before mutation to prevent focus drift
* refactor: Extract MCP_CARD_VISIBLE_TOOL_LIMIT to shared constants
* refactor: Support arbitrary authorization header schemes
* refactor: Consolidate MCP recommendations dismissal into existing storage key
* fix: Use case-insensitive comparison for MCP server ID prefix check
* refactor: Centralize MCP server visibility logic and extract recommendations hook
* refactor: Cleanup
* llama : add llama_model_ftype_name()
Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Export enum
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* s/llama_model_ftype_name/llama_ftype_name/
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Move "(guessed)" to the front in llama_ftype_name
Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Add LLAMA_FTYPE_PREFIX
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Dont check for model
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* fix: Prevent tool messages from incorrectly appending to other conversations
* ui: prevent agentic loop from poisoning another conv's currNode
* ui: make editedContent a so background recompute does not wipe in-progress edits
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* common,server: handle bracketed IPv6 literals in URL authority
Parse the [host]:port form (RFC 3986) and bracket IPv6 hosts when
formatting a URL authority: listening log, proxy Host header, proxy
log, client rebuild. The per-request remote_addr stays bare.
* common: restore unsupported scheme throw in url parser
Address @ngxson review: keep the explicit reject in port resolution so
the block stays self-contained. Non-http(s) schemes still throw (also
gated at the top of common_http_parse_url).
Tests are generally prefixed with -test, so rename export-graph-ops
accordingly.
rpc-server is probably too generic a name for /usr/bin. Because it
should work with any ggml application, it is renamed to ggml-rpc-server.
* server: SSE replay buffer, survives client disconnect
Opt in on POST /v1/chat/completions when the client sends
X-Stream-Resume: 1 and a non empty X-Conversation-Id. The conv id is
the session identity end to end, no extra opaque token. The drain
runs detached server side and buffers SSE bytes, the generation
survives HTTP disconnect, F5, or lets users switch from iOS Safari
to another app without losing the actively generated response.
Routes:
GET /v1/stream/<conv_id>?from=N replay
GET /v1/streams[?conversation_id=X] list, drives sidebar spinners
DELETE /v1/stream/<conv_id> Stop, idempotent
Router parent fans out to children for list and delete, probes on GET
to route to the owner, fans out DELETE on POST so "one session per
conv" holds across model swaps.
WebUI: the layout snapshots /v1/streams at mount and on
visibilitychange, the sidebar reflects live inferences across all
convs. The chat page reattaches on mount, append vs fresh is detected
from existing content so continue mid stream keeps its prefix.
update_slots: on llama_memory_seq_rm refusal at a deep position, full
clear of the seq and reprefill from zero instead of GGML_ABORT.
OAI strict path unchanged when the opt in headers are absent.
* server: create stream session only after post_tasks succeeds
* server, ui: drop X-Stream-Resume, X-Conversation-Id alone enables the replay buffer
* server: drop magic 17, derive the X-Conversation-Id header length from sizeof at build time
* refactor: address review feedback from ngxson
* server-context: cleaning
* server-stream: fix use-after-free on rd
Guard stop_producer with a shared alive flag, flipped by on_stream_end
before rd dies. Prevents a late cancel (session eviction by a later
POST on the same conv_id, or a DELETE arriving after the producer
ended) from touching a destroyed rd.
* ui: fix cross-conversation contamination
Scope streaming flags per conv so one finishing does not unflag the
others, guard discoverActiveStream against concurrent runs to avoid
duplicate attaches, and stop racing syncRemoteRunningStreams for the
sidebar set.
* server-http: keep request alive in detached SSE drain
The response next() lambda may reach into *request via &req long
after on_complete reset the request shared_ptr. Capture request in
the detached thread so it outlives the drain.
* ui: address review feedback from coder543
Forward Authorization to /v1/stream and /v1/streams fetches, the resumable routes
must obey --api-key like the rest of the API.
Wrap reader.read() in a try/catch, the underlying connection drop rejects with
TypeError instead of resolving done=true, treat it as a premature end of stream
so the existing resume loop kicks in.
Freeze the model at session start in chatStreamingStates.model and thread it
through cancel and resume, the dropdown selection may have changed since the
POST and the server side identity is fixed at that time.
* format
* ui: remove unused selectedModelName
* server-stream: poll session->is_cancelled() in stream_aware_should_stop
Address review feedback from coder543. The cancel propagation through
rd.stop() relies on the slot eventually processing the cancel task and
posting a result that notifies the recv condvar, remove_waiting_task_ids
does not notify directly. Add a defensive poll on session->is_cancelled()
so the producer-side next() loop exits on its next iteration after
cancel() without waiting for the cancel task to round trip through a slot.
* server-stream, ui: replace GET /v1/streams with POST /v1/streams/lookup
Address review feedback from coder543. Listing live sessions leaks the
conversation_id of every concurrent user, which defeats the random UUID
unguessability. The new route takes {conversation_ids: [...]} in the
body and returns matches only for the ids the caller already owns, so
foreign UUIDs stay private. The router fans out the same POST to every
child and aggregates, the WebUI passes the convs visible in its sidebar.
* ui: read conv ids from IndexedDB in syncRemoteRunningStreams
The conversations store is not hydrated yet at +layout onMount, so the
sidebar spinners stayed off for background convs until the user clicked
on them. Read straight from the DB to dodge the init race.
* server-models: deduplicate stream lookup timeouts behind one constant
* ui: extract visibility kick grace into a stream constant, bump to 1000 ms
* make it safer & more simple
* server-stream: survive client disconnect via stream_pipe::finish_producer
After the RAII rewrite the generation stopped the moment the client
disconnected. httplib bails its content provider on the is_peer_alive
check at the top of write_content_chunked, so returning true from the
provider never keeps it producing: the response resets, rd is destroyed
and its task gets cancelled.
Reinstate the disconnect survival inside the pipe. stream_pipe gains
finish_producer, which pumps the response next() into the ring buffer
until the generation ends, and mark_producer_done for the clean wire
end. server-http only triggers them: mark before sink.done on a clean
close, finish in on_complete when the peer left early. No detach, no
stream logic in server-http beyond the trigger, and the strict OAI path
is untouched when no pipe is attached.
Known limitation: finish_producer pumps synchronously on the http
worker, so a disconnected stream keeps its worker busy until the
generation ends. A follow-up will move the drain off the http worker so
no worker is held.
* server-stream: drain disconnected streams on a manager owned thread
The previous commit pumped the post disconnect drain synchronously in
on_complete, on the http worker, so a disconnected stream kept its
worker busy until the generation ended. Under a wave of reloads or tab
closes that pins workers from the pool.
Move the drain off the http worker. on_complete now hands the response
to stream_session_manager::adopt_orphan, which pumps it to completion on
a manager owned thread and releases the worker at once. One thread per
disconnected stream still generating, stored in a list, joined and
reaped on the next adopt, by the GC, and at shutdown. No detach, the
thread lifecycle is fully owned by the manager. needs_drain gates the
handoff so a cleanly finished stream never spawns a thread, and the
strict OAI path stays untouched when no pipe is attached.
stop_gc now cancels sessions before finalizing them, so an in flight
drain sees is_cancelled and exits instead of blocking the shutdown join
until the generation ends naturally.
* ui: add missing JSDoc
* server-stream: drain on the http worker, drop the manager thread
Address @ngxson review: httplib runs a large dynamic pool and a worker
blocked in next() sits on a condvar instead of burning cpu, so draining
the rest of the generation on that worker is fine and much simpler than
a dedicated thread.
on_complete calls finish_producer directly again. Removes adopt_orphan,
the orphan thread list and its reaping, the stop_gc session cancel that
only existed to unblock those threads, and the now dead drain_shutdown
flag.
* server-stream: split stream_pipe into producer and consumer classes
Address @ngxson review: one class covering both ends was messy. stream_pipe
is now a base holding the session and is_cancelled, with stream_pipe_producer
(write, mark_producer_done, finish_producer, cleanup, finalizes on destruct)
and stream_pipe_consumer (read only, no finalize) deriving from it.
Drops the is_producer_ discriminator and its runtime guards, the type now
encodes the role. res.spipe is retyped to shared_ptr<stream_pipe_producer>
since it is only ever a producer. No behavior change.
* server-stream: rename producer methods to unix pipe semantics
Address @ngxson review: mark_producer_done becomes done(), finish_producer
becomes close(), matching a unix pipe write end. The producer_done_ member
follows as done_. write() is unchanged. No behavior change.
* server, ui: route resumable streams via a conv map, persist resume identity
Address ngxson review: drop the polling probe, proxy_post records a conv_id ->
model map and the stream routes resolve the owning child with one lookup. The
map is the single source of truth, the ::model suffix stays for child session
uniqueness but the router never parses it.
UI: the server keys a session by the POST time identity (conv::model), but reload
probed with the bare conv id and missed model tagged sessions, so F5 stopped the
stream and sidebar spinners stayed off. Persist the model and rebuild the exact
identity on resume, single conv and bulk sidebar both send it.
Add unit coverage for the identity round trip.
* ui: resolve continue target by id to stop cross-conversation flash on switch
* ui: skip stream resume when the abort is intentional
* server: move the conv id to model map into a self contained tracker
Address review from ngxson: server_models held two mutexes side by side, the
global one and a bare conv_model_mu guarding a loose map, which made the locking
hard to follow. Wrap the map and its lock in a small conv_model_tracker struct
that owns its mutex, one mutex per struct. The remember, lookup and forget
methods move inline into the tracker, server_models exposes a single conv_models
member and the routes call models.conv_models.lookup and friends. No behavior
change, the map stays the single source of truth for routing resumable streams
to a child.
* ui: replace stream magic values with enums and shared constants
Address review from allozaur: lift the inline literals around the resumable
stream code into named symbols so the intent is explicit and reusable.
* ui: fold the stream resume and discovery helpers into ChatService
Address review from allozaur: drop the two standalone stream-*.service files.
They were used only by the chat service and store, carried no shared state, and
did not follow the static class pattern the other services use, so a separate
abstraction was not warranted. Move the helpers onto ChatService as static
methods. No behavior change, tests now exercise them through ChatService.
* docs: document the SSE replay buffer in server README-dev
Add the resumable streaming section, list stream_session_manager in the
backend component inventory, and link PR 23226 in the related PRs.
* ui: align attachServerStream call with onCompletionId param in handleStreamResponse
* server-http: rename del_ to del to match get and post
* ui: address review feedback from allozaur
* ui: drop duplicate SSE constants, keep sse.ts canonical
* ui: use svelte:document for the visibilitychange listener
address review from allozaur: replace the manual document.addEventListener
in onMount with a declarative <svelte:document onvisibilitychange>. svelte
handles attach, detach and SSR, so the typeof document guard and the onMount
cleanup go away. onMount keeps only the first load snapshot.
* server: trim redundant stream drain comments
Address review from ngxson
* server: balance and clean up stream comments
remove redundant comments and tighten the verbose ones across the resumable
stream code, keeping the concurrency and lifetime rationale that is not obvious
from the code. also fix two stale comments in server.cpp and server-models.h
that still described the old ::model suffix probe and fan out routing, now
replaced by the conv_id -> model map
Address review from ngxson
* ui: balance and clean up stream comments
dedup repeated rationale (frozen conv::model identity, the lookup privacy note,
the abort patterns) down to one canonical spot, tighten the verbose blocks, and
keep the concurrency and resume-offset reasoning. fix stale comments in
stream-identity.ts and chat.service.ts that still described the old loopback
probe and fan out routing, now the conv_id -> model map.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
