Swap the JSON encoder/decoder used across the backend from stdlib json to
orjson when ENABLE_ORJSON is set — HTTP request bodies, JSONResponse
bodies, upstream provider responses, SSE chunks, and socket.io/Redis
payloads.
The flag defaults to off, in which case the app uses stdlib json and
engineio's codec verbatim, so default behaviour is unchanged.
- json_codec exports JSONCodec (stdlib json or the orjson codec) and
SOCKETIO_JSON (engineio's codec or the orjson codec); call sites import
JSONCodec and stay implementation-agnostic
- apply_orjson_http_json() is a no-op when the flag is off, leaving
starlette's Request.json / JSONResponse.render untouched
- the orjson codec falls back to the stdlib for inputs orjson rejects
(non-str dict keys, ints beyond 64 bits, NaN literals)
- orjson is imported only when the flag is on
- FastAPI(default_response_class=...) is deliberately not used: an
explicit default disables the Pydantic direct-to-bytes fast path for
response_model routes
* fix: fetch terminal system prompt per request with TTL cache
The system prompt was only fetched once in set_terminal_servers (startup
or connection save) with a 3s timeout, using a synthetic 'system' user.
That snapshot silently stays empty when the fetch races a cold-started
orchestrator instance, and goes stale when instances are reprovisioned
with a changed OPEN_TERMINAL_SYSTEM_PROMPT — recovering only after a
restart or a manual connection re-save.
- Fetch /system during get_terminal_tools with the user's own
credentials via a central TTL-cached method (5 min per server+user;
failures cached 60s so a dead instance doesn't stall every request),
falling back to the cached snapshot.
- Raise the fetch timeout from 3s to 30s so cold-provisioned instances
can answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KjnQJNKozp47vTB13pRyYs
* refac: fetch the terminal system prompt per request without a cache
Drop the module-level TTL cache and fetch the system prompt directly in
get_terminal_tools, gathered with the existing uncached per-request cwd
fetch that already follows this pattern. The fetch uses the user's own
credentials and falls back to the set_terminal_servers snapshot, so a
cold or unreachable instance degrades to the previous behaviour instead
of needing an error cache. Also restore the 3s timeout: on the request
path a 30s wait would stall chat completions, and a cold instance is
covered by the snapshot fallback until it warms up.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix: preserve system prompt across tool calls when memories are enabled
The native tool-call loop runs generate_chat_completion with
bypass_system_prompt=True, so the provider layer does not re-apply the
model's default system prompt on tool-call iterations. It relies instead
on metadata['system_prompt'], captured in process_chat_payload, to carry
the full system prompt forward and restore it after RAG injection.
That capture read the model default system prompt from
form_data['params']['system'], but apply_params_to_form_data had already
popped 'params' from form_data, so model_system_prompt was always empty.
metadata['system_prompt'] therefore only captured whatever was already
materialized in the messages. With memories enabled, that is the injected
<memory_context> system message, so tool-call requests were restored with
memory-only system content and the model's system prompt was dropped.
Without memories there was no system message to capture at all.
Capture the model default system prompt from form_data['params'] before
apply_params_to_form_data pops it, and use that value when building
metadata['system_prompt'].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Z4L51mvxDJCx1EDxP2vFN
* refactor: condense system prompt capture comment to a single line
Replace the four-line explanation above the model_system_prompt capture with a one-line note. The variable name and the surrounding code already convey what happens; the comment only needs to state why the capture sits before apply_params_to_form_data.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Previously, when web search retrieved pages successfully but saving them to the vector DB failed (for example an unreachable or misconfigured embedding endpoint), process_web_search swallowed the exception at debug log level and still returned status: True with the collection name. The chat then showed "Searched N sites" followed by "No sources found" at retrieval time, hiding the actual misconfiguration from the user and making the failure look like a search bug.
process_web_search now logs the failure at exception level and raises an HTTPException with an actionable message pointing at the embedding configuration in Admin Settings > Documents. chat_web_search_handler surfaces the detail of any HTTPException raised during the search in the emitted error status, so the real cause (embedding misconfiguration, search engine errors, no results) is shown in the chat UI instead of the generic "An error occurred while searching the web". Non-HTTP exceptions keep the generic message, so raw internal error strings are not exposed.
Ref #26750, #25038
The legacy features block in process_chat_payload honoured client-supplied features.image_generation and features.web_search flags and dispatched to the image generation/edit provider and the web-search provider without re-checking the per-user permission that the direct /images routes and the native function-calling path enforce. A user denied features.image_generation or features.web_search could still trigger billable server-side image generation or web search via POST /api/chat/completions with params.function_calling set to legacy. Gate both branches on admin-or-has_permission before invoking chat_image_generation_handler / chat_web_search_handler, matching the existing code_interpreter gate, so a forged flag from an unpermitted user is ignored. Normal completions and permitted users are unaffected.
The chat completion entry point fetched the model row and then check_model_access immediately fetched the exact same row again. Inside the check, the direct grant lookup and every hop of the base-model chain each refetched the caller's group memberships, because neither call passed user_group_ids even though both AccessGrants.has_access and has_base_model_access already accept it.
check_model_access now takes an optional prefetched model_info (used only when its id matches the requested model, so stale callers cannot bypass the lookup) and resolves the caller's group ids once, sharing them across the direct check and the whole base-model chain. The group fetch is skipped entirely for the owner-with-no-base-chain case, which previously needed no groups either.
DB round trips for one completion-entry access check (non-owner model with one base-model hop):
| queries | before | after |
| --- | --- | --- |
| model row SELECTs | 3 | 2 |
| group membership SELECTs | 2 | 1 |
For deeper base-model chains the before column grows by one group SELECT per hop; the after column stays at one.
Functionally verified with stubbed model, group and grant accessors: owner fast path issues no group or grant queries; a non-owner with a base chain resolves groups once and passes the same set to every hop; a prefetched matching model_info skips the duplicate row fetch while a mismatched one is refetched; denial and unknown-model cases still raise; the arena path is unchanged.
convert_payload_openai_to_ollama deep-copied the entire request payload on every completion routed to an Ollama model, and again on every tool-call iteration. The cost of that copy scales with the number of messages and nested content parts in the history, so long chats pay the most, purely as CPU work before the request even leaves the server.
The function only ever mutates two things: it deletes keys on the top-level dict and on the nested options dict. convert_messages_openai_to_ollama already builds fresh message dicts. Shallow-copying exactly those two levels therefore preserves behavior while removing the whole-tree copy.
Benchmark (per conversion call):
| payload | before | after | speedup |
| --- | --- | --- | --- |
| 200-message text chat (~180 KB) | 0.22 ms | 0.057 ms | 4x |
| 20-message chat + 1 MB base64 image | 0.41 ms | 0.38 ms | 1.1x |
The image row barely moves because deepcopy shares immutable strings; the win comes from container-heavy histories, which are exactly the payloads that grow over a conversation's lifetime.
The output is byte-identical to the previous implementation (verified against it, including dict key order, root parameter hoisting, max_tokens remapping, stop handling and response_format precedence), and the caller's payload is left unmodified exactly as before.
stream_wrapper without a content handler iterates aiohttp's response.content, which reads line by line: every line costs a buffer scan, a slice, a bytes concat, a generator resume and its own ASGI response message. A typical SSE event is two lines (the data line and the blank separator), so every upstream token event became two yields and two transport writes even on routes where the body is never inspected.
stream_wrapper now takes passthrough=True, which iterates response.content.iter_any(): the exact same bytes, one yield per network read, no line scanning. It is applied only to routes no internal consumer parses line-by-line: the ollama pull/push/create/generate proxies and its v1 completions, chat completions, messages and responses endpoints, plus the openai embeddings, responses and catch-all proxies. The two internally consumed chat routes keep line iteration, which the streaming middleware and the Ollama-to-OpenAI converter require; the ollama send_request signature documents that constraint.
Benchmark (local aiohttp SSE server, 500 events, consumed through stream_wrapper):
| metric | before (readline) | after (iter_any) |
| --- | --- | --- |
| stream consumption time | 1.46 ms | 0.62 ms |
| generator yields + response writes per stream | 1000 | 1 |
The single yield is a loopback artifact (the whole body arrives in one buffered read); over a real network it becomes one yield per TCP read instead of two per SSE event.
Functionally verified: line mode and passthrough mode produce byte-identical output for the same stream, and passthrough always yields fewer, larger chunks.
_process_picture_url validated the picture URL with validate_url() but then fetched it with a plain aiohttp session that resolves the hostname again at connect time, leaving a DNS-rebinding TOCTOU window (the same gap already closed for the RAG loader, the content probe, the image fetches and webhook delivery). Routing the fetch through get_ssrf_safe_session() pins the connect-time resolution via _SSRFSafeResolver and rejects non-global addresses, so a rebinding host can no longer redirect the fetch to loopback, RFC1918 or cloud-metadata endpoints. It also stops the forwarded OAuth access_token from leaking to a rebound internal target.
Custom per-connection headers can now forward the user's groups to
upstream backends via two new template placeholders:
- {{USER_GROUPS}}: comma-separated group names
- {{USER_GROUP_IDS}}: comma-separated group ids
The group lookup is async, so get_custom_headers becomes an async
wrapper around the sync template substitution (parse_custom_headers)
and fetches groups lazily — only when a header value actually
references a groups placeholder. The external document loader path
runs in a worker thread without an event loop, so Loader.aload
prefetches the groups before offloading and passes them through to
ExternalDocumentLoader.
Claude-Session: https://claude.ai/code/session_01EbBEfTyu8fFJmC13rnQthT
Co-authored-by: Claude <noreply@anthropic.com>
