Raising GLOBAL_LOG_LEVEL to WARNING buys quieter output but not less work: 241 INFO call sites interpolate their payload into an f-string before the logging call gets to drop it. The heaviest is get_doc, which logs every chunk id and metadata dict in a collection, so on the full-context retrieval path that is the entire knowledge base, once per chat request.
That one line at WARNING, CPython 3.12:
| knowledge base | payload | before | after |
| -------------- | ------- | -------- | ------- |
| top-k of 3 | 1.2 kB | 3.8 us | 0.07 us |
| 500 chunks | 201 kB | 583.6 us | 0.08 us |
| 5000 chunks | 2.0 MB | 5.8 ms | 0.15 us |
The lazy form log.info('query_doc:result %s %s', result.ids, result.metadatas) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. Output at INFO is byte-identical. Two sites that already built their message eagerly, one str concat and one % operator, move to the same lazy form.
GLOBAL_LOG_LEVEL defaults to INFO, so every log.debug(...) in the backend is discarded, but the message is built first: 187 call sites interpolate their payload into an f-string before the logging call runs, so the work happens on every request and the result is thrown away. The worst one sits in process_chat_payload and stringifies the whole request body, full conversation history included, once per chat completion.
That one line with DEBUG disabled, CPython 3.12:
| conversation | payload | before | after |
| ------------ | ------- | -------- | ------- |
| 4 messages | 1.2 kB | 3.4 us | 0.07 us |
| 20 messages | 17 kB | 24.8 us | 0.07 us |
| 60 messages | 123 kB | 216.6 us | 0.07 us |
The lazy form log.debug('form_data: %s', form_data) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. With DEBUG enabled the emitted lines are byte-identical, f'{x=}' sites included: those map to %r. MistralLoader._debug_log callers get the same treatment, since that wrapper already forwards *args.
`_json_to_metadata` parses the metadata of every result row returned by search and get. It now goes through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.
The text it parses is produced by Oracle's own `JSON_SERIALIZE`, and the column is a native `JSON` type, so the database normalises whatever was written and the reader never depends on the writer's escaping.
The matching `_metadata_to_json` write deliberately keeps stdlib `json`: it passes `default=self._decimal_handler`, orjson accepts none of stdlib's keyword arguments, and dropping the handler would turn a currently successful insert of a `Decimal` into a hard failure. The read side has no such constraint.
With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
Fourteen modules import `json` without using it. Ruff flags every one with F401, and a word-boundary search for `json` in each file matches only the import line itself, including inside strings, comments and annotations.
Two exclusions, both deliberate. Migration files are left alone: the import is equally dead there, but those files are frozen history and not worth the churn. `models/chats.py` has the same dead import and is handled in its own change, so it is skipped here to avoid two changes touching the same line.
No behaviour change.
The Valkey backend serializes chunk metadata on every insert and parses it back on every result row in `get` and `query`. Both directions now go through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.
The stored `metadata_json` field is never matched against as text. `_build_filter_expression` only emits TAG predicates, and the TAG fields are `id`, `hash`, `file_id`, `source` and `knowledge_base_id`; `metadata_json` appears only as a return field that is immediately re-parsed. So rows written with escaped non-ASCII and rows written raw are indistinguishable to every reader, and no migration is needed.
`process_metadata` already stringifies datetimes and strips null bytes and lone surrogates before the write, so the two backends cannot disagree about what is serializable here.
Both read `except` clauses widen from `(json.JSONDecodeError, TypeError)` to `(ValueError, TypeError)`. The codec falls back to engineio's codec, which installs `parse_int=_safe_int` and raises a bare `ValueError` for integer literals longer than 100 characters; the narrower clause would have let that escape and abort a search instead of yielding empty metadata. `json.JSONDecodeError` is a `ValueError` subclass, so this is a strict superset. That removes the module's last use of stdlib `json`, so the import goes with it.
With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
RAG vector search runs in a thread pool, but then calls `future.result()` on the event loop thread, so the whole worker freezes until every collection answers. Every other user's token stream stops for that long. It's the default retrieval path.
Now `asyncio.gather` over `asyncio.to_thread`, matching what `routers/retrieval.py:2779` already does for the same call.
Measured with 3 queries across 4 collections, 60 ms search, and a second request wanting a turn every 5 ms:
| | before | after |
|---|---|---|
| RAG call | 62.0 ms | 61.2 ms |
| other request's turns | 0 | 7 |
| its worst stall | 62.5 ms | 16.0 ms |
Same results, same order, same `(result, error)` contract. Cancellation now lands mid-search instead of after every thread finishes. Threads move from an unbounded per-call pool to the loop's bounded shared one.
The web-search-* namespace was the one collection namespace filter_accessible_collections admitted unconditionally for any non-admin user, on both read and write, unlike file-*, user-memory-* and knowledge bases which are owner-scoped. process_web_search now mints these ephemeral per-query collections as web-search-{user.id}-<hash>, and the access helper only admits web-search-{requester.id}-* names, so a web-search collection is readable and writable only by the user who created it (admins keep their bypass). The collections hold transient public web-search results and their names are non-enumerable query hashes, so there was no demonstrated cross-user access path; this removes the namespace exception so the per-user scoping the other namespaces enforce also covers web-search.
Co-authored-by: rexpository <rexpository@users.noreply.github.com>
The .msg branch routed to langchain's OutlookMessageLoader, which requires the extract_msg package. extract_msg pins beautifulsoup4<4.14, but we pin unstructured==0.22.31 (needs beautifulsoup4>=4.14.3) and beautifulsoup4==4.14.3, so extract_msg can never be installed alongside the current dependency set. As a result the .msg path could not function on any supported install: uploads failed at runtime with an ImportError, and adding the missing package broke the build with an unsatisfiable resolver error.
Switch to UnstructuredEmailLoader, which parses .msg through unstructured's partition_msg (backed by python-oxmsg). Both are already shipped, so .msg uploads work with no new dependency and no version conflict. Attachment partitioning is disabled to preserve the previous body-only extraction behaviour.
Fixes#26690
Custom per-connection headers can now forward the user's groups to
upstream backends via two new template placeholders:
- {{USER_GROUPS}}: comma-separated group names
- {{USER_GROUP_IDS}}: comma-separated group ids
The group lookup is async, so get_custom_headers becomes an async
wrapper around the sync template substitution (parse_custom_headers)
and fetches groups lazily — only when a header value actually
references a groups placeholder. The external document loader path
runs in a worker thread without an event loop, so Loader.aload
prefetches the groups before offloading and passes them through to
ExternalDocumentLoader.
Claude-Session: https://claude.ai/code/session_01EbBEfTyu8fFJmC13rnQthT
Co-authored-by: Claude <noreply@anthropic.com>
Add self-hosted OpenSERP as a web search engine option. OpenSERP
provides browser-rendered search across Google, Bing, Yandex, Baidu,
DuckDuckGo, and Ecosia with no API keys required.
- New module: retrieval/web/openserp.py (async, uses aiohttp session pool)
- Config: OPENSERP_BASE_URL env var (defaults to http://localhost:7070)
- Routing: search_web() dispatch for 'openserp' engine
- Follows existing patterns (searxng, brave)
Co-authored-by: crustopher-lgtm <crustopher-lgtm@users.noreply.github.com>
`_detect_text_encoding()` hands the complete file to `chardet.detect()`. chardet is pure Python and costs roughly 1.3 seconds per megabyte, so uploading a large non-UTF-8 text file stalls for seconds inside encoding detection alone. A 4 MiB Shift-JIS file spends 6.4 seconds there. The UTF-8 fast path above it means only non-UTF-8 files reach this, which in practice are exactly the CJK documents the surrounding code was written to handle, so the slow case and the case that matters are the same case.
Detection does not need the whole file. It needs the bytes that are actually not UTF-8, and `UnicodeDecodeError.start` from the fast-path decode already says where those begin, so this samples a 256 KiB window around that offset.
Two things make that safe rather than merely fast.
Centring the window on the first non-UTF-8 byte instead of the file head is what keeps the common case correct. A plain head sample makes chardet report ascii for a file that is ASCII for its first few hundred KiB and only turns CJK later, and the method then falls through to latin-1 instead of the right codec.
The window still cannot help when a stray byte, a pasted Windows-1252 artifact for example, sits hundreds of KiB ahead of the real payload: the sample is then almost pure ASCII and carries no signal. So when the sample holds almost no non-ASCII bytes and is a strict subset of the file, detection falls back to the whole buffer. That case pays the old cost, which is the right trade, because it is precisely the case where sampling would otherwise be wrong. Without this guard a Cyrillic document with a stray leading byte was detected as ISO-8859-1 rather than windows-1251, which is silent mojibake.
Measured, with the encoding returned identical in every case:
| file | before | after |
|---|---|---|
| shift_jis 4 MiB | 6402ms | 755ms |
| gb18030 4 MiB | 3199ms | 449ms |
| big5 4 MiB | 2926ms | 413ms |
| euc-jp 4 MiB | 2456ms | 413ms |
| euc-kr 4 MiB | 2382ms | 468ms |
| latin-1 4 MiB | 1902ms | 394ms |
| gb18030 1 MiB | 807ms | 376ms |
| ascii head then gb18030 tail | 533ms | 294ms |
| stray byte then cp1251 payload | 496ms | 1051ms |
| any UTF-8 file | 8ms | 0ms |
29 cases, all returning an identical encoding before and after: six encodings at 100 KiB, 1 MiB and 4 MiB, three layouts where the non-UTF-8 bytes only begin beyond the window, four where a stray byte is separated from the payload, plus plain UTF-8, UTF-8 CJK and an empty file. The stray-byte rows are slower than before because they scan twice, once over the window and once over the whole buffer. They are the pathological shape, and correctness wins there.
The residual time is now the decode-and-validate loop below, which walks the file once per candidate codec, and `_has_cjk_characters`, which is a per-character Python loop over the decoded text. Both are the same "full scan for a detection decision" pattern and could take a bounded prefix too. That is left alone here.
`alazy_load()` builds every BeautifulSoup tree inline in an async function, so a web search that pulls in ten pages stops the entire worker for the whole time it spends parsing. Nothing else on that worker runs during it: not other users' token streams, not health checks, not socket.io traffic. Parsing is CPU work and it belongs in a thread.
Measured over 37 real pages, 13.5 MiB total, with a 5ms ticker sampling event-loop lag:
| | wall | worst loop stall | ticker fired |
|---|---|---|---|
| inline, html.parser (today) | 1793.8ms | 1788.8ms | 1 time |
| offloaded, html.parser | 1872.9ms | 82.9ms | 88 times |
| inline, lxml | 1346.7ms | 1341.8ms | 1 time |
| offloaded, lxml | 1445.4ms | 37.0ms | 118 times |
Today the loop is not merely slow during a batch, it is gone: a 5ms timer fired exactly once across 1.8 seconds. After the change it fires normally and the worst single stall drops by a factor of 20 to 36. The cost is 4 to 7 percent more wall time for the batch itself, from the thread handoffs, which is the right trade for a server handling more than one user.
Three details behind the shape of the change:
`get_text()` is only 2 percent of the cost (34ms against 1706ms of parsing over the corpus), so the whole per-page unit moves into the thread rather than the parse alone. Splitting them measured worse on both axes.
The offload is per page, not per batch. Handing the whole batch to one thread measured worse than either (2081ms wall, 235ms worst stall), so the loop is yielded to between pages.
The metadata block in `alazy_load()` was a duplicate of the module-level `extract_metadata()`, field for field, and `lazy_load()` was already using the shared helper. The new helper calls it too, which is why the diff removes more lines than it adds. The `ascrape_all()` override goes with it: it was a verbatim copy of the inherited implementation and `alazy_load()` was its only caller, so anything still calling it now gets the identical parent method, which resolves `self._unpack_fetch_results` to the override this class keeps.
Verified by feeding the real loader a 37 page corpus and comparing every resulting Document against the implementation this replaces:
```
PASS one Document per url (37)
PASS every Document identical to the pre-change implementation (0 differ)
PASS parsing ran off the main thread
PASS event loop kept running during parsing (90 ticks)
```
Both `page_content` and `metadata` are byte-identical on all 37 pages. This is independent of the parser in use and composes with switching the default parser to lxml: that change makes the stalls shorter, this one takes them off the loop.
Every page pulled in by web search and web RAG is parsed with BeautifulSoup's `html.parser`, a pure-Python parser. It is the slowest option bs4 offers, and it is being handed 300 KiB to 1.5 MiB documents, several per query. `SafeWebBaseLoader` inherits `default_parser = "html.parser"` from langchain's `WebBaseLoader` and never overrides it, so this is an upstream default carried by accident, not a decision anyone made for Open WebUI.
`default_parser` is the single chokepoint for both the sync `_scrape()` path and the async `ascrape_all()` path, so one `setdefault` covers everything and an explicit caller override still wins.
lxml is already in the tree as a transitive hard dependency of ddgs, python-pptx and unstructured, so nothing new enters the image and `uv.lock` already resolves it at 6.1.1. The pin makes it explicit and closes a latent failure: bs4's `"xml"` feature, already used for `.xml` URLs in `_unpack_fetch_results()`, requires lxml and would raise `FeatureNotFound` the day that transitive dependency moves.
## Benchmarks
37 real pages, 13.8 MiB of HTML, median of 5 runs each. The timed operation is `BeautifulSoup(html, parser)` plus `get_text()` plus `extract_metadata()`, which is exactly what the loader does per page. bs4 4.14.3, lxml 6.1.1, CPython 3.12.
| | html.parser | lxml | |
|---|---|---|---|
| 37 pages, 13.8 MiB total | 1611.0ms | 1151.3ms | 1.4x faster, 460ms saved |
Largest pages:
| page | size | html.parser | lxml | speedup |
|---|---|---|---|---|
| pypi.org/project/aiohttp/ | 1259 KiB | 243.75ms | 180.51ms | 1.4x |
| gnu.org/software/bash/manual/bash.html | 1017 KiB | 257.97ms | 178.99ms | 1.4x |
| rfc-editor.org/rfc/rfc9110.html | 1157 KiB | 205.94ms | 154.87ms | 1.3x |
| docs.aiohttp.org/en/stable/client_reference.html | 403 KiB | 108.62ms | 84.93ms | 1.3x |
| ollama.com/library | 779 KiB | 117.55ms | 73.64ms | 1.6x |
| theregister.com | 1052 KiB | 88.32ms | 60.12ms | 1.5x |
| kubernetes.io/docs/concepts/services-networking/service/ | 563 KiB | 72.18ms | 43.43ms | 1.7x |
| docs.python.org/3/library/socket.html | 301 KiB | 71.88ms | 49.04ms | 1.5x |
Ranges from 1.1x to 1.7x, and the win grows with page size. A ten result web search sheds roughly 125ms of parsing. Because the async path builds its soups inline in `_unpack_fetch_results()`, that is 125ms the event loop spends parsing HTML instead of serving other users' streams. Pages under about 10 KiB are marginally slower under lxml due to fixed setup cost, which is worth nothing either way.
## Output verification
The risk in changing parser is silently different extracted text, so that was measured rather than assumed. Across all 37 real pages:
- **Zero characters of text were lost.** Every diff opcode against html.parser output was an insertion. Not one page dropped content under lxml.
- 659 characters were added, all on one page (docs.docker.com), where an inline Alpine.js `@click` handler containing a regex confuses libxml2's attribute handling and leaks a 73-character JS fragment into the text nine times. That is 659 characters of script noise in 27,206 characters of extracted text, with no content affected.
- Metadata (`title`, `description`, `language`) was identical on 35 of 37 pages. The two exceptions are 141-byte Wikipedia bot-block stubs with no `<html>` element, where lxml's fragment auto-wrapping adds `language: "No language found."`. Both parsers extract the same text from them.
Large documents were checked separately because libxml2 carries internal size caps. A 12 MiB single text node, 12 MiB spread across 400k nodes, a 3 MiB attribute value and 50k sibling elements with a trailing marker all produced byte-identical text under both parsers, with no truncation.
Malformed markup was checked too. lxml and html.parser diverge on unterminated comments, bare CDATA and duplicated `<html>` elements, all cases where both parsers are guessing and neither is correct. None of those shapes appeared in the 37 page corpus.
`backend/open_webui/env.py:184` also uses `html.parser`, on the local CHANGELOG at import time. That is trivial input on a startup path and is deliberately left alone.
* refac: use MilvusClient instead of deprecated ORM-style PyMilvus APIs
PyMilvus 2.6 emits a PyMilvusDeprecationWarning for every ORM-style call (`connections.connect`, `utility.*`, `Collection` and its methods) and will remove those APIs in PyMilvus 3.1. Both Milvus backends still used them, so a running instance floods its logs with deprecation warnings during indexing and retrieval, and would break outright once PyMilvus 3.1 lands.
Both vector clients now go through `MilvusClient`:
- `milvus_multitenancy.py`: collection creation, index creation, has_collection, insert, search, query iteration, delete and reset.
- `milvus.py`: the remaining ORM calls in `query()` (`connections.connect`, `Collection(...).load()`, `Collection.query_iterator`), plus the now-unused `FieldSchema` import.
Behaviour is unchanged: same schema, same index parameters and the same two-step scalar-index fallback, same filter expressions, same result shapes. Verified against embedded Milvus (milvus-lite, pymilvus 2.6.14) with a functional harness over both clients: insert, get, query by string/int/bool metadata filters, vector search, tenant isolation, oversized-text truncation, delete by id and by filter, delete_collection and reset all return identical results before and after, while the deprecation warnings drop from 57 to 0 for the multi-tenancy client and from 16 to 0 for the standard one.
One Milvus Lite nuance worth recording: `MilvusClient` sends index build parameters (`M`, `efConstruction`, `nlist`) as flat keys rather than as a nested `params` blob. A Milvus server accepts both forms, Milvus Lite only reads the nested one, so those tuning values are ignored on Lite. `MilvusClient` offers no way to send the nested form, and `milvus.py` already built its index parameters this way, so both backends are now consistent.
Fixes#26978
* refac: correct the Milvus scalar-index comment
The comment claimed that embedded Milvus Lite requires an explicit scalar index type. It does not: Milvus Lite rejects `create_index` on a VARCHAR field outright ("create_index only supports vector fields"), for every index type and with or without a metric type, so neither the parameterless call nor the explicit INVERTED fallback can succeed there. Filtered queries on `resource_id` still work on Lite, just unindexed.
Only the accurate half is kept, which is the reason the parameterless call is deliberate rather than an omission.
`from datetime import datetime, time, timedelta` shadows the `time` module, so `RateLimitMixin._sync_wait_for_rate_limit` calls `datetime.time.sleep` and raises `AttributeError: type object 'datetime.time' has no attribute 'sleep'` whenever it actually has to wait.
Every synchronous loader path that paces requests hits this. `SafeFireCrawlLoader.lazy_load` calls the limiter directly, and Tavily, Microsoft Web IQ and Playwright reach it through `_safe_process_url_sync`. The exception is raised inside their per-URL `try`, so with `continue_on_failure=True` (the default) the URL is logged as a per-URL failure and dropped instead of being scraped. This is live by default: `WEB_LOADER_CONCURRENT_REQUESTS` is passed as `requests_per_second` and defaults to 10, so any URL whose predecessor finished within 100ms takes the sleep branch and is lost. Tavily and Microsoft Web IQ report it as "SSL verification failed", which points at the wrong cause.
`_wait_for_rate_limit` uses `asyncio.sleep` and is unaffected, but `SafeMicrosoftWebIQLoader.alazy_load` runs `lazy_load` in a threadpool, so its async entry point is affected too.
`datetime.time` is not used anywhere in the file, so importing the `time` module instead is enough.
The per-URL `continue` half of #26079 landed in 6f8221df5, which also added the `_sync_wait_for_rate_limit()` call to the Firecrawl loop. This makes that call work rather than throw.
Fixes#26079
`SafePlaywrightURLLoader` opened a new Playwright page for every URL and never closed it, and it only closed the browser after the URL loop finished normally. Pages therefore piled up for the whole batch, and any early exit (a raised error with `continue_on_failure=False`, or the caller abandoning/cancelling the generator mid-search) skipped `browser.close()` entirely.
With `PLAYWRIGHT_WS_URL` pointing at a remote Playwright server this leaks sessions on that server: navigation and route timeouts on slow or bot-protected pages leave pages and browser connections open until the server is restarted, which degrades every later web search.
Both `lazy_load()` and `alazy_load()` now scope the page to the per-URL loop body and the browser to the whole loop using their context managers, so each page is closed as soon as its URL is done and the browser is closed on success, on failure, and on cancellation. Closing a page also disposes the context implicitly created by `new_page()`. Exception handling is unchanged: a close error raised while `continue_on_failure` is set is still caught, logged, and the loop continues.
Fixes#25880
When `RAG_DOCUMENT_LOADER_ENGINE` is set to `paddleocr_vl`, the dispatch branch in `Loader._get_loader` checked only the engine name and a non-empty token, so every uploaded file was handed to the PaddleOCR-VL loader regardless of its type. Text based uploads such as `.md`, `.txt` and `.csv` were base64 encoded and posted to the `/layout-parsing` endpoint tagged as PDFs, and the API rejected them with `422 Unprocessable Entity` ("PDFium: Data format error"), so those files never indexed at all.
The loader already knows which extensions it can handle: it tags images with `fileType: 1` and treats everything else as a PDF. That list is now a module level constant, and the dispatch branch gates on `['pdf'] + images`, the same way `mistral_ocr`, `datalab_marker`, `document_intelligence` and `mineru` already limit themselves. Deriving the gate from the loader's own list keeps the two in sync, so a file can never be admitted by the gate and then mislabelled as a PDF on the wire. Everything outside that set falls through to the default loader chain, so `.md` and `.txt` load as text, `.csv` through `CSVLoader`, `.docx` through `Docx2txtLoader`, and so on.
The branch also never checked `PADDLEOCR_VL_BASE_URL`. With the URL cleared, `PaddleOCRVLLoader` raised `ValueError` from its constructor and the upload failed outright instead of falling back. Both settings are now required for the branch to be taken, matching how the other engines guard their own configuration.
Fixes#24988Fixes#26759
Since the config refactor, get_web_loader dispatched on the WEB_LOADER_ENGINE module constant, which is read from the environment once at import time. The engine selected in the Admin UI is stored under web.loader.engine in the config table but was never consulted, so UI-configured loader engines (external, playwright, firecrawl, tavily, microsoft_web_iq) were silently ignored and the built-in SafeWebBaseLoader always fetched pages directly. The same applied to the per-engine settings such as the external web loader URL and API key. This breaks egress-restricted deployments that rely on an external web loader: pages are fetched directly from the container and fail with errors like "Network is unreachable" even though an external loader is configured.
Pass the DB-backed loader settings into get_web_loader from both call sites, web search in process_web_search and web fetch via get_loader, and resolve every engine setting from them, keeping the module-level env constants as the fallback for keys that were never saved. Also initialise WebLoaderClass so an unknown engine raises the intended ValueError instead of an UnboundLocalError.
Fixes#26747
_ssrf_safe_new_conn unpacks each entry of self.socket_options straight into socket.setsockopt(), which accepts exactly 3 positional arguments. urllib3-future, a drop-in fork that shadows the urllib3 package whenever it is installed (for example as a dependency of niquests pulled in through a tool or function's requirements), declares its default socket options with a per-protocol 4th element: [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1, "tcp")]. Its own _set_socket_options() strips that element before calling setsockopt(), but our override does not, so with urllib3-future present every synchronous web fetch (fetch_url, web search loading) fails on connect with "TypeError: setsockopt() takes exactly 3 arguments (4 given)" and returns empty content.
Mirror urllib3-future's handling in the override: for 4-element options whose last element is a protocol string, apply "tcp" options truncated to the first 3 elements and skip "udp" options (all sockets created here are SOCK_STREAM). Plain 3-element options, and any other shapes stock urllib3 would accept, are passed through unchanged, so behavior with stock urllib3 (which only ever uses 3-element tuples) is identical.
Verified locally: with urllib3-future installed the loader previously raised the TypeError on every URL and now fetches successfully; with stock urllib3 2.3.0 and 2.7.0 fetches behave the same before and after.
Note: #26015 reported this same crash but attributed it to stock urllib3 2.x, which only uses 3-element tuples; the 4-element form comes from urllib3-future shadowing urllib3.
Fixes#26791
SafeWebBaseLoader._unpack_fetch_results assigned the resolved parser to the parser parameter itself, so the None check only ran for the first URL. In a mixed batch every later document was parsed with whatever the first URL happened to select: an .xml feed first meant all following HTML pages went through the xml parser (broken text extraction), and an HTML page first meant .xml URLs were parsed as HTML. Web search regularly fetches mixed batches, so this silently degraded extraction quality depending on result order.
The parser is now resolved per URL; an explicitly passed parser still applies to the whole batch as before. Verified with mixed xml/html batches in both orders and with an explicit parser override.
Mirrors the ENABLE_FORWARD_USER_INFO_HEADERS pattern already used by
the audio/TTS and external document loader integrations, so the
Mistral OCR backend can identify the requesting user the same way.
Co-authored-by: andrep <vpham@aut.ac.nz>
Enabling ENABLE_MILVUS_MULTITENANCY_MODE with the default MILVUS_URI (embedded Milvus Lite at DATA_DIR/vector_db/milvus.db) fails on the first embedding write: _create_shared_collection calls collection.create_index(RESOURCE_ID_FIELD) with no index params. A Milvus server auto-selects a scalar index type in that case, but Milvus Lite rejects the call with "create_index missing required 'index_type' parameter", so shared collection creation raises and every embedding write 500s (memory add, file upload, knowledge writes).
Keep the parameterless call as the first attempt so behavior on Milvus servers is unchanged, fall back to an explicit INVERTED scalar index, and if that also fails log a warning and continue. The scalar index only accelerates resource_id filters; inserts and filtered queries work without it, so a missing index must not break collection creation.
Verified against embedded Milvus Lite: shared collections now create (with the warning), and memory add, file upload and memory query succeed end to end. Against a Milvus server the first attempt is identical to the current code, so nothing changes where it works today.
Audit of asyncio.sleep vs time.sleep and event-loop-blocking calls:
- utils/plugin.py: run pip `install_frontmatter_requirements`
(subprocess.check_call) via asyncio.to_thread in load_tool_module_by_id,
load_function_module_by_id, and install_tool_and_function_dependencies.
- retrieval/utils.py: move the synchronous SSRF-guarded requests probe and
loader.load() in get_content_from_url into a sync helper run via
asyncio.to_thread.
- routers/audio.py: write uploaded audio to disk off the event loop in
transcription().
- routers/pipelines.py: write uploaded pipeline file off the event loop in
upload_pipeline().
The existing time.sleep call sites are all in genuinely synchronous
functions (sync requests/DB drivers/daemon threads) with async
counterparts that already use asyncio.sleep, so no time.sleep -> asyncio.sleep
changes were needed.
Claude-Session: https://claude.ai/code/session_01LXR5bYfsfSS42RGHQZu2Ta
Co-authored-by: Claude <noreply@anthropic.com>
Web search results stored as a vector collection were silently dropped
before retrieval when BYPASS_RETRIEVAL_ACCESS_CONTROL is False (the
default). The server-generated web_search file item carries a
'collection_name' but its 'web_search' type is not matched by any
explicit dispatch branch in get_sources_from_items, so it fell through
to the untrusted client-supplied collection_name branch and was ignored.
Add an explicit branch for type == 'web_search' items so the collection
is queried again. Access control is preserved: the collection still
passes through filter_accessible_collections, which already allowlists
web-search-* and bypasses only for admins.
Regression introduced when the retrieval access-control hardening gated
the bare collection_name fallback behind BYPASS_RETRIEVAL_ACCESS_CONTROL.
