upstream-archive/open-webui
1
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-08-06 08:00:51 -05:00
Code Issues Packages Projects Releases Wiki Activity

perf: stop resolving DNS on the thread pool (add aiodns) (#27440)

Browse Source 
aiohttp resolves every hostname with ThreadedResolver unless the aiodns package is importable, and ThreadedResolver runs socket.getaddrinfo on asyncio's default ThreadPoolExecutor. That executor is capped at min(32, cpu_count + 4) threads and is shared with every other piece of blocking work posted to it, so DNS is currently a bounded blocking resource sitting in front of every model call, every web search fetch, every RAG page load and every tool call. In plain terms: once that pool is busy, requests wait on name lookups that should never have occupied a thread at all.

This is a dependency-only change. aiohttp sets `DefaultResolver = AsyncResolver` as soon as aiodns is importable (aiohttp/resolver.py), so resolution moves onto the event loop via c-ares with zero application code touched. That is deliberate rather than lazy: there are 50 `aiohttp.ClientSession(...)` construction sites in the backend, most building a fresh default connector per call, and the alternative of passing `resolver=aiohttp.AsyncResolver()` explicitly would mean touching all of them and re-touching every future one. The shared pool in `utils/session_pool.py` does set `ttl_dns_cache`, but that only helps the shared pool. Every per-request session, including `SafeWebBaseLoader._fetch()` which builds a new session per URL, starts with a cold DNS cache and resolves from scratch.

It also unbreaks a code path that is dead today. `backend/open_webui/retrieval/loaders/mistral.py:480` constructs `aiohttp.AsyncResolver()` unconditionally, and `AsyncResolver.__init__` raises `RuntimeError("Resolver requires aiodns library")` when aiodns is absent, so the Mistral OCR content extraction engine fails on a stock install. This supplies the dependency that line already assumes. Once it is present that kwarg is redundant, since it now names the default, and dropping it is a reasonable follow-up. Reproduced by blocking the aiodns import:

```
aiodns importable: False
DefaultResolver: ThreadedResolver
AsyncResolver(): RuntimeError: Resolver requires aiodns library
```

## Benchmarks

Both sides run the real aiohttp resolver classes. The DNS wire time is replaced by an identical fixed 50ms delay on both sides, so the only variable measured is where that delay is spent. 24 cores, so the default executor holds 28 threads. `exec_max` is the worst latency an unrelated `run_in_executor` job suffered while the lookups were in flight.

Concurrent lookups, wall time:

| concurrent lookups | ThreadedResolver | AsyncResolver | speedup | exec_max before | exec_max after |
|---|---|---|---|---|---|
| 16 | 54.3ms | 41.0ms | 1.3x | 2.1ms | 1.9ms |
| 32 | 101.9ms | 50.6ms | 2.0x | 36.6ms | 1.8ms |
| 64 | 152.5ms | 43.2ms | 3.5x | 88.4ms | 2.0ms |
| 128 | 254.9ms | 44.5ms | 5.7x | 190.3ms | 2.2ms |
| 256 | 508.2ms | 50.7ms | 10.0x | 443.8ms | 2.4ms |
| 512 | 965.2ms | 47.8ms | 20.2x | 900.2ms | 2.6ms |

ThreadedResolver scales linearly with concurrency because it can only run 28 lookups at a time. AsyncResolver stays flat at roughly the cost of one lookup.

The reverse direction is worse and is not hypothetical. Open WebUI already posts long blocking jobs to that same executor (`retrieval/vector/dbs/pinecone.py:323` batch upserts, `retrieval/loaders/youtube.py:156` transcript loads). With 28 such jobs holding the pool, a single DNS lookup waits for them to finish:

| | one DNS lookup |
|---|---|
| ThreadedResolver | 1989.7ms |
| AsyncResolver | 58.9ms |

A Pinecone bulk upsert currently stalls name resolution for every other user on the instance. After this change it cannot.

At low concurrency on a real network the two are equivalent, as expected: 8 concurrent lookups against disjoint cold hostname sets landed within noise of each other in both directions.

## Behaviour verification

Checked against Open WebUI's own code, not in isolation:

- c-ares reads the system hosts file. Verified against a machine whose hosts file maps `adobe.io` to `0.0.0.0`, an address real DNS never returns for that name: c-ares returned `0.0.0.0`. `host.docker.internal`, compose `extra_hosts` and Kubernetes `hostAliases` keep working.
- `_SSRFSafeResolver` subclasses `aiohttp.resolver.DefaultResolver`, so this change swaps its base class from ThreadedResolver to AsyncResolver at runtime. It still resolves public hosts, still returns entries with the `host`/`port` keys the SSRF check reads, and still raises on a private address: resolving `localhost` raised `ValueError: The URL you provided is invalid.`
- A real fetch through `get_ssrf_safe_session()` returned 200.
- NXDOMAIN still surfaces as `aiohttp.ClientError` (`ClientConnectorDNSError`), not a c-ares specific exception, so existing error handling is unaffected.

Known limit: c-ares reads `/etc/resolv.conf` and the hosts file but not the rest of `nsswitch.conf`. Names served only by an NSS module, such as `.local` via avahi/mDNS, NIS/LDAP backends or Windows NBNS, will resolve differently or not at all. On a multi-homed test machine the local hostname returned two addresses through the system resolver and one through c-ares. Deployments pointing Open WebUI at an mDNS or NetBIOS hostname are the group affected. Resolver failures also arrive as plain `OSError` rather than `socket.gaierror`, which no code in this repo catches today.
This commit is contained in:
Classic298
2026-07-27 05:29:52 +02:00
committed by GitHub
parent f32b19c1f6
commit 2196b4e1ff
3 changed files with 51 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
backend/requirements.txt
View File

@@ -14,6 +14,7 @@ joserfc==1.7.4
requests==2.34.2
aiohttp==3.13.5 # do not update to 3.13.3 - broken
aiodns==4.0.4 # makes aiohttp resolve DNS on the event loop instead of the threadpool
async-timeout==5.0.1
aiocache==0.12.3
aiofiles==25.1.0

1
pyproject.toml
View File

@@ -22,6 +22,7 @@ dependencies = [
"requests==2.34.2",
"aiohttp==3.13.5", # do not update to 3.13.3 - broken
"aiodns==4.0.4", # makes aiohttp resolve DNS on the event loop instead of the threadpool
"async-timeout==5.0.1",
"aiocache==0.12.3",
"aiofiles==25.1.0",

49
uv.lock generated
View File

@@ -37,6 +37,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/37/d7/15d67e05b235d1ed8c3ce61688fe4d84130e72af1657acadfaac3479f4cf/aiocache-0.12.3-py2.py3-none-any.whl", hash = "sha256:889086fc24710f431937b87ad3720a289f7fc31c4fd8b68e9f918b9bacd8270d", size = 28199, upload-time = "2024-09-25T13:20:22.688Z" },
]
[[package]]
name = "aiodns"
version = "4.0.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pycares" },
]
sdist = { url = "https://files.pythonhosted.org/packages/9b/22/a2d928e0e42baad0471d12ec44c71152ac870486e8298dddb2893b888c29/aiodns-4.0.4.tar.gz", hash = "sha256:cb10e0c0d2591636716ad2fe402e977c16d71bdaf76bb8cb49e8a6633596f736", size = 29918 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/7f/70/72e4ab117425ccdc4d10bd523a94c1baa051a15586057d64a4c6888f9e3f/aiodns-4.0.4-py3-none-any.whl", hash = "sha256:c24dd605bac70a1676ce503f967a98483ff163507198557d8e9db16267e6cfd2", size = 12696 },
]
[[package]]
name = "aiofiles"
version = "25.1.0"
@@ -2999,6 +3011,7 @@ source = { editable = "." }
dependencies = [
{ name = "accelerate" },
{ name = "aiocache" },
{ name = "aiodns" },
{ name = "aiofiles" },
{ name = "aiohttp" },
{ name = "aiosqlite" },
@@ -3134,6 +3147,7 @@ dev = [
requires-dist = [
{ name = "accelerate", specifier = "==1.13.0" },
{ name = "aiocache", specifier = "==0.12.3" },
{ name = "aiodns", specifier = "==4.0.4" },
{ name = "aiofiles", specifier = "==25.1.0" },
{ name = "aiohttp", specifier = "==3.13.5" },
{ name = "aiosqlite", specifier = "==0.22.1" },
@@ -4031,6 +4045,41 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ee/2b/e18ee7c5ee508a82897f021c1981533eca2940b5f072fc6ed0906c03a7a7/pybase64-1.4.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:debf737e09b8bf832ba86f5ecc3d3dbd0e3021d6cd86ba4abe962d6a5a77adb3", size = 36134, upload-time = "2025-12-06T13:26:47.35Z" },
]
[[package]]
name = "pycares"
version = "5.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cffi" },
]
sdist = { url = "https://files.pythonhosted.org/packages/df/a0/9c823651872e6a0face3f0311de2a40c8bbcb9c8dcb15680bd019ac56ac7/pycares-5.0.1.tar.gz", hash = "sha256:5a3c249c830432631439815f9a818463416f2a8cbdb1e988e78757de9ae75081", size = 652222 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/87/78/43b09f4b8e5fb8a6024661b458b48987abdb39304c78117b106b10a029f1/pycares-5.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c29ca77ff9712e20787201ca8e76ad89384771c0e058a0a4f3dc05afbc4b32de", size = 136177 },
{ url = "https://files.pythonhosted.org/packages/19/05/194c0e039ff52b166b50e79ff166c61f931fbca2bf94fc0dbaaf39041518/pycares-5.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f11424bf5cf6226d0b136ed47daa58434e377c61b62d0100d1de7793f8e34a72", size = 130960 },
{ url = "https://files.pythonhosted.org/packages/0d/84/5fce65cc058c5ab619c0dd1370d539667235a5565da72ca77f3f741cdc70/pycares-5.0.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d765afb52d579879f5c4f005763827d3b1eb86b23139e9614e6089c9f98db017", size = 220584 },
{ url = "https://files.pythonhosted.org/packages/f6/74/d82304297308f6c24a17961bf589b53eefa5f7f2724158c842c67fa0b302/pycares-5.0.1-cp311-cp311-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ea0d57ba5add4bfbcc40cbdfa92bbb8a5ef0c4c21881e26c7229d9bdc92a4533", size = 252166 },
{ url = "https://files.pythonhosted.org/packages/39/a2/0ead3ba4228a490b52eb44d43514dae172c90421bb30a3659516e5b251a2/pycares-5.0.1-cp311-cp311-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ae9ec2aa3553d33e6220aeb1a05f4853fb83fce4cec3e0dea2dc970338ea47dc", size = 239085 },
{ url = "https://files.pythonhosted.org/packages/26/ad/e59f173933f0e696a6afbbd63935114d1400524a72da4f2cbafc6002a398/pycares-5.0.1-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5c63fb2498b05e9f5670a1bf3b900c5d09343b3b6d5001a9714d593f9eb54de1", size = 222936 },
{ url = "https://files.pythonhosted.org/packages/98/fa/d85bfe663a9c292efd8e699779027612c0c65ff50dc4cc9eb7a143613460/pycares-5.0.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:71316f7a87c15a8d32127ff01374dc2c969c37410693cc0cf6532590b7f18e7a", size = 223506 },
{ url = "https://files.pythonhosted.org/packages/2a/6b/4c225a5b10a4c9f88891a20bfe363eca1b1ce7d5244b396e5683c6070998/pycares-5.0.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:a2117dffbb78615bfdb41ad77b17038689e4e01c66f153649e80d268c6228b4f", size = 251633 },
{ url = "https://files.pythonhosted.org/packages/26/ce/ba2349413b5197b72ec19c46e07f6be3a324f80a7b1579c7cbb1b82d6dc2/pycares-5.0.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:7d7c4f5d8b88b586ef2288142b806250020e6490b9f2bd8fd5f634a78fd20fcf", size = 237703 },
{ url = "https://files.pythonhosted.org/packages/84/2f/1fd794e6fca10d9e20569113d10a4f92cc2b4242d3eb45524419a37cca6b/pycares-5.0.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:433b9a4b5a7e10ef8aef0b957e6cd0bfc1bb5bc730d2729f04e93c91c25979c0", size = 222622 },
{ url = "https://files.pythonhosted.org/packages/c9/07/7db7977649b210092a7e02d550fcebdfa69bc995c684a3b960c88a5dc4ce/pycares-5.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:cf2699883b88713670d3f9c0a1e44ac24c70aeace9f8c6aa7f0b9f222d5b08a5", size = 117438 },
{ url = "https://files.pythonhosted.org/packages/fc/ca/f322ddaa8b3414667de8faeea944ce9d3ddfaf1455839f499a21fcea4cec/pycares-5.0.1-cp311-cp311-win_arm64.whl", hash = "sha256:9528dc11749e5e098c996475b60f879e1db5a6cb3dd0cdc747530620bb1a8941", size = 108920 },
{ url = "https://files.pythonhosted.org/packages/75/67/e84ba11d3fec3bf1322c3b302c4df13c85e0a1bc48f16d65cd0f59ad9853/pycares-5.0.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2ee551be4f3f3ac814ac8547586c464c9035e914f5122a534d25de147fa745e1", size = 136241 },
{ url = "https://files.pythonhosted.org/packages/ce/ae/50fbb3b4e52b9f1d16a36ffabd051ef8b2106b3f0a0d1c1113904d187a9d/pycares-5.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:252d4e5a52a68f825eaa90e16b595f9baee22c760f51e286ab612c6829b96de3", size = 131069 },
{ url = "https://files.pythonhosted.org/packages/0e/ea/f431599f1ac42149ea4768e516db7cdae3a503a6646319ae63ab66da1486/pycares-5.0.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8c1aa549b8c2f2e224215c793d660270778dcba9abc3b85abbc7c41eabe4f1e5", size = 221120 },
{ url = "https://files.pythonhosted.org/packages/6e/4f/0a7a6c8b3a64ee5149e935c167cd8ba5d1fdd766ec03e273dbc7502f7bea/pycares-5.0.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:db7c9c9f16e8311998667a7488e817f8cbeedec2447bac827c71804663f1437e", size = 252228 },
{ url = "https://files.pythonhosted.org/packages/49/3d/7f9fd20e97ee30c4b959f87ab26e47ddcef666e5e7717e45f2245fe9d70a/pycares-5.0.1-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4b9c4c8bb69bab863f677fa166653bb872bfa5d5a742f1f30bebc2d53b6e71db", size = 239473 },
{ url = "https://files.pythonhosted.org/packages/a4/d0/c67967a10abd89529cb9aded9d73f43e5de00cf21243638ef529f6757262/pycares-5.0.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:09ef90da8da3026fcba4ed223bd71e8057608d5b3fec4f5990b52ae1e8c855cc", size = 223831 },
{ url = "https://files.pythonhosted.org/packages/4f/9a/94aacaf22a20b7d342c8f18bf006be57967beef6319adc668d4d86b627be/pycares-5.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ce193ebd54f4c74538b751ebb0923a9208c234ff180589d4d3cec134c001840e", size = 223963 },
{ url = "https://files.pythonhosted.org/packages/e6/e1/3666aab6fc5e7d0c669b981fe0407e6a4b67e4e6a37ac429d440274663d5/pycares-5.0.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:36b9ff18ef231277f99a846feade50b417187a96f742689a9d08b9594e386de4", size = 251813 },
{ url = "https://files.pythonhosted.org/packages/94/44/ddab5fbc16ad0084a827167ae8628f54c7a55ce6b743585e6f47a5dd527e/pycares-5.0.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:5e40ea4a0ef0c01a02ef7f7390a58c62d237d5ad48d36bc3245e9c2ac181cc22", size = 238181 },
{ url = "https://files.pythonhosted.org/packages/66/27/05467933e0e5c4e712302a2d7499797bc3029bf4d0d8ffbfe737254482b7/pycares-5.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3f323b0ddfd2c7896af6fba4f8851d34d3d13387566aa573d93330fb01cb1038", size = 223552 },
{ url = "https://files.pythonhosted.org/packages/3e/e2/14f3837e943d46ee12441fe6aaa418fdb2f698d42e179f368eaa9829744b/pycares-5.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:bdc6bcafb72a97b3cdd529fc87210e59e67feb647a7e138110656023599b84da", size = 117478 },
{ url = "https://files.pythonhosted.org/packages/d3/c3/3284061f18188d5085338e1f1fd4f03d9c135657acf16f8020b9dd3be5fc/pycares-5.0.1-cp312-cp312-win_arm64.whl", hash = "sha256:f8ef4c70c1edaf022875a8f9ff6c0c064f82831225acc91aa1b4f4d389e2e03a", size = 108889 },
]
[[package]]
name = "pyclipper"
version = "1.4.0"