ARO-4202 · Odin MCP Load-Test

The verdict

Five things, backed by direct measurements on real Odin (Cloudflare bypassed to hit the pod).

1 · Not volume — cost × concurrency.
A pre-fix /_mcp request held a worker ~7 s (it re-scanned all of src/ every time). With ~2 workers, a handful saturates; hundreds of ~0.5 s API calls don't.

2 · The scan fix worked.
On current master, every tools/call succeeds up to 48 concurrent. The catastrophic ~50% failures Benjamin hit on 07-29 (pre-fix) are gone.

3 · text/html is real & load-independent — on prod too.
odin.alpy.com/_mcp returns errors as text/html, not JSON-RPC. Strict clients reject before reading the body.

4 · Cloudflare Access is not prod's cause.
Only *.alpy.fun previews sit behind Access. Prod /_mcp reaches Odin directly.

5 · A residual race remains.
~2–4% of tools/list under load are dropped and returned as empty 202 text/html — a non-atomic Redis session-store bug. Worker mode does not fix it.

6 · Worker mode ≈ classic at equal CPU.
Post scan-fix, kernel-boot isn't the bottleneck anymore — CPU is. ARO-4256's real win is isolation + autoscaling replicas, not the worker loop itself.

The saturation mechanism

Why a few calls break it while hundreds are fine.

Think of /_mcp as a shop with ~2 cashiers (FrankenPHP workers). Each request occupies one cashier start-to-finish. Whether the shop copes = time-per-customer × customers-at-once vs number of cashiers.

Cheap REST

~0.5 s each. 2 cashiers clear hundreds fast. Line barely forms. ✅

→

Pre-fix /_mcp

~7 s each (full src/ re-scan). 3rd customer waits 7 s, 4th 14 s… line explodes. ❌

→

Result

Requests behind the queue exceed the timeout → come back as 202/no-session/text-html.

scan_dirs fix — cut each /_mcp from ~7 s to ~0.5 s. A handful no longer explodes the line. But still ~2 cashiers, one customer each → big bursts still grow latency.

Worker mode — cashiers are permanent & warm (kernel booted once, not per request) instead of re-hired every customer. Helps when kernel-boot is the cost… which, post scan-fix, it no longer is.

Classic vs worker mode — matched resources

Both: 1 pod, 500m CPU / 2Gi, ~2 workers, same DB/OAuth/harness. The only variable is classic-per-request vs FrankenPHP worker loop (ARO-4256). Benjamin's exact sequence — initialize → tools/list → tools/call ×4 — per session, ramped 1→48 concurrent.

Median latency

p95 latency

tools/list drop %

Concurrency	Classic p50	Worker p50	Classic drop	Worker drop	tool-calls OK

Worker mode is within noise of classic here (~13% faster only at c≥32) and the ~3% tools/list drop rate is unchanged — because the race lives in the shared Redis session store, and CPU (not kernel-boot) is the binding constraint once the scan fix is in.

The collapse, measured directly

Hitting Odin's kernel-booting /api path directly (Access-exempt) on the classic pod. Latency scales ~linearly with concurrency (requests serialize through ~1–2 workers), then the pod collapses.

Harness sanity check (local mock, 2 threads · 7 s discovery)

MCP agent pattern: 40% fail REST baseline: 0% fail

Reproduces Benjamin's exact text/html+202 symptom at ~his observed ~50% — proving the mechanism when per-request cost is high.

Timeline
2026-07-29 17:16 UTC — Benjamin re-reports failures.
2026-07-30 07:45 UTC — scan_dirs fix merged to master.
→ On 07-29 prod still did the ~7 s scan. His failures fit the mechanism exactly.

The text/html bug — root cause in code

Two independent paths on /_mcp leak text/html instead of JSON-RPC. Strict clients (Zendesk) reject on content-type before reading the body → "unsupported content type text/html".

① The empty 202 (the dropped response)

// vendor/mcp/sdk/.../StreamableHttpTransport.php
$out = getOutgoingMessages($sessionId);
if (empty($out)) {
  // no Content-Type set →
  return createResponse(202);
}

Response is queued into a Redis session blob then read back via a separate reload. If the read is empty → header-less 202 → Symfony defaults to text/html, empty body.

② The 401 (auth failure)

HTTP/2 401
content-type: text/html; charset=UTF-8
www-authenticate: Bearer ...

Full authentication is required to
access this resource.

From Symfony's security firewall (before the MCP controller). Also HTML — nobody set application/json.

Why it's a race — the outgoing queue lives inside a session serialized as one JSON blob and written with a plain Redis SET (Psr16SessionStore). No atomic ops, no lock, no WATCH/MULTI — last-write-wins. Under concurrency, an interleaved session save clobbers the queued response → empty read → 202 text/html. Hits tools/list hardest (biggest payload, every handshake).

Recommended fixes

Make /_mcp always answer JSON — never text/html.P1 · load-independent
A kernel.response listener on ^/_mcp forcing application/json (covers the header-less 202), plus a JSON auth entry-point for the 401. Turns an opaque failure into a well-formed, retryable JSON. This is Benjamin's #1 — fully in Odin's control.
Fix the dropped-response race at the store.P2 · durable
Deliver the outgoing queue with atomic Redis ops (RPUSH/LPOP or a dedicated key) instead of read-modify-write of the whole session blob — or wrap session load-modify-save in a Symfony Lock keyed by session id. Eliminates the lost update.
Confirm scan_dirs is on prod & re-test.P3
The catastrophic ~7 s/request saturation is already gone in current code — have Benjamin re-test the current build.
Ship ARO-4256 for isolation + scale, not the worker loop alone.P3
At equal CPU the worker loop is ~a wash; the win is the dedicated pool's own CPU + autoscaling replicas so MCP bursts don't starve the main app. Pair it with the race fix.