6. Compounding error, measured
Demo4/scripts/run_reliability.py:
40 cases × 3 repeat(s) × 2 configurations =
240 agent runs in 311.2s, planner
offline-lexical-router. A multi-run experiment is too slow to do in front of a
room, so it ships as an artefact and this button regenerates it.
Per-step success, all six steps agentic
| Step | What it has to get right | Marginal | Conditional on the earlier steps |
|---|---|---|---|
s1_parse | Extract the machine variant from the request | 100.0% | 100.0% (n=120) |
s2_history | Retrieve comparable ticket history | 100.0% | 100.0% (n=120) |
s3_diagnose | Choose the fault code the symptom implies | 47.5% | 47.5% (n=120) |
s4_part | Turn the fault plus variant into a part number | 55.0% | 100.0% (n=57) |
s5_stock | Decide source-from-stock or order, on lead time | 85.0% | 78.9% (n=57) |
s6_compose | Compose the work order | 45.0% | 100.0% (n=45) |
Read the two right-hand columns against each other. The product of the marginal rates is 10.0%. The observed end-to-end rate is 37.5%. They differ because the steps are not independent: a run that gets the fault wrong tends to get the part wrong too, so the failures overlap instead of multiplying. The product of the conditional rates is 37.5%, which matches the observed rate exactly — that is the chain rule, and it is the only version of this arithmetic that is actually true of a real pipeline.
Published figures, and what each one actually counts
| Source | Figure | What it measures |
|---|---|---|
| MCP-Universe (Salesforce AI Research, arXiv:2508.14704), GPT-5 | 43.72% | per task, end to end, 6 domains, 11 real MCP servers |
| MCP-Universe, Grok-4 | 33.33% | per task |
| MCP-Universe, Claude-4.0-Sonnet | 29.44% | per task |
| τ²-bench leaderboard pass^1 (retrieved 9 Aug 2026), Qwen3.5-397B-A17B | 87.9% | per task; grading version matters, v1.0.1 re-graded earlier submissions |
| τ²-bench pass^1, Claude Opus 4.5 | 85.3% | per task |
| BFCL v4 overall (official Gorilla table, last confirmed 16 Dec 2025), Claude-Opus-4-5 (FC) | 77.47% | unweighted mean across sub-categories including multi-turn — not a clean per-step rate |
The arithmetic that makes the point. MCP-Universe's GPT-5 result of 43.72% end-to-end over multi-step tasks against real MCP servers is almost exactly what a per-step reliability of 0.90 over eight steps predicts: 0.908 = 43.0%. That is a coincidence of two numbers, not a proof, and it is labelled as such — but it is the cleanest way to state the thesis: a 90%-reliable step is a 43%-reliable workflow.
To derive a per-step rate from a task-level one honestly, invert it: p = pass11/n̄ where n̄ is the mean number of tool-calling steps. Claude Opus 4.5's 85.3% τ²-bench pass^1 at an assumed n̄ = 5 implies p = 0.8531/5 = 96.9%; at n̄ = 8 it implies 98.1%. The n̄ assumption is the whole ballgame and has to be on the slide.
What pn does, including the rate measured here
| per-step p | n=3 | n=5 | n=6 | n=8 | n=10 |
|---|---|---|---|---|---|
| 0.99 | 97.0% | 95.1% | 94.1% | 92.3% | 90.4% |
| 0.95 | 85.7% | 77.4% | 73.5% | 66.3% | 59.9% |
| 0.9 | 72.9% | 59.0% | 53.1% | 43.0% | 34.9% |
| 0.85 | 61.4% | 44.4% | 37.7% | 27.2% | 19.7% |
| 0.475 | 10.7% | 2.4% | 1.1% | 0.3% | 0.1% |
Where the runs actually failed
Step 3 — turning a symptom in the reporter's words into a fault code — is the weak link, at 47.5%. Everything downstream inherits it. That is what screen 7 is about: the remedy is not a better prompt.
For comparison, the deterministic classifier on screen 7 scores 90.0% on the 10 held-out symptom phrasings taken on their own.
Demo4/pipeline/.