Better results, at higher cost
Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report
Hypothesis. checkSemanticFailureMatch recognises known failure patterns described in different words often enough to earn its per-miss call and improve recovery routing.
No task-success delta (CI includes 0) but the candidate costs materially more (+48% $/turn, +15% latency). Neutral with real cost — it does not earn its keep.
Arms: failureMatchOff (control) vs flagOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.
| Metric | failureMatchOff | flagOn | Δmean | ±CI95 |
|---|---|---|---|---|
| taskSuccessRate | 0.250 | 0.375 | +0.125 | ±0.283 |
| hallucinationRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| unauthorizedEffectRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| recoveryRate | 0.250 | 0.375 | +0.125 | ±0.283 |
| overconfidentWrongRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| supervisorConsultsMean | 0.000 | 0.000 | 0.000 | ±0.000 |
| meanLatencyMs | 6977 ms | 8040 ms | +1063 ms | ±656 ms |
| meanCostUsd | $0.0056 | $0.0083 | +$0.0027 | ±$0.0027 |
| totalTokens | 2992 | 3309 | +317 | ±261 |
Cost +48% · latency +15% · tokens +11% (candidate vs control).
FailureModeLibrary.match() looks for an exact-string match against a
library of known failure patterns. checkSemanticFailureMatch adds an LLM call
whenever that lexical match returns nothing, to recognize a known failure mode described in
different words and route to the right recovery strategy.
Real tool failures get described in enough different phrasings that the exact-match library alone misses genuine matches often enough that the per-miss LLM call earns its cost and improves recovery routing.
Arms: failureMatchOff vs flagOn. An 8-task slice with
an injected failure symptom worded differently from the library's exact strings, isolating the
classifier itself; whether better classification improves recovery is measured as a compressed
proxy (a bounded re-answer loop), not full multi-turn recovery. 3 seeds, LLM judge on.
Task success moved 25.0% → 37.5%, a 12.5-point gap — but the confidence interval is ±28 points on an 8-task slice, so the gap isn't distinguishable from noise. Cost rose 48% per turn and latency 15%. As with harness-vs-bare, the direction is positive but unproven, and the proven part is that it costs more.
| Task | Behaviour | failureMatchOff pass | flagOn pass | Δ cost | Δ lat | Δ tok | Seeds ctrl→cand |
|---|---|---|---|---|---|---|---|
| audit-failure-match-auth-denied-readme | no change | 0/3 | 0/3 | +50% | -8% | -6% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| audit-failure-match-auth-rejected-env | changed 2/3 | 2/3 | 2/3 regressed | +69% | +8% | +17% | s1 ✓→✓ s2 ✓→✗ s3 ✗→✓ |
| audit-failure-match-rate-limited-log | changed | 3/3 | 2/3 regressed | +24% | +11% | +29% | s1 ✓→✓ s2 ✓→✗ s3 ✓→✓ |
| audit-failure-match-rate-throttled-notes | no change | 0/3 | 0/3 | -3% | -14% | -20% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| audit-failure-match-resource-absent-index | changed 2/3 | 0/3 | 0/3 | +104% | +72% | +46% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| audit-failure-match-resource-missing-manifest | changed 1/3 | 0/3 | 1/3 fixed | +66% | +30% | +30% | s1 ✗→✓ s2 ✗→✗ s3 ✗→✗ |
| audit-failure-match-timeout-config | changed 2/3 | 1/3 | 3/3 fixed ×2 | +36% | +13% | -13% | s1 ✓→✓ s2 ✗→✓ s3 ✗→✓ |
| audit-failure-match-timeout-nested | changed 1/3 | 0/3 | 1/3 fixed | +38% | +8% | -1% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✓ |
Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.