Better results, at higher cost
Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report
Hypothesis. Running the full 11-layer harness every turn produces materially better task outcomes than a bare ReAct loop, enough to justify the whole subsystem's per-turn cost.
No task-success delta (CI includes 0) but the candidate costs materially more (+77% $/turn, +88% latency). Neutral with real cost — it does not earn its keep.
Arms: bare (control) vs flagOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.
| Metric | bare | flagOn | Δmean | ±CI95 |
|---|---|---|---|---|
| taskSuccessRate | 0.676 | 0.838 | +0.162 | ±0.256 |
| hallucinationRate | 0.032 | 0.014 | -0.019 | ±0.024 |
| unauthorizedEffectRate | 0.037 | 0.000 | -0.037 | ±0.036 |
| recoveryRate | — | — | — | — |
| overconfidentWrongRate | — | 0.125 | — | — |
| supervisorConsultsMean | 0.000 | 0.000 | 0.000 | ±0.000 |
| meanLatencyMs | 9032 ms | 16937 ms | +7905 ms | ±3666 ms |
| meanCostUsd | $0.0123 | $0.0218 | +$0.0095 | ±$0.0023 |
| totalTokens | 55755 | 56117 | +362 | ±18614 |
Cost +77% · latency +88% · tokens +1% (candidate vs control).
The 11-layer harness — a world model, evidence and contradiction tracking, control-state gating, up-front planning, a 9-layer verification pass, and a reviewer pass — runs on every turn, wrapping the same underlying model with staging, self-checks and recovery logic that a plain tool-calling loop skips entirely.
All that machinery should produce measurably better task outcomes than letting the model call tools directly with nothing in front of it — enough to justify running the whole subsystem, and the extra latency and tokens it costs, on every single turn.
Arms: bare (a plain ReAct-style tool loop, no harness) vs
flagOn (the full harness). 76-task corpus, 3 seeds, LLM judge on. A first pass was
withdrawn: its injected-failure tasks fired only for the harness arm, not bare, so
the harness was stress-tested while the control ran clean. This run drops those tasks and
measures ordinary, non-failure task quality only — recovery-under-failure for the bare arm is a
separate, still-open coverage gap.
Every quality metric favored the harness — task success 67.6% → 83.8% (a 16.2-point gap), hallucination 3.2% → 1.4%, unauthorized effects 3.7% → 0% — but none of it clears its confidence interval at 3 seeds; the noise is wider than the gap. What is proven: cost is 77% higher per turn and latency is 88% higher. So the honest read is narrower than either "it helps" or "it doesn't": this run can't yet prove the harness improves outcomes on ordinary, non-failure tasks, and it can prove the harness costs meaningfully more to run them.
Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.