Better results, at higher cost
Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report
Hypothesis. The checkForContradictions LLM call catches enough real paraphrased / cross-framing belief contradictions that the always-on lexical negation-pair check misses to pay for its per-belief-set-growth cost.
The candidate beats the control on task success (CI clears 0) and the gain justifies its higher cost (+29% $/turn).
Arms: contradictionOff (control) vs flagOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.
| Metric | contradictionOff | flagOn | Δmean | ±CI95 |
|---|---|---|---|---|
| taskSuccessRate | 0.857 | 0.929 | +0.071 | ±0.000 |
| hallucinationRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| unauthorizedEffectRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| recoveryRate | — | — | — | — |
| overconfidentWrongRate | 0.128 | 0.077 | -0.051 | ±0.050 |
| supervisorConsultsMean | 0.000 | 0.000 | 0.000 | ±0.000 |
| meanLatencyMs | 21958 ms | 22286 ms | +329 ms | ±464 ms |
| meanCostUsd | $0.0210 | $0.0270 | +$0.0060 | ±$0.0063 |
| totalTokens | 16127 | 15858 | -269 | ±738 |
Cost +29% · latency +1% · tokens -2% (candidate vs control).
An always-on lexical pass catches contradictions that are simple negation pairs
("X is true" vs "X is not true"). checkForContradictions adds one LLM call on top
of that, run whenever the belief set grows, to catch contradictions that are paraphrased or
reframed rather than a literal negation.
Real conversations restate beliefs in different words often enough that the lexical-only check misses genuine contradictions — enough that the extra per-belief-set-growth LLM call pays for itself.
Arms: contradictionOff vs flagOn. A 14-task slice built
specifically for this: a belief stated one turn, unrelated beliefs in between, then a paraphrased
or cross-framed contradiction on a later turn — plus control tasks (a legitimate change over
time) that must not be false-flagged. 3 seeds, LLM judge on.
Task success went 85.7% → 92.9%, a 7.1-point gain whose confidence interval clears zero — the one result in this audit that's proven, not just suggested. Cost went up 29% per turn with latency flat. The gain is real and it clears the cost: this is the one feature kept on by default so far.
Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.