Feature Value Audit

Semantic contradiction check

Better results, at higher cost

Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report

Hypothesis. The checkForContradictions LLM call catches enough real paraphrased / cross-framing belief contradictions that the always-on lexical negation-pair check misses to pay for its per-belief-set-growth cost.

3-seed numbers

Arms: contradictionOff (control) vs flagOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.

Metric	contradictionOff	flagOn	Δmean	±CI95
taskSuccessRate	0.857	0.929	+0.071	±0.000
hallucinationRate	0.000	0.000	0.000	±0.000
unauthorizedEffectRate	0.000	0.000	0.000	±0.000
recoveryRate	—	—	—	—
overconfidentWrongRate	0.128	0.077	-0.051	±0.050
supervisorConsultsMean	0.000	0.000	0.000	±0.000
meanLatencyMs	21958 ms	22286 ms	+329 ms	±464 ms
meanCostUsd	$0.0210	$0.0270	+$0.0060	±$0.0063
totalTokens	16127	15858	-269	±738

Cost +29% · latency +1% · tokens -2% (candidate vs control).

Case study

What it does

An always-on lexical pass catches contradictions that are simple negation pairs ("X is true" vs "X is not true"). checkForContradictions adds one LLM call on top of that, run whenever the belief set grows, to catch contradictions that are paraphrased or reframed rather than a literal negation.

Why we hypothesized it would help

Real conversations restate beliefs in different words often enough that the lexical-only check misses genuine contradictions — enough that the extra per-belief-set-growth LLM call pays for itself.

How it was tested

Arms: contradictionOff vs flagOn. A 14-task slice built specifically for this: a belief stated one turn, unrelated beliefs in between, then a paraphrased or cross-framed contradiction on a later turn — plus control tasks (a legitimate change over time) that must not be false-flagged. 3 seeds, LLM judge on.

What we found

Task success went 85.7% → 92.9%, a 7.1-point gain whose confidence interval clears zero — the one result in this audit that's proven, not just suggested. Cost went up 29% per turn with latency flat. The gain is real and it clears the cost: this is the one feature kept on by default so far.

Runs — 14 tasks × 3 seeds, 84 runs, one row per task

Task	Behaviour	contradictionOff pass	flagOn pass	Δ cost	Δ lat	Δ tok	Seeds ctrl→cand
adv-contradiction-mt-control-complementary	changed	3/3	3/3	+66%	+19%	+15%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-mt-control-migration	changed	3/3	3/3	+29%	+2%	-3%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-mt-crosslang	changed	2/3	3/3 fixed	+18%	-1%	-4%	s1 ✓→✓ s2 ✗→✓ s3 ✓→✓
adv-contradiction-mt-indirect	changed	3/3	3/3	+27%	+2%	-4%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-mt-scale	changed	3/3	3/3	+52%	+5%	+9%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-mt-units	changed	0/3	0/3	+21%	+1%	+1%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗
adv-contradiction-semantic-control-scope	changed	3/3	3/3	+12%	-12%	-13%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-control-temporal	changed	3/3	3/3	+11%	-11%	-11%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-crosslang	changed	3/3	3/3	+31%	+13%	+12%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-currency	changed	2/3	3/3 fixed	—	-1%	—	s1 ✓→✓ s2 ✓→✓ s3 ✗→✓
adv-contradiction-semantic-indirect	changed	3/3	3/3	+28%	+11%	+12%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-paraphrase	changed	3/3	3/3	+14%	-11%	-11%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-timezone	changed	3/3	3/3	+20%	+0%	-0%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
adv-contradiction-semantic-units	changed	2/3	3/3 fixed	-5%	-15%	-28%	s1 ✗→✓ s2 ✓→✓ s3 ✓→✓

Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.