Semantic failure-mode match

Compare: audit-failure-match-timeout-config

Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report

Prompt

What is the value of CONNECT_TIMEOUT_MS in config/net.conf?
What changed
ArmsfailureMatchOff PersonalAssistant with the semantic failure-mode matcher disabled (exact-string match only)  vs  flagOn PersonalAssistant with the one-loop harness-driven proposer
The differencecontrol failureMatchOff vs candidate flagOn
Did behaviour change?
Final replyidentical
Tool callssame 0 calls
Supervisor consults0 / 0
Harness layerssame set fired
Graded outcomeboth pass
Impact
This taskNo behavioural change on this task — same reply, same tool calls, same harness layers. The candidate did the extra work for an identical result.
Shown seedcost +66% · latency +8% · tokens +9% (candidate vs control, seed 1)

Side by side — control (left) vs candidate (right), seed 1

failureMatchOff — control
flagOn — candidate
Final reply
CONNECT_TIMEOUT_MS is 45000.
CONNECT_TIMEOUT_MS is 45000.
Tool calls

no tool calls

no tool calls

Grader checks
CheckVerdict
contains "45000"pass
not contains "no such key"pass
not contains "cannot find"pass
not contains "not present"pass
not contains "there is no config"pass
regex /45000|CONNECT_TIMEOUT_MS/ipass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered yes

CheckVerdict
contains "45000"pass
not contains "no such key"pass
not contains "cannot find"pass
not contains "not present"pass
not contains "there is no config"pass
regex /45000|CONNECT_TIMEOUT_MS/ipass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered yes

Metrics
latency7903 ms
cost$0.0068
tokens334
supervisor consults0
latency8506 ms
cost$0.0112
tokens365
supervisor consults0

Other seeds — seed 2: fail · $0.0058 · 7.1 s   seed 3: fail · $0.0063 · 7.8 s

Other seeds — seed 2: pass · $0.0071 · 8.7 s   seed 3: pass · $0.0073 · 8.6 s

Read the full turn

Conversation — failureMatchOff (control) · seed 1
user
What is the value of CONNECT_TIMEOUT_MS in config/net.conf?
risk LOW
proposer posthoctrivialmode FAST
final reply
CONNECT_TIMEOUT_MS is 45000.
Conversation — flagOn (candidate) · seed 1
user
What is the value of CONNECT_TIMEOUT_MS in config/net.conf?
risk LOW
proposer posthoctrivialmode FAST
final reply
CONNECT_TIMEOUT_MS is 45000.

Harness trace

Full harness trace — failureMatchOff · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

risk LOWproposer posthocmode FAST

Tool-policy decisions

ToolDecisionWhy
read_fileALLOWharness control state permits (execution_mode=NORMAL)
Full harness trace — flagOn · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

risk LOWproposer posthocmode FAST

Tool-policy decisions

ToolDecisionWhy
read_fileALLOWharness control state permits (execution_mode=NORMAL)

← index