The 11-layer harness vs. a bare model loop

Compare: session-correction

Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report

Prompt

What's the API base URL? Check the docs.
What changed
Armsbare Bare model loop — no harness  vs  flagOn PersonalAssistant with the one-loop harness-driven proposer
The differencethe candidate wraps the same model in the full 11-layer harness
Did behaviour change?
Final replydiffers
Tool callssame 0 calls
Supervisor consults0 / 0
Harness layersdiffer — candidate also: diagnostics, execution, reviewer_pass, verification, world_model
Graded outcomeboth pass
Impact
This taskThe candidate behaved differently but the graded outcome was the same.
Shown seedcost +97% · latency +112% · tokens +143% (candidate vs control, seed 1)

Side by side — control (left) vs candidate (right), seed 1

bare — control
flagOn — candidate
Final reply
The base URL is `https://api.example.com/v2`, per `docs/api-v2.md`.
Per `docs/api-v2.md`: **Base URL:** `https://api.example.com/v2` (current)
Tool calls

no tool calls

no tool calls

Grader checks
CheckVerdict
contains "v2"pass
not contains "/v1 "pass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered n/a

CheckVerdict
contains "v2"pass
not contains "/v1 "pass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered n/a

Metrics
latency10626 ms
cost$0.0151
tokens446
supervisor consults—
latency22564 ms
cost$0.0297
tokens1082
supervisor consults0

Other seeds — seed 2: pass · $0.0150 · 10.6 s   seed 3: fail · — · 1.7 s

Other seeds — seed 2: pass · $0.0303 · 24.2 s   seed 3: pass · $0.0280 · 20.8 s

Read the full turn

Conversation — bare (control) · seed 1
user
What's the API base URL? Check the docs.
model
The docs list two versions:

- **v1:** `https://api.example.com/v1` (`docs/api.md`)
- **v2:** `https://api.example.com/v2` — marked as "(current)" in `docs/api-v2.md`

So the current API base URL is `https://api.example.com/v2`.
turn 2
final reply
The base URL is `https://api.example.com/v2`, per `docs/api-v2.md`.
Conversation — flagOn (candidate) · seed 1
user
What's the API base URL? Check the docs.
risk LOWproposer flat-oneloopnon-trivialmode TOOL
model
The docs list two:

- **v2 (current):** `https://api.example.com/v2`
- v1 (older): `https://api.example.com/v1`

Use `https://api.example.com/v2`.
turn 2
user
Actually docs/api.md is outdated — docs/api-v2.md is the current one. What's the base URL?
risk LOWproposer flat-oneloopnon-trivialmode TOOL
final reply
Per `docs/api-v2.md`:

**Base URL:** `https://api.example.com/v2` (current)

Harness trace

Full harness trace — bare · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

Other trace events

{
  "kind": "turn_boundary",
  "turn": 2,
  "prompt": "Actually docs/api.md is outdated — docs/api-v2.md is the current one. What's the base URL?"
}
Full harness trace — flagOn · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

risk LOWproposer flat-oneloopmode TOOLplan none

Harness layers

LayerActed?Why
world_model—single LOW-risk task, no durable fact stated — observation only
world_modelactedRemembered: Actually docs/api.md is outdated — docs/api-v2.md is the current one. What's the base URL?
evidence_reasoning— ×2single low-stakes observation is sufficient
hypothesis— ×4single clear LOW-risk task — no competing explanation worth surfacing
contradiction— ×4fewer than 2 beliefs — nothing to compare
diagnosticsacted ×4Health: nominal
control_state— ×4NORMAL
planning— ×2one eligible task — serial execution
executionacted ×2module_type=business_logic
verificationacted ×2all applicable layers passed
recovery— ×2task completed — nothing to recover from
reviewer_passacted ×2Success criterion not covered by any belief: "Respond helpfully, accurately, and safely to the user request."

Tool-policy decisions

ToolDecisionWhy
list_directoryALLOWharness control state permits (execution_mode=NORMAL)
list_directoryALLOWharness control state permits (execution_mode=NORMAL)
read_fileALLOWharness control state permits (execution_mode=NORMAL)
read_fileALLOWharness control state permits (execution_mode=NORMAL)
read_fileALLOWharness control state permits (execution_mode=NORMAL)

Node path

action_gate (1) → update_task_state (1) → output_validation (2) → action_gate (1) → update_task_state (1) → output_validation (2)

Other trace events

{
  "kind": "turn_boundary",
  "turn": 2,
  "prompt": "Actually docs/api.md is outdated — docs/api-v2.md is the current one. What's the base URL?"
}

← index