Better results, at higher cost
Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report
Hypothesis. When a multi-turn task stalls after context has built up, the supervisor redirects, reframes or asks the user and the run recovers, at a cost that is negligible on tasks that never stall.
The candidate beats the control on task success (CI clears 0) and the gain justifies its higher cost (+43% $/turn).
Arms: supervisorOff (control) vs flagOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.
| Metric | supervisorOff | flagOn | Δmean | ±CI95 |
|---|---|---|---|---|
| taskSuccessRate | 0.375 | 0.625 | +0.250 | ±0.000 |
| hallucinationRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| unauthorizedEffectRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| recoveryRate | — | — | — | — |
| overconfidentWrongRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| supervisorConsultsMean | 0.000 | 0.750 | +0.750 | ±0.000 |
| meanLatencyMs | 13454 ms | 17842 ms | +4388 ms | ±257 ms |
| meanCostUsd | $0.0094 | $0.0134 | +$0.0040 | ±$0.0004 |
| totalTokens | 4663 | 6590 | +1926 | ±256 |
Cost +43% · latency +33% · tokens +41% (candidate vs control).
A slow-loop meta-controller that wakes up only when a run has stopped making progress. It reads a digest of the failure trajectory and returns one directive: redirect the strategy, reframe the plan, gather evidence, ask the user a targeted question, or stop. Between those moments it stays out of the way.
Our earlier single-turn run of this feature came out negative, but that corpus stalled on turn one with no context behind it. The bet here is narrower: when a stall hits late in a multi-turn task, after the assistant has already done useful work, a targeted directive recovers runs that would otherwise fail, and costs almost nothing on tasks that never stall.
Eight multi-turn tasks. Six have a persistent tool failure injected on a later turn (a pointer chain to follow, a file that only points elsewhere, a nested lookup, a stale newer-file override, a transitive import, and one where the user has to supply a missing detail on the following turn). Two are clean controls with no stall, where the supervisor should never fire. The control arm is the same assistant with the supervisor switched off.
With a stall injected late in a multi-turn task, the supervisor recovered runs that otherwise failed. Task success rose from 37.5% to 62.5% (+25 points; all three seeds gave the same outcomes, so the interval is zero). Two of the six stalled tasks were recovered only with the supervisor on: the one where a newer file overrides the obvious answer, and the one where the named file only points elsewhere. The task where the user supplies the missing detail on the next turn was recovered by both arms. Three tasks were recovered by neither: the pointer chain, the nested lookup and the transitive import, where the injected failure lasted two or three iterations.
The price is real. Cost per task rose 43% ($0.0094 to $0.0134), latency 33% (13.5 s to 17.8 s) and tokens 41%. On the two clean controls the supervisor was never consulted and both arms scored 100%.
Two caveats. The stalls are simulated: a persistent tool failure is injected for a fixed number of iterations, so this measures recovery from that kind of stall, not stalls that occur on their own. And an earlier single-turn run of this feature came out negative (about +2 points at +22% cost, graded by word matching, a method since replaced), so the result depends on the task shape: it helped where a stall arrived after work was already under way and a redirect had something to act on.
| Task | Behaviour | supervisorOff pass | flagOn pass | Sup. | Δ cost | Δ lat | Δ tok | Seeds ctrl→cand |
|---|---|---|---|---|---|---|---|---|
| sup-mid-ask-user | changed | 3/3 | 3/3 | 3 | +26% | +29% | +38% | s1 ✓→✓ s2 ✓→✓ s3 ✓→✓ |
| sup-mid-config-chain | changed | 0/3 | 0/3 | 3 | +43% | +43% | +59% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| sup-mid-control-clean-a | no change | 3/3 | 3/3 | 0 | +0% | -1% | +0% | s1 ✓→✓ s2 ✓→✓ s3 ✓→✓ |
| sup-mid-control-clean-b | changed | 3/3 | 3/3 | 0 | +36% | +2% | +6% | s1 ✓→✓ s2 ✓→✓ s3 ✓→✓ |
| sup-mid-nested-lookup | changed | 0/3 | 0/3 | 3 | +33% | +33% | +53% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| sup-mid-newer-status | changed | 0/3 | 3/3 fixed | 3 | +151% | +115% | +118% | s1 ✗→✓ s2 ✗→✓ s3 ✗→✓ |
| sup-mid-transitive-import | changed | 0/3 | 0/3 | 3 | +18% | +23% | +24% | s1 ✗→✗ s2 ✗→✗ s3 ✗→✗ |
| sup-mid-wrong-file | changed | 0/3 | 3/3 fixed | 3 | +124% | +95% | +106% | s1 ✗→✓ s2 ✗→✓ s3 ✗→✓ |
Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.