No measurable improvement, at extra cost
Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report
Hypothesis. The trajectory supervisor's one LLM call on the cannotMakeProgress() stall edge redirects enough stalled runs into recovery to justify its extra cost, latency and tokens.
No task-success delta (CI includes 0) but the candidate costs materially more (+22% $/turn, +40% latency). Neutral with real cost — it does not earn its keep.
Arms: flagOn (control) vs supervisorOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.
| Metric | flagOn | supervisorOn | Δmean | ±CI95 |
|---|---|---|---|---|
| taskSuccessRate | 0.648 | 0.667 | +0.019 | ±0.036 |
| hallucinationRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| unauthorizedEffectRate | 0.000 | 0.000 | 0.000 | ±0.000 |
| recoveryRate | 0.056 | 0.000 | -0.056 | ±0.109 |
| overconfidentWrongRate | 0.040 | 0.000 | -0.040 | ±0.040 |
| supervisorConsultsMean | 0.000 | 0.333 | +0.333 | ±0.000 |
| meanLatencyMs | 9674 ms | 13577 ms | +3903 ms | ±427 ms |
| meanCostUsd | $0.0105 | $0.0127 | +$0.0023 | ±$0.0023 |
| totalTokens | 8676 | 13798 | +5122 | ±596 |
Cost +22% · latency +40% · tokens +59% (candidate vs control).
A slow-loop meta-controller that wakes up only when the run has measurably stopped making progress. It reads a digest of the failure trajectory and returns one directive — redirect the strategy, reframe the plan, spawn a bounded read-only investigation, ask the user a targeted question, or abort. Between those moments it stays out of the way: the main agent keeps every tactical decision.
In August 2026 NVIDIA published AVO, an agent architecture that scored a perfect 100.00 across all 25 ARC-AGI-3 environments and ran a seven-day GPU-kernel optimisation that beat cuDNN and FlashAttention-4. Their writeup credits two components for sustaining that kind of long-horizon autonomy: a supervisor that "monitors the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent toward alternative strategies," and a persistent trajectory memory that lets the agent "resume from the current state rather than repeatedly reconstructing the search."
That shape mapped almost one-to-one onto our own recovery path — except ours was a blind ladder: on a stall it just advanced to the next strategy archetype without reading why it was stuck. Making that branch intelligent looked like free upside.
One caveat we wrote down before starting: AVO's report contains no controlled ablation of the supervisor — the authors say plainly that their experiment "does not isolate its individual contribution." A perfect score on a seven-day task is a strong signal for the architecture as a whole, not evidence that this one part carries its weight. Which is exactly what our benchmark is for. The bet: a stalled run recovers more often, or more cheaply, with a targeted directive than by blindly advancing to the next strategy archetype in a fixed ladder.
The arm is the assistant with the supervisor flag on versus off — nothing else changes. The original stress corpus was six lookup and pivot tasks that inject a persistent tool failure, so the run is forced to the stall edge where the supervisor is actually consulted (later expanded — see the timeline below). Everything is graded mechanically and re-run across three independent seeds with an LLM judge on.
Task success moved +2.8% (CI ±10.9%) — noise. Worse, the supervisor couldn't have helped: its two commonest directives can't re-queue a single-node task graph, so on a one-shot assistant turn there was nothing for them to act on. The measurement apparatus, not the idea, was the blocker.
Four changes: the recovery path now re-queues a failed leaf task after a redirect or an investigation; the investigation sub-agent can now read workspace files (it was limited to web search); its findings are spliced into the model's context; and the failure-injection harness allows a real second attempt over the new evidence. Now the supervisor runs a full recovery cycle on every stall.
With the supervisor able to act: task success 72.2% → 68.5% (Δ −3.7%, CI ±3.6% — just outside the band). Latency +34%, cost +32%, tokens +48%, all far outside their intervals. And zero additional stalls recovered — the assistant already recovers one of the six on its own, and routing that recovery through the supervisor's extra model call converts none.
| Metric | Δ (RUN 2) |
|---|---|
| Task success | −3.7% |
| Stall recoveries | 0 extra |
| Mean latency | +34% |
| Mean cost | +32% |
| Tokens per turn | +48% |
Per-seed task success — off: 72.2 / 72.2 / 72.2; on: 72.2 / 66.7 / 66.7. No change in hallucination or unauthorized actions on either arm.
The 6 lookup/pivot -stall tasks were joined by a new supervisor_conversation
slice (6 tasks, 2 turns) — turn 1 stalls, the supervisor asks the user a targeted question, turn
2 the user answers and the run should recover — a shape the original 12-task slice couldn't
exercise (ASK_USER just ends a turn with no follow-up). Re-run across all 18 tasks,
3 seeds: task success 64.8% → 66.7% (a small, unproven gain — CI ±3.6pt), cost
+22%/turn, latency +40%. Even with a task shape built for the supervisor's most distinctive
move, it still didn't clear its cost — this is the run behind the numbers table above.
The idea didn't transfer. AVO's supervisor earns its keep across a seven-day search
with thousands of actions and long plateaus to detect. An assistant turn resolves in one or two
iterations — there is almost nothing for a stagnation monitor to monitor, and when a stall
is forced, the deterministic recovery already handles the one case in six that's
recoverable at all. Adding a model call on top of that bought nothing and cost a third more per
turn — and it didn't change even after giving the supervisor's most distinctive move, asking the
user, a task shape where it could actually pay off. HARNESS_TRAJECTORY_SUPERVISOR
stays off by default; the code stays merged and flag-gated. One open question remains — a
decomposed, multi-step planning path, closer to AVO's own setting, where a redirect or a
re-framed plan has downstream tasks to affect — and that needs its own benchmark slice before
the idea is closed for good.
Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.