The Trajectory Supervisor is one of the newest parts of the harness, and it cannot be judged on a one-shot question: it needs a stall with work already behind it. It was run on a purpose-built multi-turn corpus, with the result decided by rules fixed in advance. On injected stalls it recovered two tasks, at a real cost. Every run has its own page.
A harness is a governance and reliability layer wrapped around a model. Everything it adds costs latency and tokens on every turn, and is only worth paying for if it changes outcomes: a stalled run that recovers.
Our first pass at this benchmark ran mostly single-turn tasks. That is not the shape the supervisor is for: it needs a stall with context behind it. So we took the results down, rebuilt the corpus, and re-ran. This page is the rebuilt result, and it is allowed to come back negative.
Modelled on how ML systems are evaluated: a fixed dataset, a mechanical grader, an isolated independent variable, and a confidence interval on the result.
Write, in one sentence, the belief the feature encodes — the thing that would have to be true for it to be worth the cost.
A benchmark arm that flips this one feature and nothing else. Everything else — model, corpus, grader — is held fixed.
A tagged set of tasks built so the feature should show its value here if it shows it anywhere. No delta on its own stress slice means no value.
Run the matrix at least three times. Report Δ task success, recovery, overconfident-wrong answers, cost, latency and tokens — each with a 95% confidence interval.
Effective, similar or lower cost: positive delta on the stress slice, no regression elsewhere, cost at or below budget. Effective, but costs more: a real gain that comes at a real cost — worth it only if the gain clears the bar. Not effective: no measurable gain, regardless of cost. Underpowered: too little data to say either way — grow the corpus and re-run.
Not just the number. Where the idea came from, what we expected, what actually happened, and why — as an entry on this page, with every run's turn-by-turn transcript published alongside it. Negative results included.
The Trajectory Supervisor, measured against the same assistant without it, three seeds, Claude Sonnet 5 as both the assistant and the judge. The verdict comes from fixed rules: a gain has to clear its confidence band to count (here every seed agreed, so the band is zero-width), and a gain that costs more has to be big enough to justify it. The result page holds the case study, the numbers and every run's transcript.
| Feature | The bet it encodes | Verdict | Evidence |
|---|---|---|---|
| Trajectory Supervisor stall-edge LLM call · Layers 7 & 10 | When a multi-turn task stalls after work is under way, a targeted directive (redirect, reframe, ask) recovers runs that would otherwise fail. | Better results, at higher cost | 8 tasks, 3 seeds. Task success 37.5% → 62.5% (+25 pts). Two stalled tasks recovered only with it; three recovered by neither. Cost +43%, latency +33%. Never consulted on the two clean controls. Stalls are injected, not organic. Both gains are identical across all 3 seeds, so the interval is zero: two tasks, not a distribution. Read the actual runs → |
The Trajectory Supervisor is the only feature measured at 3 seeds on a purpose-built corpus. Other layers have single-conversation, one-seed demonstrations, read turn by turn. Shown to change the outcome: semantic contradiction, model-inferred facts, semantic failure matching and its system-error retry, stuck-plan resume, the tool-policy gate (bounded work, not a better answer), mid-turn steering, next-step options, the structured ask-question mechanism and semantic compaction. Fires but shows no outcome difference, because the base model already does it: change review, source reliability, hypotheses, injection detection, verification, code plan templates and reviewer revision. Not demonstrable through the assistant: planning decomposition, diagnostics, and the harness-level control-state resolver. None of these has been run across 3 seeds.
Eight tasks, including controls where the feature should do nothing. The tasks are small on purpose, so a grader can check them without a model's opinion. What makes them relevant is their shape.
Multi-turn tasks with a tool failure injected on a later turn, after useful work: pointer chains, a file that only points elsewhere, nested lookups, a stale answer overridden by a newer file, and a turn where the user must supply a detail. Two clean sessions check it stays quiet.
Every result on this page links to its per-run transcripts: the full turn-by-turn conversation for each task, arm and seed, with the model it executed on and the judge's verdict and reasoning. The assistant and harness under test are the open-source code in the product repo.
Because “11 layers, thousands of tests” proves we built something, not that it works. A harness is only worth its latency and token cost if it changes outcomes. The honest way to know is a benchmark that can come back negative or unproven, and we publish it either way.
No. It stays merged and flag-gated, and the result is recorded rather than overridden. If a later test on a different task shape shows value, the flag can flip. The default reflects the evidence we have.
Claude Sonnet 5, with Claude Sonnet 5 as the LLM judge. Both ids are named on every transcript page.
The architecture page is the map of what the harness contains, read straight from the source. This page is the record of which of those parts we have put on the scale, and what the scale said.
Aielia ships the harness — and the benchmark that keeps it honest. Every layer is a drawable canvas node if you want to build and evaluate your own.