The architecture page shows every layer and backstop we built. This one shows which of them we've proven earns its latency and token cost — and which we've measured, found wanting, and switched off. Every result links to its own page with the full story — what the feature does, why we tried it, and what we found.
A harness is a governance and reliability control plane wrapped around a model. Every layer it adds costs latency and tokens on every turn. That cost is only worth paying if the layer actually changes outcomes — catches a contradiction the free lexical check missed, recovers a run that would otherwise stall, stops a wrong answer going out.
The honest way to know is a controlled benchmark that is allowed to come back negative. So we built one, and we run it against our own features. When a feature doesn't earn its cost, we say so here and switch it off — the code stays, flag-gated, so a later test on a different task shape can flip it back if the evidence changes.
Almost every candidate has the same shape: an expensive per-turn LLM call layered on top of a free lexical or mechanical check, added on the bet that the cheap check misses real cases often enough to pay for itself. That bet is exactly what the benchmark tests.
Modelled on how ML systems are evaluated: a fixed dataset, a mechanical grader, an isolated independent variable, and a confidence interval on the result.
Write, in one sentence, the belief the feature encodes — the thing that would have to be true for it to be worth the cost.
A benchmark arm that flips this one feature and nothing else. Everything else — model, corpus, grader — is held fixed.
A tagged set of tasks built so the feature should show its value here if it shows it anywhere. No delta on its own stress slice means no value.
Run the matrix at least three times. Report Δ task success, recovery, overconfident-wrong answers, cost, latency and tokens — each with a 95% confidence interval.
Effective, similar or lower cost: positive delta on the stress slice, no regression elsewhere, cost at or below budget. Effective, but costs more: a real gain that comes at a real cost — worth it only if the gain clears the bar. Not effective: no measurable gain, regardless of cost. Underpowered: too little data to say either way — grow the corpus and re-run.
Not just the number. Where the idea came from, what we expected, what actually happened, and why — as an entry on this page, plus a row in the benchmark doc and the machine report JSON in the repo. Negative results included.
Six features decided. This table grows as each audit lands, positive and negative alike. The 11-layer harness vs. a bare loop, and the one-loop proposer, both had a first pass whose result we withdrew — the failure-injection tasks were being applied to one arm and not the other — and were re-run on a corrected corpus (below).
| Feature | The bet it encodes | Verdict | Evidence |
|---|---|---|---|
| Trajectory Supervisor HARNESS_TRAJECTORY_SUPERVISOR · Layers 7 & 10 | A stalled run recovers better with a meta-controller that reads the failure trajectory and issues a targeted directive than by walking the deterministic strategy ladder. | Worse results, at higher cost | 3-seed, 18-task, judge-on. Task success −3.7% (CI ±3.6%); latency +34%, cost +32%, tokens +48%; zero extra stall recoveries. Read the actual runs → |
| Semantic contradiction backstop +1 LLM call per belief batch · Layer 4 | An LLM contradiction check catches paraphrased / cross-framing belief conflicts that the always-on lexical negation-pair check provably can't — often enough to pay for the call. | Better results, at higher cost | 3-seed, 14-task, judge-on. Task success +7.1% (85.7% → 92.9%, CI clears 0); overconfident-and-wrong 12.8% → 7.7%; cost +29%/turn, latency flat. The gain clears the cost. Read the actual runs → |
| LLM injection detection on tool output +1 LLM call per fetched page / shell result | An LLM classifier on every tool output catches prompt-injection the deterministic pattern pass misses, without over-flagging benign instruction-like content. | No measurable improvement, at extra cost | 3-seed, 10-task, judge-on. No task-success delta (both 80%); hallucination 20% → 10% but cost +21%/turn for no success gain. Neutral with real cost. Read the actual runs → |
| Semantic failure-mode backstop +1 LLM call per failure-library miss · Layer 10 | An LLM matcher recognises known failure patterns described in different words often enough to earn its per-miss call and improve recovery routing. | Better results, at higher cost | 3-seed, 8-task, judge-on. Task-success delta +12.5% but CI ±28% — not significant; cost +48%/turn, latency +15%. Neutral with real cost. Read the actual runs → |
| The 11-layer harness vs. a bare ReAct loop · whole subsystem | The full 11-layer harness (staging, control-state, verification, reviewer, memory) produces materially better task outcomes than a bare ReAct loop — enough to justify running the whole subsystem every turn. | Better results, at higher cost | 3-seed, 76-task, judge-on. Task success +16.2pt (67.6% → 83.8%, CI ±25.6pt — includes 0, unproven); cost +77%/turn, latency +88%. Can't prove it helps; can prove it costs more. Read the actual runs → |
One-loop proposer
ASSISTANT_ONE_LOOP · default since 2026-09-06
|
Letting the harness drive tool calls in-loop, one iteration at a time, beats post-hoc bookkeeping over an already-finished reply — validating the 2026-09-06 default flip. | No measurable difference, but cheaper | 3-seed, 76-task, judge-on. Task success +0.5pt (81.0% → 81.5%, CI ±2.6pt) — flat; cost −18%/turn, latency flat. Neither helps nor hurts measurably; cheaper either way. Read the actual runs → |
Ordered by how much new scaffolding a verdict needs.
| Feature | Per-turn cost | Batch |
|---|---|---|
| The reviewer passLayer 11 · 3 lenses | Consistency, adversarial and abstraction-fit review on every harness turn. | Batch C |
| The verification layerLayer 9 · 9 sub-checks | Always-on mechanical verification of every result before it's accepted. | Batch C |
| Up-front decomposition + task reframingLayer 5 | Breaking a request into a task graph and rewriting each task to lead with its subject. | Batch C |
| Model-inferred factsnever auto-promoted to memory | The classifier's free-form read of what the user implied — recorded for the session, never kept. | Batch C |
| Semantic criterion coverageLayer 11 | +1 LLM call per plan success-criterion the reviewer's substring check couldn't match. | Batch C |
| Semantic review-checkerLayer 11 | +1 LLM call per proposed change, checking for a real conflict with a held belief. | Batch C |
| AnswerClaim attributionevidence / confidence line | Cheap to compute — a UX surface. Its value is a trust question, not a benchmark one. | Open — not benchmark-shaped |
The task corpus, the mechanical grader, the arm definitions and the multi-seed runner all
live in the product repo under packages/aielia/eval/. Every result on this page
links to its machine report JSON, and each run records the model it executed on. One command re-runs the
supervisor matrix:
npx tsx scripts/run-harness-benchmark.ts \
--arms=flagOn,supervisorOn \
--slice=supervisor_lookup,supervisor_pivot \
--seeds=3Because "11 layers, 27 node types, thousands of tests" proves we built something, not that it works. A harness is only worth its latency and token cost if it changes outcomes. The honest way to know is a benchmark that can come back negative — and when it does, we say so and switch the feature off.
No. A feature that doesn't earn its cost stays merged and flag-gated off. The negative result is recorded, not overridden. If a later test on a different shape of task shows value, the flag can flip back on — but the default reflects the evidence we actually have.
The internal gate every behaviour-changing change must clear: a measured delta on a fixed task corpus, multi-seed with a reported confidence interval, before it becomes a default. A feature that shows no positive delta on its own stress slice does not ship on by default.
Each run writes the model id it actually executed on — and the LLM judge's model — into its machine report, and every transcript page names both, so a published number can always be traced back to the model that produced it.
The architecture page is the map of everything the harness contains, read straight out of the source. This page is the running record of which of those parts we've put on the scale — and what the scale said.
Aielia ships the harness — and the benchmark that keeps it honest. Every layer is a drawable canvas node if you want to build and evaluate your own.