The 11-layer harness vs. a bare model loop

Compare: session-revised-plan

Model: Claude Sonnet 5 (claude-sonnet-5), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report

Prompt

I need to pick a caching layer. Requirements: must support TTLs.
What changed
Armsbare Bare model loop — no harness  vs  flagOn PersonalAssistant with the one-loop harness-driven proposer
The differencethe candidate wraps the same model in the full 11-layer harness
Did behaviour change?
Final replydiffers
Tool callssame 0 calls
Supervisor consults0 / 0
Harness layersdiffer — candidate also: diagnostics, execution, reviewer_pass, verification
Graded outcomeboth pass
Impact
This taskThe candidate behaved differently but the graded outcome was the same.
Shown seedcost +39% · latency +55% · tokens +48% (candidate vs control, seed 1)

Side by side — control (left) vs candidate (right), seed 1

bare — control
flagOn — candidate
Final reply
Redis. It's the only one of the three in `options.md` that offers persistence, and all three support TTLs — so it's the only option that meets both of your requirements ("support TTLs" + "survive a restart"). Practical guidance: - Turn on **AOF** (`appendonly yes`). Use `appendfsync everysec` for a good durability/performance balance (max ~1s loss on a crash), or `always` if you can't tolerate any loss. - Running both AOF + RDB snapshots is the typical production setup. - If the restart requir
**Redis.** It's the only option in `options.md` that satisfies all three requirements: TTLs, persistence (survives restart), and multi-node sharing. For the persistence config, enable **both AOF and RDB** — AOF (`everysec`) for a small data-loss window on recovery, RDB for fast restarts and backups. If you're using a managed service and Multi-AZ durability matters, consider AWS MemoryDB; otherwise plain Redis/ElastiCache is fine as long as you still treat it as a cache, not the source of truth.
Tool calls

no tool calls

no tool calls

Grader checks
CheckVerdict
contains "redis"pass
not contains "memcached is the"pass
not contains "use memcached"pass
not contains "in-process"pass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered n/a

CheckVerdict
contains "redis"pass
not contains "memcached is the"pass
not contains "use memcached"pass
not contains "in-process"pass

success yes  ·  hallucination no  ·  unauthorized effect no  ·  recovered n/a

Metrics
latency32132 ms
cost$0.0471
tokens1739
supervisor consults
latency49762 ms
cost$0.0655
tokens2569
supervisor consults0

Other seeds — seed 2: fail · $0.0588 · 38.2 s   seed 3: fail · — · 2.9 s

Other seeds — seed 2: fail · $0.0689 · 51.5 s   seed 3: pass · $0.0648 · 50.9 s

Read the full turn

Conversation — bare (control) · seed 1
user
I need to pick a caching layer. Requirements: must support TTLs.
model
Nearly every mainstream caching layer supports TTLs, so that requirement alone won't narrow things much. Here's a quick rundown:

**Redis** – Per-key TTLs (`EXPIRE`, `SET ... EX`), plus active and lazy expiration. Rich data structures, persistence options, pub/sub, Lua scripting, clustering. The default choice unless you have a reason to go elsewhere.

**Memcached** – Per-item TTLs. Simpler, pure in-memory key/value (string/blob values only), multithreaded, very low overhead. Good when you want a plain LRU cache and nothing more. No persistence, no replication.

**In-process caches** (no network hop, single-node only):
- **Caffeine** (Java) – `expireAfterWrite`, `expireAfterAccess`, custom `expireAfter`.
- **`cachetools`** (Python) – `TTLCache`.
- **go-cache / Ristretto** (Go) – per-item TTLs.
- **`lru-cache`** (Node) – `ttl` option.

**Hazelcast / Apache Ignite** – Distributed in-memory data grids with per-entry TTL and eviction policies; heavier, more feature-rich (compute, SQL).

**Cloud-managed** – ElastiCache (Redis/Memcached), MemoryDB (Redis, durable), Google Memorystore, Azure Cache for Redis. Same engines, less ops burden.

Better discriminators to decide on:
- **Shared vs. per-instance?** Multiple app servers needing a consistent view → Redis/Memcached. Single service → in-process is faster and simpler.
- **Just key/value or need structures/atomic ops?** → Redis.
- **Need persistence/failover?** → Redis or MemoryDB.
- **Eviction behavior under memory pressure** matters more than TTL for most workloads.

If you have no strong constraints, Redis is the safe, standard pick.
turn 2
model
`options.md` is brief — it lists three options with just two attributes each:

- **memcached** – TTLs: yes; persistence: no
- **redis** – TTLs: yes; persistence: yes
- **in-process LRU** – TTLs: yes; persistence: no; single-node only

So all three meet your TTL requirement. The file's only real discriminators are persistence (redis is the only one that has it) and the note that in-process LRU can't be shared across nodes.
turn 3
model
That eliminates two of the three. Per `options.md`:

- **memcached** – persistence: no → out
- **in-process LRU** – persistence: no → out
- **redis** – persistence: yes → the only option that qualifies

So with "must support TTLs" + "must survive a restart," **redis** is the pick.

One caveat worth knowing: Redis persistence isn't automatic-perfect. You choose between:
- **RDB snapshots** – periodic point-in-time dumps; a crash can lose everything since the last snapshot.
- **AOF (append-only file)** – logs every write; with `appendfsync everysec` you lose at most ~1 second, with `always` essentially nothing (slower).
- **Both** together is the common production setup.

If "survive a restart" means a planned restart, any config handles it (Redis writes on graceful shutdown). If it means surviving a crash with zero data loss, use AOF with `appendfsync always`, or consider MemoryDB, which is Redis-compatible with durable multi-AZ replication.
turn 4
model
Redis.

It's the only one of the three in `options.md` that offers persistence, and all three support TTLs — so it's the only option that meets both of your requirements ("support TTLs" + "survive a restart").

Practical guidance:
- Turn on **AOF** (`appendonly yes`). Use `appendfsync everysec` for a good durability/performance balance (max ~1s loss on a crash), or `always` if you can't tolerate any loss.
- Running both AOF + RDB snapshots is the typical production setup.
- If the restart requirement really means "zero data loss across an unexpected crash / AZ failure," consider **MemoryDB** (Redis-compatible, durable multi-AZ) instead of self-managed Redis.
final reply
Redis.

It's the only one of the three in `options.md` that offers persistence, and all three support TTLs — so it's the only option that meets both of your requirements ("support TTLs" + "survive a restart").

Practical guidance:
- Turn on **AOF** (`appendonly yes`). Use `appendfsync everysec` for a good durability/performance balance (max ~1s loss on a crash), or `always` if you can't tolerate any loss.
- Running both AOF + RDB snapshots is the typical production setup.
- If the restart requir
Conversation — flagOn (candidate) · seed 1
user
I need to pick a caching layer. Requirements: must support TTLs.
risk LOWproposer flat-oneloopnon-trivialmode TOOL
model
Almost every caching layer supports TTLs, so that requirement alone doesn't narrow it much. The real decision driver is usually topology:

**In-process (single app instance, fastest, no network hop)**
- **Caffeine** (JVM) — expireAfterWrite/expireAfterAccess, excellent hit ratios
- **`node-cache`** or **lru-cache** (Node) — per-key TTL
- **cachetools** `TTLCache` (Python)

**Distributed / shared across instances**
- **Redis** — per-key TTL (`EXPIRE`, `SET ... EX`), the default choice; also does eviction policies, persistence, pub/sub
- **Memcached** — per-item TTL, simpler and slightly cheaper on pure LRU cache workloads, but no persistence or rich data types
- **Managed**: ElastiCache, MemoryDB, Upstash, Momento (Momento has TTL as a first-class required param)

**Default recommendation:** Redis if the cache is shared across multiple processes/servers or needs to survive restarts; a local in-process cache (Caffeine / lru-cache / cachetools) if it's a single instance and you just want to cut latency.

To give a sharper pick, tell me:
1. Language/runtime?
2. Single instance or multiple app servers sharing the cache?
3. Rough data size and does it need to survive restarts?
turn 2
user
What does options.md say about each?
risk LOWproposer flat-oneloopnon-trivialmode TOOL
model
`options.md` covers three:

- **memcached** — TTLs: yes. Persistence: no.
- **redis** — TTLs: yes. Persistence: yes.
- **in-process LRU** — TTLs: yes. Persistence: no. Single-node only.

All three meet your TTL requirement, so it comes down to whether you need persistence and multi-node sharing (only Redis gives you both).
turn 3
user
New requirement: it also has to survive a restart (persistence).
risk LOWproposer flat-oneloopnon-trivialmode TOOL
model
That eliminates two of the three in `options.md`:

- **memcached** — no persistence → out
- **in-process LRU** — no persistence (and single-node only) → out
- **redis** — TTLs: yes, persistence: yes → the only fit

So it's **Redis**. Note its persistence has two modes worth deciding on:

- **RDB snapshots** — periodic point-in-time dumps; fast restarts, but you can lose the writes since the last snapshot.
- **AOF (append-only file)** — logs every write; much smaller data-loss window (typically ≤1س with `everysec` fsync), slightly slower and larger on disk.
- You can run **both** (recommended for durability) — AOF for recovery, RDB for fast restarts/backups.

One caveat: Redis persistence protects against process/host restarts, but treat it as a cache, not a system of record — if it's a managed service (ElastiCache, MemoryDB, Upstash), check the durability guarantees. MemoryDB is the one AWS option that's Multi-AZ durable by design if that matters.
turn 4
user
So which one should we use?
risk LOWproposer flat-oneloopnon-trivialmode TOOL
final reply
**Redis.** It's the only option in `options.md` that satisfies all three requirements: TTLs, persistence (survives restart), and multi-node sharing.

For the persistence config, enable **both AOF and RDB** — AOF (`everysec`) for a small data-loss window on recovery, RDB for fast restarts and backups. If you're using a managed service and Multi-AZ durability matters, consider AWS MemoryDB; otherwise plain Redis/ElastiCache is fine as long as you still treat it as a cache, not the source of truth.

Harness trace

Full harness trace — bare · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

Other trace events

{
  "kind": "turn_boundary",
  "turn": 2,
  "prompt": "What does options.md say about each?"
}
{
  "kind": "turn_boundary",
  "turn": 3,
  "prompt": "New requirement: it also has to survive a restart (persistence)."
}
{
  "kind": "turn_boundary",
  "turn": 4,
  "prompt": "So which one should we use?"
}
Full harness trace — flagOn · seed 1

The harness runs on every turn. Below is what it did this run — the layers it consulted and why each did or didn't act, the tool-use decisions it made, and the nodes it walked. Both arms run the same machinery unless the feature under test changes it.

risk LOWproposer flat-oneloopmode TOOLplan none

Harness layers

LayerActed?Why
world_model×4single LOW-risk task, no durable fact stated — observation only
evidence_reasoning×4single low-stakes observation is sufficient
hypothesis×8single clear LOW-risk task — no competing explanation worth surfacing
contradiction×8fewer than 2 beliefs — nothing to compare
diagnosticsacted ×8Health: nominal
control_state×8NORMAL
planning×4one eligible task — serial execution
executionacted ×4module_type=business_logic
verificationacted ×4all applicable layers passed
recovery×4task completed — nothing to recover from
reviewer_passacted ×4Success criterion not covered by any belief: "Respond helpfully, accurately, and safely to the user request."

Tool-policy decisions

ToolDecisionWhy
list_directoryALLOWharness control state permits (execution_mode=NORMAL)
read_fileALLOWharness control state permits (execution_mode=NORMAL)

Node path

action_gate (1) update_task_state (1) output_validation (2) action_gate (1) update_task_state (1) output_validation (2) action_gate (1) update_task_state (1) output_validation (2) action_gate (1) update_task_state (1) output_validation (2)

Other trace events

{
  "kind": "turn_boundary",
  "turn": 2,
  "prompt": "What does options.md say about each?"
}
{
  "kind": "turn_boundary",
  "turn": 3,
  "prompt": "New requirement: it also has to survive a restart (persistence)."
}
{
  "kind": "turn_boundary",
  "turn": 4,
  "prompt": "So which one should we use?"
}

← index