Studies
What Lectern's orchestration actually costs and delivers, measured. Every number traces to a machine-readable run report; the method, the harness, and the raw traces are public.
Methodology · Harness & tasks · reproduce with python3 bench/runner.py
Does persistent memory change the outcome?
2026-07-12model claude / sonnet · convention tasks · subscription
On convention tasks whose correctness depends on a project rule held only in Lectern's memory, the same model passes 8/8 with the brain on and 0/8 with it off — and bare Claude Code fails 0/4. The memory is the only variable that changes the outcome.
Pass rate — same model, only the brain differs
The grader requires arbitrary catalog codes that live only in a seeded skill — unguessable and absent from the workspace, so a capable model can't reach them without the brain. Building a valid brain-off control also surfaced and fixed a real bug: the no-brain switch had left skills materialized where Claude Code reads them, so “brain off” still leaked them; it now stops that too.
- Narrow by design: 2 conventions × 2 arms × 2 reps on Claude/Sonnet. The effect is categorical (8 vs 0), but breadth — more conventions and domains, larger N — is the next step.
- This measures persistent-memory value on convention-dependent work, a specific capability — not a general “smarter” claim. It sits beside the low-overhead studies, where the brain is correctly neutral.
- Same fresh workspace and prompt across arms; the only variable is the brain. Deterministic grader, exit 0 = pass, run in the workspace after the agent.
Single model vs the Conductor
2026-07-06model opencode/deepseek-v4-flash-free · 2 runs per task per arm · $0 — free tier only
On tasks a single call already solves, the Conductor adds no success and costs +32% tokens and ~3.5x wall time. The overhead tracks decomposition: +1% on single-step tasks, +84% on multi-step ones.
Tokens per task
Where the overhead concentrates
Conductor token overhead vs single, grouped by how far the task decomposes.
| task | plan | single tok | conductor tok | overhead | single wall | conductor wall | passed |
|---|---|---|---|---|---|---|---|
| cross-file-slug | 3 | 14,336 | 35,732 | +149% | 10.1s | 43.1s | 4/4 |
| dedup-list | 1 | 13,796 | 13,887 | +1% | 5.7s | 22.4s | 4/4 |
| fix-off-by-one | 2 | 14,220 | 21,622 | +52% | 10.7s | 40.9s | 4/4 |
| fizzbuzz | 1 | 13,830 | 13,951 | +1% | 5.8s | 20.9s | 4/4 |
| json-config | 1 | 13,891 | 13,835 | +0% | 6.9s | 19.5s | 4/4 |
| refactor-counter | 2 | 14,194 | 21,252 | +50% | 8.7s | 25.4s | 4/4 |
| stack-class | 1 | 13,896 | 14,152 | +2% | 6.4s | 20.6s | 4/4 |
| temp-convert | 1 | 13,782 | 13,761 | +0% | 5.2s | 16.8s | 4/4 |
- No success headroom: the free model passes everything either way, so this measures the Conductor's cost, not its benefit.
- review_steps under-reports — the review runs on file-modifying tasks but emits no routing event; read review cost from the token delta.
- tool_calls/changes read 0 on the opencode backend (it edits in place); tokens and grading are accurate.
- 2 repetitions on a free model — directional, not definitive.
Harder tasks — Lectern vs the raw agent
2026-07-066 harder tasks · 5 arms · subscription CLIs, no API keys · graders validated against reference solutions
Same tasks, same Claude Code subscription, with and without Lectern: 6/6 both, +1.0% tokens, same wall time. The engine layer — indexing, brain recall, session capture, the Apply pipeline — is effectively free on top of the agent it drives.
The Conductor's per-step routing demonstrably fires: quick steps went to Haiku, the main step to Sonnet — two models inside one task, all six tasks passing fully routed.
At this difficulty orchestration still shows cost, not success gain: strong single calls pass everything, so plan-and-review can only add overhead. Its success case needs task classes where single calls genuinely fail.
Same model, with and without Lectern
| arm | passed | mean tokens | mean wall |
|---|---|---|---|
| free single ×2 · deepseek free tier | 11/12 | 15,252 | 40.8s |
| free conductor ×2 · deepseek free tier | 11/12 | 32,655 | 76.7s |
| raw Claude Code · claude -p, no Lectern | 6/6 | 11,557 | 34.4s |
| Lectern + Claude Code · lectern run | 6/6 | 11,671 | 35.1s |
| Conductor, routed · Haiku/Sonnet per step | 6/6 | 4,425 † | 113.4s |
† cache-accounting artifact — not comparable across backends; read cost from wall time.
- Cross-backend token totals are not comparable: Claude Code reports usage excluding prompt-cache reads; opencode reports fuller totals. The Conductor-routed arm's low token figure is a cache-accounting artifact — read its cost from wall time.
- Subscription arms ran once each (bounded deliberately); free arms twice. Directional, not definitive.
- One free-single run timed out at 240s (free-tier flakiness) and counts as a failure.
Next: task classes where a single call genuinely fails — long-horizon, large-repo, cross-session work — and cache-aware token accounting so cross-backend costs compare fairly. New results land here and in the repo as they run.