DispatchBench: which model should a coding task go to?

WINNER

With Codex quota, use sol61-high ⚡.

Cheapest without misses: haiku-high.
For speed, pick sonnet-medium.
Run sample tests; send a failed check to haiku-low.

Picks and model names

Four configs passed every try in every set. They were haiku-high, sonnet-medium, sol61-high and astra-medium. Repo tasks counted as task complete: hidden and visible tests both pass.

haiku-high is the cheapest, at $0.0028 per pass.

Cheapest main-ladder pass-rate matches, Claude → Codex: haiku-low → terra61-high. haiku-medium, haiku-high and sonnet-medium → terra61-low.

Cheapest main-ladder pass-rate matches, Codex → Claude: terra61-low → haiku-high; terra61-high → haiku-medium; sol61-medium → haiku-medium; sol61-high → haiku-high.

Against haiku-high, sol61-high takes 1.9× the time and ~7× the $ per pass. ⚡ makes that 1.2× the time at ~13× the $.

Numbers: the 12-problem main ladder, 60 tries per config. Time = median seconds. ~ = estimated from tokens; Claude costs are measured.
Use case	Pick	Pass	$ per pass	Time	Why
1. Default	sol61-high ⚡	60/60	~$0.036	17.4 s	No misses. $0 on the Claude bill (spends Codex quota). ⚡ saves 10 s. astra ⚡ is faster but uses 4× the quota
2. Budget	haiku-high	60/60	$0.0028	14.7 s	Cheapest with no misses. haiku-low costs less but misses 3 of 60
3. Speed	sonnet-medium	60/60	$0.012	10.1 s	No misses. 4.6 s faster than haiku-high.

⚡ = fast mode: same pass rate, about 2× list price, faster.

Caveat: sonnet-medium's code read worst in a blind review (see Notes).

Model names

Config = one model at one effort level, e.g. haiku-low.
Claude models: haiku and sonnet.
Codex models: luna, sol and astra (OpenAI).
Suffix = effort: low / medium / high / xhigh (a step above high).
luna = gpt-6-luna, sol61 = gpt-6.1-sol, astra = gpt-6-astra. luna56 = gpt-5.6-luna, sol56 = gpt-5.6-sol (older generation).

LeetCode-style problems, written fresh for this bench. All calls ran 2026-10-22 to 2026-10-28.

Results

sol61-high ⚡ is the Codex pick. haiku-high is the cheapest config with no misses on the main ladder.

Verdict key: ✓ pick = a Winner pick. 0 misses / missed N = raw miss count, no ranking. beaten = another row is as cheap and passes as often. trade-off = pricier but passes more. avoid = below 90%.

config	passed	$ per pass	Claude $	median s	verdict
Main ladder: 12 problems × 5 tries = 60 calls
haiku-low	95% (57/60)	$0.0011	$0.0011	10.0	trade-off
haiku-medium	98% (59/60)	$0.0015	$0.0015	13.0	missed 1
haiku-high	100% (60/60)	$0.0028	$0.0028	14.7	✓ pick (budget)
sonnet-medium	100% (60/60)	$0.012	$0.012	10.1	✓ pick (speed)
sol61-medium	97% (58/60)	~$0.015	$0	22.6	beaten
luna-high	92% (55/60)	~$0.0010	$0	12.2	trade-off
luna-medium	78% (47/60)	~$0.00089	$0	8.2	avoid
luna-low	77% (46/60)	~$0.00090	$0	7.2	avoid
astra-medium	100% (60/60)	~$0.075	$0	20.9	0 misses
astra-medium ⚡	100% (60/60)	~$0.15	$0	9.9	0 misses
sol61-high	100% (60/60)	~$0.019	$0	27.4	0 misses
sol61-high ⚡	100% (60/60)	~$0.036	$0	17.4	✓ pick
sol56-high	100% (60/60)	~$0.042	$0	14.5	0 misses
luna56-high	93% (56/60)	~$0.0030	$0	11.4	beaten
luna56-low	80% (48/60)	~$0.0022	$0	9.2	avoid
terra61-high	97% (58/60)	~$0.0079	$0.0079	11.9	beaten
terra61-low	100% (60/60)	~$0.010	$0.010	10.3	0 misses
Retest set: 7 problems × 5 tries = 35 attempts
luna-low	60% (21/35)	~$0.0013	$0	8.5	missed 14
haiku-low	91% (32/35)	$0.0016	$0.0016	13.1	missed 3
haiku-medium	100% (35/35)	$0.0019	$0.0019	14.8	0 misses
haiku-high	100% (35/35)	$0.0035	$0.0035	17.4	0 misses
sonnet-medium	100% (35/35)	$0.015	$0.015	11.4	0 misses
sol61-medium	94% (33/35)	~$0.018	$0	25.6	missed 2
sol61-high	100% (35/35)	~$0.022	$0	33.4	0 misses
sol61-high ⚡	100% (35/35)	~$0.043	$0	18.9	0 misses
astra-medium	100% (35/35)	~$0.086	$0	26.0	0 misses
astra-medium ⚡	100% (35/35)	~$0.17	$0	10.9	0 misses
sol56-high	100% (35/35)	~$0.051	$0	19.5	0 misses
luna56-high	91% (32/35)	~$0.0036	$0	14.1	missed 3
luna56-low	83% (29/35)	~$0.0025	$0	12.3	missed 6
luna-low→H fail	97% (34/35)	~$0.0022	$0.0013	7.9	missed 1
luna-high-bare→H fail	91% (32/35)	~$0.0014	$0.00030	10.6	missed 3
plan→luna-low	86% (30/35)	~$0.0034	$0.0023	22.1	missed 5
plan→luna-high-bare	89% (31/35)	~$0.0036	$0.0022	27.4	missed 4
luna-high-bare→retry	91% (32/35)	~$0.0011	$0	10.6	missed 3
luna-low→H all	97% (34/35)	~$0.0039	$0.0031	23.1	missed 1
luna-high-bare→H all	100% (35/35)	~$0.0035	$0.0025	26.1	0 misses
Devin: 1 repeat, free tier, no token data
devin16slow-default	50% (6/12)	–	–	10.5	not a delegate (free tier)
Low context (trimmed prompt): luna-low-bare passed 44/60 on the main ladder. luna-high-bare passed 31/35 on the retest set.

Full table: intervals and slow-call time

config	pass rate 95% CI (function tests)	slow-call time s	total $ (all calls)
Main ladder
haiku-low	85%–100%	16.0	$0.064
haiku-medium	93%–100%	26.8	$0.088
sonnet-medium	≥78% (all passed)	18.1	$0.75
luna-low	53%–95%	11.6	~$0.041
luna-medium	57%–97%	11.8	~$0.042
sol61-medium	87%–100%	36.2	~$0.87
astra-medium †	≥78% (all passed)	35.6	~$4.52
astra-medium ⚡ †	≥78% (all passed)	13.6	~$9.12
devin16slow-default †	25%–75%	15.8	unmetered
haiku-high †	≥78% (all passed)	37.6	$0.17
luna-high †	82%–100%	22.9	~$0.058
luna56-high †	83%–100%	35.5	~$0.17
luna56-low †	60%–97%	17.4	~$0.10
sol56-high †	≥78% (all passed)	48.4	~$2.51
sol61-high †	≥78% (all passed)	50.2	~$1.12
sol61-high ⚡ †	≥78% (all passed)	27.9	~$2.15
terra56-high †	≥78% (all passed)	34.1	~$0.22
terra56-low †	50%–100%	22.3	~$0.16
terra61-high †	88%–100%	19.5	~$0.46
terra61-low †	≥78% (all passed)	20.9	~$0.62
Low context: trimmed prompt
luna-low-bare †	52%–92%	11.1	~$0.035

⚡ Codex fast mode: seconds and $ per pass

Normal and fast mode

ladder · sol61-high27.4s · $0.019

ladder · sol61-high ⚡17.4s · $0.036

ladder · astra-medium20.9s · $0.075

ladder · astra-medium ⚡9.9s · $0.15

L7 · sol61-high52.7s · $0.024

L7 · sol61-high ⚡29.4s · $0.044

L7 · astra-medium30.6s · $0.083

L7 · astra-medium ⚡15.4s · $0.16

How to read this data

Ladder problems: p01–p12. Rungs: L1–L6.
Low-context rows use trimmed prompts. Other variants: Notes → Everything we tried.

Table columns

Claude $ column: all = every dollar of that row went to Claude.
Pipeline rows show only the Haiku share.
Click a header to sort within a group. Symbols: see Notation.

Retest set

Retest set = the 7 problems a cheap luna config missed. Its single-model rows reuse earlier attempts.
Not listed: →H fail rows for sol61-high and astra-medium (normal and ⚡). Haiku never ran, so their results match the plain rows.

Pass rate and time

Pass rate is pass@1: the share of single attempts that pass every hidden test.
Every pass rate names its rule. Function track: function tests (reference + brute force). Repo tasks use one headline rule, task complete: complete = hidden and the run's own visible suite both green; hidden = our hidden suite only. Counts read like "7/12 complete (12/12 hidden)".
The 95% range is the pass-rate uncertainty shown in the table.
Slow-call time: the time within which 90% of calls finished.
The lower end of the 95% range is shown as ≥N% when all passed.

Most configs ran 60 calls (terra56-low 12 calls; terra56-high 12 calls; devin16slow-default 12 calls; sol61-xhigh 35 calls; the Grok configs are counted in the Grok section). † = exploratory, added after seeing the main-run results.

Fast mode

Pairs compare sol61-high and astra-medium at normal speed and ⚡.
The model and effort stay the same. The main ladder and L7 each passed every try.
Fast-mode $ uses 2× list price. This multiplier comes from the pricing page.

Codex $

Codex $ is an estimate: tokens × list price, not a bill. ~ marks estimates.
On a subscription it uses quota instead. See Notation.

Grok and Devin

Grok got through all six rungs, L1–L6. Every config missed somewhere. grok47-high passed 57/60 and missed at L4 and L6. grok47-low passed 53/60 and missed at L3 and L6. grok47fast-low passed 51/60 and missed at L3, L4 and L6. L1, L2 and L5 were clean for all three.

For grok47-high, the closest lane by pass rate in each family is Claude haiku-low (57/60, 10.0 s, $0.0011 per pass) and Codex luna56-high (56/60, 11.4 s, $0.0030 per pass). grok47-high itself: 57/60, 105.7 s, $0.13 per pass. Against the closest lane of each full-run Grok config (grok47-high, grok47-low and grok47fast-low), Grok takes 3.4–10.6× as long and costs 27.4–160.6× as much per pass.

Where does Grok get to?

config	L1	L2	L3	L4	L5	L6	passes (function tests)	median s	$ per pass	coverage	total $
grok47-high	10/10	10/10	10/10	9/10	10/10	8/10	57/60	105.7	$0.13	12 problems	$7.52
grok47-low	10/10	10/10	7/10	10/10	10/10	6/10	53/60	39.4	$0.082	12 problems	$4.33
grok47fast-low	10/10	10/10	6/10	9/10	10/10	6/10	51/60	34.0	$0.18	12 problems	$9.19
grok47-medium	10/10	7/7	3/4	–	–	–	20/21	32.6	$0.066	6 problems	$1.33

Each full-run config ran 12 problems × 5 repeats (main + exploratory phases, repeats 1–5). 87 of the 201 Grok calls come from the first run. grok47-medium ran p01–p06 only, as before; it was not extended, so it is not in the closest-lane comparison. 1 call hit the time limit and counts as a miss: grok47-high on p12 repeat 5. A killed call has no bill, so that config's $ per pass is a floor. luna-high and sol61-high-fast did not run every problem in the main and exploratory phases, so they are left out of the comparison. CLI-reported cost; list price not used for Grok. Codex dollars use tokens × list price with the fast-mode multiplier. Closest lane = nearest pass rate among the full-ladder lanes, then nearest median time, then cost per pass.

Closest lanes by pass rate

Grok config	family	closest lane	its passes (function tests)	its median s	its $ per pass	Grok time / lane	Grok cost / lane
grok47-high	Claude	haiku-low	57/60	10.0	$0.0011	10.6×	117.7×
grok47-high	Codex	luna56-high	56/60	11.4	$0.0030	9.3×	44.3×
grok47-low	Claude	haiku-low	57/60	10.0	$0.0011	4.0×	72.9×
grok47-low	Codex	luna56-high	56/60	11.4	$0.0030	3.5×	27.4×
grok47fast-low	Claude	haiku-low	57/60	10.0	$0.0011	3.4×	160.6×
grok47fast-low	Codex	luna56-low	48/60	9.2	$0.0022	3.7×	82.6×

Every lane on the same problems

config	family	passes (function tests)	median s	$ per pass
astra-medium ⚡	Codex	60/60	9.9	$0.15
sonnet-medium	Claude	60/60	10.1	$0.012
terra61-low	Codex	60/60	10.3	$0.010
sol56-high	Codex	60/60	14.5	$0.042
haiku-high	Claude	60/60	14.7	$0.0028
astra-medium	Codex	60/60	20.9	$0.075
sol61-high	Codex	60/60	27.4	$0.019
haiku-medium	Claude	59/60	13.0	$0.0015
terra61-high	Codex	58/60	11.9	$0.0079
sol61-medium	Codex	58/60	22.6	$0.015
haiku-low	Claude	57/60	10.0	$0.0011
grok47-high	Grok	57/60	105.7	$0.13
luna56-high	Codex	56/60	11.4	$0.0030
grok47-low	Grok	53/60	39.4	$0.082
grok47fast-low	Grok	51/60	34.0	$0.18
luna56-low	Codex	48/60	9.2	$0.0022
luna-medium	Codex	47/60	8.2	$0.00089
luna-low	Codex	46/60	7.2	$0.00090
luna-low-bare	Codex	44/60	6.8	$0.00080

Is Grok better at anything?

Not on these problems. Every Grok config is beaten on pass rate, speed and cost at once: grok47-high by 10 lanes (for example haiku-high, 60/60, 14.7 s, $0.0028 per pass). grok47-low by 11 lanes (for example haiku-high, 60/60, 14.7 s, $0.0028 per pass). grok47fast-low by 12 lanes (for example haiku-high, 60/60, 14.7 s, $0.0028 per pass). Grok passes more than luna-low, luna-low-bare, luna-medium, luna56-high and luna56-low. Every one of those lanes is faster and cheaper per pass. Grok costs less per pass than only astra-medium ⚡. Each of those passes more than the Grok config that undercuts it. Grok is faster than no lane.

First run (2 repeats, p01–p06)

In the first run (p01–p06 only), Grok was accurate but slow and mid-priced. grok47-high passed 18 of 18 but took 41 s per call at the median. Every existing lane at 100% finished in 6.9 to 16.9 s.

In this first run, grok47-low, grok47-medium and grok47fast-low each missed one call at L3 (3/4), so the pre-registered stop rule fired for all three; the full run above supersedes it.

In this first run, cost per pass was $0.054 to $0.111. Claude haiku and the Codex luna lanes are under $0.003. Only the Codex astra lanes cost as much.

Key numbers

grok47-low, grok47-medium and grok47fast-low cleared L1–L2. Each first missed at L3.
grok47-high cleared L1–L3. It was not run further.
Against the closest lanes, Grok is slower and costs more per pass. The tables below give the multiples.
Repo tasks: grok47-high 8/8 complete (8/8 hidden) at 442.1 s median. grok47-low 8/8 complete (8/8 hidden) at 98.9 s median.
The repo sample is small and ran on a shared host.
Per the routing rule, Claude Haiku stays the first fallback when Codex is out. These runs show Grok is usable after that on small repo tasks.

Ranges across every lane

grok47-low, grok47-medium and grok47fast-low (cleared through L2)

Against Claude haiku-medium, Grok takes 1.8–2.5× as long and costs 50.1–103.4× as much per pass. grok47-medium (p01–p06 only) is left out of this range.
Against Codex luna-high, Grok takes 2.2–2.9× as long and costs 58.7–121.1× as much per pass. grok47-medium (p01–p06 only) is left out of this range.
Across all Claude 100% lanes, Grok takes 1.6–4.0× as long and costs 5.4–169.3× as much per pass.
Across all Codex 100% lanes, Grok takes 1.1–4.7× as long and costs 0.4–9.2× as much per pass.

grok47-high (cleared through L3)

Against Claude haiku-high, Grok takes 3.5× as long and costs 33.1× as much per pass.
Against Codex sol61-high, Grok takes 2.4× as long and costs 4.7× as much per pass.
Across all Claude 100% lanes, Grok takes 3.5–5.0× as long and costs 7.2–109.3× as much per pass.
Across all Codex 100% lanes, Grok takes 2.4–6.0× as long and costs 0.6–5.9× as much per pass.

grok47-high on repo tasks

Across all Claude 100% lanes, Grok takes 7.6–15.9× as long and costs 10.0–105.5× as much per pass.
Across all Codex 100% lanes, Grok takes 3.7–6.7× as long and costs 1.2–6.7× as much per pass.

grok47-low on repo tasks

Across all Claude 100% lanes, Grok takes 1.7–3.6× as long and costs 3.5–37.0× as much per pass.
Across all Codex 100% lanes, Grok takes 0.8–1.5× as long and costs 0.4–2.4× as much per pass.

First run: where it got to

A lane is an existing config: one model at one effort. Rungs L1–L6 get harder step by step. Cleared through L2 means every call passed on L1 and L2.

config	cleared through	L1	L2	L3	passes (function tests)	median s	$ per pass	coverage	total $
grok47-low	L2	10/10	10/10	3/4	23/24	26.5	$0.054	6 problems	$1.24
grok47-medium	L2	10/10	7/7	3/4	20/21	32.6	$0.066	6 problems	$1.33
grok47-high	L3 (not run further)	10/10	4/4	4/4	18/18	41.0	$0.072	6 problems	$1.29
grok47fast-low	L2	10/10	10/10	3/4	23/24	19.4	$0.11	6 problems	$2.56

Totals differ (18, 21, 24) because the run was shortened from 5 to 2 repeats partway through L2 and extra repeats already run were kept; the earlier full-coverage lanes have 30 = 6 problems × 5 repeats; read the rung reached, not the total. The function misses were on p06_calc.

First run: closest lanes

Grok config	family	closest lane	Grok time / lane	Grok cost / lane
grok47-low	Claude	haiku-medium	2.5×	50.1×
grok47-low	Codex	luna-high	2.9×	58.7×
grok47-medium	Claude	haiku-medium	3.1×	61.8×
grok47-medium	Codex	luna-high	3.6×	72.3×
grok47-high	Claude	haiku-high	3.5×	33.1×
grok47-high	Codex	sol61-high	2.4×	4.7×
grok47fast-low	Claude	haiku-medium	1.8×	103.4×
grok47fast-low	Codex	luna-high	2.2×	121.1×

Each family shows the lane with the closest pass rate. Ties use the closest median time, then cost per pass.

All comparable lanes

A lane is comparable when it clears the same rungs and first misses at the same rung on the 12-problem main ladder. If a Grok config never missed through L3, every lane that clears through L3 counts, with its later results shown.

Same-problem comparisons use p01–p06 (6 of 12 problems), main + exploratory phases only. Warmup, calibration, fastmode, escalation and hard phases are excluded. In this first run Grok was run only through L3, so cleared L3 is a floor here; the full-ladder runs above go to L6 (only grok47-medium stays on p01–p06). CLI-reported cost; list price not used for Grok. Codex dollars use tokens × list price with the fast-mode multiplier.

Grok config	family	comparable lane	cleared through	first miss	same-problem passes (function tests)	median s	$ per pass	Grok time / lane	Grok cost / lane
grok47-low	Claude	haiku-medium	L2	L3	29/30	10.6	$0.0011	2.5×	50.1×
grok47-low	Codex	luna-high	L2	L3	24/25	9.0	$0.00092	2.9×	58.7×
grok47-medium	Claude	haiku-medium	L2	L3	29/30	10.6	$0.0011	3.1×	61.8×
grok47-medium	Codex	luna-high	L2	L3	24/25	9.0	$0.00092	3.6×	72.3×
grok47-high	Codex	astra-medium	L6	none	30/30	12.7	$0.064	3.2×	1.1×
grok47-high	Codex	astra-medium ⚡	L6	none	30/30	6.9	$0.13	6.0×	0.6×
grok47-high	Claude	haiku-high	L6	none	30/30	11.8	$0.0022	3.5×	33.1×
grok47-high	Claude	haiku-low	L4	L5	30/30	8.2	$0.00066	5.0×	109.3×
grok47-high	Codex	sol56-high	L6	none	30/30	10.0	$0.032	4.1×	2.3×
grok47-high	Codex	sol61-high	L6	none	30/30	16.9	$0.015	2.4×	4.7×
grok47-high	Codex	sol61-high ⚡	L6	none	20/20	9.8	$0.026	4.2×	2.8×
grok47-high	Codex	sol61-medium	L5	L6	30/30	15.4	$0.012	2.7×	5.9×
grok47-high	Claude	sonnet-medium	L6	none	30/30	9.5	$0.010	4.3×	7.2×
grok47fast-low	Claude	haiku-medium	L2	L3	29/30	10.6	$0.0011	1.8×	103.4×
grok47fast-low	Codex	luna-high	L2	L3	24/25	9.0	$0.00092	2.2×	121.1×

First run: was Grok better at anything?

grok47-high is cheaper per pass than astra-medium ⚡ (fast mode, 2× list price) on functions (0.55×).
grok47-low is cheaper per pass than plain astra-medium on repo tasks (0.42×) while 1.5× slower.
grok47-low takes 0.84× as long as sol61-high on repo tasks.
No Grok config beats a comparable function lane on pass rate.
No Grok config beats a comparable function lane on time.
Codex dollars are list-price proxies.

Repository tasks

Grok is a usable fallback for these small repo tasks when Codex is out. Claude is faster and cheaper. grok47-low 8/8 complete (8/8 hidden) at 98.9 s median. Today’s Claude fallback lanes (haiku-high 12/12 complete (12/12 hidden), haiku-low 12/12 complete (12/12 hidden), sonnet-medium 12/12 complete (12/12 hidden)) took 27.7–58.4 s median.

Counts use the repo-task marker defined under Method (Repo tasks): complete first, our hidden suite alone in brackets. Red-suite runs (task_complete false) that the audit session has not labelled show 'under audit' beside their cells. Counts are raw: no run is dropped or reclassified until the audit labels it in results/annotations.json.

Grok config	family	closest lane	Grok time / lane	Grok $ per pass / lane
grok47-high	Claude	haiku-high	7.57×	44.34×
grok47-high	Codex	sol61-high	3.75×	5.50×
grok47-low	Claude	haiku-high	1.69×	15.55×
grok47-low	Codex	sol61-medium	1.17×	2.36×

Each family shows the lane with the closest pass rate. Ties use the closest median time, then cost per pass.

All comparable lanes

Grok config	existing lane	Grok time / lane	Grok $ per pass / lane
grok47-high	astra-medium	6.74×	1.20×
grok47-high	haiku-high	7.57×	44.34×
grok47-high	haiku-low	15.63×	105.53×
grok47-high	luna-low	18.33×	95.85×
grok47-high	sol61-high	3.75×	5.50×
grok47-high	sol61-medium	5.23×	6.74×
grok47-high	sonnet-medium	15.94×	9.97×
grok47-low	astra-medium	1.51×	0.42×
grok47-low	haiku-high	1.69×	15.55×
grok47-low	haiku-low	3.50×	37.01×
grok47-low	luna-low	4.10×	33.62×
grok47-low	sol61-high	0.84×	1.93×
grok47-low	sol61-medium	1.17×	2.36×
grok47-low	sonnet-medium	3.57×	3.50×

Run conditions and measurements

4 tasks (t1_discount, t2_cache_bugs, t3_config_refs, t4_retry_queue); Grok: 2 repeats per task and config, 4 parallel lanes, tools on, no web or subagents, 1,500 s timeout; all 16 runs retained with no performance stop rule. Existing lanes: 3 repeats per task; clean-home Codex replacements are used where available. Hidden tests run after the agent exits.

config	task complete (hidden alone in brackets)	median wall s	$ per complete task	median turns
grok47-high	8/8 complete (8/8 hidden)	442.1	$0.54	9
grok47-low	8/8 complete (8/8 hidden)	98.9	$0.19	5
astra-medium	12/12 complete (12/12 hidden)	65.6	$0.45	1
haiku-high	12/12 complete (12/12 hidden)	58.4	$0.012	10
haiku-low	12/12 complete (12/12 hidden)	28.3	$0.0051	5
luna-low	7/12 complete (12/12 hidden) (under audit)	24.1	$0.0056	1
sol61-high	12/12 complete (12/12 hidden)	117.9	$0.097	1
sol61-medium	12/12 complete (12/12 hidden)	84.5	$0.079	1
sonnet-medium	12/12 complete (12/12 hidden)	27.7	$0.054	3

Grok total measured spend: $5.78 of the $15 budget. Grok and Claude costs are CLI-reported; Codex costs are token-price proxies. Multiples use median wall time and total cost per passing attempt, respectively. Turns use each CLI’s convention and are not directly comparable across CLIs. This is a small exploratory sample on a shared host, not a general repo-work guarantee. Sources: results/2026-10-22b-agentic.jsonl, results/2026-10-25-isolated-agentic.jsonl, results/2026-10-27-grok-agentic.jsonl.

Devin on the free tier

Devin ran each of the 12 main-ladder problems once on swe-1-6-slow. It passed 6/12 (pass rate 50%), with a median 10.5 s per call. Timeouts: p04 taller_within, p11 best_average. Wrong answers: p06 calc, p07 weighted_distance_sums, p09 k_machine_schedule. Crashes: p12 sheet. It reports no tokens and no cost, so it has no $ per pass and appears on no $ chart, in no Winner pick and on no best-trade-offs line.

rung	passed (function tests)	what happened
L1	2/2	all passed
L2	1/2	p04 taller_within timeout
L3	1/2	p06 calc wrong
L4	1/2	p07 weighted_distance_sums wrong
L5	1/2	p09 k_machine_schedule wrong
L6	0/2	p11 best_average timeout, p12 sheet crash

Only swe-1-6-slow ran. The account is on the free tier, and every other Devin-native model (adaptive, swe-2-medium, swe-2-high, swe-2-max and the -lightning variants) returned exactly: Error: Upgrade to Pro to access this model (https://devin.ai/pricing)
The rows bypass devin-delegate, which has no model passthrough, wraps the prompt in an envelope, falls back to Codex on failure, injects auto-context and caches results. The bench called the devin CLI directly in an empty directory, in auto permission mode, one repeat.
Stage 2 (repeats 2 to 5) needed at least 9 of 12 on stage 1. It got 6, so it did not run.

Verdict: not a delegate on the free tier. The Results table and the failure grid list it as Devin (free tier, 1 repeat).

Everything we tried

Every config with rows in results/, grouped by kind.
Each row pools all of a config's runs. Pass, $ and seconds can differ from Results.
Codex dollars are list-price estimates. Dash means the full cost is unavailable.
Hover over a name for its source file.
Pilot repo pipelines: dollars include builder and review; time is builder-only.

exploratory: 4 configs, $22.37 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
grok47-low	Grok grok-4.7, low effort; grok CLI, measured total_cost_usd; exploratory, added 2026-10-26 at operator request; function track p01–p12 (L1–L6), 2026-10-28 full spectrum: 5 repeats per problem (L2–L3 had been shortened to 2 repeats on 2026-10-26)	53/60	$0.082	39.4	$4.33	charts and tables
grok47-medium	Grok grok-4.7, medium effort; grok CLI, measured total_cost_usd; exploratory, added 2026-10-26 at operator request; function track p01–p06 (L1–L3) only, not extended (PLAN.md deviation 23); 6 of 12 problems	20/21	$0.066	32.6	$1.33	charts and tables
grok47-high	Grok grok-4.7, high effort; grok CLI, measured total_cost_usd; exploratory, added 2026-10-26 at operator request; function track p01–p12 (L1–L6), 2026-10-28 full spectrum: 5 repeats per problem (L2–L3 had been shortened to 2 repeats on 2026-10-26)	57/60	$0.13	105.7	$7.52	charts and tables
grok47fast-low	Grok grok-4.7-build-fast, low effort; grok CLI, measured total_cost_usd; exploratory, added 2026-10-26 at operator request; function track p01–p12 (L1–L6), 2026-10-28 full spectrum: 5 repeats per problem (L2–L3 had been shortened to 2 repeats on 2026-10-26)	51/60	$0.18	34.0	$9.19	charts and tables

How it was run

Every config (model + effort) got the same new problems and hidden tests. Tools were off: no code runs, file reads or search. Logs confirm it for 938 of 938 Codex calls. Terra (added later) is the exception: 4 of 144 calls logged one item that was not a message or reasoning, so a tool may have run. 4 of those 4 passed. The raw item type was not kept.

Problems by rung (Easy/Medium/Hard)

We wrote these problems for this bench.
Each LeetCode link is the closest analogue I found by hand.
Pilot links say "source": those problems come from LeetCode.
Rungs L1–L6 follow this bench's own difficulty order.
Rung and LeetCode difficulty can disagree:
- p01 overlap_minutes (L1) is Medium.
- p06 calc (L3) is Hard.
- p08 recur (L4) is Medium.
Pilot: well-known LeetCode problems. Every config passed. It ranked nothing, so we exclude its scores.
- E Fizz Buzz source
- M Course Schedule II source
- H Candy source
- H Reconstruct Itinerary source
- H Minimum Window Substring source
- H Wildcard Matching source
Main ladder: 12 new problems.
- Problem numbers: p01–p12.
- Rungs: L1–L6, easiest first.
- Each rung has 2 problems.
- Most configs ran it; the exceptions: terra56-low 12 calls; terra56-high 12 calls; devin16slow-default 12 calls; sol61-xhigh 35 calls; the Grok configs are counted in the Grok section.
- p01 overlap_minutes (L1) M
  like My Calendar IIIminutes with at least k meetings at once; half-open intervals, sweep line
- p02 squash (L1) E
  like String Compressionrun-length encoding with escaped digits and backslashes
- p03 ledger (L2) M
  like Simple Bank Systembank operations with validation rules, frozen accounts and transfers
- p04 taller_within (L2) M
  like Count of Smaller Numbers After Selfcount taller people among the k in front; sliding window
- p05 pass_route (L3) M
  like Cheapest Flights Within K Stopscheapest route when zone passes can be bought on the way; state-space shortest path
- p06 calc (L3) H
  like Basic Calculatorinteger expression parser: precedence, right-assoc power, unary minus, errors
- p07 weighted_distance_sums (L4) H
  like Sum of Distances in Treeweighted distance sums from every tree node; rerooting
- p08 recur (L4) M
  no close LeetCode analoguecalendar recurrence rules (daily, weekly, monthly) expanded to dates
- p09 k_machine_schedule (L5) H
  like Maximum Profit in Job Schedulingbest total pay for jobs on k printers; weighted interval scheduling
- p10 avoid_count (L5) H
  like Find All Good Stringscount strings avoiding banned substrings; automaton DP, modulo
- p11 best_average (L6) H
  like Maximum Average Subarray IIlargest average of a subarray of length at least k, as an exact fraction
- p12 sheet (L6) H
  like Design Excel Sum Formulaspreadsheet evaluator with references, ranges, division and cycle errors
L7 set: extra-hard problems.
- Problem numbers: p13–p15.
- Only luna-low and luna-medium broke by L6. We added L7 to find the others' limit. It has 3 problems instead of the usual 2. We score it separately.
- p13 xor_ring (L7) H
  no close LeetCode analogueXOR cellular automaton on a ring for t ≤ 10^18; doubling
- p14 bounce_board (L7) H
  like Snakes and Laddersexpected turns on a bouncing board game with jumps, as an exact fraction
- p15 paint_erase (L7) H
  like Range Modulepaint and erase ranges where higher colours win; segment tree

Each config tries each problem 5 times, tools off.
Total spend, by kind: Claude $4.18 (CLI-reported), Codex $48.84 (list-price proxy), Grok $28.16 (CLI-reported). Includes pilot and appendix arms: Claude $1.72, Codex $11.12 ($5.64 pilot, $5.48 appendix and trim).

How to read this data: sets, tests, prices, intervals

Planted wrong answers must fail the hidden tests. Sample tests = the worked examples the model sees.
Cost totals leave out the pilot, runner (harness) bias, trim and retest set pipeline runs.
Sets:
- main ladder has 12 problems. Each gets 5 tries. Total: 60 calls per config that ran it in full.
- L7 set has 3 problems. Each gets 5 tries. Total: 15 calls.
- Retest set covers the 7 main-ladder problems. They are the ones that luna-low, luna-medium or luna-low-bare missed. They were chosen after the main run. Each config made 35 calls.
- Repo tasks: 4 edits. Each gets 3 tries. Total: 12 calls.
Hidden tests: fixed cases, random cases checked against a reference, and large cases for speed.
Claude $: what the CLI reports.
The 95% range uses 10,000 bootstrap resamples. It shows how much a rerun's pass rate can vary.
Run by Wren, a Claude Opus 5.5 orchestrator.

Models and versions

Main bench, Claude Code 2.1.308: claude-haiku-5-5, claude-sonnet-5-5.
Main bench, Codex CLI 0.163.0: gpt-5.6-luna, gpt-5.6-sol, gpt-6-astra, gpt-6-luna, gpt-6.1-sol. Terra rows, Codex CLI 0.164.0: gpt-5.6-terra, gpt-6.1-terra.
Whose bench? also used Codex CLI 0.164.0 and Claude Code 2.1.293.

Notation

⚡: Fast mode: 2× list price at the same model and effort.
-bare: Codex's system prompt and tool text are trimmed.
H: Haiku at low effort (haiku-low).
→H fail: Haiku fixes answers only when a sample test fails.
→H all: Haiku checks and fixes every answer.
→retry: The same model tries again if a sample test fails.
plan→: Haiku writes a plan before the named model answers.
†: An exploratory config added after the main run.
✓ pick: A Winner pick.
~: Codex cost estimated from tokens and list prices.
–: No Claude step, unavailable cost, or an unrun failure-grid cell.
$0: The Haiku step never ran, or subscription quota covers the Codex call.
≥N%: The lower 95% bound when every attempt passed.
slow-call time: The time within which 90% of calls finished.

Where they fail

p12 sheet (L6) has misses in 9 of 18 full-ladder Claude and Codex configs.

How to read this data

Cells = passed/attempted. Green: all passed. Amber: more than half. Red: half or fewer.
Dash: not run. Scroll sideways for more configs.

E/M/H = my LeetCode difficulty guess: Easy, Medium, Hard.

problem

haiku-low

haiku-medium

haiku-high

sonnet-medium

sol61-medium

luna-high

luna-medium

luna-low

luna-low-bare

astra-medium

astra-medium ⚡

sol61-high

sol61-high ⚡

sol56-high

luna56-high

luna56-low

terra61-high

terra61-low

grok47-low

grok47-medium

grok47-high

grok47fast-low

devin16slow-default

p01 overlap_minutes (L1)M

5/5

1/1

p02 squash (L1)E

5/5

1/1

p03 ledger (L2)M

5/5

4/5

5/5

3/5

5/5

4/4

5/5

1/1

p04 taller_within (L2)M

5/5

1/5

5/5

4/5

3/5

5/5

3/3

5/5

0/1

p05 pass_route (L3)M

5/5

4/5

5/5

4/5

5/5

2/2

5/5

1/1

p06 calc (L3)H

5/5

4/5

1/5

5/5

2/5

1/2

5/5

1/5

0/1

p07 weighted_distance_sums (L4)H

5/5

3/5

5/5

0/0

4/5

0/1

p08 recur (L4)M

5/5

4/5

5/5

0/0

5/5

1/1

p09 k_machine_schedule (L5)H

3/5

5/5

4/5

5/5

2/5

5/5

0/0

5/5

0/1

p10 avoid_count (L5)H

5/5

4/5

5/5

4/5

5/5

4/5

5/5

0/0

5/5

1/1

p11 best_average (L6)H

5/5

2/5

4/5

2/5

5/5

0/0

5/5

0/1

p12 sheet (L6)H

4/5

5/5

3/5

0/5

5/5

3/5

1/5

5/5

1/5

0/0

3/5

1/5

0/1

L7 set

p13 xor_ring (L7)H

5/5

–

5/5

4/5

3/5

–

p14 bounce_board (L7)H

5/5

4/5

5/5

–

5/5

3/5

2/5

–

p15 paint_erase (L7)H

2/5

4/5

5/5

–

5/5

3/5

–

grok47-medium covers p01–p06 only, so its cells for the other problems are dashes. A dash means no attempt, not a failure.

p04 taller_within (L2): luna-low, luna-medium and luna-low-bare each passed only 1 of 5 tries.
haiku-low never gave a wrong answer.
Its 3 main-ladder misses: 2 too slow. 1 crash.
Its 3 L7 misses: all too slow.

Baseline input tokens per call

Trimming luna-low's Codex system prompt and tool features cuts mean input by 58%.

Mean input tokens per call

haiku-low4,498

sonnet-medium3,719

luna-low, Codex's normal prompt14,688

sol61-medium15,380

astra-medium ⚡15,316

sol61-high ⚡15,380

Low context

luna-low, trimmed prompt6,144

Solid = cached input. Faded = uncached input.

How to read this data

Baseline input tokens = text Codex's runner adds before the task. This includes the system prompt and tool descriptions.
Bars show mean input per call, including the task.
See Notation.

Pipelines: plan, check, fix

A pipeline chains steps. Usually a single model writes, then Haiku checks. The best, luna-high-bare →H all, passed 35/35. It costs $0.0035 per pass. haiku-medium alone also passed every try. That pipeline cost 1.8× as much.

Pass rate, $ per pass, median seconds

Bar length = passes out of 35. Text by each bar = passes · $ per pass · median seconds.

Retest set

HaikuSonnetCodexolder Codexpipeline⚡ fast mode

Single models

luna-low21/35 · $0.0013 · 9s

haiku-low32/35 · $0.0016 · 13s

haiku-medium35/35 · $0.0019 · 15s

haiku-high35/35 · $0.0035 · 17s

sonnet-medium35/35 · $0.015 · 11s

sol61-medium33/35 · $0.018 · 26s

sol61-high35/35 · $0.022 · 33s

sol61-high ⚡35/35 · $0.043 · 19s

astra-medium35/35 · $0.086 · 26s

astra-medium ⚡35/35 · $0.17 · 11s

sol56-high35/35 · $0.051 · 19s

luna56-high32/35 · $0.0036 · 14s

luna56-low29/35 · $0.0025 · 12s

Low context

luna-high-bare31/35 · $0.0010 · 13s

Build, then Haiku fixes it only if a sample test fails (→H fail)

sol61-high→H fail35/35 · $0.022 · 33s

sol61-high ⚡→H fail35/35 · $0.043 · 19s

astra-medium→H fail35/35 · $0.087 · 21s

astra-medium ⚡→H fail35/35 · $0.17 · 11s

luna-low→H fail34/35 · $0.0022 · 8s

luna-high-bare→H fail32/35 · $0.0014 · 11s

Haiku checks every answer (→H all)

luna-low→H all34/35 · $0.0039 · 23s

luna-high-bare→H all35/35 · $0.0035 · 26s

Plan first, then build (plan→)

plan→luna-low30/35 · $0.0034 · 22s

plan→luna-high-bare31/35 · $0.0036 · 27s

Retry once if a sample test fails (→retry)

luna-high-bare→retry32/35 · $0.0011 · 11s

How to read this data

Gates

→H fail: Haiku fixes an answer only if a sample test fails. →H all: Haiku checks every answer.
sol61-high and astra-medium never failed a sample test, so Haiku never ran.
luna-low failed sample tests more often: 13 repairs, so the gate helped it more.

Scope

Pipelines ran only on the retest set.
Fast-mode pipelines equal their builders because every example check passed. Haiku made 0 repairs.
⚡ pipeline rows reuse cached ⚡ answers. Other pipeline rows are new calls.

Router: per main-ladder problem, the cheapest config that passed tries 1–3. Router evaluation tries: 4–5. The router passed 22/24. That cost ~40% of haiku-high's $ per pass.

See Notation.

Whose bench?

Does a Codex-built bench favour Codex models? Slightly: the gap moved 13 points. haiku-low still led on both, so no pick changes.

6 hard problems, 10 tries each. Scroll sideways on narrow screens.

model	bench	passed (function tests)	$ per pass	time per pass (s)
luna-low	Claude's bench	36/60	~$0.0013	12.7
luna-low	Codex's bench (Codex CLI)	47/60	~$0.0040	34.8
haiku-low	Claude's bench	57/60	$0.0025	18.8
haiku-low	Codex's bench (Claude Code)	60/60	$0.0027	17.6

How to read this data

Our one-shot runner (Claude's bench): a single answer per try, tools off.

Codex CLI loop: runs code, retries (Codex's bench).

The benches

Claude designed and ran this bench; Codex built only its own harness.
Codex's bench: each model runs in its vendor's agent CLI, which runs code and retries.
luna-low ran in Codex CLI, haiku-low in Claude Code: each vendor's cheapest model.
sol61-high was skipped: every try already passed.

What changed

luna-low went 36→47. haiku-low went 57→60.
Running code cut luna-low's wrong outputs from 14 to 6. Crashes fell from 10 to 7.
luna-low cost 3.1× as much per pass. Time per pass was 2.7× as long. haiku-low cost 1.1× as much per pass. Time per pass was 0.9× as long. Overall, Codex's bench cost 1.6× as much per pass.

The gap moved 13 points, below our 15-point bias threshold. Confidence level: 95%. The range is 3 to 23 points.

Luna-low variants B1 and B2 are listed under Everything we tried. luna-low-noshell-B1 removes shell context and asks for edge cases first. luna-low-noshell-B2 also asks for a detailed reasoning summary. On the retest set, B1 passed 25/35. Plain luna-low passed 21; B2 passed 19.

Time per pass = summed call seconds divided by passes.

How Codex built its bench

gpt-6.1-sol high designed and wrote it.
Its dry runs missed two sandbox bugs. A live call found them; Codex fixed both.
A final "plan and reorient" nudge changed no code.
It first replayed 12 of our haiku-low and sol61-medium results. All passed on both.
We lifted its repeat cap, a two-line change, then ran 240 calls.
Repeat-cap change source: c6528128a7a66fdc0ecd4031a9228ce8b53f5a98 (bench/codex_harness.py).
It finished with 0 runner errors and no timeouts, in about 74 minutes.

CONCLUSION

Four configs passed every try in every set. They were haiku-high, sonnet-medium, sol61-high and astra-medium. Repo tasks counted as task complete: hidden and visible tests both pass. haiku-high is the cheapest, at $0.0028 per pass.

With Codex quota left, send each task to sol61-high ⚡.
Then run the sample tests. If one fails, haiku-low fixes it. Tested with plain sol61-high: 35/35 on the retest set, one run; sample tests are a weak gate. Haiku never ran.
With Codex quota gone, use haiku-high. It is the cheapest config with no misses.

If speed matters most, use sonnet-medium. It takes 10.1 s to haiku-high's 14.7 s. sonnet-medium costs 4.4× as much per pass.
Don't use luna-low or luna-medium alone: they miss about 1 answer in 4.
luna-low passed 11 more on Codex's bench; haiku-low gained 3, near its ceiling. Bench and harness both differ, so the cause is not isolated. No pick changes.

Limitations

A routing signal from a small, self-written set, not a general ranking. Read the intervals first.

What could mislead

Scope: 12 main problems, written for this bench.
Most have a LeetCode analogue, and LeetCode code is widely public.
The problems ship with this repo, so later models can read them.
The 95% range resamples whole problems, since tries on the same problem are related.
L7 has 3 problems. Perfect score: 15/15. That only shows a pass rate of at least 37%.
Repo tasks: luna-low: 7/12 complete (12/12 hidden) in the clean-home rerun and every other config completes every run. With 4 tasks × 3 repeats = 12 runs per config they cannot rule out skill differences.
Paired gaps of 3 points or less showed no separation; 18-point gaps did.
PLAN.md was committed before the main run; a same-day pilot came first.
15 configs were added after seeing the main-run results (†). These include the haiku-high and sol61-high picks:
- astra-medium
- astra-medium ⚡
- devin16slow-default
- haiku-high
- luna-high
- luna-low-bare
- luna56-high
- luna56-low
- sol56-high
- sol61-high
- sol61-high ⚡
- terra56-high
- terra56-low
- terra61-high
- terra61-low
A Codex call's estimated $ drops up to 5.8× when the prompt cache hits.
Latency is CLI wall clock.
We ran up to 4 calls at once on one host.
One run in time; hosted models change.
Codex needed our help to build its bench: see Whose bench?

Reproduce

Cite as: DispatchBench (2026), github.com/chimera-defi/dispatchbench.
Code, problems and raw results: dispatchbench on GitHub, MIT licence.
Numbers come from results/dataset.json, which is built from the results/*.jsonl run files.

More charts

luna-low and luna-medium break at L6. luna-low-bare breaks at L2; see the grid.

Repo tasks, task complete (hidden in brackets): luna-low 7/12 complete (12/12 hidden) in the clean-home rerun (11/12 complete (12/12 hidden) in the original run); every other config completed every run. Every task prompt told agents to add tests and that all tests must pass. All 6 failing tests in the 5 clean-home red runs are tests luna-low itself added, none a shipped test, and 0 runs edited the shipped tests. The audit reads 5 of the 6 red runs (clean-home and original) as wrong tests (the cache tests insert both entries at the same time, so both expire; the config tests expect behaviour the task does not ask for) and 1 as wrong code that the hidden suite missed (t1_discount accepts a boolean percentage; see the stricter-suite regrade below). The classification is still under audit until results/annotations.json carries the labels. Costs ranged from $0.0033 to $0.45 per run; per complete task luna-low costs $0.0056 against haiku-low's $0.0051.

Stricter hidden suite (t1_discount now rejects a boolean percentage): 16 stored t1 patches regraded with no model calls (12 in the table; the other 4 are in results/2026-10-28-t1-hidden-v2-regrade.jsonl and none flips); 1 run flips from hidden pass to fail: luna-low repeat 2 (clean-home rerun), which leaves luna-low at 11/12 hidden in the clean-home rerun. The 21 t1 runs of the 10-22 main run stored no patch and cannot be regraded. Headline counts stay on the suite each run was graded with. The task text says "pct: int", which does not say whether a boolean counts; the reference rejects it, so the gap was in our hidden suite.

config	run set	t1 runs regraded	hidden as graded	hidden under stricter suite	task complete under stricter suite
astra-medium	clean-home rerun	3	3/3	3/3	3/3
luna-low	clean-home rerun	3	3/3	2/3	2/3
sol61-high	clean-home rerun	3	3/3	3/3	3/3
sol61-medium	clean-home rerun	3	3/3	3/3	3/3

Correction, 2026-10-28. A reader pointed out that "passed every hidden test" hid luna-low's visible-test misses. The headline is now task complete, with the hidden-suite count beside it.

Explorer and more charts

Explorer plots any pair of measures.

The interactive explorer is in Cost vs accuracy.

Pass rate by difficulty rung. Scroll sideways on narrow screens.

Repo tasks: task complete vs $ per run (filled: task complete; ring: hidden suite only). No ⚡ repo-task data. Scroll sideways on narrow screens.

See Notation.

Notes: code quality, fast mode, appendix

Sonnet's code read worst in a blind review.

Code quality

config	passing solutions	median LOC	median max complexity	lint warnings per solution	time vs ref	worse growth	mem vs ref
haiku-low	57	38	11	0.11	1.13	3	1.00
haiku-medium	59	46	10	0.15	1.10	2	1.00
sonnet-medium	60	46	14	0.25	1.05	4	0.99
luna-low	46	37	9	0.13	1.12	4	1.00
luna-medium	47	47	14	0.09	1.10	6	1.00
sol61-medium	58	38	14	0.00	1.08	0	1.00
astra-medium †	60	46	16	0.00	0.70	0	1.00
haiku-high †	60	43	14	0.02	0.95	3	1.00
luna-high †	55	39	9	0.00	1.13	2	1.00
luna-low-bare †	44	37	11	0.11	1.13	6	1.00
luna56-high †	56	44	14	0.00	0.85	0	1.00
luna56-low †	48	38	14	0.02	0.80	1	1.00
sol56-high †	60	46	14	0.03	0.83	1	1.00
sol61-high †	60	51	16	0.03	0.72	0	1.00

sol61-high ⚡ and astra-medium ⚡: main-ladder solutions not scored.
sol61-high ⚡'s 35-attempt retest run is scored below; astra-medium ⚡ has no retest scores.

Codex fast mode and xhigh

On the retest set, fast mode kept every pass and was 37% faster. Median times: 18.9 s vs 30.1 s. Price: 2× list, not a measured bill. xhigh effort (above high): same passes, 1.5× the cost, slower, no better code.

config	passed (function tests)	median s	median output tokens	$ per pass (proxy)	median LOC	median max complexity	lint per solution	time vs ref
sol61-high, default tier (separate control run, repeats 6-10)	35/35	30.1	1,092	$0.0212	63	22	0.00	0.73
sol61-high ⚡	35/35	18.9	1,063	$0.0435	62	22	0.09	0.71
sol61-xhigh, default tier	35/35	44.3	1,662	$0.0326	64	23	0.00	0.72

The blind read covered 20 solutions. A code-only reviewer would refuse to merge 3, all sonnet-medium. It scored 4.0/10. Other configs scored 6.75–7.5 with none blocked. One reviewer and one solution per cell: a sign of bad code, not a ranking.

Reader: claude-fable-5-1, one blind pass, 2026-10-23. Method: One passing solution per config per problem, shuffled and labelled A–E (seed 7). Score: 1–10 readability and maintainability. 4 solutions per config. Would block = would not merge as is.

config	solutions read	mean score (1 to 10)	would block
astra-medium	4	7.50	0
sol61-high	4	7.25	0
haiku-medium	4	7.00	0
sol61-xhigh	4	6.75	0
sonnet-medium	4	4.00	3

How to read this data

Code quality columns

Complexity = cyclomatic complexity (branch count) of the most complex function, via radon.
Lint = ruff style warnings per solution.
Worse growth = time-growth exponent more than 0.3 above the reference's.
Time vs ref = run time divided by the reference solution's.
Mem vs ref = memory divided by the reference solution's.
Below 1.00 is faster or leaner.

Code quality scope

Scored: main-ladder passing solutions, 770 rows.
All 1166 solutions are in results/solutions.
On p08 recur (L4), 11 solutions run slower than the reference. Slower means at least 10× longer. The smallest slowdown is 29×. They come from 7 configs. The worst is a sol56-high solution at 2453×.

Fast-mode control

The default-tier control is a separate 35-try run. Its 30.1 s differs from the Results row.

Appendix

Headline numbers: see How it was run.

Codex CLI in the main benchmark

Codex also answered through its own CLI in the main benchmark.
We ignored personal settings and disabled tools.
The luna-*-bare rows test how much Codex's system prompt and tool text matter.

Repo-task rows were rerun with a clean Codex home.

A clean rerun checked 995 other calls.

It passed 889 versus 890 originally, so those rows were kept.

Failures and later runs

haiku-low misses:

set	tries	too slow	crashes	wrong outputs
main ladder	60	2	1	0
L7 set	15	3	0	0

Too slow = exceeding the grader's time limit for that case.

L7: haiku-low has misses; sonnet-medium passed every try.
- Each problem had 5 tries.
- Problems:
  - p13 xor_ring (L7)
  - p14 bounce_board (L7)
  - p15 paint_erase (L7)
- haiku-low passed 12/15. It cost $0.0023 per pass.
- sonnet-medium passed 15/15. It cost $0.018 per pass.
Escalation: luna-high fixes most of luna-medium's misses.
- We reran weak problems at higher effort.
- Problems:
  - p04 taller_within (L2)
  - p11 best_average (L6)
  - p12 sheet (L6)
- luna-high passed 13/15.
- luna-medium had passed 3/15.
Retry vs Haiku: luna-high-bare retries matched Haiku fixes.
- Haiku fixed failed sample tests; the pipeline passed 32/35.
- It cost $0.0014 per pass.
- Same-model retries passed 32/35.
- They cost $0.0011 per pass.
- These rows use the retest set, with 35 attempts each.
Sample tests miss some bad answers.
- A fresh luna-high-bare run had 5 bad answers.
- Checks caught 2. Haiku fixed both.
- Checks missed 3.
- The run without checks had 4 bad answers.
Planning: Haiku plans cost more than Haiku checks.
- luna-low passed 21/35 without a plan.
- It passed 30/35 with a Haiku plan.
- luna-high-bare passed 31/35 either way.
- Both Haiku-plan pipelines cost more than Haiku checking after.
Pilot: every config passed, so the pilot ranked nothing.
- The pilot used 6 well-known LeetCode problems.
Fast-mode pilot: sol61-low got faster; luna-low did not.
- The pilot used 3 problems across 3 rounds.
- It used 2× list price.
- sol61-low fell from 7.4 s to 5.2 s.
- luna-low stayed at 4.5 s.
- This pilot supplies the only fast-mode data at low effort.
Original repair run: we lost its prompt but reran committed code.
- The original luna-low → Haiku-fixes-all run passed 35/35.
- The rerun from committed code passed 34/35.
- Both use the retest set, with 35 attempts per row.

Everything we tried

Every config with rows in results/, grouped by kind.
Each row pools all of a config's runs. Pass, $ and seconds can differ from Results.
Codex dollars are list-price estimates. Dash means the full cost is unavailable.
Hover over a name for its source file.
Pilot repo pipelines: dollars include builder and review; time is builder-only.

main ladder: 6 configs, ~$2.72 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
haiku-low	Claude Haiku, low effort, tools off	79/85	$0.0013	11.3	$0.10	charts and tables
haiku-medium	Claude Haiku, medium effort	72/75	$0.0018	14.7	$0.13	charts and tables
sonnet-medium	Claude Sonnet, medium effort	85/85	$0.014	11.4	$1.15	charts and tables
luna-low	Codex gpt-6-luna, low effort, stock Codex harness	46/60	~$0.00090	7.2	~$0.041	charts and tables
luna-medium	Codex gpt-6-luna, medium effort	47/60	~$0.00089	8.2	~$0.042	charts and tables
sol61-medium	Codex gpt-6.1-sol, medium effort	83/85	~$0.015	22.6	~$1.26	charts and tables

Grok: see the Grok section

exploratory: 12 configs, ~$13.51 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
haiku-high	Claude Haiku, high effort (added after the main run)	75/75	$0.0029	17.1	$0.22	charts and tables
luna-high	gpt-6-luna, high effort (added after the main run)	70/75	~$0.0012	13.3	~$0.081	charts and tables
luna56-low	gpt-5.6-luna, low effort	56/75	~$0.0025	10.1	~$0.14	charts and tables
luna56-high	gpt-5.6-luna, high effort	65/75	~$0.0036	14.1	~$0.24	charts and tables
sol56-high	gpt-5.6-sol, high effort	73/75	~$0.047	18.6	~$3.40	charts and tables
sol61-high	gpt-6.1-sol, high effort (default tier)	110/110	~$0.020	30.5	~$2.21	charts and tables
astra-medium	gpt-6-astra, medium effort	75/75	~$0.077	22.7	~$5.76	charts and tables
terra61-low	gpt-6.1-terra, low effort (added 2026-10-28, operator request; main ladder p01–p12, 5 repeats)	60/60	~$0.010	10.3	~$0.62	charts and tables
terra61-high	gpt-6.1-terra, high effort (added 2026-10-28, operator request; main ladder p01–p12, 5 repeats)	58/60	~$0.0079	11.9	~$0.46	charts and tables
terra56-low	gpt-5.6-terra, low effort. Extra data: one repeat on p01–p12 (the operator meant gpt-6.1-terra)	9/12	~$0.018	14.1	~$0.16	charts and tables
terra56-high	gpt-5.6-terra, high effort. Extra data: one repeat on p01–p12 (the operator meant gpt-6.1-terra)	12/12	~$0.018	17.5	~$0.22	charts and tables
devin16slow-default	Devin swe-1-6-slow (the only model the free tier allows), default effort; devin CLI, no token or cost data; exploratory, added 2026-10-28 at operator request; function track p01–p12, 1 repeat (PLAN.md deviation 24)	6/12	–	10.5	–	charts and tables

fast mode and xhigh: 3 configs, ~$15.54 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
sol61-high ⚡	sol61-high on Codex priority tier (fast mode). Same model and effort, about twice the price.	75/75	~$0.037	18.2	~$2.81	charts and tables
astra-medium ⚡	astra-medium on the Codex priority tier (fast mode)	75/75	~$0.15	10.7	~$11.59	charts and tables
sol61-xhigh	gpt-6.1-sol at xhigh effort	35/35	~$0.033	44.3	~$1.14	charts and tables

retest pipeline: 11 configs, ~$4.65 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
luna-low-selftest	Arm C: luna-low with shell on, told to test its code first	29/35	~$0.0036	21.9	~$0.11	only here
luna-low+haiku-low-repair	Arm E (original run, prompt not kept): luna-low answers, haiku-low reviews and repairs	35/35	~$0.0035	27.4	~$0.12	only here
plan→luna-low	Arm F: haiku-low writes a plan, luna-low builds from it	30/35	~$0.0034	22.1	~$0.10	charts and tables
plan→luna-high-bare	Arm F: haiku-low plans, luna-high (stripped) builds	31/35	~$0.0036	27.4	~$0.11	charts and tables
luna-high-bare→H fail	Arm G: luna-high builds; haiku-low repairs only if a sample test fails	32/35	~$0.0014	10.6	~$0.043	charts and tables
luna-high-bare→retry	Arm G: luna-high builds; luna-high retries only if a sample test fails	32/35	~$0.0011	10.6	~$0.037	charts and tables
luna-low→H fail	Arm G: luna-low builds; haiku-low repairs only if a sample test fails	34/35	~$0.0022	7.9	~$0.073	charts and tables
astra-medium→H fail	Arm G: astra-medium builds; haiku-low repairs only if a sample test fails	35/35	~$0.087	21.4	~$3.04	charts and tables
sol61-high→H fail	Arm G: sol61-high builds; haiku-low repairs only if a sample test fails	35/35	~$0.022	32.7	~$0.76	charts and tables
luna-high-bare→H all	Arm G: luna-high builds; haiku-low reviews every answer	35/35	~$0.0035	26.1	~$0.12	charts and tables
luna-low→H all	Arm G: luna-low builds; haiku-low reviews every answer (arm E rerun from committed code)	34/35	~$0.0039	23.1	~$0.13	charts and tables

repo tasks: 7 configs, ~$8.36 spent

name	what it was	task complete (hidden in brackets)	$ per complete task	median s	spend	where it appears
haiku-low	Repo tasks with tools: Claude Haiku, low	12/12 complete (12/12 hidden)	$0.0051	28.3	$0.061	repo-task chart
haiku-high	Repo tasks: Claude Haiku, high	12/12 complete (12/12 hidden)	$0.012	58.4	$0.14	repo-task chart
sonnet-medium	Repo tasks: Claude Sonnet, medium	12/12 complete (12/12 hidden)	$0.054	27.7	$0.64	repo-task chart
luna-low	Repo tasks: gpt-6-luna, low	7/12 complete (12/12 hidden)	~$0.0056	24.1	~$0.039	repo-task chart
sol61-medium	Repo tasks: gpt-6.1-sol, medium	12/12 complete (12/12 hidden)	~$0.079	84.5	~$0.95	repo-task chart
sol61-high	Repo tasks: gpt-6.1-sol, high	12/12 complete (12/12 hidden)	~$0.097	117.9	~$1.17	repo-task chart
astra-medium	Repo tasks: gpt-6-astra, medium	12/12 complete (12/12 hidden)	~$0.45	65.6	~$5.35	repo-task chart

trim: 3 configs, ~$0.098 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
luna/base	Codex harness floor probe: luna, stock harness	9/9	~$0.00047	4.2	~$0.0042	only here
terra/base	terra, stock harness	9/9	~$0.010	5.6	~$0.091	only here
haiku	haiku-low through the same probe, tools off (reference)	9/9	$0.00028	5.0	$0.0025	only here

pilot: 19 configs, ~$5.88 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
haiku-low	Pilot (well-known problems): Claude Haiku, low	19/19	$0.00037	4.9	$0.0070	only here
haiku-medium	Pilot: Claude Haiku, medium	19/19	$0.00041	5.3	$0.0078	only here
sonnet-low	Pilot: Claude Sonnet, low	10/10	$0.0086	5.7	$0.086	only here
sonnet-medium	Pilot: Claude Sonnet, medium	19/19	$0.0060	5.8	$0.11	only here
haiku-low-agent	Pilot: Haiku low with tools on	10/10	$0.0020	7.0	$0.020	only here
sonnet-medium-agent	Pilot: Sonnet medium with tools on	9/9	$0.037	8.6	$0.33	only here
sol61-low-prio (pilot first run)	Pilot: gpt-6.1-sol, low, priority tier	10/10	~$0.068	4.7	~$0.68	only here
sol61-medium-prio	Pilot: gpt-6.1-sol, medium, priority tier	10/10	~$0.053	5.4	~$0.53	only here
astra-low	Pilot: gpt-6-astra, low	9/9	~$0.24	5.6	~$2.14	only here
luna-low	Pilot: gpt-6-luna, low, priority tier	18/18	~$0.0021	4.4	~$0.038	only here
luna-low-defa	Pilot: gpt-6-luna low, default tier	9/9	~$0.00089	4.5	~$0.0080	only here
luna-high-prio	Pilot: gpt-6-luna high, priority tier	9/9	~$0.0024	4.8	~$0.022	only here
terra56-low-prio	Pilot: gpt-5.6-terra low, priority tier	9/9	~$0.049	4.8	~$0.44	only here
sol6-low-prio	Pilot: gpt-6-sol low, priority tier	9/9	~$0.025	4.8	~$0.22	only here
sol61-low-prio	Pilot: gpt-6.1-sol low, priority tier	9/9	~$0.069	5.2	~$0.62	only here
sol61-low-defa	Pilot: gpt-6.1-sol low, default tier (the fast-mode control)	9/9	~$0.035	7.4	~$0.31	only here
pipeline A	Pilot repo task: gpt-6.1-sol builds, a reviewer checks	2/2	~$0.051	61.5	~$0.10	only here
pipeline B	Pilot repo task: gpt-6-luna builds, a reviewer checks	2/2	~$0.0054	20.3	~$0.011	only here
pipeline C	Pilot repo task: Claude Sonnet builds, no review	2/2	$0.092	23.1	$0.18	only here

harness-bias: 5 configs, ~$1.29 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
haiku-low / Claude's bench	Harness-bias check: haiku-low on hard problems on Claude's one-shot bench	57/60	$0.0025	13.6	$0.14	only here
haiku-low / Codex's bench (Claude Code)	Harness-bias check: haiku-low in Claude Code (Codex's bench)	126/126	$0.0027	13.6	$0.33	only here
luna-low / Claude's bench	Harness-bias check: luna-low on Claude's one-shot bench	36/60	~$0.0013	7.5	~$0.047	only here
luna-low / Codex's bench (Codex CLI)	Harness-bias check: luna-low in Codex CLI (Codex's bench)	99/126	~$0.0040	21.1	~$0.40	only here
sol61-medium / Codex's bench (Codex CLI)	Harness-bias check: sol61-medium in Codex CLI (Codex's bench)	6/6	~$0.062	57.9	~$0.37	only here

Low context (prompt or tool text trimmed): 17 configs, ~$0.96 spent

name	what it was	pass	$ per pass	median s	spend	where it appears
sol61-low-prio-lean	Pilot: gpt-6.1-sol low with --ignore-user-config only	9/9	~$0.027	7.0	~$0.24	only here
sol61-low-prio-bare	Pilot: gpt-6.1-sol low, stripped harness, priority tier	9/9	~$0.031	5.4	~$0.28	only here
luna-low-noshell-B1	Arm B1: luna-low without shell, nudged to list edge cases and trace the sample tests first	25/35	~$0.00097	8.6	~$0.024	only here
luna-low-noshell-B2	Arm B2: B1 plus a detailed reasoning summary	19/35	~$0.0012	8.9	~$0.022	only here
luna-medium-bare	Arm D: luna-medium, stripped harness	21/35	~$0.0012	8.4	~$0.025	only here
luna-high-bare	Arm D: luna-high, stripped harness	31/35	~$0.0010	13.2	~$0.033	charts and tables
luna/nofeat	luna, tool features stripped	9/9	~$0.00098	4.4	~$0.0089	only here
luna/noshell	luna, features, shell and base instructions stripped	9/9	~$0.00030	3.8	~$0.0027	only here
luna/mininst	luna, minimal instructions (one call)	1/1	~$0.00033	6.5	~$0.00033	only here
luna/noctx	luna, no project context (one call)	1/1	~$0.00043	7.0	~$0.00043	only here
luna/nodoc	luna, no docs (one call)	1/1	~$0.00045	5.9	~$0.00045	only here
terra/nofeat	terra, tool features stripped	9/9	~$0.021	5.2	~$0.18	only here
terra/noshell	terra, features and shell stripped	9/9	~$0.0080	6.8	~$0.072	only here
terra/mininst	terra, minimal instructions (one call)	1/1	~$0.013	4.8	~$0.013	only here
terra/noctx	terra, no project context (one call)	1/1	~$0.0092	5.1	~$0.0092	only here
terra/nodoc	terra, no docs (one call)	1/1	~$0.0096	4.9	~$0.0096	only here
luna-low-bare	luna-low with the Codex system prompt and tool features stripped (trim level noshell)	44/60	~$0.00080	6.8	~$0.035	charts and tables

Category totals cover the rows listed here.

Interesting findings

sol56-high is a generation older than sol61-high. Both pass every main-ladder try. sol56-high takes 14.5 s at ~$0.042 per pass. sol61-high takes 27.4 s at ~$0.019. ⚡ cuts that to 17.4 s, at ~$0.036. So the newer model is cheaper but slower. ⚡ does not close the gap.
Sol 6.1 high on fast was slower than Sol 5.6 high — does Terra high match or beat Sol 5.6 high on speed and pass rate? It is faster but misses. terra61-high takes 11.9 s to sol56-high's 14.5 s, and passes 58/60 to 60/60. The 2 misses are on p04 taller_within (repeats 1, 3). terra61-low matches sol56-high at 60/60 and is faster, 10.3 s. Cost per pass, list-price estimates: terra61-low ~$0.010, terra61-high ~$0.0079, sol56-high ~$0.042. Fast mode was not tried. The trigger was a low/high gap of 10 points or more, or terra high outside 0.5–2× of sol56-high's median. The gap is 3.3 points and the ratio is 0.82×, so it did not fire. Extra data, one repeat each: gpt-5.6-terra passed 12/12 at high and 9/12 at low.
luna-low is cheaper per pass than haiku-low: ~$0.00090 to $0.0011. But it passes 77% to 95%.
SemiAnalysis: Anthropic plans give ~5× OpenAI's API-priced value at the Opus/Sol tier. We price list-rate calls, not plan quotas, so we can’t test that. At list price, haiku-high beats sol61-high ~7× per pass; tiers differ. On repo tasks the gap is ~8×. luna-low costs least per run, but it completes 7/12 (12/12 hidden), so per complete task haiku-low is cheapest: $0.0051 against luna-low's $0.0056.
On Codex's bench, both families gained. luna-low rose from 36 to 47 passes. It cost ~3× per pass and took ~3× the median time. haiku-low rose from 57 to 60 passes. It cost ~1.1× per pass, with the same median time.
Codex's bench dry runs missed 2 sandbox bugs; a live call found them. Its "plan and reorient" nudge changed no code.
On L7, luna-high passed 15/15; sol56-high missed. luna-high cost ~$0.0015 per pass versus sol56-high's ~$0.069.
Disabling luna-low's tool features cut input 31%, yet raised estimated cost 2.1×.
On repo edits, sol61-high took 118 s; astra-medium took 66 s. sonnet-medium and haiku-low took about 28 s. Both Codex picks took over twice as long. All four completed every run.

Codex vs Claude

All 4 Claude configs have Codex matches. 3 of 4 cost more at list price; terra61-low costs less than sonnet-medium. 2 of 4 also run slower. terra61-low runs faster than haiku-medium and haiku-high. All 14 Codex configs have Claude matches. Claude is cheaper for 10. Only luna undercuts its Claude match, with more misses.

Config → cheapest match	passed (function tests)	$ per pass	median s
luna-low-bare → haiku-low	73% → 95%	~$0.00080 → $0.0011	7 → 10
luna-low → haiku-low	77% → 95%	~$0.00090 → $0.0011	7 → 10
luna-medium → haiku-low	78% → 95%	~$0.00089 → $0.0011	8 → 10
luna56-low → haiku-low	80% → 95%	~$0.0022 → $0.0011	9 → 10
luna-high → haiku-low	92% → 95%	~$0.0010 → $0.0011	12 → 10
luna56-high → haiku-low	93% → 95%	~$0.0030 → $0.0011	11 → 10
terra61-high → haiku-medium	97% → 98%	~$0.0079 → $0.0015	12 → 13
sol61-medium → haiku-medium	97% → 98%	~$0.015 → $0.0015	23 → 13
terra61-low → haiku-high	100% → 100%	~$0.010 → $0.0028	10 → 15
sol61-high → haiku-high	100% → 100%	~$0.019 → $0.0028	27 → 15
sol61-high ⚡ → haiku-high	100% → 100%	~$0.036 → $0.0028	17 → 15
sol56-high → haiku-high	100% → 100%	~$0.042 → $0.0028	14 → 15
astra-medium → haiku-high	100% → 100%	~$0.075 → $0.0028	21 → 15
astra-medium ⚡ → haiku-high	100% → 100%	~$0.15 → $0.0028	10 → 15
haiku-low → terra61-high	95% → 97%	$0.0011 → ~$0.0079	10 → 12
haiku-medium → terra61-low	98% → 100%	$0.0015 → ~$0.010	13 → 10
haiku-high → terra61-low	100% → 100%	$0.0028 → ~$0.010	15 → 10
sonnet-medium → terra61-low	100% → 100%	$0.012 → ~$0.010	10 → 10

How to read this data: matches

Match = the cheapest other-vendor config passing at least as often.
Fastest Codex at 100%: astra-medium ⚡, 10 s. It costs $0.15 per pass.
At haiku-low's price, the best Codex is luna-high: 92% to 95%.
All rows use the main ladder.

WINNER

Cost vs accuracyOne chart, three trade-offs

astra · 6.x

grok

haiku

luna · 5.6

luna · 6.x

sol · 5.6

sol · 6.1

sonnet

terra · 5.6

terra · 6.1

haiku-low

haiku-medium

sonnet-medium

luna-low

luna-medium

sol61-medium

haiku-high

luna-high

luna56-low

luna56-high

sol56-high

sol61-high

astra-medium

terra61-low

terra61-high

terra56-low

terra56-high

grok47-low

grok47-medium

grok47-high

grok47fast-low

devin16slow-default

sol61-high ⚡

astra-medium ⚡

sol61-xhigh

luna-low-bare

🎯 Pass rate vs $ per pass

Speed and bare prompts

Points and labels

Key numbers

Sets and pipelines

Results

⚡ Codex fast mode: seconds and $ per pass

Table columns

Retest set

Pass rate and time

Fast mode

Codex $

Grok and Devin

Where does Grok get to?

Closest lanes by pass rate

Is Grok better at anything?

First run (2 repeats, p01–p06)

Key numbers

First run: where it got to

First run: closest lanes

First run: was Grok better at anything?

Repository tasks

Devin on the free tier

How it was run

Where they fail

Baseline input tokens per call

Pipelines: plan, check, fix

Pass rate, $ per pass, median seconds

Gates

Scope

Whose bench?

The benches

What changed

CONCLUSION

Limitations

More charts

Notes: code quality, fast mode, appendix

Code quality columns

Code quality scope

Fast-mode control

Codex CLI in the main benchmark

Failures and later runs