The benchmark that didn't pick a winner

The sweep used 25 incidents with known causes (the suite has 33 now; the eight added later are below). Twenty-two are written as code, each around a mistake a responder could make under pressure, like a deploy that lands just before a vendor outage. The other three are real failed CI runs from sous, another of my repos.

The scenarios plant decoys against the obvious heuristic, "blame the last deploy before the first error." That baseline gets three of them right. The ground truth stays out of every fixture the agent can read. The grader checks the category, the service, that every cited id exists in the scenario's telemetry and at least one is the expected evidence, and that no rollback hit an innocent deploy.

One good run proves little, so the sweep runs every scenario three times per setting, interleaved across settings. Harness failures (a crash, a timeout, a run served by the wrong model) are retried, never counted as wrong answers. Each setting gets a 95% interval on its pass rate from resampling incidents with their runs kept together, pass^3 (all three attempts right, which is what you want from something on call), and cost per solved incident, which includes what the failed runs spent.

The sweep covered seven settings: Opus 5.5 at high, medium and low effort, Sonnet 5.5 and Haiku 5.5 at high, and two retrieval variants of Opus at high effort, one with BM25 search and one with a hybrid. So five model and effort settings plus two retrieval variants, 75 runs each, all through Claude Code's own agent loop.

Cost separated them, by about 25x: from $0.0033 per solved incident for Haiku to $0.081 for Opus at high effort. Accuracy did not. Every setting landed between 77% and 84%, and no pair differed detectably. That is a statement about the design as much as the models: paired by incident, Opus at high effort minus Haiku is +1.3 points (-12.0 to +13.3), and the smallest gap this design would detect is about 19 points for that pair and 13 for Opus against Sonnet. Smaller differences are unresolved, not shown to be zero. Opus at medium effort was at least as accurate as Opus at high, for less; the extra reasoning bought nothing I could detect.

That result also depends on my grader. Two scenarios fail in every run and make up 42 of the 107 failures, and 53 failures are wrong on the service alone. Drop the service check and Opus at high effort scores 93%, Haiku 85% and Sonnet 83%, with the Opus-Sonnet gap at 3 to 20 points when I resample scenarios. That comparison was chosen after seeing the results and is not corrected for the others, so it is a hint, not a finding. Either way, no detected difference did not show the models are equal.

The hybrid-search row has the highest point, and it is tempting to credit retrieval for it. I can't. With the search tool available and named in the brief, Opus called it in 3 of 75 runs in each retrieval setting. Hybrid minus BM25 even has a resampled interval that excludes zero, but only 4 incidents differ, and it is one of 21 pairwise comparisons; after a Holm correction it is nothing.

A separate 300-run comparison held the model at Opus 5.5 high and changed the harness instead: my framework-free loop against the same investigation rebuilt as a LangGraph graph, each run as a single agent and as an orchestrator with workers. Everything else was shared, down to the grader.

The orchestrator with workers cost about three times as much per solved incident ($0.3745 against $0.1149 in my loop, $0.3430 against $0.1114 in LangGraph) and took about twice as long. Its accuracy did not differ detectably from a single agent: single minus orchestrator is +3.3 points (-4.0 to +12.0) pooled over both harnesses, and the design detects only gaps above about 12 points pooled and 15 per harness, so a smaller difference either way is not ruled out. I found no detectable difference between the two frameworks either: my loop minus LangGraph, single agent, is -2.7 points (-8.0 to +2.7), and that design detects only gaps above about 8 points.

Before the agent ever sees a search tool, I score the retrievers by themselves over a corpus of 33 past postmortems. With the alert plus the first error log line as the query, plain BM25 reached recall@5 of 0.94 and MRR of 0.83. Both embedding models and a fused hybrid scored lower on MRR, the transformer model at roughly a thousand times the query latency. Incident text is full of exact tokens like x509 or "No space left on device", and lexical matching handles those directly. The corpus and the scenarios also share an author, which favors it.

I also mutation-tested the investigator's brief with the operators from litmus, my mutation tester for eval suites: delete an instruction, invert a rule, make a fact wrong, truncate it. 0 of 12 mutants were killed: 11 survived and 1 was inconclusive. The 11 survivors had pass rates between 72% and 84%, against 79% for the original. The inconclusive one, an emptied brief, could not run because the API rejects an empty system prompt. Two survivors touch a rule the harness already enforces through its output schema, so they could not fail.

That says more about the suite than about the brief. No scenario depends on any single instruction, so the suite could not catch a regression in one.

So I built the suite the first run called for. Eight new scenarios, each named for one rule of the brief and designed so that rule decides the outcome. The grader now accepts a set of defensible services per scenario, written before any model answered, with the old single-service grade kept behind a flag. Three scenarios are held out, never used to change the brief or the grader. Every new scenario requires the agent to search past postmortems, so search is a measured step.

Run once at three model tiers, 72 runs: Opus 24 of 24, Sonnet 24 of 24, Haiku 21 of 24. The suite does not separate the top two tiers, and Haiku's three misses were all a wrong category with the right service and evidence. Every run searched, but only because the brief now says to and the grader requires it, so that shows instruction-following, not that search helps.

Then I mutation-tested the retrieval brief against the new scenarios: 112 runs, 14 mutants. 2 were killed, 11 survived and 1 was inconclusive. Both kills are the search requirement: delete the search line and the grader fails all eight scenarios, truncate the brief past it and it fails seven, because a search it demands is no longer asked for. Deleting or inverting any behavioural rule, such as gather evidence first or finish with the structured answer, survived. Scenarios built around single rules did not catch a single rule being broken. The original problem is half fixed: the suite now catches a brief that stops asking for search, and still cannot catch a regression in a behavioural instruction.

The 75 runs per setting are not 75 independent trials. Repeats of one incident mostly agree, and most incidents pass or fail all three times, so 75 runs carry the information of 29 to 32 independent ones (intraclass correlation 0.69 to 0.78). The intervals here resample incidents with their runs kept together, which makes them 1.5 to 1.6 times as wide as intervals that treat every run as independent. Paired by incident, 25 incidents can detect only gaps above about 13 to 19 points between model tiers; detecting a 10-point gap between Opus and Haiku would take about 84 incidents. So the accuracy result is unresolved, not a tie. Cost is the finding that holds.

The held-out split in the follow-up controls for tuning, not authorship: the scenarios, the corpus and the labels share an author, which limits how far any of this generalizes. A later set of eight incidents adapted from public postmortems, with answers committed before any run, put Haiku 5.5 at 50% (21% to 79%) against 77% on the main suite. With eight incidents that is a rough range, not a test of a difference, and its telemetry still has the same author. Every live run went through Claude Code, with costs as Claude Code reports them at API list prices; the native Anthropic API adapter is tested only against a mocked transport. In the 525-run sweep, some failures are disagreements with a strict grader that accepts one service per scenario, not clear misses; the accepted-service grade came after, and I did not widen those scenarios, since their answers were already published.

If the accuracy your task needs is what this grader scores, cost is the only axis this benchmark resolves, and the cheapest setting here is Haiku 5.5, at about a twenty-fifth of the cost of Opus at high effort. If you need the category right more than the exact service, Opus earns more of its price. Run the comparison with your own grader before deciding.

A suite that can tell settings apart is harder than it looks. I built the pieces: scenarios aimed at single rules, a pre-registered multi-service grader, a held-out split and a required search step. Mutation testing says it still does not catch a broken behavioural rule. The next try needs harder traps, several runs per mutant, and a weaker model as the baseline.