rootcause
An incident agent built without a framework, and measured · AI Agents
An agent that finds the root cause of production incidents and failed CI runs. The loop, tool execution, approval gate, crash recovery and multi-agent orchestration are written without a framework over MCP tools, and its performance claims come from evals scored against known causes, repeated and reported with intervals and cost. Those evals are 30 written incidents and 3 recorded CI failures; it has not run against a live system, and the Azure deploy in the repo has never been dispatched.
View source on GitHubresults
| setting | pass rate, 60% to 100% | cost per solved |
|---|---|---|
| Opus 5.5, high | 79% (63-92) | $0.0810 |
| Opus 5.5, medium | 81% (67-95) | $0.0686 |
| Opus 5.5, low | 79% (64-92) | $0.0468 |
| Sonnet 5.5, high | 80% (65-93) | $0.0325 |
| Haiku 5.5, high | 77% (63-91) | $0.0033 |
| Opus 5.5, high + BM25 search | 77% (61-91) | $0.0822 |
| Opus 5.5, high + hybrid search | 84% (71-96) | $0.0750 |
Source: rootcause results/sweeps/models-and-retrieval-2026-10-15. No pair of settings differed detectably, but paired by incident the design detects only gaps above about 13-19 points between model tiers, so smaller accuracy differences are unresolved, not zero.
how it was built
Problem
A responder under pressure blames the last deploy before the errors. That heuristic is often wrong, and an agent can make the same mistake with more confidence. Most agent demos show one run that worked and say nothing about how often it would.
Decisions
- Wrote the loop instead of using a framework, so retries, approvals and crash recovery are explicit and testable.
- Built each scenario around a mistake a responder could make, several with a planted decoy that defeats the last-deploy heuristic, and kept the ground truth out of everything the agent can read.
- Kept harness failures out of the pass rate and checked that each run was served by the model it asked for, so the numbers measure the model and not the plumbing.
- Measured retrieval on its own first. Plain BM25 beat both embedding models on incident text, which is full of exact error strings.
Outcome
A comparison of models, effort levels and retrieval that a reviewer can rerun from the repo, with every result file committed and every known limitation written down.
Key features
- A framework-free tool loop with an append-only event log, so a crashed run resumes without repeating a side effect
- Approval gate for production actions, a check in the loop that rejects answers citing evidence no tool returned, and a prompt-injection scenario where obeying a planted instruction fails the run
- Sweeps that repeat every scenario per model and effort level, with intervals from resampling incidents, pass^k and cost per solved incident
- Retrieval over past postmortems, from BM25 to embeddings to a fused hybrid, evaluated on its own before it reaches the agent
Impact
Decisions about model, effort and retrieval rest on measured reliability and cost, not on one good-looking run.
Tech stack
Python, Claude API, Claude Code, MCP, OpenTelemetry, pytest, GitHub Actions