Litmus
Mutation testing for eval suites · Evals and Retrieval
Asks whether an eval suite can fail. It breaks the skill under test on purpose (deletes an instruction, inverts a rule, swaps tool names, plants a wrong number), runs the evals against each broken copy, and reports how many they caught.
View source on GitHubproject pagedecision recordslimitations
how it was built
Problem
An eval suite that never goes red tells you nothing. A grader such as "tool used at least zero times" passes on any run, and a case can stay green on an empty reply, so a suite can look complete while testing very little.
Decisions
- Used deterministic mutation operators that skip code fences, so the same mutants can be replayed byte for byte after a model release.
- Counted a mutant as caught only when a trusted run shows it. A baseline that was never green stops the run instead of producing a score.
- Put the free checks (vacuity, and audit of results you already have) first, so they fit on every PR, and left paid mutation runs for a schedule.
Outcome
I ran its operators on the rootcause investigator brief: the suite caught 0 of 12 broken copies (11 survived, 1 inconclusive). Eight scenarios built around single rules later caught 2 of 14, both through the search requirement, none through a behavioural rule. On cited's suite it killed 5 of 14. Every run used one run per mutant, so run-to-run variance is not measured, and every suite scored so far is mine; there is no score on a third-party suite yet.
Key features
- litmus vacuity finds graders that cannot fail, offline and for free, before any run is paid for
- litmus mutate reports a mutation score and lists every surviving mutant with its diff, so you see which part of the prompt nothing tests
- A crashed, partial or rate-limited run is inconclusive and never counts as a catch
- Runs on top of claude plugin eval; it does not replace the runner
Impact
A green eval suite has to show it would have gone red on a broken skill.
Tech stack
Python, Claude Code plugin evals