litmus · mutation testing for eval suites
Can your eval fail?
An eval suite that never goes red tells you nothing. If you delete half of a skill's instructions and every case still passes, the suite was not testing those instructions. It was testing something else, or nothing at all.
litmus checks this by breaking the thing under test on purpose. It deletes an instruction, inverts a rule, swaps two tool names, truncates the prompt or plants a wrong number, then runs your evals against each broken copy. The mutation score is the share of broken copies your evals caught. Every one they missed is listed with its diff.
A mutant only counts as caught when a trusted run shows it. A crashed or partial run, or a baseline that was never green, is inconclusive and never counts in your favor. litmus runs on top of claude plugin eval and also finds graders that cannot fail at all, offline and for free.
Latest result: my own plugin
cited, a Claude Code plugin that checks every claim is on the page it cites. Third run: 3 eval cases, 14 mutants (7 of its skill, 7 of its check command), 1 run each, 2026-10-09.
- KILLEDCommand: emptied the whole body
- KILLEDCommand: cut the second half
- KILLEDCommand: replaced the description with "General helper."
- KILLEDSkill: swapped the Read and Bash tool names
- KILLEDSkill: cut the second half
- SURVIVEDSkill: emptied the whole body
- SURVIVEDSkill: replaced the description with "General helper."
- SURVIVEDSkill: deleted the paragraph that says what it is
- SURVIVEDSkill: inverted "cannot" to "can" in the failure it defends against
- SURVIVEDSkill: changed "before" to "after" in that paragraph
- SURVIVEDCommand: deleted its opening line
- SURVIVEDCommand: inverted "do not" to "do" in one input rule
- SURVIVEDCommand: swapped the Write and Bash tool names
- SURVIVEDCommand: changed "before" to "after" in its first line
The suite catches damage to the command, which is what the agent actually follows, and mostly misses damage to the skill. Deleting the whole skill body changed nothing. Either that body is redundant with the command, or the suite needs a case where the skill does work the command does not. That is the kind of answer a suite that always stays green cannot give you.
Getting a trustworthy run took three tries, and each fix is in the open. litmus now keeps installed plugin CLIs off the eval's PATH and copies the plugin once per run, so an edit in flight cannot leak into some mutants. The case that tests the method now checks against recorded copies of the cited pages instead of the live network, which the eval sandbox kept breaking. One run per case per mutant is still noisy: a single flaky run can fake a kill or a survival.
Full report with every diff · JSON
Second run, superseded
Second run: 3 eval cases, 8 mutants of its skill, 1 run each, 2026-10-09.
Same score, same eight survivors as the first run. This time I know why:
both runs mutated a file the eval agent never read. The agent
uses cited's check command, and litmus was told to mutate only the
skill file. Emptying a file nobody reads changes nothing, so every mutant
survived. So the survivors say less about the suite than they seem to: the
mutants hit the wrong file, and a whole body deleted with nothing going red is
what gave it away.
Between runs I fixed two setup problems. litmus now keeps installed plugin CLIs off the eval's PATH. And the one case that tests the method was red because the eval sandbox only reaches the network through a proxy that cited ignores unless told otherwise. I put that instruction in the skill file, which is exactly the file the agent never read, so that case was red again. The fix is now in the command too, but no paid run has measured it yet.
One more disclosure: I edited the command file while the run was going, then reverted it, and one mutant started within seconds of the edit. Its networked case was red at baseline, so it cannot count either way, but its score in the report is not evidence about that mutant. The third run above replaces this one.
Full report with every diff · JSON
First run, for the record
cited, a Claude Code plugin that checks every claim is on the page it cites. 3 eval cases, 8 mutants of its skill, 1 run each, 2026-10-09.
- SURVIVEDEmptied the whole skill body
- SURVIVEDCut the second half of the body
- SURVIVEDReplaced the description with "General helper."
- SURVIVEDSwapped the Read and Bash tool names
- SURVIVEDInverted "cannot" to "can" in the failure it defends against
- SURVIVEDDeleted the paragraph that says what the skill is
- SURVIVEDChanged "before" to "after" in that paragraph
- SURVIVEDDeleted the pointer to the methodology reference
The one case that tests the skill's method was already red, because the eval sandbox could not reach the cited site. The two green cases pass on the plugin's CLI and on the skill staying quiet, neither of which the prose controls. So the suite looked healthy and could not tell a working skill from an empty one. The run also had a setup leak: an installed copy of the cited CLI was on the eval's PATH, so mutants could still reach unmutated code. One run per case is noisy. Treat this as a first reading; the second run above is what that rerun found.