Three checks that passed when they shouldn't have
scoped · cited · signal-scout
A green check means one of two things: the code works, or the check cannot see the problem. These are three times it was the second, all in my own repos, all caught after the check had been passing for a while.
1. The concurrency test that ran on one connection
scoped stops two Claude Code sessions from editing the same file: an MCP lock server plus a hook that blocks the edit. Its test suite had a concurrency test, and it passed.
The test ran every "session" over a single connection, so the requests were serialized before they reached the lock. It could not observe a race, because none was possible. When I raced real processes instead, two sessions edited the same file. The fix addressed both causes of the race, and the test now uses separate processes, so it can fail.
2. The made-up funding round that matched
cited re-fetches every page an LLM cites and flags claims the page does not support. A claim about a funding round that never happened scored as a near match, because most of its words were on the page. The words that made it false, the amount and the name, were the ones the fuzzy match forgave.
Numbers and names in a claim now have to appear on the page. On cited's own 29-claim benchmark it catches 10 of 15 unsupported claims; swapped ordinary words still get through, and the README says so.
3. The eval job that graded its own answers
signal-scout's CI had an eval job, and it was green. It graded hand-written example outputs, not outputs the pipeline produced, so no change to the pipeline could make it fail.
It now runs the pipeline on fixtures. To prove the job can fail, I planted three bugs, one at a time, and each one turns CI red.
The rule
Before I trust a check, I make it fail once. For code that means a planted bug that has to turn CI red. For prompts and evals it means litmus, which mutates the prompt under test and lists the broken copies the suite missed. Pointed at my incident agent's eval suite, it found that the suite killed 0 of 12 mutants. That one is in the benchmark write-up.
My coding agents run inside sous, which applies the same rule to them: a green run that executed no tests, or weakened one, fails.