Back to work
cited
Checks cited claims against their sources · AI Infrastructure
A citation-checking gate for anything an LLM writes with citations: fetches every cited page and flags any claim whose words, numbers or names it cannot find there, so a reviewer looks at those before anything is published. It checks that a claim is on the page, not that the page is true.
View source on GitHubhow it was built
Problem
Signal Scout's source-verification step proved the pattern worked for one skill, and every other project generating AI text with citations needed the same check.
Decisions
- Generalized the containment-checking methodology out of signal-scout into a standalone tool, so any LLM-writing pipeline can adopt it as a dependency instead of reimplementing it.
- Made it run before publication rather than as an audit afterwards: flagged claims go to a person to confirm or cut before anything ships.
- Kept the check narrow and deterministic: re-fetch the source, confirm the claim is actually in it, rather than asking another model to grade the first model's honesty.
Outcome
In cited#1 a made-up funding round scored as a near match, so numbers and names in a claim now have to be on the page.
Key features
- Re-fetches every source an LLM cites and checks that the claim's key words, numbers and names appear in it
- Flags unsupported citations for a person to confirm or cut. On its own benchmark (29 claims about one page, labelled by me) it flagged 10 of 15 unsupported claims: recall 0.67, 95% interval 0.41-0.90, precision 0.67 (0.42-0.91)
- Runs before publication, not as an audit afterwards
Impact
"The model cited a source" gets checked against the actual page before it ships, not assumed.
Tech stack
Python