Verify Before Ship
Checks Cited Claims Against Their Sources · AI Infrastructure
A fact-checking gate for anything an LLM writes with citations: re-fetches every cited page and flags any claim it cannot find there, so a reviewer looks at those before anything is published.
View source on GitHubHow it was built
Problem
Signal Scout's source-verification step proved the pattern worked for one skill, but every other project generating AI text with citations needed the same guarantee, and copy-pasting the check into each one meant fixing the same fabrication bugs repeatedly.
Decisions
- Generalized the containment-checking methodology out of signal-scout into a standalone tool, so any LLM-writing pipeline can adopt it as a dependency instead of reimplementing it.
- Made it run before publication rather than as an audit afterwards: flagged claims go to a person to confirm or cut before anything ships.
- Kept the check narrow and deterministic: re-fetch the source, confirm the claim is actually in it, rather than asking another model to grade the first model's honesty.
Outcome
Any project that generates cited claims can now pull in a reusable pre-publish citation check instead of bolting a one-off script onto a single skill.
Key features
- Re-fetches every source an LLM cites and checks that the claim's key words, numbers and names appear in it
- Flags unsupported citations for a person to confirm or cut
- Runs before publication, not as an audit afterwards
Impact
"The model cited a source" gets checked against the actual page before it ships, not assumed.
Tech stack
Python