Can your eval fail?

cited · adapter claude-plugin-eval · litmus 0.3.0 · 2026-10-09T15:21:06+00:00
0%mutation score: 0 of 8 mutants, 1 run per case
0killed
8survived
0inconclusive (not killed)
0vacuous graders + null-green cases
2 of 3cases green at baseline (only these can kill)

Baseline (unmutated copy)

casescorecounts toward kills
catch-fabricated-citation0.4no, not green at baseline
invalid-claims-file1.0yes
no-urls-no-check1.0yes

Graders and cases on do-nothing runs

statuscase / gradertypewhy
DISCRIMINATEScatch-fabricated-citation / ran-citedtool_used
DISCRIMINATEScatch-fabricated-citation / skill-firedtool_used
UNKNOWNcatch-fabricated-citation / verdictsllm llm grader needs a judge model
DISCRIMINATESinvalid-claims-file / ran-citedtool_used
UNKNOWNinvalid-claims-file / reportllm llm grader needs a judge model
UNKNOWNno-urls-no-check / rewritellm llm grader needs a judge model
NULL_PASSno-urls-no-check / skill-not-firedtool_usedNULL_PASS (guard) passes on probe(s): silent, done, refusal, echo
OKcatch-fabricated-citationcase [ablation-none]silent: RED, done: RED, refusal: RED, echo: RED
JUDGE_ONLYcatch-fabricated-citationcase [with-without]silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN
OKinvalid-claims-filecase [ablation-none]silent: RED, done: RED, refusal: RED, echo: RED
JUDGE_ONLYinvalid-claims-filecase [with-without]silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN
JUDGE_ONLYno-urls-no-checkcase [ablation-none]silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN
JUDGE_ONLYno-urls-no-checkcase [with-without]silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN

Mutants

verdictmutantdetail
SURVIVEDdelete-body:skills/cited/SKILL.md:0:47cee39c
empty the body; keep only the frontmatter
every baseline-green case stayed green
catch-fabricated-citation: 0.4, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -5,35 +5,2 @@
 ---
 
-# cited
-
-The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention.
-
-This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
-
-Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
-
-## When to use this
-
-Any time an agent is about to hand a human a document, list, or report where individual line items are attributed to a specific source (a URL) and the human is expected to trust the attribution without independently re-checking it themselves. Examples: a lead-gen report ("this company posted this job"), a research digest ("this podcast guest said this number"), a competitive battlecard ("competitor X does Y, per their pricing page"), a citation-heavy summary.
-
-Do **not** reach for this to fact-check claims that don't cite a specific URL — it verifies containment (is the claim on that page), not truth (is the page itself honest), and it has nothing to check without a `source_url`.
-
-## Workflow
-
-1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag.
-3. Read the tier written back onto each entry:
-   - `verified` — substantially quoted from the live (or archived) page. Ship it.
-   - `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
-   - `snippet_only` — the platform blocked automated fetching (Reddit, X, LinkedIn, etc.) and no archived copy exists. Your call whether the original discovery snippet is trustworthy enough to keep; disclose the unverified status if you do.
-   - `unsupported` — the page loaded fine and the claim is **not on it**, or the claim's numbers or capitalized names (an amount, an investor, a product) are not on it. This is the fabrication signal. Drop the claim or find a claim the page actually supports.
-   - `broken` — the source URL doesn't resolve, even via the Wayback Machine fallback. Drop it.
-   - `unverified` — the claim was never checked: the page was a PDF or other binary, too thin to read (a JavaScript app, a consent wall), too large to read in full, or missing from an `--offline` cache. Check it by hand, or cite an HTML page that carries the same fact.
-4. A non-zero exit means do not ship as is; `cited --help` lists what each code means. Wire that into a CI gate or a pre-ship check if your artifact pipeline has one. A run that fails only on `low_match` is waiting on human review, not proven wrong.
-5. On the shipped artifact, disclose the result: "N of M claims verified against source." That single line is what turns a report into something a skeptical reader can trust without redoing your research.
-
-## What this does and doesn't prove
-
-It proves a claim's text is (or isn't) present on the page it cites, checked independently of whatever model wrote the claim. It does **not** prove the page itself is honest, that a correctly-quoted line means what your artifact says it means, or that the page hasn't changed since — every checked entry is stamped `verified_at` so staleness is visible later, not hidden.
-
-See [references/methodology.md](references/methodology.md) for the full tiering rules, the Wayback Machine fallback behavior, and how bot-walled platforms (Reddit, X, LinkedIn) are handled without either failing the run or pretending they were checked.
SURVIVEDdelete-instruction:skills/cited/SKILL.md:0:a557d428
delete line 6: This skill is a mechanism, not a vertical: it takes any list of `(cla...
every baseline-green case stayed green
catch-fabricated-citation: 0.2, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -9,5 +9,4 @@
 The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention.
 
-This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
 
 Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
SURVIVEDinvert-rule:skills/cited/SKILL.md:0:b268f9d4
line 4: 'cannot' -> 'can' in: The core failure mode this defends against: an agent writes a plausib...
every baseline-green case stayed green
catch-fabricated-citation: 0.2, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -7,5 +7,5 @@
 # cited
 
-The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention.
+The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading can catch self-invention.
 
 This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
SURVIVEDswap-tool-names:skills/cited/SKILL.md:0:f76b1374
swap tool names Read <-> Bash
every baseline-green case stayed green
catch-fabricated-citation: 0.2, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -11,5 +11,5 @@
 This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
 
-Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
+Bash [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
 
 ## When to use this
@@ -22,6 +22,6 @@
 
 1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag.
-3. Read the tier written back onto each entry:
+2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Read tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag.
+3. Bash the tier written back onto each entry:
    - `verified` — substantially quoted from the live (or archived) page. Ship it.
    - `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
SURVIVEDtruncate:skills/cited/SKILL.md:0:8031024e
keep the first 17 of 35 body lines
every baseline-green case stayed green
catch-fabricated-citation: 0.4, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -21,19 +21,2 @@
 ## Workflow
 
-1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag.
-3. Read the tier written back onto each entry:
-   - `verified` — substantially quoted from the live (or archived) page. Ship it.
-   - `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
-   - `snippet_only` — the platform blocked automated fetching (Reddit, X, LinkedIn, etc.) and no archived copy exists. Your call whether the original discovery snippet is trustworthy enough to keep; disclose the unverified status if you do.
-   - `unsupported` — the page loaded fine and the claim is **not on it**, or the claim's numbers or capitalized names (an amount, an investor, a product) are not on it. This is the fabrication signal. Drop the claim or find a claim the page actually supports.
-   - `broken` — the source URL doesn't resolve, even via the Wayback Machine fallback. Drop it.
-   - `unverified` — the claim was never checked: the page was a PDF or other binary, too thin to read (a JavaScript app, a consent wall), too large to read in full, or missing from an `--offline` cache. Check it by hand, or cite an HTML page that carries the same fact.
-4. A non-zero exit means do not ship as is; `cited --help` lists what each code means. Wire that into a CI gate or a pre-ship check if your artifact pipeline has one. A run that fails only on `low_match` is waiting on human review, not proven wrong.
-5. On the shipped artifact, disclose the result: "N of M claims verified against source." That single line is what turns a report into something a skeptical reader can trust without redoing your research.
-
-## What this does and doesn't prove
-
-It proves a claim's text is (or isn't) present on the page it cites, checked independently of whatever model wrote the claim. It does **not** prove the page itself is honest, that a correctly-quoted line means what your artifact says it means, or that the page hasn't changed since — every checked entry is stamped `verified_at` so staleness is visible later, not hidden.
-
-See [references/methodology.md](references/methodology.md) for the full tiering rules, the Wayback Machine fallback behavior, and how bot-walled platforms (Reddit, X, LinkedIn) are handled without either failing the run or pretending they were checked.
SURVIVEDwrong-fact:skills/cited/SKILL.md:0:91bfedff
line 6: 'before' -> 'after' in: This skill is a mechanism, not a vertical: it takes any list of `(cla...
every baseline-green case stayed green
catch-fabricated-citation: 0.4, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -9,5 +9,5 @@
 The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention.
 
-This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
+This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — after the artifact carrying those claims ships to a human.
 
 Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
SURVIVEDdrop-description:skills/cited/SKILL.md:0:3c7b9e87
replace the frontmatter description with 'General helper.' (tests triggering)
every baseline-green case stayed green
catch-fabricated-citation: 0.4, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -1,5 +1,5 @@
 ---
 name: cited
-description: Before an agent ships any artifact that cites sources (a report, a lead list, a research summary, a changelog claim), re-fetch every cited source and confirm the claim is actually there — not just self-graded by the model that wrote it. Use when an agent's output includes claims attributed to a URL and those claims need to be trustworthy before a human sees them.
+description: General helper.
 license: MIT
 ---
SURVIVEDdelete-instruction:skills/cited/SKILL.md:1:474b72cb
delete line 8: Read [references/methodology.md](references/methodology.md) before wi...
every baseline-green case stayed green
catch-fabricated-citation: 0.4, invalid-claims-file: 1.0, no-urls-no-check: 1.0
diff
--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -11,5 +11,4 @@
 This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
 
-Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
 
 ## When to use this

How this was run

claude plugin eval cited --ablation none --runs 1 --threshold 1.0 --trust-plugin --no-publish --json <out>/result.json --output-dir <out>/eval-output --allow-tools Bash Write WebFetch(domain:www.rfc-editor.org) WebFetch(domain:archive.org)

Mutants from deterministic operators: delete-body, delete-instruction, invert-rule, swap-tool-names, truncate, wrong-fact, drop-description. INCONCLUSIVE never counts as killed. Probe runs are synthetic and judge graders are not called on them.