Can your eval fail?
cited-evals · adapter claude-plugin-eval · litmus 0.3.0 · 2026-10-09T17:16:40+00:00
0%mutation score: 0 of 8 mutants, 1 run per case
0killed
8survived
0inconclusive (not killed)
0vacuous graders + null-green cases
2 of 3cases green at baseline (only these can kill)
Baseline (unmutated copy)
| case | score | counts toward kills |
|---|---|---|
| catch-fabricated-citation | 0.5 | no, not green at baseline |
| invalid-claims-file | 1.0 | yes |
| no-urls-no-check | 1.0 | yes |
Graders and cases on do-nothing runs
| status | case / grader | type | why |
|---|---|---|---|
| DISCRIMINATES | catch-fabricated-citation / disclosed | regex | |
| DISCRIMINATES | catch-fabricated-citation / ran-cited | tool_used | |
| DISCRIMINATES | catch-fabricated-citation / skill-fired | tool_used | |
| UNKNOWN | catch-fabricated-citation / verdicts | llm | llm grader needs a judge model |
| DISCRIMINATES | invalid-claims-file / ran-cited | tool_used | |
| UNKNOWN | invalid-claims-file / report | llm | llm grader needs a judge model |
| UNKNOWN | no-urls-no-check / rewrite | llm | llm grader needs a judge model |
| NULL_PASS | no-urls-no-check / skill-not-fired | tool_used | NULL_PASS (guard) passes on probe(s): silent, done, refusal, echo |
| OK | catch-fabricated-citation | case [ablation-none] | silent: RED, done: RED, refusal: RED, echo: RED |
| OK | catch-fabricated-citation | case [with-without] | silent: RED, done: RED, refusal: RED, echo: RED |
| OK | invalid-claims-file | case [ablation-none] | silent: RED, done: RED, refusal: RED, echo: RED |
| OK | invalid-claims-file | case [with-without] | silent: RED, done: RED, refusal: RED, echo: RED |
| JUDGE_ONLY | no-urls-no-check | case [ablation-none] | silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN |
| JUDGE_ONLY | no-urls-no-check | case [with-without] | silent: UNKNOWN, done: UNKNOWN, refusal: UNKNOWN, echo: UNKNOWN |
Mutants
| verdict | mutant | detail |
|---|---|---|
| SURVIVED | delete-body:skills/cited/SKILL.md:0:47cee39cempty the body; keep only the frontmatter | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -5,35 +5,2 @@
---
-# cited
-
-The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention.
-
-This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
-
-Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
-
-## When to use this
-
-Any time an agent is about to hand a human a document, list, or report where individual line items are attributed to a specific source (a URL) and the human is expected to trust the attribution without independently re-checking it themselves. Examples: a lead-gen report ("this company posted this job"), a research digest ("this podcast guest said this number"), a competitive battlecard ("competitor X does Y, per their pricing page"), a citation-heavy summary.
-
-Do **not** reach for this to fact-check claims that don't cite a specific URL — it verifies containment (is the claim on that page), not truth (is the page itself honest), and it has nothing to check without a `source_url`.
-
-## Workflow
-
-1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag. If `HTTPS_PROXY` is set, as it is in Claude Code's sandboxed Bash, add `--proxy-from-env`: the sandbox has no direct route out, so without it every source fails to resolve and comes back `broken`. Do not read those `broken` results as dead links.
-3. Read the tier written back onto each entry:
- - `verified` — substantially quoted from the live (or archived) page. Ship it.
- - `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
- - `snippet_only` — the platform blocked automated fetching (Reddit, X, LinkedIn, etc.) and no archived copy exists. Your call whether the original discovery snippet is trustworthy enough to keep; disclose the unverified status if you do.
- - `unsupported` — the page loaded fine and the claim is **not on it**, or the claim's numbers or capitalized names (an amount, an investor, a product) are not on it. This is the fabrication signal. Drop the claim or find a claim the page actually supports.
- - `broken` — the source URL doesn't resolve, even via the Wayback Machine fallback. Drop it.
- - `unverified` — the claim was never checked: the page was a PDF or other binary, too thin to read (a JavaScript app, a consent wall), too large to read in full, or missing from an `--offline` cache. Check it by hand, or cite an HTML page that carries the same fact.
-4. A non-zero exit means do not ship as is; `cited --help` lists what each code means. Wire that into a CI gate or a pre-ship check if your artifact pipeline has one. A run that fails only on `low_match` is waiting on human review, not proven wrong.
-5. On the shipped artifact, disclose the result: "N of M claims verified against source." That single line is what turns a report into something a skeptical reader can trust without redoing your research.
-
-## What this does and doesn't prove
-
-It proves a claim's text is (or isn't) present on the page it cites, checked independently of whatever model wrote the claim. It does **not** prove the page itself is honest, that a correctly-quoted line means what your artifact says it means, or that the page hasn't changed since — every checked entry is stamped `verified_at` so staleness is visible later, not hidden.
-
-See [references/methodology.md](references/methodology.md) for the full tiering rules, the Wayback Machine fallback behavior, and how bot-walled platforms (Reddit, X, LinkedIn) are handled without either failing the run or pretending they were checked.
|
| SURVIVED | delete-instruction:skills/cited/SKILL.md:0:387d9f1bdelete line 6: This skill is a mechanism, not a vertical: it takes any list of `(cla... | every baseline-green case stayed green catch-fabricated-citation: 0.8333333333333334, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md +++ b/skills/cited/SKILL.md @@ -9,5 +9,4 @@ The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention. -This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human. Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits. |
| SURVIVED | invert-rule:skills/cited/SKILL.md:0:a51a449bline 4: 'cannot' -> 'can' in: The core failure mode this defends against: an agent writes a plausib... | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md +++ b/skills/cited/SKILL.md @@ -7,5 +7,5 @@ # cited -The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention. +The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading can catch self-invention. This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human. |
| SURVIVED | swap-tool-names:skills/cited/SKILL.md:0:b39811f3swap tool names Read <-> Bash | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -11,5 +11,5 @@
This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human.
-Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
+Bash [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits.
## When to use this
@@ -22,6 +22,6 @@
1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag. If `HTTPS_PROXY` is set, as it is in Claude Code's sandboxed Bash, add `--proxy-from-env`: the sandbox has no direct route out, so without it every source fails to resolve and comes back `broken`. Do not read those `broken` results as dead links.
-3. Read the tier written back onto each entry:
+2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Read tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag. If `HTTPS_PROXY` is set, as it is in Claude Code's sandboxed Read, add `--proxy-from-env`: the sandbox has no direct route out, so without it every source fails to resolve and comes back `broken`. Do not read those `broken` results as dead links.
+3. Bash the tier written back onto each entry:
- `verified` — substantially quoted from the live (or archived) page. Ship it.
- `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
|
| SURVIVED | truncate:skills/cited/SKILL.md:0:8031024ekeep the first 17 of 35 body lines | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md
+++ b/skills/cited/SKILL.md
@@ -21,19 +21,2 @@
## Workflow
-1. Produce your artifact's claims as a flat JSON array: `[{"id": "...", "claim": "...", "source_url": "..."}, ...]`. `id` is whatever your artifact uses to key this claim back to its full record (a lead's name, a section heading — anything stable).
-2. Run `cited claims.json --annotate-out verified.json`. The `cited` command is on the Bash tool's PATH while the plugin is enabled; without the plugin, run `python3 scripts/verify_claims.py` from this skill directory with the same arguments. Add `--json` for a machine-readable report on stdout, and `cited --help` lists every flag. If `HTTPS_PROXY` is set, as it is in Claude Code's sandboxed Bash, add `--proxy-from-env`: the sandbox has no direct route out, so without it every source fails to resolve and comes back `broken`. Do not read those `broken` results as dead links.
-3. Read the tier written back onto each entry:
- - `verified` — substantially quoted from the live (or archived) page. Ship it.
- - `low_match` — the claim shares vocabulary with the page but is not quoted from it. This is not evidence the claim is on the page: a made-up claim about a real company also shares the company's name with its site. Do not ship it as is. A human must check the claim against the page, or you tighten the claim to wording the page actually contains and re-run.
- - `snippet_only` — the platform blocked automated fetching (Reddit, X, LinkedIn, etc.) and no archived copy exists. Your call whether the original discovery snippet is trustworthy enough to keep; disclose the unverified status if you do.
- - `unsupported` — the page loaded fine and the claim is **not on it**, or the claim's numbers or capitalized names (an amount, an investor, a product) are not on it. This is the fabrication signal. Drop the claim or find a claim the page actually supports.
- - `broken` — the source URL doesn't resolve, even via the Wayback Machine fallback. Drop it.
- - `unverified` — the claim was never checked: the page was a PDF or other binary, too thin to read (a JavaScript app, a consent wall), too large to read in full, or missing from an `--offline` cache. Check it by hand, or cite an HTML page that carries the same fact.
-4. A non-zero exit means do not ship as is; `cited --help` lists what each code means. Wire that into a CI gate or a pre-ship check if your artifact pipeline has one. A run that fails only on `low_match` is waiting on human review, not proven wrong.
-5. On the shipped artifact, disclose the result: "N of M claims verified against source." That single line is what turns a report into something a skeptical reader can trust without redoing your research.
-
-## What this does and doesn't prove
-
-It proves a claim's text is (or isn't) present on the page it cites, checked independently of whatever model wrote the claim. It does **not** prove the page itself is honest, that a correctly-quoted line means what your artifact says it means, or that the page hasn't changed since — every checked entry is stamped `verified_at` so staleness is visible later, not hidden.
-
-See [references/methodology.md](references/methodology.md) for the full tiering rules, the Wayback Machine fallback behavior, and how bot-walled platforms (Reddit, X, LinkedIn) are handled without either failing the run or pretending they were checked.
|
| SURVIVED | wrong-fact:skills/cited/SKILL.md:0:b187c833line 6: 'before' -> 'after' in: This skill is a mechanism, not a vertical: it takes any list of `(cla... | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md +++ b/skills/cited/SKILL.md @@ -9,5 +9,5 @@ The core failure mode this defends against: an agent writes a plausible-sounding claim, attributes it to a source it read (or half-read, or invented from a search snippet), and grades its own confidence — using the same model that might have made the claim up. Self-grading cannot catch self-invention. -This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human. +This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — after the artifact carrying those claims ships to a human. Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits. |
| SURVIVED | drop-description:skills/cited/SKILL.md:0:c09e3832replace the frontmatter description with 'General helper.' (tests triggering) | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md +++ b/skills/cited/SKILL.md @@ -1,5 +1,5 @@ --- name: cited -description: Before an agent ships any artifact that cites sources (a report, a lead list, a research summary, a changelog claim), re-fetch every cited source and confirm the claim is actually there — not just self-graded by the model that wrote it. Use when an agent's output includes claims attributed to a URL and those claims need to be trustworthy before a human sees them. +description: General helper. license: MIT --- |
| SURVIVED | delete-instruction:skills/cited/SKILL.md:1:5dbd49a5delete line 8: Read [references/methodology.md](references/methodology.md) before wi... | every baseline-green case stayed green catch-fabricated-citation: 0.5, invalid-claims-file: 1.0, no-urls-no-check: 1.0 diff--- a/skills/cited/SKILL.md +++ b/skills/cited/SKILL.md @@ -11,5 +11,4 @@ This skill is a mechanism, not a vertical: it takes any list of `(claim, source_url)` pairs and re-fetches each source independently, checking whether the claim is actually contained in the page — before the artifact carrying those claims ships to a human. -Read [references/methodology.md](references/methodology.md) before wiring this into your own artifact type — it explains the tiering logic and where this mechanism is honest about its limits. ## When to use this |
How this was run
claude plugin eval cited-evals --ablation none --runs 1 --threshold 1.0 --trust-plugin --no-publish --json <out>/result.json --output-dir <out>/eval-output --allow-tools Bash Write WebFetch(domain:www.rfc-editor.org) WebFetch(domain:archive.org) WebFetch(domain:*.archive.org)
Mutants from deterministic operators: delete-body, delete-instruction, invert-rule, swap-tool-names, truncate, wrong-fact, drop-description. INCONCLUSIVE never counts as killed. Probe runs are synthetic and judge graders are not called on them.