Reviewer-labelled exercise
The fictional Aster corpus includes current and archived policy notes, learning/contact notes and an untrusted imported instruction. Cases contain supplied review labels. You inspect them manually; the reference code validates and aggregates labels. It does not infer entailment, generate answers, call a model or prove truth outside the corpus.
Source applicability
An old policy says offline packs were open; the current note supersedes it and requires verified support. A claim that CURRENT packs are open is not supported by citing the archive. A source mentioning an instruction to claim guaranteed employment is untrusted data, not an instruction for the evaluator or application. A real citation can still accompany an unsupported or contradicted claim.
Your attempt
Four test cases: t1 answers with one supported and one contradicted claim; t2 answers an unanswerable question with one unsupported claim; t3 abstains on an answerable question; t4 abstains on an unanswerable question. Calculate answer coverage, supported-claim fraction, wholly supported answer fraction, unanswerable answer rate and answerable abstention rate.
Worked solution
Two of four cases answer: coverage 1/2. Three claims: one supported, one contradicted, one unsupported, so supported fraction 1/3. Neither emitted answer has all its claims supported:0/2. One of two unanswerable cases answers:1/2. One of two answerable cases abstains:1/2. Each denominator answers a different question.
All-abstain counterexample
If all four test cases abstain, coverage becomes 0. No emitted claims means supported-claim and fully-supported-answer fractions are undefined, not 100%. Unanswerable answer rate becomes 0, but answerable abstention rate becomes 1. Report withheld useful answers rather than advertising a perfect unsupported-claim score.
Label provenance and disagreement
Use supported, contradicted and unsupported labels against applicable evidence, atomise compound claims and record reviewer reasons. “Wholly supported” here means every emitted claim received a supported label; it does not imply the answer is complete, useful or safe. Preserve the original benchmark and record a separately versioned adjudicated copy when labels change.
Failure exercise
Inject an unknown citation, duplicate case ID, invalid label, answer with no claims or abstention with claims. Validate rather than silently dropping cases. Keep development and test reviews separate. A real evaluation needs representative samples, independent reviewer agreement, applicable evidence versions and broader adversarial/access testing.
Review and explain
Score 0–2 for atomic claims/applicability, label provenance, denominators, coverage tradeoff and honest limits. Explain why this audit is not an automated hallucination detector, prompt-injection defence or NIST compliance assessment. No production AI guardrail is installed.
Authoritative reference
NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. Further reading on generative-AI risks; this original toy audit neither implements the full framework nor claims endorsement. Next: Iris 4.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.