6. Review factual support claim by claim

Objective & prerequisites

Distinguish supported, contradicted and insufficient claims with a reproducible rubric. Prerequisites: Python functions, JSON, lists and the existing API/testing foundations. Extract the full kit and open a terminal in applied-ai-kit. The core lab uses Python 3.10+ with no third-party dependencies or network. Actual new-text semantic inference is a separate optional local setup, tested on Python 3.12/macOS arm64. All CedarLab policies, invoices and review cases are fictional teaching material; they are not DataCareerHub service terms. No payment, registration, live chatbot or visitor-data collection is activated. These free companions do not change course progress or protected access.

python3 lab.py
python3 -m unittest discover -s tests -v

Understand the concept

Factual support is a relation between a particular claim and eligible evidence. Split compound claims into atomic statements, locate the relevant passage and compare entities, numbers, negation, time, conditions and quantifiers. Use supported only when the evidence establishes the claim; contradicted when it conflicts; insufficient when evidence is missing, outdated or cannot establish the whole assertion. The exact-quote locator checks whether text occurs in a current cited source. It intentionally returns human_review_required rather than a semantic verdict. A real quote can accompany a contradicted interpretation. Human reference labels are authored teaching judgments, not an unbiased model benchmark. The FActScore paper provides background on atomic factual evaluation; this three-label teaching rubric is not an implementation or validation of FActScore. For a real evaluation, define reviewer training, independent labels, disagreement handling and provenance.

Worked example

C1’s fourteen-day claim is supported by D1. C2 cites the same current document and quotes its true fourteen-day sentence, but claims thirty days: contradicted. C3’s job-offer promise has no evidence in a project-report passage. C4 adds guaranteed renewal to a source requiring review: insufficient for the compound claim. C5 reverses a negation; C6 contradicts the privacy instruction. C7 invents a quote and C8 cites a retired draft. The supplied imperfect human-review example gets 6/8 labels right, but only one of its three supported predictions is correct: supported precision is 1/3. This is not an LLM accuracy score.

from lab import load,evidence_checks,evaluate_reviews
case=load("claims.json")[1]
assert evidence_checks(case,load("corpus.json"))["quote_found"]
assert case["gold"]=="contradicted"
r=evaluate_reviews(load("claims.json"),load("reviews-example.json"))
assert r["accuracy"]==0.75
assert r["supported_precision"]==1/3

Try it yourself

Copy reviews-blank.json and label all eight cases before looking at gold labels/reasons. For each, write an atomic claim, current evidence quote and reason. Compare your labels with claims.json, investigate disagreements, then run evaluate_reviews. Explain why overall accuracy can look acceptable while unsupported claims are incorrectly approved. Write your prediction, inspect the source and record actual results. Keep a copy of the original fixtures before making changes.

Knowledge check

Can an exact quotation in a real current citation prove that an accompanying paraphrase preserves its meaning?

Evidence, mistakes & handover

Record the executed mode: actual local encoder, saved-vector replay, deterministic extraction or human factual review. They establish different things. Keep model/version, source hashes, commands, expected outcomes, failures and limitations with your work. Never turn a schema pass, cosine score or quotation into a claim of proven truth. The download contains a worksheet, independent controls, worked solutions, a factual-review rubric, privacy/cost contract and mini-project checklist. Credit the supplied starting point and describe your own contribution.

Authoritative reference

Read the official reference ↗

— official source for the underlying mechanism. Teaching prose and CedarLab cases are original. Model/runtime compatibility, licensing and operating limits require review for any later deployment; this course does not activate a production AI service.