The review meeting
A teammate says,“The candidate wins on development, so the assistant is ready.” Prepare a one-page review: corpus/judgment versions, development choice, frozen test, failure slices, answer evidence, abstention tradeoff and remaining evidence. No real assistant is deployed by this phase.
Write before reading the reference
Separate the ranking experiment from the human-labelled answer audit. What was actually computed, and what was supplied? Which findings can be reproduced locally, which require human judgment, and which production requirements were not tested? Do not merge these two fixtures into an invented end-to-end service result.
Reference solution
The retrieval candidate wins the stated development comparison, but the frozen test misses a second required evidence item. The separate answer fixture includes outdated evidence, unsupported claims and unnecessary abstention. These observations justify targeted improvements and a new evaluation plan, not a claim of a trustworthy production RAG system. No model training, live API latency, cost, user outcomes or access-isolation evaluation was performed.
An evidence table
Keep per-query/case failures beside aggregates. State where judgments came from and how benchmark changes are versioned. Never replace denominators and failed cases with a single quality badge.
evaluation | result | limit
retrieval development | candidate AP@3 1 | two authored queries
retrieval frozen answerable test | AP@3 5/8 | two queries, one multi-evidence miss
separate answer audit | support 1/3 claims; fully supported 0/2 answers | supplied human labels
abstention audit | 1/2 answer coverage | missed useful answer and no-answer mistakePlan the next evidence
Propose representative multi-hop, no-answer, stale-source, permission-boundary and adversarial cases with documented reviewers. For a real application, validate authorisation before retrieval and keep untrusted source instructions separate from application instructions. These are review tasks, not guarantees provided by this fixture. If a configuration changes after test inspection, use fresh final evidence.
Failure rehearsal
A source page is replaced, a citation points to the wrong version, or a query set changes. Preserve the evaluated snapshot/IDs and rerun appropriate regression checks with an explicit version. Do not fetch arbitrary live links from an unreviewed corpus as part of this offline exercise. No confidential user data is necessary for these projects.
Honest portfolio statement
“Built deterministic evaluators for supplied fictional retrieval runs and a reviewer-labelled answer-evidence fixture, with metric contracts, no-answer slices and abstention analysis.” Do not claim you trained embeddings, built production RAG, automated semantic fact checking, achieved NIST compliance or delivered measured business improvements.
Rubric and interview defense
Score 0–2 for component separation, reproducible metrics, judgment/version provenance, failure-oriented next steps and truthful limits. Answer: why can MRR 1 miss needed evidence; why can citation presence mislead; why does all-abstain not earn 100%; which decisions used development rather than final test?
Authoritative reference
Christopher D Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval. Public references remain public; supporter value is the authored offline workflow and additional review material, not exclusive access to a public book.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.