Your assignment
Original fictional Aster study-desk assistant. Write an evaluation contract: corpus snapshot, allowed documents, query population, relevance judgments, answerable/no-answer cases, evidence version, ranking cutoff and review process. Prerequisites: Python dictionaries, ratios and ranked lists. No live assistant, embedding model or external API is built in this phase.
Separate the layers
Retrieval asks whether useful evidence was returned and ranked well. Answer review asks whether emitted claims are supported by applicable evidence. Answerability asks whether the authorised corpus contains sufficient information for the question. A correct citation format, fluent response or high recall does not establish factual support, completeness or usefulness.
Your attempt
A query asks what kind of project data is used AND how results should be checked. The corpus has one document on fictional data and another on checking results. A returned top document covers only the first part. Which metric might look encouraging while the answer remains incomplete? What must the relevance judgment define?
Worked solution
Reciprocal rank can be 1 if the first document is relevant even though the second needed document is missing. Recall over the two known relevant documents would be 1/2 if only one is retrieved. Neither metric alone measures whether a generated answer includes both supported claims. Record required evidence, per-query judgments and answer criteria independently.
Judgment and split contract
Keep development queries for choosing prompts, rankers or thresholds; freeze the configuration before final test inspection. Preserve query IDs, corpus IDs/versions, judgment provenance and reviewer disagreements. A tiny handpicked test is not a representative benchmark. In a real partially judged collection, unjudged does not automatically mean irrelevant; this phase explicitly uses complete judgments within a tiny fictional corpus.
No-answer and permissions review
An unsupported guaranteed-job question has no answer in the teaching corpus. Report no-answer cases separately rather than manufacturing a relevant document. A real system must restrict retrieval before generation to documents the requester may access; quality metrics do not grant access or test authorisation. These local fixtures contain only fictional non-sensitive text.
Failure taxonomy exercise
Classify missing relevant evidence, relevant evidence at poor rank, outdated source, unsupported claim, contradicted claim, unnecessary abstention and source text mistaken for an instruction. More than one failure may occur in a case. Keep original test labels when investigating disagreements; version an adjudicated copy separately.
Review and explain
Score 0–2 for layer separation, query/corpus contract, judgment provenance, split discipline and no-answer/access reasoning. Explain what additional independent review would be needed for a real service. Self-practice is unsaved and awards no credential.
Authoritative reference
Christopher D Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval. Public reference for retrieval concepts; all Iris corpus and exercises are original. Next: Iris 2.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.