The declared metric contract
Use binary relevance and k=3. Precision@3 is relevant hits divided by 3 even when fewer results return. Recall@3 divides hits by all known relevant documents for the query. Reciprocal rank is 1/first relevant rank, or 0 if none. Truncated AP@3 sums precision at relevant ranks through 3 and divides by ALL known relevant documents. This declared convention is not untruncated full-ranking AP.
Small worked fixture to attempt
Known relevant docs are d4,d5. The stored ranked results are d7,d4,d8. Predict all four metrics. No model runs here: these are supplied fictional rankings, allowing you to audit the evaluator independently.
rank | document | relevant
1 | d7 | no
2 | d4 | yes
3 | d8 | no
Known relevant set: d4,d5Worked solution
One hit gives P@3=1/3 and Recall@3=1/2. The first hit is rank 2, so RR=1/2. The only precision contribution is 1/2; divided by 2 known relevant documents, truncated AP@3=1/4. Dividing only by retrieved relevant count would overstate the metric. Duplicating d4 cannot earn another hit; reject duplicate document IDs in a ranked list.
Development selection
The offline fixture has two stored runs. Development baseline macro AP@3=2/3; candidate macro AP@3=1. Select candidate using development only, tie by run name, then freeze before test. An empty ranked list remains a valid zero-retrieval result; unknown IDs or missing query entries are input failures, not silent dropped cases.
Frozen test solution
Candidate has two answerable test queries. The first has P 1/3,R 1/2,RR 1/2,AP 1/4; the second P 1/3,R 1,RR 1,AP 1. Macro over these answerable queries is P 1/3,R 3/4,RR 3/4,AP 5/8. Report the query count and per-query failures. Do not tune the candidate again after seeing test losses while still calling this an untouched holdout.
No-answer contract
A third test query has no relevant document in this corpus. Recall/AP are undefined under this contract; RR 0 and P@3=0 if no hit. Keep it separate from answerable macro metrics. The fact that a retriever returns three documents does not mean an answer was emitted or that those documents support an answer.
Mutation exercise
Move a relevant item earlier and predict which metrics change. Shorten a list to one relevant result: P@3 stays 1/3. Change only test rankings: the development-based choice must not change. Add a second copy of the same doc ID and confirm validation fails. Do not add test judgments to ranker training to improve reported test results.
Interview defense and rubric
Score 0–2 for metric definitions, input validation, per-query arithmetic, split discipline and no-answer reporting. Explain why aggregate retrieval quality cannot certify answer grounding or production readiness. The supporter pack includes both frozen runs and a checked reference implementation.
Authoritative reference
Christopher D Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval: ranked-retrieval evaluation. Our cutoff/AP denominator convention is explicitly defined. Next: Iris 3.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.