3. Evaluate retrieval and unsupported questions

Objective & prerequisites

Use relevance labels, ranking metrics and separate abstention cases. Prerequisites: Python functions, JSON, lists and the existing API/testing foundations. Extract the full kit and open a terminal in applied-ai-kit. The core lab uses Python 3.10+ with no third-party dependencies or network. Actual new-text semantic inference is a separate optional local setup, tested on Python 3.12/macOS arm64. All CedarLab policies, invoices and review cases are fictional teaching material; they are not DataCareerHub service terms. No payment, registration, live chatbot or visitor-data collection is activated. These free companions do not change course progress or protected access.

python3 lab.py
python3 -m unittest discover -s tests -v

Understand the concept

An evaluation set states which passages are relevant to which queries. A hit at k means at least one labelled relevant passage appears in the first k results. Reciprocal rank is 1 divided by the first relevant position; mean reciprocal rank averages this across queries. These fixture queries each have one relevant passage, so the report calls its metric hit rate rather than implying general multi-label recall. Keep development examples separate from test examples, freeze relevance labels and avoid tuning thresholds repeatedly against the same test labels. Seven authored questions cannot estimate performance for a real customer population. Include unsupported questions separately: a nearest-neighbor search always returns something when candidates exist, even when none answers the question.

Worked example

Three development and three answerable test questions are supplied. At k=3, both methods hit all three relevant test passages; semantic MRR@3 is 1.000 and keyword MRR@3 is about 0.611. Thus their hit rates match while ranking quality differs on this fixture. Q7, the breakfast question, has no relevant source and is excluded from answerable-query ranking metrics. Semantic retrieval still returns D6 first. That is not permission to answer. No score threshold was validated for automatic abstention; the reference reports this gap instead of inventing one.

from lab import load,fixture_rank,retrieval_metrics
qs=[q for q in load("queries.json") if q["split"]=="test" and q["answerable"]]
ranks={q["id"]:[r["id"] for r in fixture_rank(q["id"])] for q in qs}
assert retrieval_metrics(ranks,qs)["mrr_at_k"]==1.0

Try it yourself

Calculate reciprocal ranks from the saved rankings by hand. Explain why reporting only hit rate hides differences. Add one new question to a separate evaluation copy, label its relevant sources before measuring and document whether it is development or a fresh test. Unknown fixture IDs must fail until actual embeddings are generated. Write your prediction, inspect the source and record actual results. Keep a copy of the original fixtures before making changes.

Knowledge check

Can a retriever returning three high-scoring results guarantee the question is answerable?

Evidence, mistakes & handover

Record the executed mode: actual local encoder, saved-vector replay, deterministic extraction or human factual review. They establish different things. Keep model/version, source hashes, commands, expected outcomes, failures and limitations with your work. Never turn a schema pass, cosine score or quotation into a claim of proven truth. The download contains a worksheet, independent controls, worked solutions, a factual-review rubric, privacy/cost contract and mini-project checklist. Credit the supplied starting point and describe your own contribution.

Authoritative reference

Read the official reference ↗

— official source for the underlying mechanism. Teaching prose and CedarLab cases are original. Model/runtime compatibility, licensing and operating limits require review for any later deployment; this course does not activate a production AI service.