Know the execution mode. The pretrained encoder was run locally. The small download replays saved vectors for supplied queries. Extraction is rule-based; support labels are authored human judgments. No live chatbot, paid provider, payment or registration is activated.
01 / Semantic retrieval, actually executed
384 dimensions · CPU inference
MiniLM maps short English text into learned vectors. For “How long can a learner get the exercise files?”, semantic search ranks the relevant download-duration passage D1 first. The keyword baseline ranks D6 first.
Pinned model revision and source hashes included. A new credit-transfer paraphrase was also encoded and ranked D3 first. These are tiny fixture results, not a general accuracy guarantee.
02 / Ranking is not answerability
Answerable test set · only three questions · k = 3| Method | Hit rate | MRR |
|---|
| Semantic | 3/3 | 1.000 |
|---|
| Keyword | 3/3 | 0.611 |
|---|
Both find relevant evidence within three results. Their ordering differs. Unsupported Q7 still returns candidates; no score threshold has been validated to decide whether an answer is supported.
03 / Structured does not mean correct
A candidate amount of 125400 is a valid integer in a valid schema. It is still wrong for I1. Source agreement is a separate check.
04 / A real quote can accompany a false claim
Claim: “Downloads last thirty days.”
Citation: D1, a current document.
Located quote: “practice downloads stay available for fourteen days after enrolment”.
Review numbers, negation, conditions and the whole claim. The quote locator returns human_review_required, never “verified true”.
The imperfect hand-authored review example labels 6/8 cases correctly, but only 1/3 of its “supported” judgments is correct. These are reviewer-fixture metrics, not LLM accuracy.
05 / Keep the operating boundary visible
First recorded encoding: about 1.81 seconds for fifteen short texts including setup and hash validation, on one local CPU run. Replay timing is not inference timing. No p95, hosted cost or privacy guarantee is inferred.
The model card declares Apache-2.0; attribution, standard license text, pinned hashes and runtime inventory accompany the kit. Retrieved instructions grant no tool authority. No production vector database, generative extraction, automatic entailment judge or hosted adapter is implemented.
06 / Make the reasoning yours
Complete the lessons, label claims before reading the answer key and add one independently checked scenario per boundary. Mark optional model work unexecuted if you cannot run it. Keep the original fixtures and identify your own contribution.
Read all nine lessons · Compare worked solutions