Freeze before opening the test data
Complete Meridian 2 first and record threshold 0.6 from its original capacity/cost contract. The following eight cases are a separate held-out batch. Evaluate the frozen policy once. If you retune after looking at these labels, this set has been consumed for selection; it can no longer serve as untouched final evidence for the revised policy.
Held-out fixture
A and B denote fictional operating workflows, not demographic groups. Labels are complete only by teaching assumption. Capacity is still five per batch, now of eight cases.
case_id,workflow,score,label
t01,A,0.90,1
t02,A,0.75,0
t03,B,0.65,0
t04,B,0.60,0
t05,A,0.55,1
t06,B,0.35,1
t07,A,0.20,0
t08,B,0.10,1Your task
Compute all four confusion cells, precision, recall, false-negative rate, selected count and cost (FP=2, FN=7). Repeat by workflow. Reconcile every cell across slices to the aggregate and state denominators before percentages. Compare descriptively with validation; do not report a significance test or claim population performance from these tiny fixtures.
Worked solution
At threshold 0.6, equality selects t04. Four of eight are selected. Total TP=1, FP=3, FN=3, TN=1, so precision=1/4 and recall=1/4. Cost is 3*2+3*7=27. A has one of every cell; B has zero TP, two FP, two FN and zero TN. Slice cells sum to the aggregate.
slice | n | TP FP FN TN | precision | recall | FNR | cost
A | 4 | 1 1 1 1 | 1/2 | 1/2 | 1/2 | 9
B | 4 | 0 2 2 0 | 0/2 | 0/2 | 2/2 | 18
ALL | 8 | 1 3 3 1 | 1/4 | 1/4 | 3/4 | 27Interpret the evidence
The sample shows lower held-out precision and recall than validation and missed positives in both workflows. It does not establish a stable workflow difference, causal mechanism, demographic bias or generalisation to a real service. Investigate label consistency, feature timing, case mix and sample construction. Keep additional policy development separate from a new untouched final evaluation.
Failure injection
Remove all labelled positives from a copied slice. Recall and FNR become undefined because their denominators vanish; precision can remain defined if selections remain. Set all scores below threshold: precision is undefined. Duplicate a case ID: reject before counting. A metric function that returns 0 for every undefined quantity hides missing evidence.
Portfolio evidence and rubric
Deliver an aggregate table, slice table, reconciliation assertion, frozen-policy note and limitations paragraph. Score 0–2 each for split discipline, equality handling, denominator reporting, cell reconciliation and cautious interpretation. The supporter offline project adds typed validation and mutation tests; this free lab is complete on-page.
Authoritative reference
scikit-learn developers: keep test evidence out of model/policy selection. Next: Meridian 4, defend a release recommendation.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.