The decision contract
Use ten fictional validation cases below. A case is selected when score >= threshold, including equality. Candidate thresholds: 0.4, 0.6, 0.8. The team can review at most five cases in this batch. FP costs 2 illustrative units; FN costs 7; TP/TN cost zero for this simplified exercise. The numbers are teaching assumptions, not measured business costs. A score is not assumed to be a calibrated probability.
Validation fixture
Treat label 1 as a case that needed review. Keep all ten rows. No supporter download is needed to complete this on-page activity.
case_id,score,label
v01,0.95,1
v02,0.85,0
v03,0.80,1
v04,0.70,0
v05,0.60,1
v06,0.50,1
v07,0.40,0
v08,0.30,0
v09,0.20,1
v10,0.10,0Your task
Build a confusion table for each threshold: TP, FP, FN, TN, selected count, precision, recall and cost. Reject over-capacity candidates before choosing minimum cost. Break equal cost by fewer selections, then higher threshold. Write down the chosen threshold before opening Meridian 3. Do not use its held-out outcomes to revise this selection.
Metric contract
Precision = TP/(TP+FP); recall = TP/(TP+FN); cost = 2*FP + 7*FN. Return undefined when a denominator is zero, rather than interpreting zero as an observed rate. Check TP+FP+FN+TN=10. Capacity is an absolute count in this batch, not a guaranteed future rate.
Worked solution
Threshold 0.4 is cheapest at 13 units but selects seven, exceeding capacity. Of the feasible policies, 0.6 costs 18 and selects five; 0.8 costs 23 and selects three. Choose 0.6 under this contract. Its precision and recall are both 3/5 = 0.60. This is not a universal optimal threshold.
threshold | TP FP FN TN | selected | cost | feasible
0.4 | 4 3 1 2 | 7 | 13 | no
0.6 | 3 2 2 3 | 5 | 18 | yes
0.8 | 2 1 3 4 | 3 | 23 | yesSensitivity exercise
Keep the original policy record. Separately change FN cost from 7 to 1, leaving FP cost and capacity unchanged. Recompute validation costs. Explain why stakeholders must agree costs before policy selection, rather than choosing costs to justify a preferred result.
Sensitivity solution
New costs are 7, 6 and 5 at thresholds 0.4, 0.6 and 0.8. The first remains infeasible, so 0.8 is selected under the changed assumption. Different assumptions can justify different actions; publish which contract produced which choice. If every candidate violates capacity, report no feasible policy rather than silently increasing capacity.
Interview defense
Explain why lower threshold can raise recall while adding false positives. Do not claim precision must always rise monotonically with threshold; it depends on which labelled cases leave the selected set. Discuss a future overflow queue, tie handling and abstention instead of claiming a static threshold guarantees workload.
Authoritative reference
scikit-learn developers: Precision-Recall. Read the metric definitions and threshold examples. Next: Meridian 3, evaluate the frozen choice.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.