Your decision meeting
A manager asks, “Can we turn Meridian on next week?” Use the original contract and the previous three labs. Do not assume a real deployed system exists. Deliver a one-page recommendation and a 90-second explanation that separates observed fixture results from unsupported real-world claims.
Write the memo before reading the reference
Use these headings: decision requested; data/availability audit; frozen policy; validation evidence; held-out evidence; workflow checks; missing evidence; proposed next action; monitoring; stop and rollback ownership. Include row counts, metric denominators and cost assumptions. Explain what would change your recommendation.
Reference solution
A defensible recommendation is not to approve autonomous rollout from this evidence. The feasible validation choice is 0.6 with five selections, cost 18 and recall 3/5. The held-out batch selects four of eight, costs 27 and recalls one of four positives. All data are synthetic and tiny; no real capacity, label maturity or operating cost is demonstrated. Propose an agreed shadow-review study with explicit human ownership and a predeclared evaluation plan, if a real team later explores this design. Shadow outputs must not silently drive decisions.
An actionable monitoring proposal
Define batch ID, input count, selected count, missing-score count, score distribution, labelled-positive count and label-arrival coverage. Review lag matters: a low observed error count before labels mature is not evidence of accuracy. Propose “pause evaluation and investigate any unknown schema/duplicate IDs; investigate workload above agreed capacity; review error metrics only on mature labelled cohorts with denominator and confidence limitations.” A qualified operating team must set real stop thresholds and approve any deployment.
Failure rehearsal
An upstream field changes from decimal 0.6 to percentage 60. Your input contract should reject it, record the batch failure and avoid publishing a misleading score. Specify who restores the known input contract, how a corrected batch is rerun without duplication, and what evidence is kept. No production infrastructure is built by this learning exercise.
Honest portfolio statement
“Built a deterministic evaluation workflow for a fictional queue-ranking score fixture: validated inputs, selected a capacity-constrained policy on validation data, reconciled held-out workflow slices and wrote a release review.” Do not claim you trained a model, reduced business loss by a measured percentage, passed an employer assessment or deployed production AI.
Interview follow-ups
Why is a test set not a dashboard for weekly policy tuning? What happens if all policies exceed capacity? Which label delays could make recall misleading? Can a score of 0.8 be called an 80% probability? What does workflow B actually establish? A strong answer names assumptions, makes one checked claim and identifies the next evidence needed.
Reviewer rubric
Score 0–2 each: numeric accuracy, assumptions vs observations, operational ownership, limits and honest communication. Rework any zero. Ask a professor or experienced practitioner to challenge your reasoning. This self-review neither certifies deployment safety nor guarantees career outcomes.
Authoritative reference
scikit-learn developers: review how threshold choice changes the evaluation question. The release memo and original fictional case are DataCareerHub-authored practice.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.