Scenario and prerequisites
Original fictional queue-review case. You know basic Python tables, binary classification and training/validation/test splits. A ticket is scored at its creation time. The target is whether specialist review was needed within seven days. No real model is trained here. Deliver a feature-availability ledger and a corrected evaluation plan, not a claim that the model is accurate.
Your evidence
A colleague proposes these features: channel_at_creation; previous_30_day_tickets calculated strictly before creation; reviewer_resolution entered after the case closes; seven_day_followups counted through day seven; customer_id. The notebook fits an imputer on all rows, randomly splits ticket rows, tries 30 thresholds against test labels, and reports the highest test recall. Several customers occur in both splits.
Attempt before the diagnosis
For each feature, record when it exists, whether it is available at scoring time, and the evidence you would request. Identify three selection/processing problems beyond future-derived fields. Decide whether random ticket splits match forecasting future tickets for existing customers or generalising to unseen customers. These are different evaluation questions.
Worked diagnosis
Keep channel_at_creation only after verifying capture timing. Previous_30_day_tickets is usable only with a strict as-of boundary and reproducible point-in-time join. Reject reviewer_resolution and seven_day_followups: they use post-score information related to the outcome. customer_id is an identifier, not automatically a valid generalisable predictor. Repeated customers may be appropriate for an existing-customer temporal scenario but require a group-isolated evaluation when the claim is unseen-customer performance. Fit imputation on training only, transform validation/test with the fitted object, select policy on validation, then evaluate once on an untouched final test. A Pipeline helps preprocessing discipline but cannot repair future features or a wrong split.
Build the availability ledger
Use one row per candidate feature. Include owner, raw timestamp, transformation cutoff, backfill risk, historical reproducibility and decision. A column named created_at is insufficient proof; compare the timestamp used in the feature join with the actual scoring deadline.
feature | available_at | cutoff_rule | historical_source | keep/reject | reason
channel_at_creation | creation | captured before score | event snapshot | conditional keep | verify
reviewer_resolution | closure | after score | final ticket row | reject | outcome leakageCounterexample and follow-up
A feature may be available today but unavailable in historical snapshots because a source was overwritten. Ask how the team reconstructs the value that would have been known then. A high shuffled-label score is a warning signal, not a complete leakage detector; no single smoke test proves absence of leakage.
Reviewer rubric
Score 0–2 each: feature timing, point-in-time joins, group/time split matched to the claim, training-only transforms, validation/test separation. Two requires a reason and an evidence request. Explain the corrected plan in 90 seconds. This is unsaved self-practice, not a certification.
Authoritative reference
scikit-learn developers: common pitfalls and recommended practices. Read the data-leakage and inconsistent-preprocessing sections; the project explanations above are original. Next: Meridian 2, choose a policy under capacity.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.