The fictional brief
A product team reports a new checkout: control had 1,000 assigned users and 100 purchasers; treatment had 1,000 assigned users and 120 purchasers. Each user is counted once over the same seven-day observation period. The team asks whether to release immediately. This constructed scenario is separate from Harbor sales data.
Your task
Calculate both purchase rates, the absolute percentage-point difference and relative lift. List evidence you need before calling the difference a campaign effect. Identify two guardrails and distinguish an observed difference from a guaranteed future improvement.
Worked solution
Control is 100/1,000=10%; treatment is 120/1,000=12%. The absolute difference is 2 percentage points, while relative lift is (12%-10%)/10%=20%. Saying “a 2% relative lift” confuses the measures. These aggregates alone do not establish a valid causal experiment or a rollout decision.
Review the design
Ask how assignment was randomized, whether eligibility and observation windows match, whether users cross groups, and whether assignment or purchase events are missing. Check the planned sample size, stopping rule, primary metric and uncertainty analysis. Equal group counts alone do not prove random assignment or sound instrumentation. Avoid choosing the analysis only after seeing a favorable result.
Communicate a decision
A defensible note is: “The observed purchase rate is 2 percentage points higher in treatment, equivalent to 20% relative lift. I would validate assignment, instrumentation, planned uncertainty analysis and guardrails before recommending rollout.” Candidate guardrails include refunds, checkout errors and latency. The counts contain no cost or margin evidence, so revenue or profit claims are unsupported.
Follow-up and self-review
What changes if treatment has 2,000 assigned users but still 120 purchasers? Its rate becomes 6%, 4 percentage points below control, or -40% relative change. Denominators reverse the conclusion. Score clarity, arithmetic, design checks, uncertainty and communication 0–2 each; explain any score below two in your next practice plan.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.