Objective & prerequisites
Define assignment, intention-to-treat analysis, outcomes, guardrails and stopping rules. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.
python3 analysis.py
python3 -m unittest discover -s tests -vUnderstand the concept
Random assignment supports a causal comparison when assignment is implemented correctly and the design’s assumptions hold. Define an eligible population, a stable randomization unit and the outcome before rollout. The lab unit is a user, assigned once to control or treatment in a planned 50:50 split. Analysis follows original assignment, including users who do not see the feature: this is an intention-to-treat comparison. Conditioning on post-assignment exposure can break comparability. Define a seven-day binary completion outcome, an error-user guardrail, a fixed observation horizon and a rule for missing telemetry. Repeated sessions are not independent randomized units. Shared-account or social spillovers may require cluster randomization and cluster-aware inference instead. Do not run a live experiment on visitors as part of this exercise.
Worked example
The kit uses a reproducible balanced shuffle to place 200 fictional IDs into two groups of 100. Outcomes are then deliberately constructed to illustrate 20 versus 28 completions; they are not measurements from a real randomized trial and do not demonstrate an actual causal effect. The assignment file has one row per user; the outcome file has one observed seven-day row per assigned user. Exposed counts are 90 and 80. Dropping non-exposed users would produce 20/90 versus 28/80 and change the estimand after treatment. A missing outcome is refused, not assumed to mean no completion.
from analysis import read_csv, experiment
r=experiment(read_csv("assignments.csv"),read_csv("outcomes.csv"))
assert r["arms"]["control"]["assigned"]==100
assert r["arms"]["treatment"]["assigned"]==100Try it yourself
Complete experiment-plan-template.txt before opening the result. State the assignment unit, primary outcome, guardrail, completeness rule, sample-size assumptions, fixed end and decision criteria. Calculate both assigned-user rates and exposed-only rates, then explain why the latter is not the primary causal estimate. Identify how you would detect crossovers or duplicate assignment. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.
Worked solution
The primary rates are 20/100=20% and 28/100=28%. Exposed-only rates are 22.22% and 35%; they select different post-assignment subsets. A duplicate user assignment, including a different arm for that user, fails validation. Before interpretation, check the planned 50:50 sample ratio, full seven-day follow-up and one observed outcome per assigned user. In real systems a validated zero from complete telemetry differs from an absent outcome row. The fixture asserts observation and does not independently prove a real tracking system’s completeness.
Knowledge check
Does randomly shuffling a file of manufactured outcomes establish that a product change works?
Check and explain
No. This is a transparent simulation for learning the analysis. Only an appropriately implemented and observed experiment can supply real causal evidence.
Evidence, mistakes & handover
Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.
Authoritative reference
— reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.