Objective & prerequisites
Calculate an effect with its uncertainty and resist premature winner claims. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.
python3 analysis.py
python3 -m unittest discover -s tests -vUnderstand the concept
Absolute effect is treatment rate minus control rate; report it in percentage points. Relative lift divides that difference by the control rate and is undefined when the control rate is zero. For this teaching fixture, a two-sided 95% unpooled normal interval is delta ±1.96×sqrt(pc(1-pc)/nc + pt(1-pt)/nt). This approximation assumes independent user outcomes and sufficiently large success/failure counts; the reference refuses fewer than 10 of either per group. It is not a general solution for sparse, clustered or sequential data. A 95% confidence procedure has repeated-sampling coverage under its assumptions; it does not assign a 95% probability to this particular fixed parameter interval. An interval spanning zero is inconclusive, not proof of no effect or equivalence. Multiple metrics, segments and repeated peeking need a predeclared multiplicity/sequential approach.
Worked example
The synthetic effect is+8 percentage points, or+40% relative lift. The approximate 95% interval is[-3.79,+19.79] percentage points, spanning harm and benefit. No winner is established. The 50:50 sample-ratio check uses an exact two-sided binomial test and a predeclared 0.001 alert threshold:100/100 gives p=1, while 130/70 triggers an investigation. Passing SRM does not prove correct randomization or logging. This fixture fixes allocation counts through a balanced shuffle; the binomial diagnostic is illustrated for a hypothetical independent 50:50 assignment scheme, not calibrated as a test of that fixed-count randomizer. A real diagnostic must match the assignment design. A normal-approximation sample-size plan for baseline 20%, an 8-point absolute increase, two-sided alpha 0.05 and 80% power is 447 users per arm, not the 100 supplied. The current fixture is intentionally smaller and inconclusive.
from analysis import difference_interval,srm,plan_sample
r=difference_interval(20,100,28,100)
assert round(r["difference"]*100)==8
assert r["ci95"][0]<0<r["ci95"][1]
assert srm(130,70)["flag_at_0_001"]
assert plan_sample()==447Try it yourself
Reproduce the difference and interval, then distinguish+8 points from+8%. Calculate the exposed-only comparison as a mistake case. Run the 130/70 SRM diagnostic. State why you should not extend a real experiment only until a conventional interval excludes zero. Write a guardrail note for 5/100 versus 7/100 error users without claiming safety from a small difference. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.
Worked solution
The standard error is about 0.06013, giving a margin of 0.11786 and the stated interval. Baseline 20% multiplied by 1.40 gives 28%; a relative 8% increase would give 21.6%. SRM flags the 130/70 mismatch before effect interpretation. Use a planned fixed horizon or separately validated sequential method rather than optional stopping. The guardrail difference is+2 percentage points, but the sparse error counts make this kit refuse its normal interval; report the counts and require a separately verified suitable method and a predeclared harm margin.447 is an approximate planning result under stated assumptions, not guaranteed power for every operating design.
Knowledge check
If the interval includes zero, can you conclude the variants are equivalent?
Check and explain
No. Equivalence needs a predeclared practical margin and an appropriate test/design. An inconclusive superiority result is not evidence of equivalence.
Evidence, mistakes & handover
Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.
Authoritative reference
— reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.