Product Analytics

Objective & prerequisites

Define the population, analysis unit, event and observation window before calculating. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

A product metric is a rule for turning evidence into a decision-relevant quantity. Begin with a question: where do new learners stop between starting a course and attempting a practice exercise? Define the unit as a stable fictional user_id rather than an event row or a browser visit. Specify the eligible population, identity policy, internal-user exclusion, starting action, conversion action, time zone, ordered sequence and observation cutoff. A rate requires both numerator and denominator. Record whether a time boundary is inclusive, whether later attempts can re-enter and whether a repeated event counts again. A tracking plan describes how evidence is collected; a metric contract describes how that evidence is interpreted. Neither automatically proves that the event reflects meaningful learning.

Worked example

The product snapshot contains 13 signup records, including one internal test user. Excluding that account leaves 12 eligible users. The event watermark is 2026-01-26 00:00 UTC, exclusive: the teaching source is assumed complete before that instant. The funnel is the earliest start, then a later lesson, then a later exercise within 72 hours of that start; only one attempt per user counts. Retention separately uses practice_active, not merely opening a page. These are declared lab rules, not claims about DataCareerHub’s real users or Amplitude’s default settings.

from analysis import read_csv, product_data
people, events, audit = product_data(read_csv("users.csv"), read_csv("events.csv"))
assert len(people) == 12
assert audit["internal_users_excluded"] == 1

Try it yourself

Complete metric-contract-template.txt for the three-step funnel and day-7 retention. Write one ambiguous definition such as “active this week,” then rewrite it to state the action, unit, exact window and eligibility rule. Explain why course completion and exercise attempt are not interchangeable. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The funnel denominator is eligible users with a first start in the observed data; in this fixture all 12 qualify and all 72-hour windows are complete. Retention uses the original signup cohort, not only users who converted. Day 7 means the elapsed interval [signup+7 days, signup+8 days), and an entire cohort row is withheld until all its members have that full interval. The watermark is a completeness assumption documented by the case; it is not inferred from the latest event timestamp.

Knowledge check

If 12 users generate 30 lesson events, is the user conversion denominator 30?

Check and explain

No. The denominator follows the declared user population. Repeated actions may describe intensity, but they do not create additional people.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://amplitude.com/docs/analytics/charts/funnel-analysis/funnel-analysis-build — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Validate identities, timestamps, repeated deliveries and event meaning. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

An event table usually has event grain: one record describes one recorded action. It is not a user table. Stable event IDs can reveal delivery retries. Remove an exact repeated event_id only when all fields agree; a repeated ID with conflicting fields is an error requiring investigation. Different IDs from the same user may be genuine repeated actions and should not be dropped merely to simplify totals. Validate known user references, allowed event names and explicit UTC timestamps. Event time and ingestion time are different concepts; this kit has only event time and an explicit completeness assumption, so it cannot diagnose late-arriving ingestion. Never copy personal details into event payloads for a teaching exercise.

Worked example

The source contains 45 CSV event rows. One exact delivery retry is removed. One event from the internal user is excluded, and two records at or after the watermark are withheld. This leaves 41 accepted event records. A lesson before the user’s first start remains valid event evidence but does not advance the ordered funnel. An exercise before its lesson is similarly retained and evaluated against the sequence rule. The validator rejects unknown users, conflicting IDs and pre-signup events instead of silently publishing a partial answer.

from analysis import read_csv, product_data
raw = read_csv("events.csv")
people, accepted, audit = product_data(read_csv("users.csv"), raw)
assert len(raw) == 45 and len(accepted) == 41
print(audit)

Try it yourself

Reconcile the 45 raw rows to 41 accepted rows. Locate the repeated event ID in a spreadsheet or with collections.Counter. In a copy, change the time on its repeated row and predict what the validator should do. Add an unknown user ID to a separate copy and explain why a join that silently drops it could hide instrumentation failure. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The reconciliation is 45 minus 1 exact retry minus 1 internal event minus 2 out-of-snapshot events = 41. The supplied audit reports each exclusion separately. A conflicting duplicate raises ValueError and no report is produced. An unknown user also fails. If a real operating policy instead quarantines records, it must publish the rejected population and completeness implications; the current reference deliberately fails closed for these contract defects.

Knowledge check

Can you infer that tracking is complete because the maximum event time looks recent?

Check and explain

No. Absence of events can mean inactivity or collection failure. Completeness needs an independent collection/watermark contract and operational checks.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://amplitude.com/docs/analytics/charts/funnel-analysis/funnel-analysis-how-amplitude-computes-conversions — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Measure step conversion without counting repeats or out-of-order actions as new users. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

A funnel represents a defined sequence. The lab uses strict chronological order: start time < lesson time < exercise time. Exercise must occur no later than first start plus 72 hours; a timestamp exactly at the deadline qualifies. The earliest start anchors one attempt; later starts do not restart the window. These rules are deliberate and may differ from a vendor’s longest-sequence, session or re-entry rules. Count each user once at each reached step. Overall conversion divides final-step users by first-step users. Conditional step conversion divides a step by the previous step. A drop-off indicates where recorded progression stops, not why it stops. Qualitative research and instrumentation review are needed before attributing frustration or lack of interest.

Worked example

Twelve users start; eight reach a later lesson and five reach a later exercise in time. Overall conversion is 5/12 = 41.67%. Start-to-lesson is 8/12 = 66.67%; lesson-to-exercise is 5/8 = 62.50%. Four people stop before a qualifying lesson; three more stop before a qualifying exercise. U03 has an exercise before its lesson and does not finish. U06’s exercise at signup+73 hours is beyond first start+72 hours. U07’s lesson precedes start, and U08 has an exercise without a lesson. Repeated U10 lessons do not add people.

from analysis import read_csv, product_data, funnel
people, events, _ = product_data(read_csv("users.csv"), read_csv("events.csv"))
result = funnel(people, events)
assert result["counts"] == [12, 8, 5]
assert round(result["overall_rate"]*100, 2) == 41.67

Try it yourself

List each user’s furthest qualifying step. Explain why a COUNT of exercise rows or an unordered intersection produces a different number. In a copy, move U06’s exercise to exactly its first start+72 hours and rerun. Do not change the production definition merely because a different boundary improves the reported rate. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The completing users are U01,U02,U04,U05,U09. The intermediate-only users are U03,U06,U10; U07,U08,U11,U12 reach only start. Moving U06 to the inclusive deadline makes final conversion 6/12 = 50%, while first and second steps stay 12 and 8. The original fixture remains 12→8→5. If your business permits repeated attempts, define and test a different metric rather than silently mixing approaches.

Knowledge check

Is 5/8 the overall conversion rate for this funnel?

Check and explain

No. It is the conditional last-step conversion. Overall conversion uses the original entrant denominator: 5/12.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://amplitude.com/docs/analytics/charts/funnel-analysis/funnel-analysis-interpret — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Group users consistently and compare the same stage of their product lifetime. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

A cohort groups units by a declared shared property. An acquisition cohort may group signup dates; a behavioral cohort may group a later action. This lesson uses signup week with Monday boundaries in UTC. Cohort membership remains fixed when later behavior changes. A cohort matrix separates calendar date from elapsed product age: read across a row to follow one cohort through time, and down a fixed-age column to compare cohorts at the same age. Newer cohorts cannot yet contribute to older intervals. Channel or device differences can explain some aggregate shifts; observational cohort comparisons do not establish a treatment effect. Selecting users based on future conversion and then comparing their earlier behavior can also introduce selection bias.

Worked example

There are three four-user cohorts starting January 5, 12 and 19. Their day-1 practice-return counts are 3,2,1, giving 75%,50%,25%. The raw numbers show a declining sequence in this invented fixture, but the cohorts are tiny and were not randomized. Day-7 values are 2/4 and 1/4 for the first two; the third row is not yet mature. Comparing the oldest cohort’s day 7 with the newest cohort’s day 1 mixes different product ages and answers no stable retention question.

from analysis import read_csv, product_data, retention
p, e, _ = product_data(read_csv("users.csv"), read_csv("events.csv"))
assert [r["size"] for r in retention(p,e,1)] == [4,4,4]
assert [r["returned"] for r in retention(p,e,1)] == [3,2,1]

Try it yourself

Build a three-row matrix with cohort size, day-1 count/rate and day-7 count/rate/status. Label the latest day-7 cell “not yet observable.” Write two plausible non-causal explanations for the day-1 pattern, and a proposed evidence check for each. Do not assert that either explanation happened. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The matrix is Jan 5:4,3/4,2/4; Jan 12:4,2/4,1/4; Jan 19:4,1/4,N/A. Possible hypotheses include different acquisition mix or changing collection quality; inspect source channel distributions or independent event-delivery checks before attributing either. The fictional channels alternate evenly, so channel imbalance is not supported by this particular source. Tiny group size and other unmeasured factors remain. The visual worked report includes both percentages and denominators.

Knowledge check

If a user changes acquisition channel later, should the original signup cohort change?

Check and explain

Not under this contract. Cohort membership uses signup attributes. A different time-varying segment needs its own definition.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://amplitude.com/docs/analytics/create-cohorts — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Distinguish exact-day return, rolling return and incomplete data. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

Exact-day retention asks whether a user performed a specified return action during one particular interval after signup. Here day 1 is [24 h,48 h) and day 7 is [168 h,192 h); these are elapsed UTC intervals, not local calendar days. Returning twice in an interval still counts once. A later return at day 8 does not count for exact day 7. Rolling or “on or after” retention is a different metric with its own bounded observation policy. An empty cell can mean not yet observable, and must not be displayed as 0%. This implementation withholds an entire cohort row unless every member’s relevant interval is complete, preserving the fixed cohort denominator. Other systems may use mature subsets, but they must label the changing denominator explicitly.

Worked example

At the exclusive January 26 watermark, day-7 intervals for January 19 signups have just begun and are incomplete. That cohort’s count and rate are null. The first two mature cohorts return 2/4 and 1/4. Pooled mature day-7 retention is (2+1)/(4+4) = 37.50%; dividing by all 12 users would incorrectly report 25%. U04 returns on day 8 and does not count. U01’s two day 1 events count as one returning user. Overall day 1 is 6/12 = 50%.

from analysis import read_csv, product_data, retention
p,e,_=product_data(read_csv("users.csv"),read_csv("events.csv"))
rows=retention(p,e,7)
mature=[r for r in rows if r["mature"]]
assert sum(r["returned"] for r in mature)/sum(r["size"] for r in mature)==0.375
assert rows[-1]["rate"] is None

Try it yourself

Reproduce pooled mature day-7 retention from counts, not an unlabelled average of percentages. Then change the latest cohort size in a paper example to 8 and show why an unweighted mean can disagree with a pooled rate. Explain why moving the watermark forward is not enough unless the event source is also complete to that new time. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

Filter to mature rows, sum returned and size, then divide:3/8. With an 8-user cohort returning 2 and a 4-user cohort returning 2, pooled return is 4/12 =33.33%, while the unweighted mean of 25% and 50% is 37.50%. Do not backfill missing event history with zeros or simply assert a later watermark. Advancing the observation boundary requires the actual additional source data and its completeness assurance.

Knowledge check

Is a null retention rate evidence that no one returned?

Check and explain

No. In this case null marks incomplete observation. A mature zero requires a fully observed interval with no qualifying return.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://amplitude.com/books/mastering-retention/appendix — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Define assignment, intention-to-treat analysis, outcomes, guardrails and stopping rules. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

Random assignment supports a causal comparison when assignment is implemented correctly and the design’s assumptions hold. Define an eligible population, a stable randomization unit and the outcome before rollout. The lab unit is a user, assigned once to control or treatment in a planned 50:50 split. Analysis follows original assignment, including users who do not see the feature: this is an intention-to-treat comparison. Conditioning on post-assignment exposure can break comparability. Define a seven-day binary completion outcome, an error-user guardrail, a fixed observation horizon and a rule for missing telemetry. Repeated sessions are not independent randomized units. Shared-account or social spillovers may require cluster randomization and cluster-aware inference instead. Do not run a live experiment on visitors as part of this exercise.

Worked example

The kit uses a reproducible balanced shuffle to place 200 fictional IDs into two groups of 100. Outcomes are then deliberately constructed to illustrate 20 versus 28 completions; they are not measurements from a real randomized trial and do not demonstrate an actual causal effect. The assignment file has one row per user; the outcome file has one observed seven-day row per assigned user. Exposed counts are 90 and 80. Dropping non-exposed users would produce 20/90 versus 28/80 and change the estimand after treatment. A missing outcome is refused, not assumed to mean no completion.

from analysis import read_csv, experiment
r=experiment(read_csv("assignments.csv"),read_csv("outcomes.csv"))
assert r["arms"]["control"]["assigned"]==100
assert r["arms"]["treatment"]["assigned"]==100

Try it yourself

Complete experiment-plan-template.txt before opening the result. State the assignment unit, primary outcome, guardrail, completeness rule, sample-size assumptions, fixed end and decision criteria. Calculate both assigned-user rates and exposed-only rates, then explain why the latter is not the primary causal estimate. Identify how you would detect crossovers or duplicate assignment. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The primary rates are 20/100=20% and 28/100=28%. Exposed-only rates are 22.22% and 35%; they select different post-assignment subsets. A duplicate user assignment, including a different arm for that user, fails validation. Before interpretation, check the planned 50:50 sample ratio, full seven-day follow-up and one observed outcome per assigned user. In real systems a validated zero from complete telemetry differs from an absent outcome row. The fixture asserts observation and does not independently prove a real tracking system’s completeness.

Knowledge check

Does randomly shuffling a file of manufactured outcomes establish that a product change works?

Check and explain

No. This is a transparent simulation for learning the analysis. Only an appropriately implemented and observed experiment can supply real causal evidence.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Calculate an effect with its uncertainty and resist premature winner claims. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

Absolute effect is treatment rate minus control rate; report it in percentage points. Relative lift divides that difference by the control rate and is undefined when the control rate is zero. For this teaching fixture, a two-sided 95% unpooled normal interval is delta ±1.96×sqrt(pc(1-pc)/nc + pt(1-pt)/nt). This approximation assumes independent user outcomes and sufficiently large success/failure counts; the reference refuses fewer than 10 of either per group. It is not a general solution for sparse, clustered or sequential data. A 95% confidence procedure has repeated-sampling coverage under its assumptions; it does not assign a 95% probability to this particular fixed parameter interval. An interval spanning zero is inconclusive, not proof of no effect or equivalence. Multiple metrics, segments and repeated peeking need a predeclared multiplicity/sequential approach.

Worked example

The synthetic effect is+8 percentage points, or+40% relative lift. The approximate 95% interval is[-3.79,+19.79] percentage points, spanning harm and benefit. No winner is established. The 50:50 sample-ratio check uses an exact two-sided binomial test and a predeclared 0.001 alert threshold:100/100 gives p=1, while 130/70 triggers an investigation. Passing SRM does not prove correct randomization or logging. This fixture fixes allocation counts through a balanced shuffle; the binomial diagnostic is illustrated for a hypothetical independent 50:50 assignment scheme, not calibrated as a test of that fixed-count randomizer. A real diagnostic must match the assignment design. A normal-approximation sample-size plan for baseline 20%, an 8-point absolute increase, two-sided alpha 0.05 and 80% power is 447 users per arm, not the 100 supplied. The current fixture is intentionally smaller and inconclusive.

from analysis import difference_interval,srm,plan_sample
r=difference_interval(20,100,28,100)
assert round(r["difference"]*100)==8
assert r["ci95"][0]<0<r["ci95"][1]
assert srm(130,70)["flag_at_0_001"]
assert plan_sample()==447

Try it yourself

Reproduce the difference and interval, then distinguish+8 points from+8%. Calculate the exposed-only comparison as a mistake case. Run the 130/70 SRM diagnostic. State why you should not extend a real experiment only until a conventional interval excludes zero. Write a guardrail note for 5/100 versus 7/100 error users without claiming safety from a small difference. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The standard error is about 0.06013, giving a margin of 0.11786 and the stated interval. Baseline 20% multiplied by 1.40 gives 28%; a relative 8% increase would give 21.6%. SRM flags the 130/70 mismatch before effect interpretation. Use a planned fixed horizon or separately validated sequential method rather than optional stopping. The guardrail difference is+2 percentage points, but the sparse error counts make this kit refuse its normal interval; report the counts and require a separately verified suitable method and a predeclared harm margin.447 is an approximate planning result under stated assumptions, not guaranteed power for every operating design.

Knowledge check

If the interval includes zero, can you conclude the variants are equivalent?

Check and explain

No. Equivalence needs a predeclared practical margin and an appropriate test/design. An inconclusive superiority result is not evidence of equivalence.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/diffprop.htm — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.

Objective & prerequisites

Combine funnel, retention and experiment evidence into an honest recommendation. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

A useful memo leads with the decision and the strength of evidence. Separate descriptive observations from causal estimates and unresolved data quality. Include population, numerator/denominator, exact window, counts, uncertainty and limitations. A funnel can guide investigation but does not explain motivation. A cohort pattern can suggest a hypothesis but does not isolate a cause. An experiment can evaluate a defined intervention only if its design and data justify interpretation. Keep visual scales and missing values clear: a narrow heatmap of tiny cohorts must not imply precise forecasts. An uncertainty interval should show the zero reference and actual units. Privacy-conscious analytics begins with necessary fields and retention rules, not collecting identifying details just because a tool allows it.

Worked example

The worked report shows 12→8→5 funnel users, mature day-7 retention 3/8, and a synthetic experiment difference+8 points with an interval crossing zero. A responsible memo proposes investigating first-step progression and validating telemetry, while describing cohort results as small-sample patterns. It does not recommend shipping the fictional treatment as a proven winner. The existing platform’s payments, registration and protected-course access are unaffected; this material adds no visitor tracking or feature-flag infrastructure.

python3 analysis.py
python3 -m unittest discover -s tests -v
# Open worked-report.html locally to compare your memo with the visual reference.

Try it yourself

Complete decision-memo-template.txt in 250–400 words. Include one chart/table from worked-report.html and the original count controls. Add a mistake diagnosis for “retention 25%, treatment lift 8%, launch now.” State the evidence needed for a real follow-up, including sufficient eligible users, observation completeness, a reviewed design and guardrail assessment. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Worked solution

The mistaken retention divides three mature returns by 12 instead of 8, turning unobservable users into non-returners. The lift label confuses 8 percentage points with 40% relative lift. The launch recommendation ignores uncertainty, small sample, incomplete guardrail evidence and manufactured outcomes. A sound memo retains 3/8=37.50%, reports the interval, requests investigation and an appropriately planned real study rather than inventing certainty. Use the capstone rubric and worked-solutions.txt to assess your evidence.

Knowledge check

Can a polished dashboard compensate for a mismatched denominator or missing follow-up?

Check and explain

No. Clear design helps people inspect evidence; it cannot repair an invalid metric or establish a causal claim.

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ — reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.