Objective & prerequisites
Combine funnel, retention and experiment evidence into an honest recommendation. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.
python3 analysis.py
python3 -m unittest discover -s tests -vUnderstand the concept
A useful memo leads with the decision and the strength of evidence. Separate descriptive observations from causal estimates and unresolved data quality. Include population, numerator/denominator, exact window, counts, uncertainty and limitations. A funnel can guide investigation but does not explain motivation. A cohort pattern can suggest a hypothesis but does not isolate a cause. An experiment can evaluate a defined intervention only if its design and data justify interpretation. Keep visual scales and missing values clear: a narrow heatmap of tiny cohorts must not imply precise forecasts. An uncertainty interval should show the zero reference and actual units. Privacy-conscious analytics begins with necessary fields and retention rules, not collecting identifying details just because a tool allows it.
Worked example
The worked report shows 12→8→5 funnel users, mature day-7 retention 3/8, and a synthetic experiment difference+8 points with an interval crossing zero. A responsible memo proposes investigating first-step progression and validating telemetry, while describing cohort results as small-sample patterns. It does not recommend shipping the fictional treatment as a proven winner. The existing platform’s payments, registration and protected-course access are unaffected; this material adds no visitor tracking or feature-flag infrastructure.
python3 analysis.py
python3 -m unittest discover -s tests -v
# Open worked-report.html locally to compare your memo with the visual reference.Try it yourself
Complete decision-memo-template.txt in 250–400 words. Include one chart/table from worked-report.html and the original count controls. Add a mistake diagnosis for “retention 25%, treatment lift 8%, launch now.” State the evidence needed for a real follow-up, including sufficient eligible users, observation completeness, a reviewed design and guardrail assessment. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.
Worked solution
The mistaken retention divides three mature returns by 12 instead of 8, turning unobservable users into non-returners. The lift label confuses 8 percentage points with 40% relative lift. The launch recommendation ignores uncertainty, small sample, incomplete guardrail evidence and manufactured outcomes. A sound memo retains 3/8=37.50%, reports the interval, requests investigation and an appropriately planned real study rather than inventing certainty. Use the capstone rubric and worked-solutions.txt to assess your evidence.
Knowledge check
Can a polished dashboard compensate for a mismatched denominator or missing follow-up?
Check and explain
No. Clear design helps people inspect evidence; it cannot repair an invalid metric or establish a causal claim.
Evidence, mistakes & handover
Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.
Authoritative reference
— reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.