4. Compare cohorts without confusing time and age

Objective & prerequisites

Group users consistently and compare the same stage of their product lifetime. Prerequisites: percentages, basic tables, Python lists/dictionaries and the earlier analysis foundations. Download and extract the full kit, then open a terminal in product-analytics-kit. Python 3.10+ and the standard library are sufficient. Spreadsheet users can inspect the CSVs and reproduce the small controls manually. All people, events and experiment outcomes are invented teaching data. No live tracking, public experiment, payment or registration is activated. These free companion lessons use the existing studio without changing paid access or course progress.

python3 analysis.py
python3 -m unittest discover -s tests -v

Understand the concept

A cohort groups units by a declared shared property. An acquisition cohort may group signup dates; a behavioral cohort may group a later action. This lesson uses signup week with Monday boundaries in UTC. Cohort membership remains fixed when later behavior changes. A cohort matrix separates calendar date from elapsed product age: read across a row to follow one cohort through time, and down a fixed-age column to compare cohorts at the same age. Newer cohorts cannot yet contribute to older intervals. Channel or device differences can explain some aggregate shifts; observational cohort comparisons do not establish a treatment effect. Selecting users based on future conversion and then comparing their earlier behavior can also introduce selection bias.

Worked example

There are three four-user cohorts starting January 5, 12 and 19. Their day-1 practice-return counts are 3,2,1, giving 75%,50%,25%. The raw numbers show a declining sequence in this invented fixture, but the cohorts are tiny and were not randomized. Day-7 values are 2/4 and 1/4 for the first two; the third row is not yet mature. Comparing the oldest cohort’s day 7 with the newest cohort’s day 1 mixes different product ages and answers no stable retention question.

from analysis import read_csv, product_data, retention
p, e, _ = product_data(read_csv("users.csv"), read_csv("events.csv"))
assert [r["size"] for r in retention(p,e,1)] == [4,4,4]
assert [r["returned"] for r in retention(p,e,1)] == [3,2,1]

Try it yourself

Build a three-row matrix with cohort size, day-1 count/rate and day-7 count/rate/status. Label the latest day-7 cell “not yet observable.” Write two plausible non-causal explanations for the day-1 pattern, and a proposed evidence check for each. Do not assert that either explanation happened. State your denominator, time window and expected result before running the reference. Use a copy for changes and keep the original evidence intact.

Knowledge check

If a user changes acquisition channel later, should the original signup cohort change?

Evidence, mistakes & handover

Keep a metric definition and an auditable count beside every percentage. Distinguish a measured zero, missing telemetry and an incomplete observation window. Record exact commands, actual outputs, assumptions and one failed case. Do not describe an observational association or a manufactured experiment fixture as proof of real business impact. The kit includes separate exercises, worked solutions, a metric contract, an experiment plan template, a decision memo template, tests and a visual worked report. Credit the supplied starting point and explain your own contribution.

Authoritative reference

Read the official reference ↗

— reference for the underlying concept; vendor defaults can differ from this lesson’s explicit definitions. Lesson prose, case data and exercises are original. No analytics vendor installation, live user telemetry or production experimentation is claimed.