Compare groups and separate composition from performance
Learn the reasoning. Try the task. Compare your answer.
Prepared for DataCareerHub · Curriculum version 2026-10-15.2
By the end of this lesson
- Calculate group-specific rates and reconcile their denominators.
- Reconstruct an overall rate using group weights.
- Use a fixed-weight comparison without calling it a causal effect.
Harbor Helpdesk · UTC · Snapshot 22 June 2026 00:00 · Prepared fictional extract · No Python or SQL required.
Understand the method
An overall rate can change because outcomes within groups change, the mix of groups changes, or both. Before explaining a headline, inspect the composition of its denominator. Channel is especially relevant in Harbor Helpdesk because email and chat have different recorded response patterns and their proportions change between the creation cohorts.
Start with counts, then rates
Week 1 contains eight email tickets with four timely responses and four chat tickets with four timely responses. Email timeliness is 50%, chat timeliness 100%, and the overall result is eight timely out of twelve. Week 2 contains four email tickets with two timely responses and eight chat tickets with eight timely responses. The same channel-specific rates combine to ten timely out of twelve. Reconcile channel ticket counts and timely counts before interpreting either rate.
Do not average 50% and 100% to report a 75% overall rate. The channel denominators are unequal in each week. Use timely totals divided by eligible observed totals, or multiply each group rate by its ticket share and add the products. Week 1 is (8/12 × 0.5) + (4/12 × 1) = 2/3. Week 2 is (4/12 × 0.5) + (8/12 × 1) = 5/6. The higher-rate channel has a larger share in Week 2.
Separate two useful summaries
The observed overall comparison describes the tickets actually present in each cohort: 66.67% and 83.33%, a 16.67 percentage-point rise. An illustrative fixed-weight comparison answers another question: what do the two sets of channel rates produce under a common chosen composition? Using Week 1 weights, two thirds email and one third chat, both results are 66.67%. Declare the reference weights and keep them fixed for both periods.
Using Week 2 weights would instead give 83.33% for both periods. Neither reference is the universally correct operational rate; each is a labelled standardized description. Do not replace the observed rates without explaining the change of question. In this deliberately simple example the within-channel rates are identical, so the arithmetic headline difference is fully accounted for by the changed channel weights. That does not identify what caused the mix, nor prove routing had no effect on any ticket.
Watch smaller groups
Segmenting further can expose useful questions, but it also produces tiny cells. Priority has only three high-priority tickets per cohort. Their timely counts are two then three. The normal-priority groups have nine tickets each, with six then seven timely. One ticket changes a three-ticket group's rate by 33.33 percentage points. That sensitivity is a reason to show counts and treat findings as exploratory, not a universal minimum-sample rule or a significance calculation.
The source has no verified routing-exposure field. Queue and priority are classifications, not substitutes for treatment assignment. A difference between queues cannot establish a routing effect merely because the intervention is in the story. Keep the analysis question visible: describe the supplied cohorts, identify alternative explanations and request stronger evidence for operational attribution. The right output can be a careful table and a follow-up question rather than a confident winner.
Practical exercise
Build a cohort-by-channel table with eligible count, timely count, rate and ticket share. Reconcile both the counts and weighted rates. Then calculate both periods using Week 1 weights and again using Week 2 weights.
- Explain why 75% is the wrong unweighted overall answer.
- Write one sentence about the observed result and one about the standardized result.
- Inspect high-priority groups; quantify the effect of one ticket.
- State why queue/priority cannot identify routing treatment.
Open the worked solution
W1: email 4/8=50%, share 2/3; chat 4/4=100%, share 1/3; total 8/12=66.67%. W2: email 2/4=50%, share 1/3; chat 8/8=100%, share 2/3; total 10/12=83.33%. The unweighted 75% gives both channels equal importance despite their unequal ticket counts.
With W1 weights, both standardized rates are 66.67%; with W2 weights, both are 83.33%. Label the chosen reference and retain observed results. “Observed timeliness rises by 16.67 percentage points in this extract. Under a fixed W1 channel composition, the channel-rate summary is unchanged.” This is descriptive decomposition, not an intervention effect estimate. High-priority timeliness is 2/3 then 3/3; a single observation changes that group's rate by 1/3, or 33.33 percentage points. Exposure needs an authoritative deployment/assignment record; classifications do not provide it.
The change in chat share is 1/3 and the gap between channel rates is 1/2. Their product is 1/6, matching the observed 16.67 percentage-point rise. This algebra reconciles the simple constructed example; it does not explain why channel choice changed or what each ticket would have experienced under a different routing process.
One more question.
How these learning materials are prepared and maintained · Report a correction: support@datacareerhub.io
Read and practice without an account. Sign in with an existing account to save lesson completion.
Next lesson →Harbor Helpdesk is a fictional teaching case. Descriptive comparisons do not establish routing causality. Portfolio practice is not a claim of client work or an accredited qualification. Explore practical lessons, worked examples and projects. Follow a published path, practise the method and build work you can explain.