Three different questions
Service monitoring asks whether requests complete within an agreed reliability/latency contract. Data monitoring checks schema, missingness, ranges and distributions. Quality monitoring compares outputs with suitable outcomes when labels mature. Healthy service can return consistently wrong predictions; a changed input distribution alone does not establish predictive failure.
Your evidence
Candidate has 20 requests,zero errors,p95 of 120 ms and zero mature labels. The observed absolute-error sum is 0 because no outcomes are labelled. Stable has 20 mature labels and error sum 40. The teaching policy needs 10 mature labels per version before quality review.
Your attempt
Calculate stable MAE and decide whether candidate MAE is 0,undefined or already acceptable. What would make a mature-label sample unrepresentative? Which facts would you record for a later quality investigation?
Worked solution
Stable MAE 40/20=2. Candidate MAE is undefined because its denominator is 0; the correct simulator action is hold_missing_mature_labels. Zero observed error sum with no labels is not perfect quality. Even ten mature labels can mislead if labels arrive preferentially for easy cases or failed outcomes are missing.
Monitoring contract
Record artifact/version and evaluation configuration, window/traffic slice, request and error counts, percentile definition, monitoring freshness, label maturity/coverage and outcome definition. Version or snapshot feature/preprocessing contracts. Avoid unnecessary raw user data in logs. A real system needs privacy/access controls and retention policies; these local fixtures contain only synthetic aggregates.
Drift versus quality exercise
A new customer mix changes a feature histogram but labelled error remains stable. Another case has unchanged input summaries but a shifted outcome relationship. Explain why the first is a review signal, not automatic proof of degradation, and the second may require mature outcomes to detect. No drift statistic or detector is implemented in Maple; the learner designs the review plan.
Counterexample and next evidence
The aggregate MAE passes while a meaningful low-volume slice fails. Ask for slice counts, comparable windows, label completeness and case review rather than only the global average. Do not choose favourable slices after seeing final outcomes and hide the rest. Define missing-data and incident owners before relying on automated gates.
Review and explain
Score 0–2 for layer separation, denominator handling, label-delay/selection risk, versioned evidence and bounded next steps. Explain why changing an alert threshold after a failure is a policy change, not evidence that the original release was good. This is unsaved self-practice.
Authoritative reference
Alec Warner and Štěpán Davidovič, Google SRE, The Site Reliability Workbook: Canarying Releases. Use the monitoring discussion as further reading; this project adds original model-label teaching scenarios without claiming production monitoring coverage. Next: Maple 4.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.