Offline gate contract
This is an original recorded-window simulation, not a real canary router or automatic deployment. Both versions require 20 requests. Service gate: error rate >0.05 or nearest-rank p95 latency >200 ms fails; equality passes the stated threshold only. One latency is supplied per request, including errors. These numbers are teaching assumptions, not universal SLOs.
Version-level fixture
Stable has 100 requests,1 error and p95 of 100 ms. Candidate has 20 requests,4 errors and p95 of 250 ms. Both have adequate mature labels and MAE 2 in this scenario. Calculate per-version and pooled error rates before deciding.
version | requests | errors | p95_ms
stable | 100 | 1 | 100
candidate | 20 | 4 | 250Your attempt
Would a pooled error rate suggest the same action as per-version gates? Which control evidence supports a candidate-specific triage path? Why is this a predeclared operational rule rather than a causal proof that the candidate caused every failure?
Worked solution
Stable error rate 1/100=0.01; candidate 4/20=0.20. Pooled 5/120≈0.0417 would hide the candidate threshold failure. Candidate also fails latency. With adequate stable evidence and a healthy baseline, the simulator emits rollback_candidate. Workload differences or shared infrastructure still need investigation; the rule is not statistical attribution.
Gate precedence
The project holds if either request count is too small. Baseline service failure goes to shared/baseline investigation before candidate attribution. Otherwise a candidate service failure yields a rollback advisory. If service passes, both versions need 10 mature labels before quality review; baseline quality failure is investigated, candidate-only quality failure yields rollback. All pass means eligible_for_human_review, never automatic promotion.
A shared-fault counterexample
If stable has 10/100 errors and p95 of 260 ms while candidate also fails, investigate shared/baseline faults under the stated policy. More than one cause can coexist. Do not relabel every coincident infrastructure problem as a candidate regression, or assume reverting the model fixes it.
Boundary exercise
Twenty requests with one error has rate 0.05, at the teaching boundary. Nearest-rank p95 over 20 sorted values uses the 19th value. A single largest outlier is not necessarily the p95. State the quantile convention, sample size and window before comparing dashboards. Sequentially checking many windows also raises inference questions not implemented by this simulator.
Rubric and interview
Score 0–2 for per-version denominators, declared percentile, gate precedence, insufficient-evidence holds and honest causal/operational limits. Explain why a process-health endpoint and a passing aggregate average cannot establish model quality. No actual service is changed by the offline program.
Authoritative reference
Alec Warner and Štěpán Davidovič, Google SRE, The Site Reliability Workbook: Canarying Releases. Further reading on release evaluation; our original fixture and policy do not implement a production SRE platform. Next: Maple 3.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.