Objective & prerequisites
Produce a handover that separates containment, code rollback, data recovery and prevention. Prerequisites: Python functions, JSON, basic SQL, Git and the existing software/testing workshops. Extract the full kit and open a terminal in cloud-operations-kit. Python 3.10+ with its standard SQLite module is sufficient for the executed labs; no cloud account, credentials, package installation or network is needed. All data and prices are invented teaching fixtures. The runnable drills use new temporary folders and databases, never the website or hosting. These free companions retain existing course progress/access. Payments and public registration remain disabled.
python3 lab.py
python3 -m unittest discover -s tests -vUnderstand the concept
An incident response begins with user impact, scope and evidence, not a confident root-cause guess. Record detection time, affected task/population, current version/configuration, trusted observations and unknowns. Stabilize through the least risky authorized action, preserve evidence and assign ownership. A code rollback can address a compatible regression; data restoration needs an explicit recovery point and review of lost changes. Validate recovery with business controls and user-visible acceptance, then document causes, contributing conditions and prevention work without blame. A tabletop is a scenario exercise; it does not establish a production response time or authorize live changes.
Worked example
The fictional window shows two failed publications and p95=1000 ms. The runbook asks you to inspect bounded error categories and recent release/configuration evidence before naming a cause. The local release drill proves pointer rollback preserves a fictional state file; the restore drill proves a separate snapshot restores 70000 cents while current source remains 75000. Neither proves that a rollback fixes the log failures. You must identify the missing causal/acceptance evidence rather than joining unrelated drills into a false end-to-end incident success.
from lab import report
r=report()
assert r["observability"]["meets_sample_slo"] is False
assert r["release"]["state_unchanged"]
assert r["restore"]["live_total_still"]==75000
assert "Docker/container runtime" in r["not_executed"]Try it yourself
Complete incident-runbook-template.txt and a 250–400 word evidence memo. Use each drill’s actual results, record unexecuted cloud/container activities and specify the checks needed to close a real incident. Include a resource cleanup review and one improvement with owner and acceptance evidence. Predict the result first, keep the original fixture, and record the exact command and observed outcome from your practice copy.
Worked solution
A sound memo distinguishes sample SLO breach, local rollback mechanics and separate recovery controls. It keeps current data intact, states the snapshot’s missing post-backup record and avoids a claimed production RTO. Follow-up may include reproducible error investigation, independent source completeness checks and a reviewed deployment/recovery exercise. No live site, account, upload, payment or registration setting is changed by this mini-project.
Knowledge check
Does a successful local rollback drill prove it fixed an unrelated set of production failures?
Check and explain
No. Recovery requires incident-specific acceptance evidence and a supported causal diagnosis. Keep tabletop and production claims separate.
Evidence, mistakes & handover
State whether an activity was executed locally, reviewed as a design, or left unexecuted. A successful local drill does not verify a cloud provider, container, hosted pipeline or production recovery. Keep an owner, acceptance criteria, known failures and rollback/cleanup evidence with the work. The download includes source fixtures, runnable drills/tests, separate exercises and solutions, a runbook, a resource plan, an optional container/CI review pack and a capstone rubric. Credit the teaching baseline and your own contribution; no certification, commercial service or employment guarantee follows.
Authoritative reference
— primary/official reference for the mechanism. Lesson explanations and fictional cases are original. Provider details can differ; this lesson does not configure or provision an account. No production access, payment, public registration or live deployment is activated.
Try the activity before reading its reference. Keep your sources, commands and assumptions with your work. Companion activities are unsaved self-practice and do not award a certificate or change lesson progress.