Model Versioning and Experiment Tracking

Define an experiment record

Define an experiment record

An experiment record ties the problem, source snapshot, feature contract, code, environment, split, model and metrics together. A run name alone cannot reproduce a result. Hash the source and canonical serialized model, and record target units and training rows. The fixed reference fits 40 telemetry rows and reports separate validation and final-test MAE. It does not choose a model from test performance. A fresh future tuning cycle needs its own selection protocol and an untouched final sample.

Follow a concrete example

Hash canonical model content as well as the original source bytes. Indentation changes can alter a file-byte hash without changing model values, so state which identity you use. The reference records a canonical serialized model hash and the telemetry source hash. Include code/version/configuration context before using the record to compare future candidates.

Further reading: Docker build guidance, Python HTTP server reference. The case and exercises are authored for DataCareerHub.

Your task

Create an experiment record and identify every input needed to reproduce the model hash.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.

Compare metrics with their populations

Compare metrics with their populations

A metric needs partition, denominator, unit and computation rule. MAE averages absolute prediction errors in hours on the same rows for all compared models. Keep validation and final-test results separate; do not average them into a development score then tune against it. A training-mean baseline should be fitted on the training population and evaluated on the same held-out rows. The reference’s small synthetic sample illustrates mechanics rather than real operating accuracy. Preserve failed or weaker candidate results in an honest comparison log.

Follow a concrete example

A validation MAE of0.5hours and a test MAE of0.7hours would be two scoped observations, not an averaged development score; these numbers are illustrative, while the supplied expected file contains executed values. Keep baseline predictions fitted on training only and evaluate them on the same rows. Denominators and units travel with every metric.

Inspect the reference implementation

Language: Python 3.12. The complete downloadable file includes the named helpers and the fresh-output command-line runner. Trace the source, transformation and result against this lesson’s contract before running it locally.

def mlops_tracking(source,out):
 model=fit_service(source);raw=json.dumps(model,sort_keys=True).encode();traces=[]
 for period in ['validation','test']:
  d=[r for r in rows(source,'telemetry.csv') if r['period']==period]
  errors=[abs(float(r['actual_hours'])-predict_payload({'distance_km':float(r['distance_km']),'load_ratio':float(r['load_ratio'])},model)['prediction_hours']) for r in d]
  traces.append({'partition':period,'n':len(d),'mae_hours':sum(errors)/len(errors),'training_mean_baseline_mae_hours':sum(abs(float(r['actual_hours'])-statistics.mean(float(t['actual_hours']) for t in rows(source,'telemetry.csv') if t['period']=='train')) for r in d)/len(d)})
 return {'experiment_id':'haven-duration-v1-reference','model_sha256':hashlib.sha256(raw).hexdigest(),'source_sha256':hashlib.sha256((Path(source)/'telemetry.csv').read_bytes()).hexdigest(),'metrics':traces,'training_rows':40,'selection_note':'Fixed model; test is final reporting only, not used for selection.'}

Further reading: Docker build guidance, Python HTTP server reference. The case and exercises are authored for DataCareerHub.

Your task

Read the partition metrics and write a baseline-comparison specification. Explain why test cannot choose a replacement version.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.

Track artifacts and promotion decisions

Track artifacts and promotion decisions

Artifact identity and release status are different concepts. A hash identifies bytes; it does not mean a model is approved. Use explicit development, reviewed, candidate and approved states under an owner’s process, and keep a rollback reference. Record why a candidate is rejected or paused. The learning reference produces a versioned record but does not register an operational model, grant service access or promote anything automatically. An audit trail should show the evidence and decision rather than a fabricated production badge.

Follow a concrete example

Write a release decision separate from the artifact identity. A model can be reproducible and still unsuitable for deployment. The record should say who reviewed the evidence, what is missing and which prior approved artifact could be restored in a real release process. This learning example has no invented operational approval or customer deployment.

Further reading: Docker build guidance, Python HTTP server reference. The case and exercises are authored for DataCareerHub.

Your task

Write a promotion decision for the supplied synthetic model and a rollback record template.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.