Neural Networks and Backpropagation

Trace a neuron and its loss

Trace a neuron and its loss

A neuron combines inputs with weights and a bias, then applies an activation. For binary classification, a logit z=x·w+b can be converted through a sigmoid to a number between zero and one. A probability-shaped output is not automatically calibrated. Binary cross-entropy compares logits with binary targets; stable implementations avoid taking logs of rounded zero probabilities. The reference uses logaddexp(0,z)−y*z on 24 synthetic training examples. No validation or test target is used in its gradient calculation.

Follow a concrete example

For a single input x=1, weight0 and bias0, the logit is0 and sigmoid is0.5. With target1, the weight gradient contribution is x*(p−y)=−0.5. A small gradient-descent step raises the weight and the predicted positive probability. This manual example illustrates direction; the packaged reference averages gradients over its24 training observations.

Further reading: PyTorch basics, NumPy 2.2 reference. The case and exercises are authored for DataCareerHub.

Your task

Write the logit and sigmoid for two features. Explain what changes when the bias increases and what the loss measures.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.

Derive and check a gradient

Derive and check a gradient

Backpropagation applies the chain rule through the computation graph. For this logistic example the weight gradient is Xᵀ(p−y)/n and the bias gradient is the mean residual probability. A finite-difference approximation perturbs one weight by a small epsilon and compares losses; it is a useful implementation check, not an efficient training method. The reference compares both gradients at fixed weights, then takes one small descent step. Keep the sample and preprocessing fixed during that comparison so the derivative refers to the same function.

Follow a concrete example

Perturb one weight by+epsilon and−epsilon while keeping all other quantities fixed. Divide the loss difference by2*epsilon and compare with the analytic coordinate. An overly tiny epsilon can amplify roundoff; an overly large one approximates a broader secant. The supplied1e−5 scale is a checked teaching choice, not a universal derivative-test parameter.

Inspect the reference implementation

Language: Python 3.12. The complete downloadable file includes the named helpers and the fresh-output command-line runner. Trace the source, transformation and result against this lesson’s contract before running it locally.

def dl_backprop(source,out):
 import numpy as np
 x,y=dl_arrays();x=x[:24];y=y[:24];w=np.array([.1,-.1]);b=.0
 def loss(w,b):
  z=x@w+b;return float(np.mean(np.logaddexp(0,z)-y*z))
 z=x@w+b;p=1/(1+np.exp(-z));grad=x.T@(p-y)/len(y);eps=1e-5
 finite=np.array([(loss(w+np.eye(2)[i]*eps,b)-loss(w-np.eye(2)[i]*eps,b))/(2*eps) for i in range(2)])
 before=loss(w,b);after=loss(w-.1*grad,b-.1*float((p-y).mean()))
 return {'training_rows':24,'loss_before':before,'loss_after_one_step':after,'analytic_gradient':grad.tolist(),'finite_difference_gradient':finite.tolist(),'max_gradient_error':float(np.max(np.abs(grad-finite))),'test_used':False}

Further reading: PyTorch basics, NumPy 2.2 reference. The case and exercises are authored for DataCareerHub.

Your task

Run the gradient lab and compare analytic_gradient with finite_difference_gradient. Predict the direction of the loss change for a sufficiently small step.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.

Separate optimization from generalisation

Separate optimization from generalisation

An optimizer updates parameters to reduce an objective; it does not choose the right task, representative data or evaluation protocol. Learning rate controls step size, while model complexity and regularization affect fit and potential overfitting. Keep train, validation and final test roles distinct. A tiny synthetic example helps inspect mathematics but cannot validate a deployed classifier. Review learning curves and error counts rather than assuming more epochs are always better. Preserve seeds and versions, while acknowledging that hardware and library changes can still affect exact numeric results.

Follow a concrete example

A gradient check verifies the derivative of the implemented objective, not whether the task or dataset is appropriate. A decreasing training loss verifies an optimization step on training data. Held-out performance and decision usefulness require separate evidence. Keep these claims at their own levels so a successful mathematical check is not overstated as deployment readiness.

Further reading: PyTorch basics, NumPy 2.2 reference. The case and exercises are authored for DataCareerHub.

Your task

Write an experiment plan with partitions, a learning-rate candidate rule and a final-test boundary. Name a sign of overfitting.

Use learner_worksheet.txt and the provided source/code files. Save your own artifact and compare the separate solution after attempting the task.