Machine Learning

Demystifying Gradient Descent in Neural Networks

This lesson breaks down how neural networks learn by conceptually bridging the loss function, the backpropagation of error, and the iterative gradient descent update rule.

Scene 1 of 5

The Network Architecture

f_\theta(x) = W_2\sigma(W_1x + b_1) + b_2

A neural network is just a function that takes inputs and maps them to outputs through layers of weights. Think of it as a complex machine with thousands of knobs and dials that we need to tune to get the right answer.

Step-by-step solver

(a) Network Definition

Define a one-hidden-layer network with weights, biases, and activation function.

z_1=W_1x+b_1, \ h=\sigma(z_1), \ \hat{y} = W_2 h + b_2

(b) Loss Definition

Define the empirical risk as the average squared error over the training set.

L(\theta) = \frac{1}{N}\sum_{i=1}^N \frac{1}{2}\|f_\theta(x_i)-y_i\|^2

Compute partial derivatives using the chain rule, moving backward from the output layer to the input.

\delta_2 = \hat{y}-y, \ \frac{\partial L}{\partial W_2} = \delta_2 h^T, \ \delta_1 = (W_2^T\delta_2) \odot \sigma'(z_1)

(d) Update Rule

Update the parameter vector $\theta$ by moving in the negative gradient direction scaled by the learning rate $\eta$.

\theta_{t+1} = \theta_t - \eta \nabla_\theta L(\theta_t)

Original question

Explain how gradient descent trains a simple neural network. Describe the loss, the gradient and the weight update rule.

Follow-up chat

Ask me anything about this lesson — I'll answer using what we just covered.