Library
Machine Learning

Gradient Descent in Plain Sight

Think of standing on a hilly surface in fog; gradient descent walks downhill by taking steps opposite to the steepest slope, scaled by a learning rate.

Scene 1 of 4
The foggy hill
J(θ)J(\theta)
Imagine you're on a hilly landscape in fog and want to reach the lowest valley. You can only feel the slope under your feet.
Step-by-step solver
1

Define the loss

We minimize a loss function J of parameters theta.

J(θ)J(\theta)
2

Compute the gradient

Find the direction of steepest increase.

g=∇J(θ)g = \nabla J(\theta)
3

Update opposite to the gradient

Subtract a scaled step in the gradient direction.

θ←θ−η g\theta \leftarrow \theta - \eta\, g
4

Repeat until convergence

Stop when the gradient is near zero or steps stop changing the loss.

∥∇J(θ)∥≈0\|\nabla J(\theta)\| \approx 0

Original question

Explain gradient descent and how it learns, with a visual analogy.

Follow-up chat

Ask me anything about this lesson — I'll answer using what we just covered.