Too big, it explodes. Too small, it crawls.

The learning rate, step by step: why one number decides whether training works at all.

Backpropagation tells every weight which way to move: its gradient. It doesn't say how far. That's the job of the learning rate, written η (eta). Every step of gradient descent is

w ← w − η ∇L

The gradient comes from the model. The learning rate is a setting you choose, and it is the first one people tune. This article shows what it does on the smallest possible example, why there is a hard limit on how big it can be, and how practitioners pick it and change it during training. The ideas follow the CS878 Deep Learning lecture on learning rates.

One weight on a bowl

Take a single weight and the loss L(w) = w², a bowl with its bottom at w = 0. The gradient is 2w, so one step is

w ← w − η · 2w = (1 − 2η) · w

Start at w = 2, where the loss is 4, and try four learning rates:

• η = 0.01: each step multiplies w by 0.98. After ten steps, w has only reached 1.634 and the loss is still 2.67. It crawls.
• η = 0.4: each step multiplies w by 0.2. Two becomes 0.4, then 0.08, then 0.016. Fast.
• η = 0.9: each step multiplies w by −0.8. The weight jumps to the other side of the bowl each time (−1.6, 1.28, −1.024, 0.819), but the jumps shrink, so it still lands.
• η = 1.1: each step multiplies w by −1.2. The jumps grow: −2.4, 2.88, −3.456, 4.147. The loss climbs 5.76, 8.29, 11.94, 17.20. It explodes.

One weight on the bowl L = w², four learning rates: crawls, fast, zig-zags, explodes.

Why the edge is at one

Everything depends on the factor 1 − 2η. After t steps, w_t = (1 − 2η)^t · w₀.

• Below η = 0.5, the factor is between 0 and 1: the weight slides in without overshooting.
• At η = 0.5, the factor is 0: one jump lands exactly on the bottom.
• Between 0.5 and 1, the factor is between −1 and 0: the weight flips sides but shrinks.
• At η = 1, the factor is −1: it bounces between 2 and −2 forever.
• Past 1, the factor is below −1: it grows every step.

Nothing here is special to w². For any bowl L = (λ/2) w², with curvature λ (the second derivative), the factor is 1 − ηλ. So training is stable only if 0 < η < 2/λ, and one jump lands at η = 1/λ. For L = w², λ = 2: the edge is 1 and the one jump is 0.5.

The factor 1 − 2η decides everything: slides in below 0.5, one jump at 0.5, zig-zags up to 1, explodes past 1.

Millions of directions

A real network has millions of weights, so the loss has millions of directions, and they are not equally steep. Take two weights and L = ½(w₁² + 10 w₂²): one direction with curvature 1 (flat), one with curvature 10 (steep). Each step multiplies w₁ by (1 − η) and w₂ by (1 − 10η).

Starting from (4, 1):

• η = 0.02: the steep direction shrinks fast (×0.8 per step) but the flat one barely moves (×0.98). After ten steps the weights are at (3.27, 0.11): done on the steep side, stuck on the flat side.
• η = 0.21: 0.21 is past 2/10, the edge of the steep direction. w₂ is multiplied by −1.1 each step: 1, −1.1, 1.21, −1.331, 1.464. It explodes.
• η = 2/11 ≈ 0.18: the best single fixed rate, 2/(λ_min + λ_max). Both directions shrink by 9/11 per step, and after ten steps the weights are at (0.54, 0.13).

The steep direction sets the speed limit, η < 2/λ_max, and the flat direction then crawls. Even the best fixed rate shrinks the error by only (κ − 1)/(κ + 1) = 9/11 per step, where κ = λ_max/λ_min = 10 is how uneven the valley is.

The valley L = ½(w₁² + 10w₂²): the steep side sets the speed limit; the best fixed step is 2/11.

Picking a starting rate

Two quick checks get you most of the way:

1. Before training, look at the first loss. For softmax with C classes and small random weights, it should be about ln C, which is 2.30 for ten classes. If it isn't, something else is wrong.
2. Try a few learning rates a factor of ten apart: 1e-1, 1e-2, 1e-3, 1e-4. Run each for about a hundred iterations and keep the one whose loss drops fast without blowing up. Too small barely moves; too big turns into NaN.

Trying 1e-4 to 1e-1 from the same starting loss of 2.30: keep the one that drops fast.

Try it in PyTorch

The bowl example, at the learning rate that explodes:

w, lr = 2.0, 1.1
for t in range(4):
    w = w - lr * 2 * w
    print(round(w, 3))

It prints −2.4, 2.88, −3.456, 4.147. Change lr to 0.5 and w lands on 0 in one step.

In a real training loop the learning rate is one argument of the optimizer. A quick sweep tries a few values a factor of ten apart:

import torch
# model = your network
for lr in (1e-1, 1e-2, 1e-3, 1e-4):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    ...            # train ~100 iterations, keep the lr whose loss drops fast

Shrinking the rate during training (decay schedules) is its own episode.

Key takeaways

• The learning rate sets how far each gradient step goes; it is chosen, not learned.
• On a bowl with curvature λ, each step multiplies the error by 1 − ηλ: stable only if η < 2/λ, one jump at η = 1/λ.
• In a real network the steepest direction sets the speed limit, so the flat directions crawl.
• Start by checking the first loss (ln C), then try rates a factor of ten apart and keep the one that drops fast.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.