Dev.to AI 🤖 Ai 👁 0 📖 7 min read

AdamW: Why Decoupling Weight Decay from the Gradient Fixes Adam

Adam is the default optimiser for almost every transformer, and for years most people used it with weight decay switched on, assuming that was the same as regularisation. It was not. The weight decay in Adam is rescaled

Adam is the default optimiser for almost every transformer, and for years most people used it with weight decay switched on, assuming that was the same as regularisation. It was not. The weight decay in Adam is rescaled by the same adaptive step sizes that make Adam work, so it behaves nothing like the weight decay in SGD. The fix, AdamW, is a one-line change with a surprisingly large effect: take the decay out of the gradient and apply it directly to the weights. This article explains why the original version was broken, what the decoupled version does differently, and when it matters in practice.

What weight decay is for

Weight decay is the simplest form of regularisation in deep learning. Add a penalty on the size of the weights to the loss:

L(θ)=Ldata​(θ)+2λ​∣θ∣22​

The penalty pulls every weight toward zero with a force proportional to its size. Large weights, which the model needs a good reason to keep, are penalised most. In a Bayesian reading, this is a Gaussian prior on the weights. In a practical reading, it is a way of telling the optimiser that a slightly worse fit with much smaller weights is usually the better model.

In stochastic gradient descent, this penalty works exactly as intended. The gradient of the penalty is λθ , so the update becomes:

θt​=θt−1​−η∇Ldata​(θt−1​)−ηλθt−1​

Every weight shrinks by the same fraction, ηλ , at every step, regardless of its gradient history. That uniformity is the whole point. Weight decay is a steady, predictable pull toward zero, and the learning rate and the decay strength are two separate knobs.

Adam in four lines

Adam replaces the single global step size with a per-parameter step size, computed from the history of each gradient. It tracks two moving averages, the first moment mt​ and the second moment vt​ :

mt​=β1​mt−1​+(1−β1​)gt​,vt​=β2​vt−1​+(1−β2​)gt2​

After bias correction, the update is:

m^t​=1−β1t​mt​​,v^t​=1−β2t​vt​​
θt​=θt−1​−ηv^t​​+ϵm^t​​

A parameter with large, noisy gradients gets a small effective step; a parameter with small, consistent gradients gets a large one. This adaptivity is why Adam trains transformers well with minimal tuning. But it also quietly breaks weight decay.

The bug: L2 inside Adam is not weight decay

When you pass weight_decay to Adam, the library adds the L2 penalty to the loss, which means it adds λθt−1​ to the gradient:

gt​=∇Ldata​(θt−1​)+λθt−1​

That gradient then flows through the whole adaptive machinery. The decay term is smoothed into mt​ , squared into vt​ , and finally divided by v^t​​+ϵ . To first order, the effective decay applied to a weight is roughly:

ηv^t​​+ϵλθt−1​​

Compare that with SGD, where the decay is ηλθt−1​ . In Adam, the decay is divided by the root-mean-square gradient of that parameter. A weight with large gradients, exactly the kind of weight that tends to grow large, gets its decay shrunk. A weight with tiny gradients gets its decay amplified. The regularisation strength now depends on the gradient scale of each parameter, which changes during training and differs across layers. Two weights with the same size but different gradient histories are regularised differently, for no good reason.

Worse, the coupling makes hyperparameter tuning misleading. In Adam with L2, the learning rate and the decay strength interact through the adaptive denominator, so tuning one changes the effective value of the other. The clean separation that makes weight decay interpretable in SGD is gone. Loshchilov and Hutter, who identified this in 2019, showed that Adam with L2 regularisation is not equivalent to Adam with weight decay at all: no choice of λ in the L2 version reproduces the behaviour of true decoupled decay, because the rescaling depends on the data.

The fix: decouple the decay

AdamW applies the decay directly to the weights, outside the adaptive update:

θt​=θt−1​−ηλθt−1​−ηv^t​​+ϵm^t​​

Read it as two separate operations. The second term is the usual Adam step, driven by the data. The first term is pure shrinkage: every weight loses the same fraction ηλ , independent of its gradient history. The decay is now a genuine prior on the weights again, and the learning rate and decay strength are separate knobs, as they are in SGD.

The change is small, but its practical consequences are not. In the original paper, AdamW generalised better than Adam with L2 on image classification benchmarks, and it changed the optimal hyperparameters: with decoupled decay, larger decay values became useful, and the best learning rate and the best decay could be tuned independently. That decoupling is the real win. A regulariser you can tune without retuning the learning rate is a regulariser you will actually tune well.

There is also a subtle interaction worth knowing. Because the decay term is multiplied by the learning rate η , a learning rate schedule also schedules the decay: under cosine decay, the effective decay fades to zero as training ends. Some implementations keep a separate decay schedule, or apply the decay with a fixed coefficient, precisely to avoid this. If your final epochs seem to overfit more than expected, the fading decay is a plausible suspect.

A minimal implementation

In PyTorch, the difference is one class name, but the names hide a real distinction:

import torch
from torch import nn

model = nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6)

# This is NOT AdamW: weight_decay here is L2 regularisation,
# added to the gradient and then rescaled by the adaptive step.
adam_l2 = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.1)

# This is AdamW: the decay is applied directly to the weights.
adamw = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)

The second detail that matters is which parameters get decayed. The standard practice is to decay only the weight matrices, not biases or normalisation parameters:

decay, no_decay = [], []
for name, param in model.named_parameters():
    if not param.requires_grad:
        continue
    if param.ndim == 1 or name.endswith("bias"):
        no_decay.append(param)
    else:
        decay.append(param)

optimizer = torch.optim.AdamW(
    [
        {"params": decay, "weight_decay": 0.1},
        {"params": no_decay, "weight_decay": 0.0},
    ],
    lr=3e-4,
    betas=(0.9, 0.95),
    eps=1e-8,
)

Biases and normalisation scales are one-dimensional, and shrinking them toward zero does not regularise the function the network computes in the way matrix decay does. Decaying them anyway is usually harmless but wasteful, and in some architectures it measurably hurts. Most training frameworks now apply this split by default, but if you build the optimiser by hand, it is worth doing explicitly.

Typical decay values for AdamW are larger than people expect from the Adam-with-L2 habit: 0.01 to 0.1 is common for transformers, where the equivalent L2 value in Adam would have been far smaller. If you switch from Adam to AdamW, do not carry the old decay value across. Retune it.

Limitations and common pitfalls

AdamW fixes the decay, not Adam. A few things it does not do:

It does not remove the need for warmup. AdamW inherits Adam's early-training instability, when the second-moment estimate is still noisy. Warm up the learning rate as before.

It does not fix Adam's other weaknesses. The non-convergence examples constructed for Adam, and the cases where SGD with momentum generalises better, are unaffected by decoupling the decay. If Adam underperforms SGD on your problem, AdamW may close part of the gap, but it is not a guarantee.

It is not a substitute for data or early stopping. Weight decay controls the size of the weights, not the information in the data. A model that memorises a small dataset will still memorise it, only with slightly smaller weights.

The decay value is still a hyperparameter. Decoupling makes it easier to tune, not free. Too much decay underfits, and the symptom, a training loss that plateaus too high, looks like a learning rate problem. If your model underfits after switching to AdamW, check the decay before the learning rate.

Watch the interaction with the learning rate schedule. As noted above, the effective decay scales with the learning rate in the standard formulation, so an aggressive cosine schedule quietly turns the regularisation off at the end of training. If that matters for your setup, use a decay schedule that is independent of the learning rate.

Takeaway

Adam with L2 regularisation applies weight decay through the adaptive denominator, so each weight is regularised by an amount that depends on its own gradient history. That is not weight decay; it is a data-dependent distortion of it. AdamW moves the decay outside the adaptive step, so every weight shrinks by the same fraction, the learning rate and the decay strength become separately tunable, and generalisation improves. It is a one-line change, and it is the reason AdamW, not Adam, is the default optimiser in most modern training setups.

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.