How does a network know it's wrong?

Loss functions turn every mistake into one number. Here's how squared and absolute loss score the same misses very differently.

A neural network can't feel embarrassed when it's wrong. It needs a way to measure its mistakes, so it knows which direction to improve.

That measurement is the loss function. It turns all of a network's errors into a single number, and training is the process of pushing that number down.

One number for "how wrong"

A loss function compares the network's predictions with the right answers and returns one number:

• Zero means every prediction is perfect.
• Bigger means worse.

That's it. Everything a network learns, it learns by trying to make this number smaller.

A simple example

Say a network draws a line to predict three data points. It doesn't fit perfectly. Its predictions miss the three points by 3, 2, and 1.

How bad is that, in one number? It depends on which loss function you pick.

The network's line misses three points by 3, 2 and 1.

Squared loss

The squared loss squares each miss, then adds them up:

• 3² = 9
• 2² = 4
• 1² = 1

The total is 9 + 4 + 1 = 14. When you average it over all the data points, this is called the mean squared error, or MSE. Here, that's 14 ÷ 3 ≈ 4.7.

Squaring makes every miss positive, and it punishes big misses much more than small ones. A miss of 3 costs nine times as much as a miss of 1.

Squared loss: the misses 3, 2 and 1 cost 9, 4 and 1, for a total of 14.

Absolute loss

The absolute loss is simpler. It just adds up the size of each miss, ignoring whether the prediction was too high or too low:

3 + 2 + 1 = 6.

Averaged over the data points, this is called the mean absolute error, or MAE. Here, that's 6 ÷ 3 = 2.

What one outlier does

Now a new point arrives, and the network misses it by 5.

• Squared, that one miss costs 5² = 25. That's more than all the other misses combined (14).
• The absolute loss only adds 5.

The totals jump from 14 to 39 for squared loss, but only from 6 to 11 for absolute loss.

So one unusual point can dominate the squared loss. If your data has a few wild outliers, a network trained with squared loss will bend over backwards to reduce them, sometimes at the expense of fitting everything else.

One new point misses by 5. Squared, it costs 25, more than the other three misses combined. Absolute only adds 5.

How training uses the loss

Training adjusts the network's weights to push the loss down, one small step at a time. Each step asks: "If I nudge this weight a little, does the loss go up or down?" Then it moves the weight in the direction that lowers the loss.

Which loss you choose shapes what the network cares about. Squared loss cares a lot about big mistakes. Absolute loss treats every unit of error the same.

Training nudges the weights to push the loss down, one small step at a time.

Quick check

Your data has a few wild outliers. Which loss stays calmer: squared, or absolute?

Answer: absolute. It grows in step with each miss, while squaring blows big misses up.

Key takeaways

• A loss function turns all of a network's errors into one number. Zero is perfect.
• Squared loss squares each miss (3, 2, 1 → 9 + 4 + 1 = 14). Averaged, it's the mean squared error.
• Absolute loss adds the miss sizes (3 + 2 + 1 = 6). Averaged, it's the mean absolute error.
• Squared loss is very sensitive to outliers: a single miss of 5 costs 25.
• Training adjusts the weights, step by step, to push the loss down.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.