Without one bend, 100 layers act like one

Why every neural network needs an activation function, explained with two simple equations.

Deep networks stack layer after layer. But there's a catch hiding in that stack. Without one small ingredient, a hundred layers can do exactly what one layer does, and nothing more.

That ingredient is the activation function. Here's why it matters, in a few lines of algebra.

What a layer does

Every layer in a neural network does two things:

1. It multiplies and adds: each input is multiplied by a weight, the results are summed, and a bias is added.
2. It passes the result through an activation function: a small bend.

The first step is simple arithmetic. The second step is the one that's easy to overlook, and it's the one that makes depth worth having.

What happens without the bend

Let's remove the activation function and watch what happens with two tiny layers.

• Layer one doubles its input and adds one: h = 2x + 1.
• Layer two triples that and subtracts four: y = 3h − 4.

Now put them together. Substitute the first into the second: y = 3(2x + 1) − 4 = 6x − 1.

Two layers, and the result is a single multiply-and-add. It's still just one straight line.

Two layers without a bend: 2x + 1, then 3h − 4. Combined, they collapse into a single line, 6x − 1.

Stacking doesn't help

This isn't a coincidence of the numbers we picked. Any multiply-and-add fed into another multiply-and-add is still a multiply-and-add.

So stack a hundred of these layers, and they still collapse into one. All that depth, and the network can only draw a straight line.

Add the bend: ReLU

Now add an activation function. The most popular one is called ReLU (rectified linear unit). Its rule is:

• Negative numbers become zero.
• Positive numbers pass straight through.

So −3 becomes 0, and 2 stays 2. That's it. It looks almost too simple to matter.

ReLU: negative inputs become zero, positive inputs pass straight through.

From bends to curves

That one kink at zero is enough to stop the collapse. The layers can't be merged into a single line anymore.

Better still, bends can be combined. Add two ReLUs, one of them flipped, and you get a peak. Combine many of them, with different positions and slopes, and they trace a curve, one straight piece at a time.

This is how networks build complicated functions out of very simple parts.

Many ReLU pieces, each a straight segment with one bend, combine to trace a curve.

Sigmoid vs ReLU

Older networks often used a smooth S-shaped curve called sigmoid, which squashes any number into the range between 0 and 1.

Today, ReLU is the standard choice in most networks, because it's simple and fast to compute. It also tends to make deep networks easier to train.

Sigmoid, a smooth S-curve, was common in older networks. ReLU is the standard choice today.

Quick check

A network has ten layers, with no activation functions. Can it draw a curve, or only a straight line?

Answer: only a straight line. Without a bend, all ten layers collapse into one multiply-and-add.

Key takeaways

• Every layer multiplies, adds, and then applies an activation function.
• Without the activation, any stack of layers collapses into a single straight line, like 2x + 1 and 3h − 4 becoming 6x − 1.
• ReLU turns negatives into zero and passes positives through. That one bend stops the collapse.
• Many bends combined can trace curves.
• Sigmoid was common in older networks. ReLU is the default today.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.