Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Scaled Dot-Product Attention: How Content Decides Where Information Flows

Every token in a sequence needs to know which other tokens matter. A word like "it" in "the animal did not cross the street because it was too tired" only makes sense once the model looks back at "animal". Convolution lo

Every token in a sequence needs to know which other tokens matter. A word like "it" in "the animal did not cross the street because it was too tired" only makes sense once the model looks back at "animal". Convolution looks at neighbours. Recurrence looks at the past one step at a time. Attention looks everywhere at once, and decides for itself where to look.

That decision - routing information between positions based on content, not on fixed distance - is the single idea that made the transformer possible. This article works through the mechanism precisely: what the query, key and value matrices actually compute, why the scaling factor matters, what multi-head attention adds, and where the whole scheme breaks down.

The routing problem

Consider a sequence of n tokens, each represented by a vector of dimension d. The task at each position is to build a new representation that incorporates information from the other positions. A fixed rule, such as averaging the neighbours, cannot adapt: sometimes the relevant token is adjacent, sometimes it is twenty positions away, and sometimes it does not exist at all.

Attention solves this by letting each position compute a set of weights over all positions, where the weights depend on the content of the tokens themselves. The output at position i is a weighted average of the values at all positions, with weights chosen by a compatibility score between position i and every other position.

Formally, stack the token vectors into a matrix X∈RnΓ—d . Attention projects X into three matrices:

Q=XWQ​,K=XWK​,V=XWV​

where WQ​,WK​,WVβ€‹βˆˆRdΓ—dk​ are learned projection matrices. Each row of Q is a query: what this position is looking for. Each row of K is a key: what this position offers. Each row of V is a value: the information this position contributes if selected.

The attention formula

Given Q, K and V, scaled dot-product attention is:

Attention(Q,K,V)=softmax(dk​​QKT​)V

Read it from the inside out. QKT is an n by n matrix of dot products: entry (i, j) measures how compatible the query at position i is with the key at position j. Divide by dk​​ , apply softmax row-wise to turn each row into a probability distribution over positions, and multiply by V to take the weighted average.

Each row of the output is therefore a convex combination of the value vectors, with the mixing weights determined by content similarity. Nothing about the mechanism fixes which positions a token may attend to: the model learns the projections, and the projections decide the routing.

Why divide by the square root of d_k

The scaling factor looks like a detail. It is not. Suppose the components of a query vector q and a key vector k are independent with mean 0 and variance 1. Then their dot product qβ‹…k=βˆ‘i=1dk​​qi​ki​ has mean 0 and variance dk​ , so its standard deviation is dk​​ . As dk​ grows, the raw dot products spread out, and the softmax saturates: one entry dominates, the rest are driven to zero, and the gradients through the softmax vanish.

Dividing by dk​​ keeps the logits at unit scale regardless of dimension, so the softmax stays in the regime where its gradients are informative. This is the same reasoning that motivates careful initialisation of neural networks: keep the scale of activations stable as dimensions change.

There is a subtlety worth noting. The independence assumption is false in a trained model - queries and keys are correlated by design, since the model wants large dot products for related tokens. The scaling is a stabiliser, not a guarantee. It prevents saturation at initialisation, which is when training is most fragile, and the learned projections take over from there.

Softmax as differentiable selection

The softmax is doing real work here. A hard selection - attend only to the single best-matching token - would be non-differentiable and impossible to train by gradient descent. Softmax is a smooth relaxation of argmax: with temperature 1 it produces a distribution that can be sharp or diffuse, and gradients flow through the weights back into the projections.

It helps to see attention as a form of kernel smoothing, the same idea behind the Nadaraya-Watson estimator in nonparametric statistics. Given query q, the output is:

Attention(q,K,V)=j=1βˆ‘nβ€‹βˆ‘l=1n​exp(qβ‹…kl​/dk​​)exp(qβ‹…kj​/dk​​)​vj​

This is a weighted average of the values with weights given by a normalised kernel of the query-key similarity. The "kernel" is the exponential of the scaled dot product, and the "bandwidth" is fixed by the scaling. Viewed this way, attention is a learned, differentiable memory lookup: the keys are the addresses, the values are the contents, and the query selects what to retrieve.

Multi-head attention

One attention mechanism can only learn one routing pattern. A token might need to attend to its syntactic parent for grammar, to a distant pronoun antecedent for coreference, and to the previous token for local fluency, all at the same time. A single softmax distribution cannot express all three.

Multi-head attention runs h attention mechanisms in parallel, each with its own learned projections, on a smaller dimension dk​=d/h :

MultiHead(Q,K,V)=Concat(head1​,…,headh​)WO​

where headi​=Attention(QWQi​,KWKi​,VWVi​) and WO​ projects the concatenated output back to dimension d.

Each head is free to specialise. Empirical studies of trained transformers find heads that track syntactic dependencies, heads that attend to the previous token, heads that focus on rare tokens, and heads that do little at all. The heads are not assigned roles by the designer; the specialisation emerges from training, because different routing patterns reduce the loss in different ways.

A minimal implementation makes the structure concrete:

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, V), weights

# Q, K, V: (batch, heads, seq_len, d_k)
# returns output and the attention weights for inspection

The returned weights are worth inspecting directly. When a model behaves unexpectedly, the attention pattern is often the first diagnostic: a head that attends uniformly is not routing at all, and a head that attends only to the first token may be doing something subtler than it looks.

What attention cannot do on its own

Attention is permutation equivariant: if you permute the input tokens, the output permutes in the same way. The mechanism has no notion of order. That is why positional encodings - sinusoidal, learned, or rotary - are added to the input before attention is applied. Without them, "dog bites man" and "man bites dog" would produce the same representations. This is also why the RoPE article in this series matters: the position encoding is where the order information lives, not in the attention operation itself.

The second limitation is cost. Computing QKT is quadratic in sequence length, both in time and in memory. A sequence of 100,000 tokens needs 10 billion attention scores per head, per layer. This is the reason long-context models rely on sparse attention, sliding windows, linear attention, or state space models: the quadratic term eventually dominates everything else.

A third, subtler failure mode is rank collapse. In deep transformers, repeated attention layers tend to make token representations more similar to each other, because every token is an average of the same set of values. Residual connections and layer normalisation slow this down, but the pressure is always there. This is one reason deep transformers need careful normalisation and why their representations can become overly smooth.

Finally, attention weights are not explanations. A large weight on a token does not prove the model used that token to make its decision, and a small weight does not prove it was ignored. The weights describe the routing of information into a representation, not the causal role of that information in the output. Treating attention maps as faithful explanations is a common pitfall; they are a diagnostic, not a justification.

When to use it, and the common pitfalls

Attention is the right tool when the relevant context for a prediction is content-dependent and potentially distant: language modelling, machine translation, protein folding, and any task where the interaction structure is unknown in advance. It is the wrong tool when the structure is known and local - a convolution will beat it on both speed and sample efficiency for images - or when sequences are very long and the quadratic cost dominates.

The most common practical pitfalls:

  • Forgetting the scale. Implementing attention without the 1/dk​​ factor trains poorly at large head dimensions, and the failure looks like a mysteriously flat loss rather than an obvious bug.
  • Masking errors. In causal language modelling, the future must be masked before the softmax, not after. Masking after the softmax leaks probability mass and produces subtly wrong distributions.
  • Attending to padding. Padding tokens must be masked out, or the model learns to route information through meaningless positions and the attention weights become hard to interpret.
  • Assuming heads are interpretable. Inspecting one head in isolation says little. The meaningful object is the combination of all heads plus the output projection.

The takeaway

Scaled dot-product attention is a differentiable routing mechanism: queries select keys, and the selected values are averaged. The scaling keeps the softmax trainable, the multiple heads let different routing patterns coexist, and the whole operation is a learned generalisation of kernel smoothing. Its limits - no sense of order, quadratic cost, and a tendency to blur representations - explain most of the architecture that surrounds it: positional encodings, sparse attention, residual connections, and normalisation.

Understand attention as routing, and the transformer stops looking like a black box and starts looking like a design: a mechanism for letting content decide where information flows, repeated at every layer.

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.