Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Transformer Architecture 101: From RNNs to Attention, Step by Step

Think of a Large Language Model (LLM) as an engine for language, and the Transformer as the internal combustion mechanism that makes it run. In this post, we'll walk step by step from absolute fundamentals to a practical

Think of a Large Language Model (LLM) as an engine for language, and the Transformer as the internal combustion mechanism that makes it run. In this post, we'll walk step by step from absolute fundamentals to a practical mental model of how Transformers operate.

Table Of Contents

  • What Is an LLM, and Why Transformers?
  • The Internal Blueprint
  • Encoder vs. Decoder Architectures
  • What Is Attention and Self-Attention?
  • Query, Key, and Value (Q, K, V)
  • Multi-Head Attention and Feed-Forward Networks
  • Why Positional Encoding?
  • The Complete Journey of a Sentence
  • Knowledge-Check Questions
  • Practical Exercises

1. What Is an LLM, and Why Do We Need Transformers?

At its core, a text-generative LLM operates on a surprisingly simple premise: next-token prediction. Given a sequence of text, it asks: "Based on all the words seen so far, what is the single most likely token to come next?"

The Problem with What Came Before (RNNs and LSTMs)

Before Transformers arrived in 2017, the standard tools for sequences were Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs).

  • The "Telephone Game" Bottleneck: RNNs read text one word at a time, left to right. To understand word #20, the model had to pass a running summary (the "hidden state") through words #1, #2, #3, and so on. Like the children's game of telephone, earlier details got compressed, corrupted, or forgotten over long distances (the vanishing gradient problem).
  • The Hardware Ceiling: Each word depended on the calculation of the word before it, so sequential processing could not be parallelized across modern GPUs. Training on trillions of words was computationally impossible with RNNs.

The Transformer Breakthrough

Transformers eliminated the sequential chain. Instead of reading word by word, a Transformer processes all words in a sequence simultaneously. It connects every token directly to every other token through a mechanism called attention, removing the memory bottleneck and unlocking massive GPU scale.

RNN / LSTM (Sequential - Slow, Forgets):
[Token 1] ──► [Token 2] ──► [Token 3] ──► [Token 4] (Early words fade)

Transformer (Parallel - Fast, All-to-All Direct Access):
[Token 1] ◄──────────────► [Token 2]
    β–²   \                  /   β–²
    β”‚     \              /     β”‚
    β–Ό       \          /       β–Ό
[Token 3] ◄──────────────► [Token 4]

2. The Internal Blueprint of a Transformer

A Transformer takes raw text and transforms it through three major stages:

  1. The Embedding Layer: Turns raw text into vectors that encode meaning and sequence position.
  2. The Transformer Blocks (repeated N times): The core engine. Multi-Head Self-Attention routes context across tokens, and Feed-Forward Networks (MLPs) refine each token's representation. Models stack dozens of these blocks (12 in GPT-2 small, 96 in GPT-3).
  3. The Output Layer (Un-embedding & Softmax): Translates the final vectors into probabilities over a vocabulary to predict the next token.

3. Encoder vs. Decoder Architectures

The original 2017 paper, Attention Is All You Need, introduced an Encoder–Decoder architecture designed for translation. Today there are three main families:

  • Encoder-Only (e.g., BERT): Processes all input tokens at once and lets tokens look in both directions. It builds deep, bidirectional representations, which suits classification, search, and sentence analysis.
  • Encoder–Decoder (e.g., T5): The encoder processes the full source sentence (say, English), and the decoder generates the target (say, French) token by token, combining self-attention with cross-attention to the encoder's output.
  • Decoder-Only (e.g., GPT, LLaMA): Most modern generative LLMs drop the encoder entirely. A single stack of blocks generates text token by token. To stop the model from "cheating" by looking at future words during training, it uses causal masking: each token can only attend to itself and earlier tokens.

4. What Is Attention and Self-Attention?

Attention in Plain English

Consider this sentence:

"The trophy did not fit in the suitcase because **it* was too big."*

What does "it" refer to? You immediately focus on "trophy". If the sentence ended with "...because it was too small", your attention would shift to "suitcase".

Attention is the mathematical mechanism that lets a word look at surrounding words, decide which ones provide essential context, and blend their information into its own representation.

Self-Attention

In older networks, words carried static meanings. Self-attention makes word representations dynamic:

  • In isolation, the word "bank" is ambiguous.
  • In "river bank", self-attention connects "bank" with "river", pulling in geographic meaning.
  • In "money in the bank", it connects "bank" with "money", pulling in financial meaning.

Every token simultaneously "attends" to every other available token, updating its own vector to reflect the exact context.

5. Query, Key, and Value (Q, K, V)

To decide which words matter, the Transformer borrows an idea from search engines:

Vector Real-World Analogy Meaning for a Token
Query (Q) The search text you type into Google "What kind of context am I looking for?"
Key (K) The title/tags of pages in the index "What information or label do I offer?"
Value (V) The actual page content "If you match with me, here is the information to pass along."

The Math Made Intuitive (Scaled Dot-Product Attention)

For every token, the network multiplies its embedding by three learned weight matrices ( WQ​,WK​,WV​ ) to produce its Q, K, and V vectors.

  Input Embedding
     β”‚   β”‚   β”‚
  xWQβ”‚xWKβ”‚xWVβ”‚ (Learned Projection Matrices)
     β–Ό   β–Ό   β–Ό
     Q   K   V
  1. Dot product ( QKT ): Take a token's Query and dot it against the Keys of all tokens. A higher score means higher relevance.
  2. Scale by dk​​ : Dividing by the square root of the key dimension keeps numbers from growing too large, which keeps training gradients stable.
  3. Apply causal masking (for text generation): Future positions are masked with βˆ’βˆž so the model can't look ahead.
  4. Softmax: Normalizes the scores into weights between 0 and 1 that sum to 1.
  5. Weighted sum of Values: Multiply the weights by each token's Value vector. Relevant tokens contribute heavily; irrelevant ones are ignored.
Attention(Q,K,V)=softmax(dk​​QKT​+M)V

6. Multi-Head Attention and Feed-Forward Networks

Multi-Head Attention: Multiple Perspectives at Once

A single attention pass gives only one perspective on relevance, but language is multi-layered.

  • Analogy: A team of reviewers reads a document at the same time. One checks grammar, another tracks pronouns and their nouns, a third monitors tone and topic.
  • In Multi-Head Attention, the vectors are split across independent "heads" (12 in GPT-2 small, 96 in GPT-3). Each head has its own learned projections and can focus on a different relationship. At the end, all head outputs are concatenated and projected back to the original dimension.

Feed-Forward Networks (MLP): Digesting Context

Attention routes information between tokens. Afterwards, each token passes through an independent Feed-Forward Network (MLP):

  • It operates on each token separately.
  • It expands the vector (usually 4Γ—, e.g., 768 β†’ 3072 in GPT-2), applies a non-linearity such as GELU or ReLU, then projects back down.
  • Intuition: If self-attention is "talking to others", the MLP is "privately digesting what was heard".

Crucial Stabilizers: Residuals and Layer Normalization

  • Residual (skip) connections: The input of each sublayer is added back to its output ( x+Sublayer(x) ), so signals and gradients flow cleanly through deep stacks.
  • Layer Normalization (LayerNorm / RMSNorm): Rescales values to keep mean and variance consistent, preventing numbers from blowing up across layers.

7. Why Do We Need Positional Encoding?

Because Transformers process every token at once, basic attention is permutation-invariant: it treats text as an unordered "bag of words". To a pure attention matrix, "the dog bit the man" and "the man bit the dog" look identical.

Positional encoding fixes this by generating a unique signal for position 0, 1, 2, and so on, and adding it to the token embeddings.

  • The original Transformer used sinusoidal (sine and cosine) functions of varying frequencies, which lets the network infer relative distances.
  • Modern models often use learned positional embeddings or rotary position embeddings (RoPE).

8. The Complete Journey: How a Sentence Travels Through a Transformer

Let's trace the prompt "The dog sat on the":

1. RAW INPUT TEXT: "The dog sat on the"
   β”‚
2. TOKENIZATION: Split into subword tokens and mapped to vocabulary IDs
   β”‚
3. EMBEDDING + POSITIONAL ENCODING:
   - Look up 768-dim semantic vectors for each token
   - Add positional vectors (Pos 0, Pos 1, Pos 2...)
   β”‚
4. TRANSFORMER BLOCK STACK (Blocks 1 through 12):
   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   β”‚ LayerNorm ──► Multi-Head Self-Attention                β”‚
   β”‚               (Tokens calculate Q, K, V;               β”‚
   β”‚                "sat" connects to "dog";                β”‚
   β”‚                Causal mask hides future words)         β”‚
   β”‚ ──► + Residual Skip Connection                         β”‚
   β”‚ ──► LayerNorm ──► Feed-Forward MLP (Processes context) β”‚
   β”‚ ──► + Residual Skip Connection                         β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   β”‚
5. FINAL LAYER & LOGITS:
   - The last token's ("the") final vector goes through the un-embedding matrix
   - Produces raw scores ("logits") over the ~50,000-word vocabulary
   β”‚
6. SAMPLING & NEXT TOKEN:
   - Scale by Temperature, filter with Top-k / Top-p
   - Softmax gives probabilities (e.g., "mat": 45%, "rug": 22%, "grass": 15%)
   - The model samples: "mat"
   β”‚
7. REPEAT AUTOREGRESSIVELY: Append "mat" and run another forward pass

Knowledge-Check Questions

  1. Why was parallelization across GPUs impossible with standard RNNs and LSTMs?
  2. How does information flow differ between an encoder-only model (BERT) and a decoder-only model (GPT)?
  3. Why do we need positional encodings if word embeddings already capture meaning?
  4. In the search engine analogy, what do Query, Key, and Value represent?
  5. Why do we divide the dot product by dk​​ ?
  6. What is the role of causal masking in decoder models during training?
  7. If self-attention shares information between tokens, what is the distinct job of the feed-forward network?
  8. What problem do residual connections solve in deep Transformers?
  9. How does multi-head attention help a model understand a sentence better than a single head?
  10. Why does standard attention become expensive for long context windows (quadratic scaling, O(N2) )?

3 Practical Exercises

Exercise 1: Trace the Winograd Schema

  • Sentence A: "The city councilmen refused the demonstrators a permit because they feared violence."
  • Sentence B: "The city councilmen refused the demonstrators a permit because they advocated violence."

For both sentences, treat "they" as the Query token. Which Key ("councilmen" or "demonstrators") should get the highest attention score in A versus B, and why?

Exercise 2: Manual Q, K, V Calculation

Two tokens, with:

  • Q1​=[1,2]
  • K1​=[2,1],Β V1​=[1,0]
  • K2​=[1,3],Β V2​=[0,1]

Compute Q1​⋅K1​ and Q1​⋅K2​ . Which Value vector contributes more to Token 1's contextual update?

Exercise 3: Architecture Detective

Which architecture (encoder-only, decoder-only, or encoder-decoder) would you pick, and why, for:

  1. Classifying customer support emails as spam vs. urgent
  2. Writing creative bedtime stories
  3. Translating legal contracts from Spanish to German

Your Turn!

Try explaining the Transformer back in your own words: write a short paragraph, describe how you visualize Q, K, and V, or answer the questions and exercises above. Share your answers in the comments. I'd love to see how you think about it!

Found this useful? Drop a ❀️ and follow for more AI fundamentals.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.