I built a tiny GPT from scratch to show my kids how chatbots really work

What actually happens inside a GPT? Everyone talks about "attention" and "transformers" and "training", but if you had to build one with nothing but arrays of numbers, where would you even start? I've been coding for over 30 years and I couldn't have told you.

So I built a small one in Python and called it PyLLM. There's no PyTorch and no TensorFlow, just numpy for the array maths. The autograd engine, the layers, attention, the tokenizer, the optimiser, the training loop and the sampling are all plain Python you can read.

pip install fromscratch-pyllm

Dream up some Pokémon

PyLLM ships with a tiny model that's already trained, so it starts writing straight away:

$ pyllm --seed 0 --max-new-tokens 40
machop
machoke
machamp
bellsprout
weepin

It wrote that one letter at a time: it guessed m, then a, then c, and fed each guess back in as the start of the next one. (weepin gets cut off because I only gave it 40 characters.) A GPT does exactly this, just with far more practice.

Chopping text into Lego bricks

A model can't read letters. First the text gets chopped into tokens, a bit like snapping a sentence into Lego bricks, and every brick gets a number. The model only ever sees the numbers:

$ pyllm tokenize --text "Pika"
'P' -> (I have not seen that character yet)
'i' -> 9
'k' -> 11
'a' -> 1

Here every character is its own brick. The alphabet was learned from lower-case Pokémon names, so a capital P simply isn't in it. (There's also a BPE tokenizer that learns bigger bricks like "ing", which is closer to what the big models use.)

The breadcrumb trail

Learning means asking every number in the model: if I nudge you up a little, does the answer get better or worse, and by how much? That's a gradient, and working it out automatically is called autograd.

The docs explain it with breadcrumbs. Every time PyLLM's Tensor does some maths, it drops a breadcrumb saying who its parents were and how to pass the blame back to them. Then backward() follows the trail in reverse:

from pyllm.autograd import Tensor

a = Tensor([2.0])
b = Tensor([3.0])
c = a * b + a      # c = 2*3 + 2 = 8
c.backward()
print(c.data, a.grad, b.grad)
# [8.] [4.] [2.]

a was used twice, once multiplied by 3 and once on its own, so its blame adds up to 4. Each op is a few lines: work out the answer, then leave a small _backward recipe behind.

How do you know the breadcrumbs are right? You check them the slow, obvious way: nudge each number a tiny bit and see how far the output moves. PyLLM's numerical_grad does exactly that, and the tests use it on every op and layer. Everything else in PyLLM sits on top of this one idea.

Re-reading a sentence

When you read "The trophy didn't fit in the case because it was too big", you glance back to work out what "it" means. Attention is the model doing the same thing. Each token asks a question (a query), every earlier token holds up a label (a key), and the good matches hand over their information (a value). A causal mask stops anyone peeking at letters that haven't been written yet.

A few attention "heads" side by side, plus a bit of private thinking (the feed-forward part), make one Transformer block. Stack a few blocks and you've got a GPT. The bundled Pokémon model has two blocks and 59,739 numbers in total. A real chatbot has billions.

The guessing game, three ways

Training is a guessing game. Guess the next letter, find out how surprised you should have been (the loss), follow the breadcrumbs to share out the blame, nudge every number a little, and repeat.

PyLLM gives you a ladder of three models, each one fixing a weakness of the one below. I trained each for 2,000 steps on the bundled list of 149 Pokémon names:

$ pyllm train --model bigram --steps 2000 --seed 0 --out bigram.npz
$ pyllm train --model mlp    --steps 2000 --seed 0 --out mlp.npz
$ pyllm train --model gpt    --steps 2000 --seed 0 --out gpt.npz

Then I asked each one for names with the same seed (pyllm --checkpoint <model>.npz --seed 3 --max-new-tokens 60):

This was the most interesting thing I learned. The bigram only sees one letter, so it babbles. The MLP makes up names that sound right, like "flearowth" and "mandshrew". The GPT has the lowest loss and mostly just recites real Pokémon in order. (It starts with "choke" rather than "machoke" because it begins with nothing to read.)

With only 149 names to learn from, the strongest model basically memorised the list, and the bundled checkpoint does too. Over 20 seeds at the default setting, only 7 of the 223 complete names it wrote weren't real Pokémon.

You can push it to be bolder with a dial called temperature. Think of the model's scores as a weighted dice where the letters it likes get bigger sides. At 0 it always picks the favourite. Turn it up and the dice flattens out, so long shots get a chance:

$ pyllm --seed 0 --temperature 1.5 --max-new-tokens 60
ndbell
victafble
vulpix
nizweepinbell
victreebel
tentacool
t

Counted the same way at 1.5, 66 of 217 names were new. "Nizweepinbell" is my favourite.

Teaching it a language with no data

The main demo is harder. Pebble is a small programming language from the same series, and there's no pile of Pebble code anywhere in the world to learn from. So PyLLM grows its own. The docs put it as planting your own garden instead of foraging for wild berries.

A generator builds random programs by rolling dice at every choice ("loop or print? add or multiply?"), but it only makes moves the grammar allows, so every program it writes is valid. Add the real snippets from Pebble's docs and you get the bundled corpus: 1,001 programs, about 354,000 characters.

Because the series has Pebble's real parser too, we can mark the model's homework honestly. Does what it wrote actually parse?

$ pyllm pebble --seed 7 --temperature 0.5 --score
let total = n * (5 + n)
        let a = 6
    }
    let a = result * (6 - result)
    }
}
...
parse rate: 0.0% of 1 programs

(Output trimmed.) It looks like Pebble, but the braces are all over the place. Over 20 seeds, only 4% of 27 programs parsed.

Call in the referee

So PyLLM brings in a referee. A football referee doesn't play and doesn't tell anyone where to kick. They know the rules, and when someone tries an illegal move they blow the whistle, the move doesn't count, and the player has to choose again.

PyLLM's referee is a small hand-written parser that answers two questions about half-written code:

from pyllm.pebble import referee

for text in ["let total = 5 +", "let total = 5 +\n", "let total = 5 + 2", "let total = = 5"]:
    print(f"{text!r:22} viable={referee.viable(text)!s:5} complete={referee.complete(text)}")

'let total = 5 +'      viable=True  complete=False
'let total = 5 +\n'    viable=False complete=False
'let total = 5 + 2'    viable=True  complete=True
'let total = = 5'      viable=False complete=False

While the model writes, every character it picks gets checked. If the code could no longer turn out valid, the whistle goes: that character is crossed off the dice and the model rolls again. Same tiny model, but now:

$ pyllm pebble --seed 3 --constrained --score --max-new-tokens 300
let a = 10

let x = calc() == 7 + 15 - (n + 97)
print(step(1 - 89 - z, y - 8))

parse rate: 100.0% of 2 programs

Over the same 20 seeds, 100% of 37 programs parsed.

There's a catch, though: parsing and working are different things. Save that program to a file and run it with Pebble itself (the pebble command comes with the [pebble] extra):

$ pebble s3.pbl
3 | let x = calc() == 7 + 15 - (n + 97)
            ^
Undeclared function 'calc'

The referee checks the grammar, not the meaning. To be fair to the model, the generated training programs call made-up functions like compute() and combine() too, so it learned to write exactly what it was shown.

What real LLMs do that this doesn't

PyLLM is built to be read, so it keeps the simple version of a lot of things.

The obvious one is scale: about 60,000 numbers here, billions in a real model, trained on a large chunk of the internet on racks of GPUs. It's the same loop at a very different size. The bundled models also read one character at a time, where real models use learned sub-word tokens (the BPE tokenizer shows the idea), and the Pokémon model can only look back 16 characters, where real ones look back hundreds of thousands of tokens.

There's no GPU and no caching of earlier attention work either, so every step is plain numpy. And real assistants get extra training to follow requests and be helpful, while PyLLM only ever learns to continue text.

The referee is the one part that matches production systems. They use the same "veto illegal tokens" trick to force valid JSON or code, and PyLLM's version is small enough to read through.

Try it

• Install: pip install fromscratch-pyllm (add 'fromscratch-pyllm[pebble]' for parse scores and the pebble command), then run pyllm
• Docs, with every idea explained simply: https://robcox2910.github.io/pyllm/
• Code: https://github.com/robcox2910/pyllm

PyLLM is part of From scratch in Python, a series of 18 projects that rebuild real systems: a database, an operating system, git, a search engine, a programming language and more. Get the whole series with pip install fromscratch-pystack, or start at https://robcox2910.github.io/

How this was made: I designed PyLLM and directed every part of it, but most of the code was written with Claude Code, under strict test-driven development, and this article was drafted with Claude's help too. Every output shown above comes from actually running it.