A GPT-2 transformer in a 1.4MB language: fixed-point parity with NumPy

We are a tiny bootstrapped team building MsrfTM - a small, strict programming language with a self-hosting compiler (no LLVM, no Rust runtime, no C runtime) that emits PE32+, ELF64 and WASM. It ships with its own AI library. This is the technical story of how we proved our fixed-point AI runtime matches full-precision floating point on a real transformer.

The goal: parity, not vibes

"Fast" and "small" mean nothing if the math is wrong. So we set a hard gate: our GPT-2-style transformer running in pure MsrfTM (Q16 fixed-point, integer arithmetic) must match an fp64 NumPy reference implementation on the loss - to well under 1%.

Result: 0.51% loss difference (5.6489 fixed-point vs 5.6201 fp64) on a real forward+loss pass. Element-level bisection localized the residual to known quantization noise: embeddings +-1 LSB, layer norm 0.3-0.4%, attention 0.3%, GELU 0.15%. Zero structural bugs.

How fixed-point held up

Q16 fixed-point replaces every float with a scaled integer. The risks are well known: overflow in accumulation, drift in softmax normalization, GELU approximation error. What made it work:

• Per-op range budgets: every intermediate is bounded before it is scaled
• Attention scores stay in Q16 throughout the softmax; the exp table is pre-scaled
• Layer norm stats computed with widening intermediates so the variance never wraps
• A bisection harness that flips one element of the reference to +-1 LSB and localizes exactly which stage diverges - so we could tell "quantization noise" from "structural bug" at every layer

The suite that keeps us honest

The whole self-host suite is 175/175 passing. Parity is a repeatable one-command run, not a one-time screenshot - and every benchmark number on our site traces to a checked-in artifact.

What is in the AI library

• MLP with backprop, conv1d/conv2d + maxpool, attention, LSTM/GRU, layer norm, softmax, GELU
• MNIST 784->64->10 trained to 74.5% test accuracy - no Python, no frameworks
• ONNX import (protobuf parser included), SPIR-V compute shader emission for Vulkan, AVX2 SIMD kernels
• A TinyStories-33M model exported and generating text autoregressively (20 tokens, anti-repetition)
• int8 quantization for tiny models, plus a neural classify/story API serving live

Why a language, not a library?

Because the AI runtime is part of the language's standard library, the same source runs on Windows (PE32+), Linux (ELF64) and the browser (WASM) with zero dependencies - nothing to install, nothing to configure. The whole toolchain is ~1.4 MB.

Try it

The Code Studio runs real MsrfTM in your browser right now: https://msrfteam.com/msrftm

We are a tiny unfunded team and we build everything in public. If you have worked with fixed-point ML or numerics - we would genuinely love your critique: what would you test that we have not?