I put a real LLM in my portfolio. It runs on your GPU, not my server.

Every AI-engineer portfolio says "I build with LLMs." I wanted mine to prove it in the first ten seconds.

So the chat box on my homepage has a button: Run a real LLM on your GPU. Click it and a 0.5B-parameter model downloads into your browser, boots on your graphics card through WebGPU, and answers questions about my work. No server. No API key. Nothing you type leaves your machine.

👉 Try it: rahul-agarwal.in (desktop Chrome or Edge works best)

Here's how it works, and the four things I got wrong before it worked well.

What happens when you click the button

1. The page checks for WebGPU and whether your GPU supports 16-bit floats.
2. It lazy-loads WebLLM from a CDN (none of this code ships until you ask for it).
3. It downloads Qwen2.5-0.5B-Instruct (about 270 MB, 4-bit quantised). Your browser caches it, so the second visit is instant.
4. Every question runs a tiny retrieval step over my notes, then the model answers using only what was retrieved.
5. Each generated token fires a pulse through the animated neural network behind the hero, so you can literally watch it think.

Loading the model (the part that's easier than you'd expect)

const adapter = await navigator.gpu.requestAdapter();
const f16 = adapter.features.has('shader-f16');
const model = f16
  ? 'Qwen2.5-0.5B-Instruct-q4f16_1-MLC'
  : 'Qwen2.5-0.5B-Instruct-q4f32_1-MLC'; // fallback for GPUs without fp16

const webllm = await import('https://cdn.jsdelivr.net/npm/@mlc-ai/[email protected]/+esm');
const engine = await webllm.CreateMLCEngine(model, {
  initProgressCallback: r => updateProgressBar(r.progress, r.text),
});

That's it. WebLLM exposes an OpenAI-style API, so streaming looks familiar:

const chunks = await engine.chat.completions.create({
  messages, stream: true, temperature: 0.2, max_tokens: 160,
});
for await (const c of chunks) {
  const token = c.choices[0]?.delta?.content || '';
  output.textContent += token;
  window.neural.fire(1); // light up the hero network
}

Two details matter: pin the CDN version (an unpinned import can change under you), and check for shader-f16 instead of assuming it. Plenty of laptop GPUs don't have it.

Grounding a 0.5B model with a 30-line RAG

A half-billion-parameter model knows nothing about me and will happily invent things. So it never answers from memory. Every question goes through a retrieval step first:

// Simplified from the real code, which also weights longer keywords higher
function retrieve(question) {
  const q = question.toLowerCase();
  return NOTES
    .map(note => ({ note, score: note.keys.filter(k => q.includes(k)).length }))
    .filter(x => x.score > 0)
    .sort((a, b) => b.score - a.score)
    .slice(0, 2)
    .map(x => x.note);
}

const SYSTEM = `Answer in 1 to 3 short sentences using ONLY the notes provided.
If the notes don't contain the answer, say you don't know.
Never invent companies, numbers, dates or links.`;

The notes are a dozen short paragraphs I wrote (projects, stack, experience). Keyword scoring sounds crude next to embeddings, but with a dozen documents it's instant, deterministic and easy to debug. Under every answer the UI shows what it used:

Q: Where does he work now?
A: Rahul is the sole engineer at Aryan News Agency, where he builds and develops websites and apps.
↳ retrieved 1 note (intro) · 22 tokens in 2.8s · generated on your GPU

And the test I cared about most:

Q: What is his favourite food?
A: Rahul is not specifically known for his favorite food.

No hallucination. Low temperature plus a strict "only the notes" prompt does most of the work.

Four things I got wrong

1. The first answer took 28 seconds

The model loaded fine, but the first question sat there "thinking." WebGPU compiles its shaders lazily, on the first real inference. The fix is a throwaway warm-up generation while the progress bar is still visible:

await engine.chat.completions.create({
  messages: [{ role: 'user', content: 'Hi' }], max_tokens: 4,
});

Now the slow part happens during loading, where users expect waiting, not after they've asked a question.

2. A text animation wrecked my layout score

My headline "decodes" from random glyphs into the real text. It looked great, and it gave the page a Cumulative Layout Shift of 1.41 (anything above 0.25 is "poor"). Random glyphs have different widths, so the line re-wrapped about 60 times a second and shoved everything below it around.

The fix: wrap each word in an inline-block, lock it to its real width, scramble inside it, then unwrap.

for (const w of words) {
  w.span.style.display = 'inline-block';
  w.span.style.width = w.span.getBoundingClientRect().width + 'px';
}
// ...animate...then restore plain text nodes
words.forEach(w => w.span.replaceWith(document.createTextNode(w.text)));

Same effect, layout shift 0.00. One more catch: measure after document.fonts.ready, or you lock in the fallback font's widths.

3. My CDN was editing my HTML

I added a strict Content-Security-Policy (hash-based, no unsafe-inline). Locally: zero violations. In production: errors. My domain was proxied through Cloudflare in front of Vercel, and Cloudflare was injecting its own inline scripts (analytics and bot detection) and rewriting my email address into an obfuscated link. That put an <a> inside my "copy email" <button>.

Switching the DNS records to "DNS only" removed the extra hop and the rewriting. The strict CSP went back on, and the page got faster. If you put a CDN in front of a platform that's already a CDN, check what it's adding to your pages.

4. Assuming everyone has WebGPU

Plenty of visitors (many phones, older browsers) can't run the model. The chat works without it: by default it streams pre-written answers token by token, clearly labelled as "scripted mode". The real model is opt-in, behind an explicit button that shows the download size. Nobody gets a 270 MB surprise on mobile data.

The numbers

No React, no build step. Every heavy thing (the model, the runtime) loads only when someone asks for it, and every animation pauses when it scrolls off screen.

What I'd do differently

• A 0.5B model is a demo, not a product. It's grounded and fast, but its phrasing is sometimes clumsy. With a bigger download budget, a 1.5B model is noticeably better.
• Keyword retrieval doesn't scale. Past a few dozen notes I'd switch to small embeddings computed in the browser.
• First load is the real UX problem. On a slow connection the download takes minutes. The progress bar helps, but caching the model with a service worker before anyone clicks would be smarter.

Have you shipped anything that runs a model fully client-side? I'd love to hear what you used and what broke. Drop it in the comments 👇

I'm Rahul, an AI engineer in Bengaluru who builds tools that ship. More of my work is at rahul-agarwal.in, and the code is on GitHub.