I built a LEGO builder for my son with Claude. The hard part was making sure no brick floats

I built Mister Blocks for my son. We have the classic giant box of mixed LEGO: sets long broken up, instructions long gone. Every time he asked "what can I build?", I didn't have a good answer.

Now he takes a photo of the pile, and a few minutes later there are a few builds made only from the bricks in that photo, with 3D step-by-step instructions he can turn, zoom and print.

The first working version came out of the very first prompt to Claude. The next few days went into the part nobody warns you about: making sure the build actually holds together in real plastic. This post is about that part.

The pipeline

There is no framework. It's a handful of Vercel serverless functions in plain Node, MongoDB for sessions, Cloudflare R2 for images and PDFs, and three.js for the 3D view:

api/sessions/[id]/recognize   photo  -> inventory (every piece: kind, size in studs, color)
api/sessions/[id]/design      inventory -> one model (parts with x/y/z positions + steps)
api/sessions/[id]/chat        "make it a tow truck" -> a changed model, same checks

Recognize. Claude (Opus 5.5) looks at the photo and returns structured JSON. Each piece gets an id (B3 is a brick, P7 a plate, W1 a wheel), its size in studs, its height in plates and its color. If it miscounts, you can correct it in the chat ("there are 48 of the 2×4s") and the inventory is reconciled without breaking the ids the models already use.

Design. Claude gets the inventory and returns a model: every part placed on a grid (x and z in studs, y in plates, where a brick is 3 plates tall), plus which step it goes in.

And that's where it gets interesting, because a language model's first answer is almost right very often. Almost right in LEGO means a brick floating in mid-air.

The stud checker

Every model goes through a checker before anyone sees it. It's one plain JS file (core.js), and the server and the browser load the exact same file, so what the page shows and what the server accepts can't disagree.

It checks the things that make a build fail in your hands:

• Every piece is held by studs. A piece's bottom cells have to land on studs of something below it, or the piece has to sit on the table. Slopes only have studs on their top row, so the checker knows that too.
• No two pieces occupy the same space.
• Only pieces you actually have. Every part has to exist in the inventory, be of the same kind, and be used only once.
• Wheels can turn. Nothing may come lower than the deck inside a tire's zone, except the mudguard over its own tire.

The checker returns problems in plain English, and they go straight back to Claude:

// lib/designer.js (simplified)
while (rounds < MAX_ROUNDS) {
  rounds++;
  const { data, message } = await askJSON({ system, messages, schema });
  model = toModel(data, inventory);
  issues = problems(model, inventory);   // "B4 is held by nothing", "P2 overlaps P5", ...
  if (!issues.length) break;
  messages.push({ role: 'assistant', content: message.content });
  messages.push({ role: 'user', content:
    `The checker found these problems:\n- ${issues.join('\n- ')}\n\nFix every one and answer again in full. Same rules.` });
}

Up to three rounds. In practice most models pass in one or two. This loop does more for quality than any amount of prompt tuning did: the model is good at fixing a specific, named problem, and much worse at avoiding every possible problem up front.

Steps you can actually build

The next surprise was in the instructions, not the model. A model can be perfectly valid at the end and still impossible to build in the given order, for example when step 3 places a piece that is only held by a plate that arrives in step 5. You'd be holding that piece in the air.

So there's a second pass (fixOrder) that walks the steps. For every group of new pieces in a step, it checks that they clip onto something already built or stand on the table. If they don't, it pulls the holding piece forward into that step, the way a real builder would put the plate on over the pieces it holds. Moving a piece earlier never breaks an earlier step, and a later step can only gain support, so the pass is safe to repeat until nothing changes.

Rules learned from a kid's truck

The prompt has a section of rules that came from real builds, mostly from my son's own truck. For example, the chassis that works: two wheel plates in a row, joined by a 2-wide strip of plates, a mudguard over each wheel set one stud outward, and the deck resting on the mudguards plus a center block. Before that rule existed, a lot of generated cars had decks that sat on the tires.

The rules are written as plain sentences, the same way you'd explain them to a person.

The viewer

The 3D viewer is a small three.js file shared by the build page, the guides and the print view. Every brick gets a gentle sheen and highlighted edges, so it reads as plastic rather than as flat boxes. You drag to turn, scroll to zoom, and step through the build one step at a time; pointing at a step in the list lights up its pieces and fades everything else back. Every guide can be printed as a booklet or saved as a PDF, and the guides you've opened work offline.

Prompt mode and bigger builds

Not everyone has a pile handy, so you can also describe a build in words. A few of my favourites, each a free guide you can turn in 3D, step through and print:

Boston Dynamics Spot, with its arm

The yellow-and-black robot dog on four backward-bending legs, reaching for a valve on a factory floor. 150 pieces, 13 steps. Build it →

SpongeBob's Pineapple House

The pineapple under the sea: round, wider in the middle, with the blue door and a crown of leaves. 237 pieces, 18 steps. Build it →

Porsche Mission S

Porsche's vision car in white, with the black canopy and the full-width red light bar. The big one: 965 pieces in 73 steps. Build it → (Want something quicker? There's also a 69-piece version.)

Mega Dragonite

A voxel sculpture of the dragon Pokémon: 1,315 pieces in 121 steps. Build it →

More in the gallery.

The large sculptures go through a different path. A 3D shape is described as ellipsoids, limbs and tubes, packed into standard bricks on a plate base, and given clear supports until it holds. Then it goes through the same stud check.

Cost

Each build takes a few minutes and costs a couple of dollars in API calls. Every Claude call's tokens and cost are logged to MongoDB, and there's a per-visitor daily limit plus a site-wide daily budget, so a busy day can't turn into a surprise bill.

Try it

It's free: misterblocks.com. Snap your pile and see what it comes up with. I'd love to hear what you build, and especially what it got wrong.

Unofficial fan project. LEGO® is a trademark of the LEGO Group, which does not sponsor or endorse this site. Spot, SpongeBob SquarePants, Porsche and Pokémon belong to their owners; these are fan builds.