Your next LLM has a "loops" knob: what looped transformers do to latency, memory and your bill

A 36-layer model just ran 96 layers per token, and nobody changed its weights.

That's what a looped transformer does, and it means a model you deploy will soon ship with a new knob. Not temperature, not max tokens. Loop count.

NVIDIA's Dory already has it. It's a 4B model whose middle block of layers can run more than once per token, and with two or more loops its authors report better average results than selected larger models across twelve benchmarks. That's a nice headline. It's also the start of a capacity-planning conversation, and this post is about that conversation.

The 30-second architecture

A standard transformer is a fixed stack. If it has 36 layers, every token goes through 36 layers, each with its own weights, each used once.

A looped transformer (you'll also see "recurrent depth") keeps some layers shared and runs them again. Dory's layout is 8 input layers, a 20-layer recurrent core, and 8 output layers:

def executed_layers(loops, prelude=8, core=20, coda=8):
    return prelude + core * loops + coda

for r in range(1, 5):
    print(r, "loop(s):", 36, "stored,", executed_layers(r), "executed")

# 1 loop(s): 36 stored, 36 executed
# 2 loop(s): 36 stored, 56 executed
# 3 loop(s): 36 stored, 76 executed
# 4 loop(s): 36 stored, 96 executed

The weights are identical on every pass. What changes is the hidden state, the activations flowing through the core. Nothing is learned during inference and nothing carries over to the next request.

That gives you the first rule of thumb: parameter count now tells you what the model stores, not what it computes. Your GPU pays for the executed layers.

What changes in production

Here's how I'd sort the effects of turning that knob from 1 to 4:

Two rows deserve a second look.

KV cache. If your serving stack keeps a cache per loop, memory grows with loops. Dory shows that it doesn't have to. Check what your runtime actually does before you size anything.

Fewer tokens. It's tempting to read "1.3–2.5× fewer generated tokens" as "cheaper." It isn't automatically. A model can write less while doing more work per token. Measure the whole request.

The question to ask before you turn the knob up

Loops add computation, not information.

Take two facts: Alice finished before Bob, Bob finished before Cara. Ask who finished first, and extra passes help, because the answer is already in the input and just needs assembling. Ask for Alice's finishing time and no number of passes will help. The fact isn't there. A retrieval call or a tool will.

I call this the Alice test: is the answer already in what the model has? Run it before you touch the knob, then sort the failure:

• has the facts, fumbles multi-step reasoning → more loops (or a bigger reasoning budget) are a fair test
• the fact isn't in the prompt or the weights → retrieval or a tool call
• arithmetic or exact strings come out wrong → a calculator or code tool
• right but slow → fewer loops
• worse the longer it thinks → stop earlier

That last line isn't hypothetical. Fastino reports that a third pass hurt their model on a causal-reasoning benchmark. More passes can improve an answer, flatten out, or make it worse. Compute only goes one way.

A small evaluation harness

If you get access to a looped checkpoint, this is the shape of the test I'd run. Same prompts, same sampling, same tool access, only the loop count changes:

results = []
for loops in [1, 2, 3, 4]:
    for task in eval_set:                      # tasks with a pass/fail check
        t0 = time.perf_counter()
        answer = model.generate(task.prompt, loops=loops, **fixed_settings)
        latency = time.perf_counter() - t0
        results.append({
            "loops": loops,
            "task_type": task.kind,
            "passed": task.check(answer),      # tests, reference, schema
            "latency_s": latency,
            "tokens_out": count_tokens(answer),
        })
# Then: pass rate vs p50/p95 latency per loop count, split by task_type.
# Pick the loop count where the marginal gain flattens. Confirm on fresh examples.

Look at the marginal return. If 1 → 2 loops helps and 2 → 4 adds little, 2 is your default. Split by task type, because an average can hide one class of task that gets worse. And confirm on examples you didn't tune on.

Where this leaves us

Looped transformers aren't magic and they aren't hype. They're a trade: smaller files, deeper computation, real latency. The papers are early (Huginn in 2025, Dory, GLiDE and a stability preprint called RLT this November), and every number above is reported by its authors for a specific setting.

The habit worth building now is to ask two questions of every model card: how many layers does it store, and how many does it run?

Question for you: would your workload take 35% more latency for half a point of quality? I'd like to know where that line sits for agents versus chat. Tell me in the comments.

This post is the developer cut of a longer guide I wrote, Looped Transformers Explained, originally published on piszczek.pl. The original has seven interactive labs, including a loop machine you can run at 1 to 4 passes and a cost ledger that updates per loop.

I drafted this with AI assistance and checked every number against the sources linked in the original.