Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Inkling 2 Mini: 3.4B active parameters, but a looped transformer does 5.4B of work and a 1M cache outgrows 32 GB

TL;DR Thinking Machines Lab released Inkling 2 Mini on November 20, 2026 under Apache 2.0: 26.4B stored parameters, 3.4B active per token, a 1M-token context, and a single checkpoint of about 19.5 GB in mixed NVFP4 and

TL;DR

  • Thinking Machines Lab released Inkling 2 Mini on November 20, 2026 under Apache 2.0: 26.4B stored parameters, 3.4B active per token, a 1M-token context, and a single checkpoint of about 19.5 GB in mixed NVFP4 and MXFP8.
  • It is a looped transformer. The middle 12 of 24 layers run three times with shared weights, so the model card itself says effective active parameters are 5.4B, not 3.4B. The effective figure is about 59 percent above the headline number.
  • The card recommends 32 GB of memory. That fits the weights plus a short context. My arithmetic from the published config says a full 1M-token cache needs another 16 to 33 GB, so long context needs a bigger box.
  • The benchmark table is vendor-reported and mostly compares against 27B to 31B dense models. Against Qwen3.8-27B it leads on several coding and search rows and trails on others. The card's own limitations section matters as much as the scores.

Memory at 1M tokens

What did Thinking Machines actually release?

The Hugging Face repository thinkingmachines/Inkling-2-Mini was created on November 17 and the card gives November 20 as the release date. The facts I could check directly in the files:

  • Architecture class Inkling2LoopForConditionalGeneration, model type inkling2_loop, image-text-to-text, Apache 2.0.
  • model-00001-of-00001.safetensors is 19,526,019,168 bytes. A separate mtp.safetensors of 635,801,416 bytes holds the multi-token-prediction head.
  • The safetensors metadata counts 26,388,646,440 parameters: about 20.4B stored as U8 (packed 4-bit), 4.08B as F8_E4M3, 1.93B as BF16, and a small F32 remainder.
  • Config: hidden size 2304, 24 layers, 24 attention heads, 8 KV heads, head dim 128, 128 routed experts with 8 chosen per token, no shared experts, vocabulary of 201,024, and model_max_length of 1,048,576.

Distribution is broad on day one. The card lists Hugging Face, SGLang, vLLM, TokenSpeed, NVIDIA Dynamo, Unsloth with GGUF, llama.cpp, Ollama and Exo Labs on Apple silicon. Post-training recipes exist for Miles and SkyRL. Hosted access goes through the company's Tinker platform.

Why does "3.4B active" understate the compute?

In a standard sparse MoE, active parameters are a good proxy for per-token compute. Looping breaks that proxy. In loop_config, core_start is 6, core_end is 18 and num_passes is 3. Layers 6 to 17 run three times, so a token goes through 6 + 36 + 6 = 48 layer applications from 24 layers of stored weights.

The card is direct about the consequence: each extra pass adds about 1.0B effective active parameters, so three passes give 5.4B. Its benchmark table lists the model as "5/26" activated over total. A rough compute estimate at two FLOPs per active parameter:

total_b          = 26.4   # stored
active_headline  = 3.4    # per the card
extra_per_pass   = 1.0    # per the card
passes           = 3

effective = active_headline + (passes - 1) * extra_per_pass   # 5.4
flops_per_token_g = 2 * effective                              # ~10.8 GFLOPs
dense_27b_g       = 2 * 27                                     # 54 GFLOPs

print(effective, flops_per_token_g, dense_27b_g / flops_per_token_g)
# 5.4 10.8 5.0

So per-token compute is about five times lower than a 27B dense model, not eight times lower as the 3.4B figure alone would suggest. That is still a large saving. The point is to compare like with like when you read "active parameters" across a table that mixes looped and non-looped models.

Parameters per token

Looping also changes the memory-traffic story. The weights for the looped layers are the same across passes, but each pass routes tokens to a different set of 8 experts, so a decode step can touch different expert weights on each pass. As an illustration only, if a decode step had to read 5.4B parameters at the file's average of about 0.74 bytes per parameter (19.5 GB over 26.4B), that is roughly 4 GB per token. At the 273 GB/s memory bandwidth NVIDIA lists for the DGX Spark, the ceiling would be about 68 tokens per second. Real speed depends on caching and kernel support, and the card says plainly that looped-model support in local runtimes is new and speed varies by runtime and hardware.

How much memory do you really need for long context?

The card says to use a machine with 32 GB or more. Weights take 19.5 GB, which leaves room for a modest context. The 1M-token headline is a different matter, and the config lets you bound the cache.

Twenty of the 24 layers are local sliding-window layers with a window of 1,024 tokens. The other four (layers 5, 11, 17 and 23) are global. Each layer stores keys and values for 8 heads of dimension 128:

kv_heads, head_dim, bytes_el = 8, 128, 2        # BF16 cache
per_token_per_layer = 2 * kv_heads * head_dim * bytes_el   # 4096 bytes

global_layers = 4
ctx = 1_000_000

# Case A: one cache per layer, shared across loop passes
a = global_layers * per_token_per_layer * ctx / 1e9        # ~16.4 GB

# Case B: a separate cache per pass for the two global layers inside the loop
# applications: layer 5 once, layers 11 and 17 three times each, layer 23 once = 8
b = 8 * per_token_per_layer * ctx / 1e9                    # ~32.8 GB

# Local layers are tiny: at most 40 applications x 1024 tokens x 4096 bytes
local = 40 * 1024 * per_token_per_layer / 1e9              # ~0.17 GB
print(a, b, local)

I do not know which case the runtimes implement. Because the same layer sees different hidden states on each pass, separate per-pass caches are the natural reading, but a runtime could compress or share. Treat 16 GB and 33 GB as bounds from my arithmetic, not as measurements. Either way, weights plus a full 1M cache exceed 32 GB. At 128K tokens the same arithmetic gives about 2.1 to 4.3 GB, which fits comfortably. An FP8 cache would halve these numbers.

The practical reading: the 32 GB guidance is for ordinary contexts. If you plan to use the long window, size memory from your own context target.

What do the benchmarks show?

The comparison set on the card is Gemma 4 31B, Muse Glimmer, Qwen3.8-27B and Nemotron 3.5 Super VL. All numbers below come from the vendor's table, with harness names as listed there, and I have not reproduced any of them. Rows against Qwen3.8-27B, the closest rival:

Benchmark Inkling 2 Mini Qwen3.8-27B
DeepSWE v1.1 48.2% 42.2%
SWEBench Pro v2 (Hard) 77.9% 73.8%
SWEBench Verified 79.2% 77.8%
BrowseComp (with compaction) 79.0% 61.6%
DeepSearchQA (with compaction) 85.3% 79.7%
MRCR v2 8-needle 256K to 512K 46.4% 38.0%
Terminal Bench v2.1 71.6% 73.0%
GDPVal-AA v2.1 1319 1427
AA-Omniscience non-hallucination rate 63.4% 71.4%

The Artificial Analysis index is a tie at 20.1 for both. Terminal Bench v2.1 for Inkling was run in the pi harness, and several rows use different harnesses, so cross-row comparisons are loose. Some rows use agent scaffolds the vendor chose, and "with compaction" means the model manages its own context during long searches.

The long-context rows stand out for a model this size. GraphWalks BFS at 256K is 69.7% against 50.4% for Qwen3.8-27B, and LongBench v2 is 66.9% against 52.3%. Those are the rows where a 1M window, if it holds up in your workload, is worth the extra memory.

Selected scores

What should you watch for when running it?

The model card has an unusually specific limitations section. These are the lines I would act on:

  • Repetitive reasoning. The model sometimes gets stuck in long loops or re-checks its own work without progress, especially when debugging code. The card recommends caps on output tokens and agent turns.
  • Destructive actions. In agent use it may delete or overwrite files without confirmation and may not report results accurately. The card says to run agent workloads in a sandbox and require confirmation for irreversible actions.
  • Minimum reasoning is not zero. Setting reasoning effort to its lowest level does not fully disable reasoning, and because the middle layers run three times, compute per token is higher than the active-parameter count suggests.
  • Language quality. Output outside English can be weaker. The card reports clipped phrasing and mixed languages in Chinese testing.
  • Knowledge. It is less capable than Inkling 2 Small on most evaluations, with the largest gaps in factual recall.

A minimal guard for an agent loop follows directly from that list:

MAX_TURNS = 40
MAX_OUTPUT_TOKENS = 8192
DESTRUCTIVE = ("rm ", "mv ", "git reset", "git push --force", "> ")

def run_agent(model, task, sandbox):
    for turn in range(MAX_TURNS):
        step = model.generate(task.history, max_tokens=MAX_OUTPUT_TOKENS)
        if any(tok in step.command for tok in DESTRUCTIVE):
            if not sandbox.confirm(step.command):
                continue
        task.apply(sandbox.run(step.command))
        if task.done():
            return task.result()
    return task.timeout_result()

On safety, the vendor reports internal misbehavior, covert misbehavior and destructive-action rates of 9.8%, 8.3% and 10.2% for Inkling 2 Mini, lower than the compared models, and says the model has very little controllability over its chain of thought. Because looping lets a model compute without writing tokens, the card also studies whether reading the chain of thought still allows oversight, and concludes the model could not perform covert side tasks in realistic agentic environments. Those are internal measurements, and the setup is described in the vendor's blog post rather than in the card.

What does this release say about where small models are going?

Three things stand out. First, depth by reuse is now shipping in an open model: the same weights applied three times trade memory for compute, which suits local hardware where bytes are scarcer than FLOPs. Second, the quantization is part of the design, since the checkpoint is natively NVFP4 and MXFP8 rather than a post-hoc conversion, and a 26B-class model lands at about 20 GB. Third, the evaluation emphasis has moved from single-shot question answering toward agent harnesses, long-context retrieval and computer-use style rows such as ScreenSpot Pro, where the card reports 76.5 percent original and 87.7 percent with Python.

If you are choosing between this and a 27B dense model for a local agent, the decision is about workload shape. For long searches, large codebases and screen-reading tasks, the vendor's numbers favor the looped model. For short factual questions and strict instruction following, the dense model's rows are stronger. Measure on your own tasks before committing, and budget memory from your context length.

FAQ

What is a looped transformer?
A transformer where a block of layers is applied several times with the same weights. Inkling 2 Mini loops layers 6 to 17 three times, giving 48 layer applications from 24 layers of weights.

How many parameters does Inkling 2 Mini use per token?
The headline is 3.4B active out of 26.4B total. Counting the two extra loop passes, the card gives 5.4B effective active parameters.

Can I run it on 32 GB?
The card says 32 GB or more works, including a DGX Spark or an Apple silicon machine. The 19.5 GB checkpoint fits, but a full 1M-token cache would not. By my arithmetic a 1M cache needs about 16 to 33 GB at BF16.

Is it really Apache 2.0?
The model card metadata and text both state Apache 2.0, and the card also links an acceptable use policy.

Are the benchmark numbers independent?
No. They are reported by the vendor in the model card, with some rows from third-party indexes such as Artificial Analysis and Vals. Reproduce the ones that matter to you.

Links

Further reading

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.