Dev.to AI 🤖 Ai 👁 0 📖 6 min read

Inkling 2 Mini GGUF on Your Own Device: Which Quant Fits 16, 24 and 32 GB, and What the KV Cache Adds

TL;DR Thinking Machines released Inkling 2 Mini on November 20, 2026 under Apache 2.0: 26.2B parameters, 3.4B active per token, a 1M-token context window, text and image input. Unsloth published GGUF builds the same we

TL;DR

Thinking Machines released Inkling 2 Mini on November 20, 2026 under Apache 2.0: 26.2B parameters, 3.4B active per token, a 1M-token context window, text and image input. Unsloth published GGUF builds the same week. This note does the memory math an edge builder needs before downloading anything. The ten GGUF files run from 7.23 GiB to 21.3 GiB, and only 4 of the 24 layers use global attention, so the cache for a 128K prompt is 2.0 GiB if the runtime shares it across the three loop passes and 4.0 GiB if it keeps one per pass. The result: a 16 GB machine can run the IQ3 and Q3 files with a mid-size context, a 24 GB machine fits Q4_K_M with 128K, and a 32 GB machine fits the Q8 file with room for the vision projector.

GGUF file size by quant

Sources: the Unsloth GGUF repository and the base model card and config. File sizes are read from the repository listing. Cache figures are our arithmetic from config.json.

What is Inkling 2 Mini, in terms an edge builder cares about?

Four facts from the card and config decide everything below:

  • Sparse MoE. 24 layers, 128 routed experts, 8 chosen per token, no shared experts. Only a small slice of the weights is read for each token, which keeps token speed reasonable on unified-memory laptops.
  • Looped middle. Layers 6 to 17 run three times with shared weights. The card says each extra pass adds about 1.0B effective active parameters, so the working figure is 5.4B, not 3.4B. Weights are stored once, so the loop costs compute but not file size.
  • Hybrid attention. 20 of the 24 layers are local with a 1,024-token sliding window. Layers 5, 11, 17 and 23 are global.
  • Multimodal. A vision tower sits beside the decoder. In GGUF form it lives in a separate mmproj file (about 0.6 GiB in BF16 or F16, 1.19 GiB in F32).

For deployment this combination is friendly: the stored weights are small for the capability, and the long-context cost is carried by only four layers.

How big is each GGUF file, and how many bits per weight is that?

The GGUF header counts 26,070,390,272 parameters. Dividing each file size by that count gives the effective bits per weight:

Quant File (GB) File (GiB) Bits per weight Same top token as Q8
UD-IQ1_S 7.76 7.23 2.38 89.4%
UD-IQ1_M 8.02 7.47 2.46 90.2%
UD-IQ2_XXS 8.49 7.90 2.60 90.6%
UD-Q2_K_XL 9.14 8.51 2.80 91.6%
UD-IQ3_XXS 11.66 10.86 3.58 94.7%
UD-Q3_K_XL 13.63 12.70 4.18 96.3%
UD-IQ4_XS 15.73 14.65 4.83 97.7%
UD-Q4_K_M 16.61 15.47 5.10 97.9%
UD-Q4_K_XL 17.63 16.42 5.41 98.0%
UD-Q8_K_XL 22.87 21.30 7.02 reference

The last column is Unsloth's own report, measured against its Q8 build. It gives a useful ordering of the files and shows a clear knee: from IQ4_XS upward the agreement stays between 97.7% and 98.0% while the file grows by 1.9 GB. Below 3 bits per weight the agreement sits at 89 to 92 percent.

Agreement with the Q8 reference

The per-weight figures sit above the nominal names (Q4_K_M is 5.10, not 4) because these are dynamic mixes: embeddings, attention and the router stay at higher precision while the many expert tensors take the low-bit types.

How much memory does the KV cache add at 32K, 128K and 1M tokens?

Per global layer, one token stores K and V for 8 heads of dimension 128. At bf16 that is 2 x 8 x 128 x 2 = 4,096 bytes. The runtime question is how many layer caches exist.

bytes_per_layer_token = 2 * 8 * 128 * 2          # K+V, 8 KV heads, dim 128, bf16
global_layers = [5, 11, 17, 23]                   # not in local_layer_ids
core = range(6, 18)                               # loop core, 3 passes
shared = len(global_layers)                       # cache reused across passes -> 4
per_pass = sum(3 if l in core else 1 for l in global_layers)   # 11 and 17 x3 -> 8

for n in (32768, 131072, 262144, 1048576):
    print(n, n * bytes_per_layer_token * shared / 2**30,
             n * bytes_per_layer_token * per_pass / 2**30)
# 32768   0.5   1.0
# 131072  2.0   4.0
# 262144  4.0   8.0
# 1048576 16.0  32.0   (GiB)

KV cache bounds by context length

The two columns are bounds, not measurements: which one a given runtime hits depends on whether it stores a separate cache for each loop pass. Plan with the larger one and any saving is a bonus. The 20 local layers add only 80 MiB (shared) to 160 MiB (per pass) in total, because the window is 1,024 tokens, so they are small next to the global cache.

Quantized cache types shrink the number further. With an 8-bit cache the per-pass 128K figure falls to about 2.1 GiB, and with a 4-bit cache to about 1.1 GiB, at the usual cost of some long-context accuracy.

Which quant fits a 16, 24 or 32 GB device?

Budget = weights + KV cache + mmproj (if you want images) + runtime buffers and the OS. A practical reserve for the OS and buffers is 2 to 4 GB on a laptop and about 1.5 GB on a headless box.

Device memory File to try Weights Cache (per-pass bound) Notes
16 GB UD-IQ3_XXS 10.86 GiB 1.0 GiB at 32K leaves about 2 GiB for mmproj and buffers
16 GB, text only UD-Q3_K_XL 12.70 GiB 1.0 GiB at 32K tight, short context
24 GB UD-Q4_K_M 15.47 GiB 4.0 GiB at 128K about 3 GiB to spare with mmproj
32 GB UD-Q8_K_XL 21.30 GiB 4.0 GiB at 128K the card's recommended class of machine
32 GB, long context UD-Q4_K_XL 16.42 GiB 8.0 GiB at 256K trade precision for window

Two practical rules fall out of the table. First, for most assistants and coding agents a prompt of 32K to 128K is already generous, and that costs 0.5 to 4 GiB of cache, so the file size is the main lever. Second, moving from Q4_K_M to Q8_K_XL costs 5.8 GiB for a 2.1 point gain in top-token agreement, which is worth it only when the memory is otherwise idle.

How do I serve it and check the numbers myself?

Use the mainline-compatible files in the Mainline_Support folder if you run stock llama.cpp, and the plain files with Unsloth Desktop. A minimal launch with an 8-bit cache:

huggingface-cli download unsloth/Inkling-2-Mini-GGUF \
  Mainline_Support/Inkling-2-Mini-UD-Q4_K_M.gguf_file mmproj-BF16.gguf \
  --local-dir ./inkling

llama-server -m ./inkling/Mainline_Support/Inkling-2-Mini-UD-Q4_K_M.gguf_file \
  --mmproj ./inkling/mmproj-BF16.gguf \
  -c 131072 -ctk q8_0 -ctv q8_0 -ngl 99 --port 8080

Check the actual footprint with nvidia-smi or Activity Monitor after the first long prompt, not at idle, since cache memory is often allocated lazily. Compare the reading with the table above to learn which cache bound your runtime follows, then keep that number for capacity planning. The .gguf_file suffix on the mainline files is how the repository ships them; point your loader at the path as shown.

What should an on-device release checklist include?

Our edge notes keep returning to the same short list, and this model fits it well:

  1. Read the file sizes from the repository listing, not from the quant name.
  2. Compute the cache for your real context length and for the worst-case runtime behavior.
  3. Include the vision projector in the budget if the product takes images.
  4. Re-test the chat template and tool-call format on the quantized file, since the template here carries a thinking-effort header that your app must set.
  5. Pick the file by measured agreement and your own task set, then freeze it.

FAQ

Does the loop make the GGUF bigger? No. The three passes reuse the same stored weights, so file size follows the 26.1B parameter count. The loop raises compute per token to about 5.4B effective active parameters.

Which file is the best default on a 24 GB machine? UD-Q4_K_M at 15.47 GiB. It is within 0.1 points of Q4_K_XL in the Unsloth agreement figure and leaves room for a 128K cache and the projector.

Can I run the 1M-token context on a laptop? The global-layer cache alone is 16 GiB (shared) to 32 GiB (per pass) in bf16, so plan for 256K or less unless you have a workstation or a quantized cache.

Do I need the mmproj file for text-only use? No. It is only for image input. Skipping it saves about 0.6 GiB.

Where do these cache numbers come from? From config.json: 8 KV heads, head dimension 128, global layers 5, 11, 17 and 23, loop core layers 6 to 17 with 3 passes. Re-derive them with the snippet above.

Is the quant ordering the same for my task? Agreement with the Q8 build is a proxy. Run a small set of your own prompts on two neighbouring files before you ship.

Further reading

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.