BPE Tokenization: A Trailing Space in My Prompt Broke 41 of 300 Labels

I spent an evening blaming a quantized 8B model for being dumb. It was classifying support tickets into four labels, and roughly one answer in seven came back as garbage: a double space, a capitalized label, a stray newline, half a word. I tried a bigger quant, a lower temperature, a stricter instruction. Nothing moved.

The bug was one character. My prompt template ended with Label: and that final space was fighting BPE tokenization. When I deleted it, 41 bad outputs out of 300 dropped to 3.

This post explains why one trailing space in a prompt can do that, how to see it in your own tokenizer in 10 lines of Python, and what "token healing" is.

TL;DR

• BPE tokenizers (GPT-2, tiktoken, Llama, Mistral and most others) attach the space to the start of the next word: " billing" is one token, and a bare " " followed by "billing" almost never appears in training data.
• If your prompt ends with a space, the tokenizer encodes that space as its own token. The model now has to continue from a state it rarely saw during training, so its next-token distribution gets weird.
• The symptoms: double spaces, wrong casing, labels split into odd pieces, newlines, or the model skipping the answer entirely.
• Fix: never end a raw completion prompt or an assistant prefill on whitespace. Put the space inside the expected answer. If you can't control the boundary, use token healing (back up one token and constrain the next one to start with the removed text).
• Chat endpoints mostly hide this, but prefills and raw /completion endpoints do not.

What did the trailing space actually break?

It broke the first generated token, and everything downstream depended on that one token. My setup was a llama.cpp server running a local 8B instruct model through the raw /completion endpoint, with a few-shot template like this:

Ticket: I was charged twice this month.
Label: billing

Ticket: The export button does nothing on Safari.
Label: bug

Ticket: {ticket}
Label: 

Notice the end. Label: with a space, because I typed the template the way a human writes a form. The few-shot examples above it all had Label: billing, where the space and the word travel together.

The outputs that failed my parser looked like " billing" (an extra leading space), "Billing", "\nbilling", "bill" followed by junk, and a few that just started a new Ticket: line. Temperature was 0, so this wasn't sampling noise. The model was confidently doing the wrong thing.

How does BPE tokenization handle spaces?

BPE tokenization treats the space before a word as part of that word. GPT-2 style tokenizers mark it with Ġ, SentencePiece models like Llama 2 mark it with ▁, tiktoken just keeps a literal leading space in the token bytes. Same idea in all of them.

You can see it with any public tokenizer:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")  # any BPE tokenizer shows the same pattern

for s in ["Label: billing", "Label: ", "Label:"]:
    ids = tok.encode(s)
    print(repr(s), [tok.decode([i]) for i in ids])

What you get back:

'Label: billing' ['Label', ':', ' billing']
'Label: '        ['Label', ':', ' ']
'Label:'         ['Label', ':']

The first line is what the model saw in training millions of times: colon, then a word that carries its own space. The second line is what my prompt sent. That last ' ' is a real token, sitting alone at the end of the sequence.

Why does a trailing space in a prompt confuse the model?

Because the model is a next-token predictor trained on text that was tokenized the normal way, and the normal way almost never produces a standalone space token followed by a word. After ':' the model has learned "the next token is a space-prefixed word". After ':', ' ' it has learned very little, because that sequence barely exists in its training data.

So the model has two bad options:

1. Emit a word without a leading space, like "billing". That token exists, but it usually appears mid-word or after punctuation, not after a space token. Its probability in this position is low.
2. Emit the token it actually wants, " billing", which produces a double space in the decoded text.

Neither is what the model would do on familiar ground, so the probability mass spreads out. Some of it goes to casing variants, some to newlines, some to fragments. At temperature 0 you get whichever of those wins, and on a hard ticket a wrong one wins more often.

You can watch it happen. llama.cpp's server returns top candidates for the next token if you ask:

import requests

def peek(prompt):
    r = requests.post("http://localhost:8080/completion", json={
        "prompt": prompt, "n_predict": 1, "temperature": 0, "n_probs": 5,
    })
    # field names inside this list differ between llama.cpp versions
    print(r.json()["completion_probabilities"])

peek(template.format(ticket=t))          # ends with "Label: "
peek(template.format(ticket=t).rstrip()) # ends with "Label:"

On my tickets, the stripped prompt put one space-prefixed label far ahead of everything else. The trailing-space prompt gave a flatter list where the "right" answer was sometimes a no-space variant competing with a newline. That flatness is the whole bug.

Why did my label parser make it worse?

My parser did out == label after a .strip(). Strip saved the double-space cases, but not "Billing", not the fragments, and not the outputs that started a new ticket. Multi-token labels were the worst: feature_request is several tokens, so a bad first token derailed the rest of the word.

After removing the space, the 3 failures left were genuinely ambiguous tickets. That's a model problem. The other 38 were a tokenizer problem dressed up as a model problem.

What is token healing?

Token healing is a fix for prompts whose last characters cut a token in half. Instead of making the model continue from an awkward boundary, you remove the last token from the prompt and force the first generated token to start with the text you removed.

The classic example is a URL. A prompt ending in href="http: sends ':' as its own token, but in training data "://" is very often one token. The model rarely produces "//" on its own, so URLs come out mangled. Token healing backs up past the : and lets the model pick "://".

Roughly:

ids = tok.encode(prompt)
tail = tok.decode(ids[-1:])     # e.g. " " or ":"
ids = ids[:-1]                  # back up one token

allowed = [
    i for i in range(len(tok))
    if tok.decode([i]).startswith(tail)
]
# step 1: sample the first token only from `allowed` (logit mask)
# step 2: generate normally after that

For my case, the tail is " ", and the allowed set includes " billing", " bug" and every other space-prefixed word. The model gets to pick the token it actually wanted. Microsoft's guidance library ships token healing for exactly this reason.

Does a trailing space break chat APIs like Claude or GPT?

Usually not in the user message, sometimes yes in a prefill. Chat endpoints wrap your message in a chat template, so after your text come special tokens for "end of user turn" and "start of assistant turn". The model generates after those, not after your trailing space.

The danger zones are the places where your text is the last thing before generation:

• Raw completion endpoints (/completion in llama.cpp, /v1/completions in vLLM, base models anywhere).
• Assistant prefill, where you start the model's reply for it. Anthropic's Messages API refuses a final assistant message that ends in whitespace, which tells you how seriously they take this.
• Hand-built JSON prefills. {"label": with a trailing space has the same problem. {"label": " ends on a quote token and is much safer.
• Few-shot templates where the examples and the final slot are formatted differently.

What should I change in my prompts?

Make the end of your prompt look like the middle of your training data. In practice:

1. rstrip() every completion prompt and prefill before sending it. It's one call and it would have saved me an evening.
2. Put the space in the answer, not the prompt. Expect " billing" and parse with .strip().
3. Make the final slot match the few-shot examples byte for byte. If examples read Label: billing\n, the slot is Label: and nothing else.
4. Tokenize your template once and look at the last three tokens. If the last one is whitespace or a partial word, fix the boundary.
5. Use constrained decoding for labels. A grammar or allowed-token list stops casing variants and fragments from ever being sampled.
6. Turn on token healing if your library supports it and you can't control where the prompt ends.

Why does a trailing space in my prompt break LLM output?

A trailing space breaks LLM output because BPE tokenization normally glues a space onto the start of the next word, so a prompt ending in a space produces a standalone space token the model almost never saw during training. From that unfamiliar state, the model has to choose between a rare no-space word token and a space-prefixed token that creates a double space, and its probability spreads into casing errors, fragments and newlines. Strip trailing whitespace from completion prompts and prefills, keep the space inside the expected answer, and use token healing or constrained decoding when you can't control the boundary. In my classifier that one change took parse failures from 41 of 300 to 3.

Written by the developer behind Preterview, an interview prep platform.