A token is not a word, and your chat box is not the bill
When a tool says a request used "1,000 tokens", it's tempting to read that as 1,000 words. It isn't. A token is the unit a model's tokenizer cuts text into. Sometimes that's a whole word, sometimes a piece of one, someti
When a tool says a request used "1,000 tokens", it's tempting to read that as 1,000 words. It isn't. A token is the unit a model's tokenizer cuts text into. Sometimes that's a whole word, sometimes a piece of one, sometimes a space glued to a number, and sometimes a few bytes that don't make a readable character on their own.
That gap matters as soon as you start budgeting context windows or reading an API invoice.
What the model actually receives
The model never sees your sentence as typed. The tokenizer splits it into chunks using a fixed vocabulary and rules, then maps each chunk to an integer ID. Those IDs are just positions in a lookup table. A bigger ID doesn't mean a more important word.
Generation runs the other way: the model produces token IDs one step at a time and the app decodes them back to text. The "typing" effect in a chat UI is also shaped by network buffering and rendering, so each visible jump on screen isn't necessarily one token.
One English word, six tokens
Storing every possible word would make the vocabulary huge and still miss new names and typos. Splitting everything into single characters makes common phrases painfully long. Most tokenizers land in between with subword pieces, and those pieces don't have to match the roots and suffixes you learned in school.
OpenAI's tiktoken cookbook uses this example. Under the cl100k_base encoding, antidisestablishmentarianism becomes six tokens:
ant | idis | establish | ment | arian | ism
Under the older r50k_base encoding, the same word is five tokens, with establishment kept as one piece. Neither result is "more accurate". They're different vocabularies, so a count from one encoding can't be reused for a model that runs on the other. BPE, WordPiece and Unigram also build vocabularies differently, which is why the only count that means anything is the one from the model's own tokenizer.
Spaces and digits aren't free
The same cookbook shows 2 + 2 = 4 (with Γ’Β marking a space):
| Encoding | Tokens | Count |
|---|---|---|
r50k_base |
2 Γ’Β + Γ’Β 2 Γ’Β = Γ’Β 4
|
5 |
cl100k_base |
2 Γ’Β + Γ’Β 2 Γ’Β = Γ’Β 4
|
7 |
Same string, different grouping of spaces and digits. A leading space, a case change or an extra newline can all shift the count, so if you strip whitespace before counting, you're no longer measuring the text you send.
The "4 characters per token" rule
You've probably seen "1 token is about 4 characters" or "about 3/4 of a word". OpenAI gives that as a rough rule of thumb for English prose. It falls apart on code, URLs, emoji and non-Latin scripts.
Chinese is a good stress test. There are no spaces between words, so a tokenizer might merge two or three common characters into one token, or spend several tokens on one rare character. In UTF-8 a Chinese character usually takes multiple bytes, and byte-level tokenizers can split it so that a single token decodes to half a character. Fewer tokens for the same text also doesn't tell you which model understands it better.
Which tokens are you being charged for?
Usage numbers can mean different things:
- Input tokens are everything the request sends: your new message, plus whatever history, system prompt, tool definitions or retrieved documents the app includes.
- Output tokens are what the model generates. Some APIs also count hidden reasoning or structural tokens here, so this won't always match the visible reply.
- Context window is how many tokens one request can hold. It isn't a character limit you can fill with the same number of characters, and it isn't long-term memory.
So typing "make it shorter" can still be an expensive turn if the app resends the whole conversation with it. Some apps trim, summarize or retrieve history instead. What you can scroll back to and what the model got this turn are two separate things.
On the API side, providers may price input, output and cached tokens separately. Chat subscriptions use their own message limits. Token counts tell you how much you used. Price and quota come from the product, the same way kilowatt-hours don't tell you your bill until you know the rate.
Counting properly
For a short chunk of plain text, use the provider's tokenizer tool or tiktoken with the right encoding for your model, and paste the text exactly, spaces and newlines included. That beats asking a chat model to guess its own token count.
For a real API request, counting the message body isn't enough. Roles, message boundaries, tool schemas, images and audio can all be counted in other ways. OpenAI's docs separate local text tokenization from full-request counting, and if you need to match your bill, check the provider's counting endpoint and usage records.
Last thing: deleting punctuation or compressing prompts into shorthand to save tokens often backfires. The count doesn't drop in proportion, and the instruction gets vaguer first. Cutting irrelevant pasted context and repeated instructions usually saves more.
Examples are from OpenAI's How to count tokens with tiktoken cookbook. Also see OpenAI's Counting tokens guide and Hugging Face's tokenizer summary.
I first wrote this up in Chinese for WarpNav, where we keep plain-language AI explainers.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.