What an Urdu sentence costs in tokens

Here are two sentences that mean exactly the same thing:

The government announced a new policy on electricity prices yesterday.

حکومت نے کل بجلی کی قیمتوں سے متعلق نئی پالیسی کا اعلان کیا۔

Ten words in English, thirteen in Urdu. If you're paying for an LLM, you'd expect them to cost roughly the same.

Measured across 30 pairs like this one, the same meanings cost 320 tokens in English and 1,999 in Urdu on GPT-2.

That's not a rounding error. That's the price of writing in a different script.

Why tokens are the wrong unit to ignore

Every LLM interaction is measured in tokens — you're billed in them, and your context window is counted in them. Words and characters are irrelevant to the meter.

Tokenizers are trained on whatever text their creators had, and that text is overwhelmingly English. When a tokenizer meets a script it barely saw during training, it falls back to breaking text into small pieces — sometimes into raw bytes.

The result is that "one page" of Urdu is not one page of tokens. It's five pages' worth.

I wanted a real number instead of a vibe, so I built a small benchmark to measure it.

The method

The measurement is called fertility: tokens divided by words. Compare the same content in two languages, and the ratio tells you how much more you pay.

The key requirement is that the two texts must mean the same thing — otherwise you're measuring content, not encoding. So I wrote a parallel corpus: 30 sentences, each written in both English and Urdu, spanning 17 everyday domains (news, medical, legal, technical, agriculture, and so on).

I couldn't use FLORES+, the standard benchmark for this work. Its access gate explicitly forbids re-hosting it anywhere web crawlers can reach, and a public Space or dataset would violate that. So the corpus is original, released CC-BY-4.0, and freely re-hostable: urdu-english-parallel.

Corpus shape: 30 pairs, 283 English words / 1,711 characters vs 328 Urdu words / 2,680 characters. All strings verified NFC, with no zero-width joiners and no Arabic presentation forms — those inflate token counts for reasons that have nothing to do with vocabulary, and they are easy to overlook.

The results

Three tokenizers, spanning a wide range of vocabulary sizes:

Same 30 sentences, same meanings, in every row.

Inflation is Urdu fertility ÷ English fertility. A value of 5.39× means the same meaning costs you more than five times as many tokens.

The number that actually made me stop

Fertility is abstract. Here it is per character:

English gets more than five characters packed into every token. Urdu gets one and a third.

That is close to character-level: on this tokenizer, Urdu is being encoded nearly one letter at a time rather than one word at a time. A thousand-character Urdu page costs about 746 tokens on GPT-2; the same number of characters in English costs about 187.

What this does to your context window

A 128,000-token context sounds enormous. Here's what it holds:

(English capacity differs slightly per row because English fertility differs slightly — 1.13 vs 1.12.)

On the English-trained tokenizer, a "128k" model holds roughly a fifth of the Urdu it holds of English. Put a 200-page Urdu document in front of it and you're truncating most of it. In a RAG pipeline, the same chunk costs five times as much, so you retrieve far less for the same budget.

And the billing follows the context window: at any per-token price, Urdu documents cost 2.7–5.4× more to process than English ones of the same substance.

A surprise: bigger vocabulary isn't the whole story

I expected this to be a pure vocabulary-size story. It isn't.

GPT-2 has 50k vocabulary and 5.39× inflation. Mistral has only 32k and still does better at 3.73×. Qwen2.5 with 151k is best at 2.72×.

So vocabulary size helps, but what the tokenizer was trained on appears to matter at least as much. My reading is that Mistral's tokenizer saw far more non-English text than GPT-2's, and that diversity outweighs an 18k-vocabulary deficit. Either way the conclusion holds: you cannot fix script fairness by scaling the vocabulary alone.

What I'd actually do about it

If you're building for Urdu users today:

1. Normalize your text first — and decide what you are normalizing. Apply NFKC and remove Arabic presentation forms. Be deliberate about zero-width joiners: they are orthographically meaningful in Urdu (they block ligatures), so they are not simply noise. Measure both ways and document which you chose.
2. Measure your own corpus. Fertility varies by domain and register. My formal corpus is not your conversational data.
3. Pick the tokenizer deliberately. If your app is Urdu-first, the tokenizer choice can cut your bill by half — that's a real product decision, not a detail.
4. Budget context accordingly. If you assume 128k tokens ≈ 128k tokens of Urdu, you'll silently truncate.

Caveats, honestly

• 30 sentences is a pilot. It supports the direction of the effect; it does not pin down a precise multiplier. Treat 5.39× as "roughly five", not a decimal.
• Formal register only. These sentences lean administrative and technical. Conversational Urdu — and Roman Urdu, which tokenizes completely differently — is not represented.
• One translator, no inter-annotator measurement.
• Each sentence was encoded independently. Where a tokenizer adds per-sentence special tokens, they are added equally to both languages, so ratios are essentially unaffected — though absolute counts may differ slightly from a one-pass measurement.

Try it yourself

Everything runs in your browser — no server, no API key, your text never leaves the page:

• Space: urdu-token-fairness
• Corpus: urdu-english-parallel (CC-BY-4.0)

The Space shows the actual subword pieces. Some are pink: those are single Urdu characters split across two tokens. That visual is the whole argument in one glance.

The general point

This isn't only about Urdu. Every language outside the tokenizer's training distribution pays a tax — in money, in context, in latency, in what fits in a prompt. The tax is invisible unless you measure it, and it compounds for exactly the users who already have less compute directed at their language.

If you speak a language that isn't well represented in tokenizers, the most useful thing you can do is measure it. The corpus format is 30 sentences and a JSON file. Add your language, run the Space, publish the number.

The tax is real. It should at least be visible.