Who Decides the Best AI, and Who Gets Left Behind?

This is my perspective on a question I think deserves more attention. It is not a criticism of any company or country. The goal is to think through a risk early, while there is still time to shape the outcome.

Every few weeks a new AI model is announced as the best. Leaderboards shift, headlines follow, and the ranking changes again. Underneath that race sits a bigger question. If some people and countries get the best AI and others do not, AI could become a new version of the gap between rich and poor.

I wanted to test one part of that question myself, so this post includes a small measurement: what the same text costs an AI model in twelve different languages. The result surprised me more than I expected.

How "best" is decided today

Model superiority is mostly decided by benchmarks and leaderboards. Public arenas let people compare answers from two models side by side and vote for the better one, producing a rating similar to a chess score. Other tests measure coding, math, reasoning, and knowledge.

The race at the top is very close. Stanford's 2026 AI Index reports that as of March 2026, the top closed model led the top open-weight model by 3.3% on the Arena leaderboard, up from 0.5% in August 2024, and six of the top ten models were closed. When the top is that tight, competition shifts away from raw capability toward cost, speed, and reliability.

Benchmarks only tell part of the story. Many become saturated as models max them out, and some reward test-taking skill more than real-world usefulness. The real advantage comes from things a leaderboard does not show: the compute to train and serve models, the energy to power them (the subject of Part 3), large and diverse data, the talent to build with it, distribution to billions of existing users, and the trust of people and governments (Part 4). None of these are spread evenly around the world.

The gap is real and growing

Microsoft's AI Diffusion reports track how many working-age people use AI tools. In the second quarter of 2026, usage reached 28.8% in the Global North and 16.2% in the Global South. The gap widened to 12.6 percentage points, from 12.1 the quarter before. At the top end, the UAE had 64.0% of its working-age population using AI by the end of 2025, with Singapore at 60.9%.

Infrastructure tells the same story. A UN concept note states that Africa accounts for less than 1% of the world's data center capacity, and in October 2026 a UN official noted that Africa and Latin America together hold just 3.3% of global capacity. The UN Secretary-General has called for a global fund to build AI capacity in developing countries.

These gaps appear between countries, between large and small companies, and between paid and free tiers. The one I find easiest to overlook, because it is built into the technology itself, is language.

A small test: what does the same text cost?

AI models do not read words. They read tokens, small chunks of text produced by a tokenizer. Prices, rate limits, and context windows are counted in tokens. If one language needs more tokens to say the same thing, its speakers pay more, wait longer, and fit less into each request.

To measure this, I took the Universal Declaration of Human Rights, which has official translations in hundreds of languages, and ran all 30 articles through two OpenAI tokenizers using the open-source tiktoken library. o200k_base is the current generation used by GPT-4o-era models, and cl100k_base is the previous one used by GPT-4 and GPT-3.5. Each number is the token count compared with the English version of the same text.

I started with Hindi and Punjabi because I speak both, and I have often noticed that AI tools feel slower and less fluent in them than in English. I wanted to see whether part of that came down to cost. I then added ten more languages to see whether the pattern held beyond the two I know.

Two things stand out.

The good news first. The newer tokenizer cut the gap dramatically for widely spoken languages. Hindi went from 5.41x to 1.57x, and Arabic from 2.66x to 1.18x. That is a design decision that made AI several times cheaper for hundreds of millions of people, and it shows this problem can be fixed.

The improvement was much smaller for languages with fewer speakers online. Punjabi still costs 2.71x, almost twice as much as Hindi, even though both are major languages in India. Yoruba costs 3.14x and Amharic 6.67x. An Amharic speaker using the same model at the same per-token price pays roughly six and a half times more for the same content, and fits only about 15% as much text into the same context window. Where free tiers and rate limits are counted in tokens, those limits run out faster too.

This finding is not new on its own. Petrov and colleagues showed in 2023 that tokenizers can produce large cost differences between languages. What this test adds is a check on the current generation: the gap has narrowed for large languages and remains wide for smaller ones. The languages that improved least are often spoken in the same regions that have the lowest AI usage and the least data center capacity. That is how a technology gap can quietly become an economic one.

Token cost is also only one layer. The quality of answers in these languages is a separate question, and one that a tokenizer count cannot answer.

Forces that could close it

The tokenizer result itself is a reason for hope. A single engineering choice moved Hindi from more than five times the English cost to about one and a half. Similar choices are being made across the stack.

Open-weight models remain close to the frontier and can be downloaded, adapted, and run locally without paying per use, which lets a university, a startup, or a government build on capable AI without owning a frontier lab. The cost of running a given level of capability keeps falling, and the token discipline discussed in Part 1 stretches every budget further. Small models that run on a phone or laptop can reach people where cloud access is limited or expensive. Community efforts to build datasets and models for local languages go straight at the gap shown in the table above. National and regional compute programs, along with proposals like the UN global fund, can give researchers and public services access they could not afford alone.

What would help, and who can do it

Benchmark and leaderboard maintainers could report cost per language alongside quality. A model that scores well in English but costs 6x more in Amharic should show both numbers. Adding a tokenizer ratio column, like the table above, takes very little effort.

Model providers could publish the token ratio of their tokenizer for major languages, and treat a large gap as a defect to be fixed, the way the move from cl100k to o200k fixed much of it for Hindi and Arabic. Pricing free tiers by message rather than by token would also remove the penalty for users of costly languages.

Developers building products for multilingual users can measure this before launch. The script I used is about 60 lines and runs in seconds. If your users write in Tamil or Yoruba, your cost and latency estimates based on English test data will be wrong by a factor of two or more.

Governments and funders can support open, local-language data and shared regional compute, and invest in the basics that every model depends on: electricity, connectivity, affordable devices, and skills. The ideas from Part 2 about sharing AI's value apply here as well. If AI creates wealth, some of it can fund access for those being left behind.

Closing

The question of which AI is best will keep changing every few months. The more important question is who gets to use it, and at what price. Leaderboards measure capability. They do not measure fairness.

The tokenizer result shows both sides of this. The same engineering effort that made AI several times cheaper for Hindi speakers has not yet reached Amharic or Yoruba speakers. The difference will not come from the models alone. It will come from the choices people make about access, cost, language, and infrastructure, and those choices are still open.

Next in this series: can AI feel, and should it predict us?

How the test was run

The text is the 30 articles of the Universal Declaration of Human Rights from the Unicode UDHR project. Only articles present in all twelve translations were compared (all 30 were). Token counts come from tiktoken 0.14.0 with the o200k_base and cl100k_base encodings. This measures token cost only, not answer quality, and covers OpenAI tokenizers only. Other providers use different tokenizers and will produce different numbers. Translations also differ slightly in length and style, so treat the ratios as approximate. The script is available as a public gist and can be rerun with any language in the UDHR collection.

Sources

• Stanford HAI: 2026 AI Index, technical performance
• Stanford HAI: 2026 AI Index report
• Microsoft: The continued state of global AI diffusion in 2026 (Q2)
• Microsoft: Global AI adoption in 2025
• UN concept note: Africa's AI opportunity
• Al Jazeera: AI's opportunity divide could deepen global inequality (October 2026)
• UN: Secretary-General calls for a global fund for AI capacity building
• World Bank: The AI wave is global. Readiness is not.
• Petrov et al. (2023): Language Model Tokenizers Introduce Unfairness Between Languages
• Unicode UDHR project (translations used in the test)
• OpenAI tiktoken