Hindi Pays a Tax on AI: I Measured It Across 10 Tokenizers and 11 Indian Languages

Ask an AI model the same question in English and in Hindi, and the Hindi one is usually billed for more. Nobody charges Hindi extra on purpose. The model reads text in pieces called tokens, and Hindi gets cut into more of them. I measured how many more on 1,012 sentences that say exactly the same thing in 16 languages, across 10 tokenizers used by real models. The answer ranges from 1.14x to 4.76x. Which end you're on depends almost entirely on which model you pick.

You don't need to know how language models work. If you have ever used ChatGPT or called an AI API, you can follow this. Every section ends with a one-line Remember, and the whole article fits in 9 lines at the end.

• Part 1 · What you're billed for: tokens, why Hindi gets more of them, and why one example sentence can't prove anything, sections 1–3.
• Part 2 · The experiment: Hindi across 10 tokenizers, how the tax fell, other Indian languages, a tokenizer built for India, and Hinglish, sections 4–9.
• Part 3 · What it means for you: the bill, the answer, the memory, the wait, and what to do about it, sections 10–11.

Four things people often say, and what the measurements show:

1. AI reads in tokens, not words

A language model never sees letters or words. Before it reads anything, a program called a tokenizer cuts the text into pieces from a fixed list, and the model works with those pieces. Each piece is a token.

Think of the tokenizer as a box of rubber stamps. A common English word like weather has its own stamp, so it costs one press. A word the box has no stamp for has to be spelled out with smaller stamps, a few letters at a time. Every press is one token.

Tokens matter for three reasons. API providers bill per token. A model's memory, the context window, is measured in tokens. And a model writes its answer one token at a time, so more tokens means a longer wait.

Pick a tokenizer and a language below. The sentence is the same in all four versions.

On GPT-4's tokenizer, the English sentence is 10 tokens and the Hindi one is 29. Many of those 29 pieces aren't even whole letters. दिल्ली (Delhi) is cut into seven bits, most of them fragments of a character that mean nothing on their own.

2. Why Hindi gets cut into more pieces

A tokenizer builds its stamp box by reading a huge pile of text and giving stamps to the pieces it sees most often. If most of that pile is English, English words get whole-word stamps and Hindi gets few. The box has a fixed size, so every stamp spent on one language is a stamp not spent on another.

When there's no stamp for a Hindi piece, the tokenizer falls back to the raw bytes of the text. This is where Hindi loses twice. An English letter is 1 byte in UTF-8, the encoding computers use for text. A Devanagari letter is 3.

So Hindi isn't long. On my 1,012 sentences, Hindi had slightly fewer characters than English. It just starts from 2.55x the bytes, and a tokenizer that hasn't learned Hindi pieces has to pay for those bytes in tokens.

3. One sentence can lie

Go back to the splitter and choose GPT-4o / GPT-5. The Hindi weather sentence is 8 tokens there, and the English one is 10. Going by that one sentence, Hindi would look cheaper.

Short, common sentences fit a tokenizer's stamps unusually well. दिल्ली, मौसम and रहेगा happen to be whole stamps in that box. Normal text has rarer words, names and longer sentences, and that's where the gap shows. Across all 1,012 sentences, Hindi on the same tokenizer costs 1.57x English.

This is why I didn't trust any single example, including mine. Screenshots of one sentence in a tokenizer playground are how most claims about this get made, and they can point either way.

4. How I measured it

The text. FLORES-200 is a public dataset made for testing translation. It has 1,012 English sentences from web articles, each translated by professional translators into 200 languages. Every language says the same thing, line for line, so the only thing that changes is the language. I used English, 11 Indian languages, and French, German, Chinese and Japanese for comparison.

The tokenizers. Ten that real models use, all run locally, with the exact versions pinned:

• OpenAI's three, from the tiktoken library: r50k_base (GPT-3 era), cl100k_base (GPT-3.5 and GPT-4), and o200k_base (GPT-4o, GPT-4.1, o3 and GPT-5).
• The published tokenizers of Llama 3, Llama 4, Gemma 3, Qwen3, DeepSeek-V3 and Mistral Small 3.1. Google's Gemma 3 report says Gemma 3 uses the same tokenizer as Gemini 2.0.
• Sarvam-1, a model built in India for Indian languages.

The count. Each sentence encoded on its own, with no extra control tokens, then summed per language. The result is the number of Hindi tokens divided by the number of English tokens: 1.57x means Hindi needs 157 tokens for every 100 in English.

This is the core of the script:

def load_encoder(kind, source, revision):
    if kind == "tiktoken":
        enc = tiktoken.get_encoding(source)
        return lambda s: len(enc.encode_ordinary(s))
    path = hf_hub_download(source, "tokenizer.json", revision=revision)
    tok = Tokenizer.from_file(path)
    return lambda s: len(tok.encode(s, add_special_tokens=False).ids)

for code, lines in text.items():
    total = sum(count(s) for s in lines)
    ratio = total / eng_total

I didn't measure Claude or current Gemini models. Their tokenizers aren't published as files, and I ran this without API keys. Both offer a token-counting endpoint if you want to check them yourself.

5. Hindi costs 1.14x to 4.76x, depending on the model

When I ran it:

Three things stand out.

The spread is huge. Same sentences, same meaning, and Hindi costs between 1.14x and 4.76x English. That spread is bigger than the gap between most model prices.

Popular open models still charge a lot. Qwen3 is one of the most widely used open models, and Hindi costs 4.42x English on its tokenizer. Llama 3, DeepSeek-V3 and Mistral Small 3.1 sit between 2.5x and 2.8x. If you self-host one of these for Hindi users, your GPU spends that many more steps on every Hindi answer.

The models built with more languages in mind do much better. Gemma 3, which shares Gemini 2.0's tokenizer, is at 1.31x. Sarvam-1 is at 1.14x.

6. The tax is falling

The big change happened in the last few years. Compare tokenizers from the same company:

For every 10 English tokens, Hindi needed 74 on GPT-3's tokenizer, 48 on GPT-4's, and 16 on GPT-4o's. Meta's move from Llama 3 to Llama 4 cut Hindi from 2.52x to 1.65x.

The main thing that changed is the size of the stamp box. cl100k_base has about 100,000 tokens; o200k_base has about 200,000. Llama 3 has 128,256; Llama 4 has 201,135. The extra room went largely to other languages, and Hindi was one of the winners.

That also means most "AI costs 4x more in Hindi" numbers you'll find online describe tokenizers from 2023.

7. Other Indian languages pay more than Hindi

Hindi is far from the worst case. Most Indian languages pay more than Hindi, and the gap depends on the tokenizer again. Pick one below to see all 16 languages:

On GPT-4o and GPT-5's tokenizer, Tamil, Telugu, Kannada and Malayalam cost about 1.9–2.0x English. Punjabi costs 2.62x. Odia costs 4.99x, close to where Hindi was on GPT-4. Switch to GPT-4 and the Dravidian languages go to 7.6–8.9x, and Odia to 12.40x.

French, German and Chinese stay between 1.25x and 1.37x on the same tokenizer. Every language pays something compared with English, but Indian languages pay more, and the less common the script in the training data, the more they pay.

8. A tokenizer built for India shows the trade-off

Sarvam-1 was built for Indian languages, and it shows: Hindi at 1.14x, Tamil 1.16x, Telugu 1.15x, Odia 1.46x. It's the only tokenizer where every Indic-script language I tested stays under 1.5x.

Now look at what it gave up:

Urdu costs 7.39x on Sarvam-1, worse than Urdu on any other tokenizer I tested, including GPT-3's (6.24x). Japanese is 5.34x. French and German are about 2.2x. Even English costs more: the same 1,012 English sentences take 31,422 tokens on Sarvam-1, against 26,873 on GPT-4o's tokenizer.

Its stamp box has 68,096 entries, about a third of GPT-4o's. It spent them on the languages it was built for. That's a sensible choice for its users. It also shows that no tokenizer is fair to everyone. It reflects whoever it was built for.

9. Hinglish: the old trick now backfires

A lot of us type Hindi in English letters: kal Delhi mein mausam kaisa rahega? An old piece of advice says this saves tokens, since tokenizers know English letters well.

To test it I needed the same Hindi sentences written both ways, by people. Google's Dakshina dataset has exactly that: 10,000 Hindi Wikipedia sentences, each typed in Roman letters by native speakers. I counted both versions on every tokenizer:

On older tokenizers the advice holds. Roman letters cut tokens by about 60% on GPT-4's tokenizer and 57% on Qwen3's. On new tokenizers it flips. Roman letters cost 8% more on GPT-4o and GPT-5's tokenizer, 23% more on Gemma 3's, and 76% more on Sarvam-1's.

The reason is the stamp box again. New tokenizers learned whole Devanagari words. Romanized Hindi has no standard spelling (mein, me, mai), so it gets cut into small English-letter pieces.

My three everyday sentences showed the same thing: on GPT-4o and GPT-5's tokenizer, the Devanagari version was 1.28x the English and the Roman version 1.41x.

10. What the tax costs you

A higher token count costs you in four ways. Pick a tokenizer and a language and move the sliders:

The bill. At the same price per token, 1.57x the tokens is 1.57x the bill. If you serve the same users in Hindi and English, the Hindi share of your bill grows by the ratio.

The answer. If you cap answers with max_tokens, a Hindi answer runs out sooner. On GPT-4o and GPT-5's tokenizer, a 500-token Hindi answer holds about what 318 English tokens would. On GPT-4's, about 105.

The memory. A 128K context window holds about 4,820 of these sentences in English and 3,062 in Hindi on GPT-4o's tokenizer. On GPT-4's, it's 1,001 in Hindi. Retrieval-based apps (RAG) that stuff documents into the prompt fit fewer Hindi documents in.

The wait. Models write one token at a time. If the speed per token is the same, a Hindi answer with 1.57x the tokens takes 1.57x as long to finish.

These are computed from the measured ratios, not from a real bill. Your users' text will differ from news sentences. The next section shows how to measure your own.

11. If you build for Indian users

Measure your own text. Take 50–100 real messages from your app, write them in each language your users use, and count. This is the whole script:

import sys
import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # swap for the tokenizer of the model you use


def tokens(path):
    with open(path, encoding="utf-8") as f:
        return sum(len(enc.encode_ordinary(line)) for line in f.read().splitlines())


english = tokens(sys.argv[1])
for path in sys.argv[2:]:
    print(f"{path}: {tokens(path) / english:.2f}x English")

When I ran it on three everyday support-style sentences:

$ python count_mine.py sample/en.txt sample/hi.txt sample/hinglish.txt
sample/hi.txt: 1.28x English
sample/hinglish.txt: 1.41x English

Compare models by cost per meaning, not per token. The real price for Hindi is price per token × the Hindi ratio. Here's an illustrative case: a model that is half the price per token but needs 3x the tokens for Hindi is 1.5x more expensive for Hindi users. Do this sum before you pick a model.

Set max_tokens per language. If you tuned the limit on English answers, Hindi answers will get cut off. Raise it for Indian languages by the ratio you measured.

Watch your rate limits. API limits are usually set in tokens per minute. Hindi traffic uses them up faster than the same number of English requests.

Don't push users into English or Roman letters to save money. On modern tokenizers, Devanagari is already the cheaper way to write Hindi, and asking users to switch costs you more than tokens. If you're tempted to translate everything to English first, measure the translation call's cost and the answer quality too.

Check the tokenizer, not only the price page. Two models with the same price per token can cost very different amounts for your users.

What this doesn't show

• Answer quality. Fewer tokens means a smaller bill, not a better answer. Whether a model answers well in Hindi is a separate question I didn't test.
• Claude and current Gemini. Not measured, as explained in section 4. Gemma 3 shares Gemini 2.0's tokenizer; I don't know if later Gemini models changed it.
• Your text. FLORES is news and encyclopedia writing. Chat messages, code-mixed text and names will give different ratios, which is why section 11 starts with measuring.
• Billed tokens. Chat APIs add a few formatting tokens per message, and those are the same in every language. Output tokens usually cost more than input tokens; the ratio applies to both.
• Romanized Hindi isn't all of Hinglish. Dakshina is Hindi sentences written in Roman letters. Real Hinglish also mixes in English words, which tokenizers handle well.

The whole article in 9 lines

1. Models read and bill text in tokens, pieces from a fixed vocabulary, not in words.
2. Hindi has the same number of characters as English but 2.55x the bytes, and gets fewer whole-word tokens.
3. One sentence proves nothing. Measure hundreds that say the same thing.
4. On 1,012 identical sentences, Hindi costs 1.14x to 4.76x English depending on the tokenizer.
5. The tax fell sharply: 4.76x on GPT-4 to 1.57x on GPT-4o and GPT-5; 2.52x on Llama 3 to 1.65x on Llama 4.
6. Other Indian languages pay more, from about 2x for Tamil to 4.99x for Odia on GPT-4o's tokenizer.
7. A vocabulary is a fixed budget: Sarvam-1 makes Hindi 1.14x and Urdu 7.39x.
8. On modern tokenizers, Hindi in English letters costs more than Hindi in Devanagari.
9. The tax hits your bill, your answer length, your context window and your latency, so measure your own traffic.

Measured on 22 October 2026 with tiktoken 0.14.0 and tokenizers 0.23.3, using FLORES-200 devtest (CC BY-SA 4.0) and Dakshina v1.0 Hindi romanized dev and test sets (CC BY-SA 4.0). Hugging Face tokenizer revisions are pinned in the lab script. All ratios and token counts are real output. The bill, answer, memory and wait figures in section 10 are computed from those ratios. The rubber-stamp box is an analogy. Real tokenizers (BPE and SentencePiece) learn their pieces by merging frequent byte pairs.