October 25, 2026 · 14 min read
Why Hindi Pays More: Inside the Tokenizer
Part 2 of the Hindi token tax experiment. I followed one sentence from the keyboard to the model, counted every stamp in 10 tokenizers' vocabularies, and trained five small tokenizers that differ only in how much Hindi they saw. The cause is how much of a script the tokenizer was built from, with bytes setting the starting line.
In part 1 I measured that the same text costs 1.14x to 4.76x more tokens in Hindi than in English, depending on the model. This part answers why. I followed one sentence through every step a tokenizer takes, counted every stamp in 10 real vocabularies, and then trained five small tokenizers myself, changing nothing but how much Hindi they saw. The answer is short: a tokenizer gives stamps to the text it was built from, and bytes decide how far behind a script starts.
You don't need to know how language models work, and you don't need to have read part 1. Every section ends with a one-line Remember, and the whole article fits in 9 lines at the end.
- Part 1 · Left to right: what happens to a sentence before the model sees it, sections 1–2.
- Part 2 · How the stamp box is built: the algorithm, the direct test, and the budget, sections 3–5.
- Part 3 · Sarvam-1, up close: what a tokenizer built for India looks like, and what it gave up, sections 6–7.
- Part 4 · What it means: tokenizer text vs training data, and what to do, sections 8–9.
Four things people often say about why Hindi costs more, and what the measurements show:
| What people say | What I measured |
|---|---|
| "Devanagari is a complex script, so it costs more." | Chinese characters are far more complex, and Chinese costs 1.00x English on Qwen3's tokenizer. Complexity isn't the cause. |
| "All Asian languages pay the same tax." | On GPT-4o's tokenizer, Chinese is 1.25x, Hindi 1.57x, Odia 4.99x. It depends on the script and the tokenizer, not the region. |
| "A bigger vocabulary fixes it." | Sarvam-1 has the smallest vocabulary of the 10 (68,096) and makes Hindi 1.14x. Qwen3's is more than twice that and makes Hindi 4.42x. |
| "It's the training data." | Close. It's the text the tokenizer was built from, which isn't always the same as what the model learned from. Section 8 explains the difference. |
Part 1
Left to right
What happens to a sentence before the model sees it.
1. Seven steps from text to bill
In part 1, I described the tokenizer as a box of rubber stamps. Common pieces of text have their own stamp. Anything without a stamp gets spelled out with smaller ones. Every press of a stamp is one token, and tokens are what you're billed for.
Here is what actually happens, left to right:
- Text. What you type: letters, plus the vowel signs and other marks that attach to Indian-script letters.
- UTF-8 bytes. The computer stores each character as one to four numbers between 0 and 255.
- Pre-split. The text is cut at spaces and punctuation, so stamps never cross a word boundary.
- Stamps. The tokenizer covers each word with the longest stamps it has.
- Token IDs. Each stamp has a number, its row in the stamp box.
- The model. It looks up one row of numbers per ID, and writes its answer one token at a time.
- The bill and the limits. Price, context window and rate limits are all counted in tokens.
Steps 2 to 5 are the tokenizer. Play them below with real output from five tokenizers.
Try Hindi on GPT-4's tokenizer, then on Sarvam-1's. The bytes in step 2 are identical, since the sentence is the same. Everything that changes happens in step 4, where one box has stamps for दिल्ली and the other has to fall back to bytes.
2. Bytes set the starting line
Most tokenizers start from bytes. A stamp box that knew nothing would charge one token per byte, so bytes are where every language starts before any learning helps.
UTF-8 gives an English letter 1 byte and a Devanagari, Tamil or Bengali letter 3. A Chinese character is also 3 bytes, but one character often carries a whole word or syllable.
Hindi has almost the same number of characters as English (0.99x) and 2.55x the bytes. Chinese has a third of the characters, so even at 3 bytes each it ends up with fewer bytes than English (0.91x). So the complexity of a script was never the right question. What matters is how many bytes it takes to say the same thing.
Part 2
How the stamp box is built
The algorithm, the direct test, and the budget.
3. How a tokenizer builds its stamp box
Almost every tokenizer in this experiment builds its stamps with byte pair encoding (BPE). It runs once, on a big pile of text, before the model is trained:
- Start with the smallest units: bytes.
- Count every pair of neighbours in the text.
- Glue the most frequent pair into a new stamp.
- Repeat until the box is full, for example 68,096 or 200,019 stamps.
The order of the merges is saved. When you type a sentence later, the tokenizer replays those merges on your text. So a script that appeared often in the pile collects many merges and gets long stamps. A script that appeared rarely gets few merges and stays in small bits.
Run it yourself. This is the same algorithm on characters instead of bytes, so you can read it. First leave the Hindi lines out and press +20 merges. Then tick the box, which resets the trainer, and press it again.
With no Hindi in the text, all 20 merges go to English and the Hindi test sentence never shrinks. With Hindi in, it shrinks fast, and the English sentence ends up with more tokens than before, because the same 20 merges are now shared.
4. The direct test: change only the Hindi
Comparing 10 real tokenizers only shows a pattern, because they differ in many ways at once: size, method, company, year. To test the cause, I trained five byte-level BPE tokenizers myself. They have the same size (4,000 stamps), the same method and the same English text. The only difference is how much Hindi is added. I trained on the first half of the FLORES sentences and measured on the second half, which none of them saw.
| Hindi in training text | English tokens | Hindi tokens | Hindi bytes per token |
|---|---|---|---|
| 0.0% | 19,953 | 167,785 | 1.00 |
| 13.5% | 20,062 | 47,607 | 3.53 |
| 35.2% | 20,125 | 44,959 | 3.74 |
| 55.9% | 20,360 | 43,783 | 3.84 |
| 71.7% | 20,658 | 42,963 | 3.92 |
With no Hindi in the training text, the tokenizer learns zero Devanagari stamps. Hindi falls back to about one token per byte: 8.41x English. Adding just 13.5% Hindi brings it to 2.37x. More Hindi keeps helping, with smaller gains each time, down to 2.08x.
Look at the English column. The box size is fixed, so every stamp Hindi gains is one English doesn't get. English tokens rise from 19,953 to 20,658, which is 3.5% more.
These are small tokenizers trained on about 500 sentences per language, so don't compare the exact ratios with production models. The direction and shape of the effect are what carry over.
5. The stamp box is a budget
Back to the 10 real tokenizers. I turned every entry in each vocabulary back into text and counted how many are written entirely in each script. Then I compared that with the tokens each language needs. Switch scripts below.
| Tokenizer | Box size | Hindi stamps | Hindi vs English |
|---|---|---|---|
| Gemma 3 | 262,145 | 13,754 | 1.31x |
| Sarvam-1 | 68,096 | 12,126 | 1.14x |
| GPT-4o / GPT-5 | 200,019 | 3,963 | 1.57x |
| Llama 4 | 201,135 | 2,677 | 1.65x |
| Mistral Small 3.1 | 131,072 | 1,555 | 2.76x |
| Llama 3 | 128,256 | 1,012 | 2.52x |
| DeepSeek-V3 | 128,815 | 297 | 2.79x |
| Qwen3 | 151,669 | 67 | 4.42x |
| GPT-4 | 100,277 | 23 | 4.76x |
| GPT-3 | 50,257 | 1 | 7.41x |
The pattern holds for every language I checked, 11 Indian languages plus Chinese: the more stamps a tokenizer has for a script, the fewer tokens that language needs. GPT-4's tokenizer has 23 Devanagari stamps and makes Hindi 4.76x. GPT-4o's has 3,963 and makes it 1.57x, which explains most of how fast the tax fell between the two. Odia has zero stamps in 5 of the 10 tokenizers, and on those it needs 8.42x to 13.26x the tokens of English.
Size alone doesn't decide it. Sarvam-1 has the smallest box here and one of the largest Devanagari sections. Qwen3's box is more than twice as big and holds 67 Devanagari stamps, while Chinese gets 24,966 and costs exactly what English does.
Two families of tokenizer also behave differently when they meet a script they don't know:
- Byte-level BPE (OpenAI, Llama, Qwen, DeepSeek, Mistral) turns everything into bytes first. It never fails on any script, but an unknown letter costs up to one token per byte, which is 3 tokens for a Devanagari letter.
- SentencePiece style (Gemma, Sarvam-1) works on characters, marks spaces with
▁, and falls back to single bytes like<0xE0>only for characters it never learned.
Part 3
Sarvam-1, up close
What a tokenizer built for India looks like.
6. A tokenizer built for Indian languages
Sarvam-1 is the clearest case of a tokenizer built for a chosen set of languages. Here is what Sarvam says about it, next to what I measured:
| Source | Claim |
|---|---|
| Sarvam says | A 2-billion-parameter model for 10 Indian languages plus English: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu. |
| Sarvam says | A custom tokenizer "optimized specifically for Indic languages", with 68,096 stamps, 4,096 of them reserved for later. |
| Sarvam says | Training corpus of about 2 trillion Indic tokens, almost evenly split across the 10 languages except Hindi at about 20%. English and code were added to train the model. |
| Sarvam says | 1.4 to 2.1 tokens per word across its languages. |
| Measured | Hindi: 1.40 tokens per word on FLORES, which matches. 1.14x English, the lowest ratio of the 10. Marathi, which shares the script, is 1.08x. |
| Measured | 12,126 Devanagari stamps, 18% of the box, many of them whole words. |
| Not known | The exact text the tokenizer was built from. Sarvam publishes the model's data mix, not the tokenizer's. |
So, was Sarvam-1's tokenizer built from more Hindi? As far as anyone outside Sarvam can show, yes. Sarvam says it built the tokenizer for these languages. The vocabulary has thousands of long Devanagari stamps. And section 4 shows that more of a script in the tokenizer's text produces exactly this.
7. What a focused box gives up
A fixed budget spent on 10 Indian languages has to come from somewhere. You can see where in the same measurements:
- Urdu: 7.39x. Urdu isn't one of Sarvam-1's 10 languages and uses a different script. It's the worst Urdu figure of any tokenizer I tested.
- Chinese: 3.82x, with zero Chinese stamps in the box.
- English: about 16% more tokens than the other tokenizers need for the same text (31,422 against 26,873 to 27,827). That's the same trade-off as the English column in section 4, at full scale.
- Hindi typed in English letters: 15 tokens for the weather sentence, against 7 in Devanagari. The box was built for the script, not for Hinglish.
That English cost also flatters Sarvam-1's ratios, because every ratio divides by that tokenizer's own English count. In raw token counts, Gemma 3 uses slightly fewer tokens for Hindi than Sarvam-1 (35,537 against 35,954).
Part 4
What it means
Tokenizer text vs training data, and what to do.
8. The tokenizer isn't the model
It's tempting to say "Hindi costs more because the model saw little Hindi." That's close, but not quite right. The tokenizer is built first, from its own pile of text, and then frozen. The model is trained afterwards, on its own data, through that frozen tokenizer.
Usually the tokenizer's text is a sample of the model's data, so the two go together. They don't have to. A model can be trained on a lot of Hindi and still read it through a tokenizer that cuts it into small pieces. Then it learns Hindi well and still charges a lot for it. The cost is decided by the tokenizer's text, and neither that text nor most models' training data is published.
9. What to do with this
Measure your users' text, not a benchmark. The ratio depends on the tokenizer and on the kind of text. Part 1 has a 10-line script for counting your own.
Compare models by cost per meaning. Price per token times the token ratio for your language is the real price. A cheap model with a stamp box that has no stamps for your users' script can cost more overall.
Watch where a model's tokenizer came from. A tokenizer built for one region's languages is cheap for those and expensive for others, as Qwen3 and Sarvam-1 show in opposite directions.
Don't confuse tokens with quality. Nothing in this article measures how good the answers are. Fewer tokens means a smaller bill and more room in the context window, not a better answer.
Remember agents. System prompts, tool definitions and tool results are usually in English, so inside an agent the language gap touches only the user's part of the context. I haven't measured how much of the tax survives there yet.
What this doesn't show
- Proof from inside the companies. No tokenizer's training text is published, so for real tokenizers the evidence is the vocabulary plus the direct test in section 4, not the text itself.
- Answer quality, speed and real bills. Only token counts were measured.
- Claude and current Gemini. Their tokenizers aren't published as files. Gemma 3 shares Gemini 2.0's tokenizer, per Google's Gemma 3 report.
- Production-size training. The tokenizers in section 4 are small. They show the direction of the effect, not the exact ratios of a real model.
The whole article in 9 lines
- A tokenizer turns text into bytes, then covers them with the longest stamps in its box. Models differ only at that stamp step.
- Before any learning, Hindi starts at 2.55x the bytes of English and Chinese at 0.91x.
- BPE builds the box by gluing the most frequent pairs in its text, so frequent scripts get long stamps.
- Changing only the Hindi in a tokenizer's text took Hindi from 8.41x to 2.08x, and made English 3.5% more expensive.
- Across 10 real tokenizers and 12 languages, more stamps for a script meant fewer tokens, every time.
- Box size doesn't decide it: Sarvam-1's small box makes Hindi 1.14x, and Qwen3's large one makes it 4.42x.
- Sarvam-1 built its box for 10 Indian languages, with thousands of whole Hindi words as single stamps.
- A focused box costs other languages more: Urdu 7.39x, Chinese 3.82x and English about 16% more on Sarvam-1.
- Cost comes from the tokenizer's text, not directly from the model's data, and fewer tokens doesn't mean better answers.
Measured on 25 October 2026 with tiktoken 0.14.0 and tokenizers 0.23.3, on FLORES-200 devtest (CC BY-SA 4.0), with Hugging Face tokenizer files pinned to fixed revisions. Token counts, stamp counts and the five trained tokenizers are real output from the lab scripts (measure.py, vocab_scripts.py, train_toy.py). A "stamp" counts for a script when every letter in it belongs to that script; stamps that hold only part of a letter are left out. Sarvam-1 facts marked "Sarvam says" come from Sarvam's Sarvam-1 announcement. The stamp box is an analogy. Real tokenizers learn their stamps by merging frequent byte pairs, as section 3 shows.