The Token Tax Index: 28 Languages, Measured
Twenty eight languages, two independent texts, one tokeniser. The same declaration that takes 33 tokens in English takes 72 in Japanese, using half the characters. Full table, method and a downloadable dataset below.
Every AI API charges per token, and almost nobody checks whether the language they write in changes how many tokens they get. I measured it properly, twice over, using two texts of different kinds so the result does not rest on one lucky sentence.
Measured on 5 September 2026 with the o200k_base encoding, the vocabulary the GPT-5.x family uses. Version 1.1.
The index
Click any column heading to sort. Both texts shown so you can see where they disagree.
What the numbers actually say
Four findings, one of which surprised me
Character count tells you nothing
Japanese writes Article 1 of the Universal Declaration in 85 characters against English’s 170, and pays 72 tokens against 33. Half the text on screen, more than double the bill. Korean does the same thing in a milder form. If you have been estimating cost from length, you have been estimating the wrong quantity.
Chinese breaks the rule everyone assumes
The lazy version of this finding is that non-Latin scripts are penalised. Chinese refutes it: 43 characters, 35 tokens, six percent above English and the most token-dense language in the set. Whether a language is expensive depends on how much of it the tokeniser saw in training, not on its alphabet.
Grammar inside words costs money
Hungarian, Finnish, Romanian, Ukrainian, Czech and Polish all land well above English. These languages carry grammar inside the word, where English uses separate words the tokeniser has seen a billion times. Turkish behaves the same way at word level: evlerinizden, meaning from your houses, is four tokens on its own.
The tax depends on the text, not just the language
This is the finding that keeps the index honest, and it is why two texts are better than one. Arabic costs 33 percent more than English on the declaration and 7 percent less on the working instruction. Swahili is 9 percent above on one and 40 percent above on the other. Anyone quoting a single per-language multiplier is overselling it. Measure your own text.
Structured data pays no tax at all. The JSON request body is the same 41 tokens whether your users write in English, Japanese or Swahili. Only the human language moves.
What changed when OpenAI doubled the vocabulary
The same 28 texts, run through the old tokeniser and the new one
The GPT-4 generation used a vocabulary called cl100k_base, with roughly 100,000 entries. The GPT-5.x generation uses o200k_base, with roughly 200,000. Doubling a vocabulary is an expensive engineering decision, and the stated reason was better handling of languages other than English. I had both tokenisers available offline, so rather than take that on trust I ran all 28 declaration texts through each.
It worked, and the effect is not small. The languages that were punished hardest gained the most, and English gained exactly nothing.
| Language | cl100k_base (GPT-4 era) | o200k_base (GPT-5.x) | Tokens saved |
|---|---|---|---|
| Hindi | 180 | 54 | 70% |
| Hebrew | 127 | 50 | 61% |
| Greek | 172 | 71 | 59% |
| Arabic | 88 | 44 | 50% |
| Russian | 74 | 41 | 45% |
| Ukrainian | 94 | 55 | 41% |
| Vietnamese | 84 | 50 | 40% |
| Korean | 85 | 51 | 40% |
| Chinese (Simplified) | 50 | 35 | 30% |
| Portuguese | 51 | 37 | 27% |
| Swahili | 48 | 36 | 25% |
| Dutch | 53 | 41 | 23% |
| Turkish | 60 | 46 | 23% |
| Japanese | 93 | 72 | 23% |
| Finnish | 68 | 54 | 21% |
| French | 50 | 41 | 18% |
| Danish | 57 | 47 | 18% |
| Indonesian | 57 | 47 | 18% |
| Czech | 60 | 50 | 17% |
| Polish | 64 | 53 | 17% |
| Swedish | 56 | 47 | 16% |
| German | 44 | 38 | 14% |
| Spanish | 44 | 38 | 14% |
| Romanian | 67 | 58 | 13% |
| Norwegian | 52 | 46 | 12% |
| Hungarian | 69 | 62 | 10% |
| Italian | 53 | 51 | 4% |
| English | 33 | 33 | 0% |
Same input as text A above, Article 1 of the Universal Declaration of Human Rights in the official translations. Sorted by how much the newer vocabulary saves.
Hindi went from 180 tokens to 54, a 70 percent cut. Hebrew saved 61 percent, Greek 59, Arabic 50, Russian 45. English went from 33 to 33.
The token tax is real, and it has been shrinking fast for exactly the languages that were paying it. Any article quoting GPT-4 era token ratios for non-English text is now badly out of date.
Two things follow. First, if you read a piece about multilingual token costs written before 2025, treat its numbers as historical rather than wrong: the gap it describes was real and has since narrowed. Second, the ranking changed as well as the size. Japanese is still expensive in absolute terms, but its 23 percent saving is below average, while Hindi moved from one of the worst penalised languages to a middling one.
Method
Everything you need to reproduce or refute this
Tokeniser. o200k_base, the vocabulary used by the GPT-5.x family, via the gpt-tokenizer package. Counts are computed at build time and baked into this page, so nothing is estimated at render.
Text A, the declaration. Article 1 of the Universal Declaration of Human Rights, in the official translations published by the UN Office of the High Commissioner for Human Rights. This text was chosen deliberately: the translations are official rather than mine, they are parallel by construction, and they are public domain, so the comparison is not an artefact of my own translation choices. Whitespace is normalised to single spaces before counting.
Text B, the working instruction. “Please summarise this customer email in three bullet points and suggest a reply.” These translations are mine, not official. They are included because they represent the kind of text people actually send to an API, but a clumsy translation would shift a count, so treat text A as the stronger evidence and text B as corroboration. Turn on “Show the source text” and check my work.
Text C, the constant. A JSON request body, identical in every row, included to show the floor: 41 tokens regardless of language.
What this does not cover. Claude and Gemini use their own vocabularies and neither publishes an offline tokeniser, so exact counts differ there. The direction holds everywhere, because every major tokeniser was trained on an English-heavy corpus. The size of the gap is what varies.
Second tokeniser. The comparison section above uses cl100k_base, the GPT-4 generation vocabulary, from the same package, on the identical text A inputs. Nothing else about the measurement changes between the two runs.
Changelog. Version 1.1, 5 September 2026: added the cl100k_base against o200k_base comparison for all 28 languages. Version 1.0, 5 September 2026: first publication.
Corrections welcome. If you run these strings and get different numbers, or if you can improve a translation in text B, tell me and I will update the page and note the change.
Cite this page
Schulz, C. (2026). The Token Tax Index: token counts for 28 languages under o200k_base, version 1.1, measured 5 September 2026. LearnAI24. https://learnai24.com/ai-tools/token-tax-index/The dataset is free to reuse with attribution. The underlying declaration text is public domain.
Now put it in money
Tokens only matter because they are billed. To turn these multipliers into a monthly figure at current published prices, use the AI API cost calculator, which carries ten vendor-verified rates.
If you want the mechanism behind all of this explained from scratch, start with what tokenization actually is, then see the eight-language walkthrough with the tokens animated one by one.
