Do Non-English Prompts Cost More Tokens? I Measured 28 Languages

Everybody who works with an AI API learns quickly that you pay per token. Almost nobody checks whether the language they write in changes how many tokens they get. I assumed the effect was small. It is not small, and the direction surprised me.

So I measured it properly. Not with a rule of thumb, not with the usual “one token is about four characters” line, but with the actual vocabulary file. Everything below comes from o200k_base, the vocabulary OpenAI maps to its GPT-4o and GPT-5 model names. Two texts, translated into 28 languages, run through the real thing. Start with the tool below, then take the full index further down.

Split a sentence yourself

Click a sample and watch it come apart, one token at a time

Fewer characters, more tokens

The result that made me check my own code twice

Look at the Japanese line. That instruction is 33 characters. The English version is 80 characters. Japanese is less than half the length on screen, and it costs 73 percent more tokens.

That is not a rounding error and it is not a translation artefact. It is how the tokenizer was trained. A tokenizer learns its vocabulary from a training corpus, and that corpus is overwhelmingly English. Common English strings like ” customer” or ” reply” earn their own single entry because they turn up billions of times. Japanese gets far less of the budget, so it falls back to smaller and smaller pieces.

You can watch this happen in the tool. Switch to the Japanese sample and look for the pink chips. Those are tokens that are not even a whole character. The tokenizer ran out of vocabulary and split a single Japanese character across two tokens. You are paying twice for one character.

What a German compound noun does to your bill

One word, nine pieces

German has a different problem, and it is a fun one to look at. Pick the compound noun sample. The word Krankenversicherungsbeitragsberechnung is one word to a German reader and to a dictionary. To the tokenizer it is nine separate pieces: Kranken, vers, icher, ungs, be, itr, ags, bere, chnung.

Every technical German document is full of words like that. Every legal one, every insurance one, every administrative one. If you are building anything that processes German business text, this is where your budget goes, and it does not show up in a word count or a character count anywhere.

What this costs in real money

Where the small percentage stops being small

A twenty percent surcharge on one sentence sounds like nothing. It stops sounding like nothing when you notice two things.

First, the tax applies to everything in that language, not just your instruction. The customer email you feed in, the documents you retrieve, the history you carry. All of it.

Second, the model answers in the same language, and output tokens cost about six times what input tokens cost. So the surcharge lands on the expensive side of the bill too.

Put those together with a support desk handling 100,000 requests a month, each one roughly 2,000 input tokens and 500 output tokens, priced at the current GPT-5.6 Terra rate of $2.00 input and $12.00 output per million:

That is $8,760 a year between the English pipeline and the Japanese one, for identical work. Nobody in that company will ever see a line item explaining it.

What this does not mean

Three honest limits

It does not mean you should prompt in English. If your users write German, your system reads German. Translating first costs its own tokens and loses meaning. This is a number to plan with, not a trick to route around.

It does not transfer perfectly to other models. These are o200k_base counts, which OpenAI maps to its GPT-4o and GPT-5 model names. Claude and Gemini each use their own vocabulary, so exact counts differ, and neither lets you check a current model offline. Google does ship a local tokenizer in the older Vertex AI SDK, vertexai.preview.tokenization in Python and cloud.google.com/go/vertexai/genai/tokenizer in Go, which downloads the Gemma vocabulary once and then counts on your machine. Its supported list in the shipping package is Gemini 1.0 and 1.5 only, and that SDK has been superseded by the Google Gen AI SDK, so for current Gemini models, as for Claude, counting means calling the vendor’s count-tokens endpoint. The pattern holds everywhere, because every major tokenizer was trained on an English-heavy corpus. The size of the gap is what varies.

It does not mean short is cheap. That is the whole point of the Japanese result. If you want the concept behind this properly explained, I wrote up what tokenization actually is separately.

The mistake that cost me more than any language did

Where my own bill actually went

I should be honest with you: the language tax is not the most expensive thing I have ever done to myself. This is.

I had a stack of PDFs analyzed, and from that analysis I built a set of skills. Then I kept having those skills reviewed and extended, round after round. The cost climbed far faster than the work justified, and it took me a while to see why. The model was not extending the skill. It was rewriting the whole thing from scratch on every pass. And because I wanted the result checked, I had several agents on it, each one reading the entire input again.

So I was not paying for the paragraph I actually wanted added. I was paying to regenerate everything I already had, several times over, every single round.

If I built the same thing today I would work with what I think of as brains: the important information parked somewhere permanent, and agents with narrow, specific jobs, so the context does not have to be read in from scratch on every run. Store the state, then send in something small that knows where to look.

That is worth saying next to all the numbers above, because a twenty percent language surcharge is a rounding error compared to an architecture that re-reads itself. Fix the architecture first. Then worry about the language.