One Emoji, 26 Tokens: What a Language Model Actually Reads

A language model does not read characters. It reads tokens, and one symbol on your screen can be one token or 26 of them. The clearest way to see that is to measure emoji, because an emoji looks like exactly one thing on your screen and behaves like anything but.

I measured 27 of them against the two OpenAI vocabularies compared here. Most got cheaper when the vocabulary changed. Three got more expensive, and those three are the same three flags.

What a token actually is

Before a model sees your text, a tokenizer cuts it into pieces from a fixed list. That list is built once, before training, by finding the most common byte sequences in a large pile of text. Everything the model later reads, pays for and reasons over is expressed in those pieces.

Two consequences follow, and both are easy to miss. What is common in the training pile becomes one cheap piece. What is rare gets shredded into fragments, sometimes down to single bytes. And because the list is fixed before training, it is a snapshot of what looked common at that moment, not a fact about language.

Emoji make this visible because they are unusually varied. Some are a single code point. Some are two characters glued with an invisible joiner. Some are a flag character followed by a region code spelled out in invisible tag letters.

The measurement

Both vocabularies run offline from the same package, so the two columns are produced the same way rather than estimated from different sources. o200k_base is the newer one, which OpenAI’s own tokenizer library maps to the GPT-4o and GPT-5 model names. cl100k_base is the older one, mapped to GPT-4 and GPT-3.5.

Twenty-two of the 27 got cheaper, two stayed the same, and three got more expensive. The three are England, Scotland and Wales.

Why one flag costs 26 tokens

A country flag is two letters in a special alphabet. Germany is regional indicator D followed by regional indicator E. Two symbols, four tokens in the newer vocabulary.

The flags of England, Scotland and Wales are built differently, because they are not countries with a two-letter country code. Unicode spells them out. The sequence starts with a black flag, then writes the region code in invisible tag characters, then closes with a cancel tag. Scotland is the black flag plus g, b, s, c, t, then the terminator: seven code points and 28 bytes for one thing you see.

Those tag characters are rare enough that neither vocabulary learned them as units. Both fall back to near-byte-level fragments. The older one happened to have a piece covering a two-byte pattern that repeats six times in this sequence, and the newer one does not, which costs six tokens. The newer one wins one token back on the black flag itself, which it covers in two pieces where the older one needs three. Six lost, one regained: five more than it used to be. The six tag characters alone are 24 tokens in the new vocabulary and 18 in the old.

The number to sit with is not the five. It is the 26. The words “hello world” cost two tokens. One Scottish flag costs thirteen times that, and more than twice the entire four-person family emoji.

What this does not mean

It does not mean emoji are expensive in money. At current input rates a 26-token flag costs a small fraction of a cent, and you would need to send it tens of thousands of times before it showed up on a bill. Anyone selling you emoji removal as a cost saving is selling you a rounding error.

It also does not mean the newer vocabulary is worse. Twenty-two of twenty-seven emoji got cheaper, several of them by half, and the family emoji dropped from 18 tokens to 11. The regression is real and it is narrow.

What it means is the thing worth carrying away: the model’s view of your text is not your view of your text. Part of the difference is real length, since one flag is 28 bytes and one heart is six. The rest is the list: what was common enough to become a single piece, and what was not. That list was frozen before the model was trained, and the same symbol changes price when the list changes.

It is also part of why models are shaky at questions about spelling and letter counts. The text arrives already cut into pieces, and the pieces rarely line up with letters.

If you want the same effect at the scale of whole languages rather than single symbols, the token index measures 28 of them, and what tokenization is covers the mechanism from the beginning.