A token is a unit of text that a language model can ingest. Before the text you type enters an LLM, it first goes through a tokenizer, which converts it into a form the LLM can understand.
These tokens are what many model providers charge you for. This means that, much as different languages can use different numbers of characters to represent the same idea, they can also use different numbers of tokens. In the former case, we don't pay much attention because we aren't explicitly charged for using more characters or words in some languages. That changes when it comes to tokens. All of a sudden, those extra pieces have a literal price.
When it comes to regular writing, those words still have a “price,” but it isn't an explicit dollar amount. Computers only understand binary, which means text needs to be mapped to a binary representation. Unicode assigns code points to characters, and UTF-8 turns those code points into bytes. The details aren't super important, but the key point is that characters in higher code-point ranges often require more UTF-8 bytes. That means more space when the text is stored or transmitted as UTF-8, though it does not automatically mean more memory inside a language model. Now we have two related axes on which to think about cost: raw UTF-8 bytes and model tokens.
01 · bytes
The storage bill
UTF-8 stores a Unicode code point using one to four bytes. This matters for files and networks, and directly affects byte-level models.
02 · tokens
The model bill
A tokenizer maps the text into learned pieces, which many model providers charge you for.
A question you can ask at this point is: are these two bills correlated at all? Take Chinese, for example. Chinese characters usually use three UTF-8 bytes, yet Chinese sentences can express the same meaning with fewer characters because the writing system is morphosyllabic: one character can encode both a syllable and a unit of meaning instead of spelling it across several letters. In the test I ran, Simplified Chinese used roughly the same total bytes as English and only 1.10 times as many Qwen tokens.
The experiment
How was this set up?
I first took 250 FLORES sentences in each of the test language-script varieties. These were translations of the same source sentences, chosen to express the same meaning across languages, though exact nuance and information density can still differ. For each of these sentences, I counted the UTF-8 bytes, graphemes, and tokens (measured across six different tokenizer families).
As a way to check whether this correlated at all with performance on one downstream task, I also used a separate test called Belebele from Meta to measure Qwen2.5-0.5B’s ability to answer the reading-comprehension questions in every language-script variety. Note that this is purely a toy run and is not intended to make any causal claims about languages being intrinsically harder. Model performance on any domain is highly dependent on how the model was trained. The toy hypothesis to test here was basically: if you pick an off-the-shelf, open-weight model, does the Qwen token cost for a language correlate at all with how the model performs on this task?
The interactive results below use Qwen’s tokenizer. The Belebele result uses all 900 questions per variety, two English format examples, and next-token A/B/C/D scoring.
Every multiplier on this page is relative to English, where English = 1.00×
Cost × capability
Not making any causal claims here but it's cool to see a correlation.
Each dot in the graph below is one language-script variety. A few reference varieties are pre-labeled to keep the graph readable. Hover over any dot to see the name of the language it represents. You can also switch between model tokens and raw UTF-8 bytes to control what the x-axis represents. Moving right means a higher cost relative to English, and moving down means lower accuracy on this particular Belebele reading-comprehension setup.
Hover over any dot to reveal its language and bring it into focus.
This last value is a packing estimate: Belebele accuracy × 8,192 ÷ average prompt tokens, normalized to English. It is not measured serving throughput.
To put some numbers to this correlation, we can calculate something called the Pearson correlation and the Spearman correlation. Pearson is a way of measuring the strength of a linear relationship between two variables. Spearman, aka rank correlation, does something similar except it checks how the rankings between the two variables move together.
Across the 29 language-script varieties, token premium and accuracy had a Pearson correlation of −0.80 and a Spearman correlation of −0.91, indicating that there was indeed a negative association. It’s super important not to think of this as proof that extra tokens lead to lower accuracy. As mentioned earlier, there are a ton of variables—most importantly tokenizer-training data and model-pretraining coverage—that could lead to both a poorly matched tokenizer vocabulary and a weaker model.
It does open the door to more downstream research, such as training matched models where the tokenizer is the main thing that changes, separating the effects of training data from vocabulary coverage, or testing how much equivalent meaning each language can fit inside the same token budget.
Writing systems
A closer look at the trends that appear if you group by writing system
One thing you might have noticed in the graph above is that writing-system group is associated with where a variety lands in this particular panel. Below, we can see what the multipliers look like for each group. Once again, all of these are relative to the English baseline.
As a quick primer on these writing systems:
- Alphabets explicitly represent both vowels and consonants.
- Abjads primarily represent consonants.
- Abugidas have consonants that carry an inherent vowel that can be modified or suppressed entirely.
- Chinese characters form a morphosyllabic writing system.
- Japanese mixes several systems: a logosyllabary (kanji), two syllabaries (hiragana and katakana), and sometimes the Latin alphabet.
For the purposes of this exercise, I’ve grouped Persian, Urdu, Pashto, and Sindhi with the abjad scripts. There are always points of contention around which categories certain writing systems fall into, which is why these groups are meant to be roughly descriptive and not super clean categories.
Token and byte premiums are geometric means within each group; accuracy is an arithmetic mean.
With respect to the language groups, there are some interesting trends with token premiums. In our sample, the Chinese/Japanese group was closest to English at 1.24×, alphabets and abjads were in the middle at about 2.3×, and abugidas were much more expensive on average at 5.39×. So, directionally, knowledge of the writing system does tell us something about the token cost. But there are definitely exceptions. For example:
- Arabic and Hebrew are close to many European languages.
- English and Armenian are both alphabetic, but Armenian used about 5.1 times as many Qwen tokens as English.
That variation probably reflects a mix of orthography, morphology, UTF-8 width, tokenizer pre-processing and fallback behavior, vocabulary allocation, and Qwen’s pretraining coverage. The writing system is part of the story, but it is not the whole story.
Hardware
What did those extra tokens do to the GPU?
There’s one level deeper we can look here, since sequence length helps determine how much work the GPU does alongside padding, batching, model architecture, and the attention implementation. I ran Qwen2.5-0.5B on an NVIDIA L4 using batches of 16 aligned FLORES sentences and measured a real KV-cache prefill without truncating the text. Each number below is the median of five trials run in randomized language order, with 20 timed iterations after warmup. Board energy includes the GPU’s active baseline, and peak allocated memory includes the resident model. Choose one of the 12 profiled varieties below to compare it with English.
For the same batch of 16 translated sentences, Burmese used 8.3 times as many input tokens as English. In this benchmark, its median measured board energy was 6.4 times the English result.
What I took away
Some reflections
This whole thing started because I could not stop thinking about how UTF-8 (shoutout to that one Tom Scott video) should absolutely have some parallel with the tokenizers we use in LLMs. The idea of storage and model-cost profiles being a function of how we encode, tokenize, or use any other word for compression that you want to throw in here is something I wanted to dive into. Also, the fact that writing systems have spent millennia balancing compactness and decoding is incredibly exciting.
Some takeaways:
In this panel, writing system matters in a way that is directional, but ultimately any writing system can be made more token-efficient if you invest in a tokenizer and corpus that represent it well.
As a quick aside, one of the best parts about BPE is how conceptually simple it is to improve—give representative patterns more merges and vocabulary capacity—yet how fundamentally difficult that is operationally for low-resource languages and low-resource domains within those languages.
The next thing I’m going to try is holding the model architecture, training data, and compute budget fixed while training matched models that differ mainly in their tokenizer. This should give us a better sense of whether what we’re seeing here generalizes to other tokenizers or is specific to Qwen.
What this does not prove
- The writing-system groups are small, curated panels, not representative samples of each group.
- The accuracy result comes from one 0.5B-parameter base model (Qwen) and one answer format.
- Aligned sentence IDs from FLORES across languages do not guarantee identical information density across translations.
- The GPU numbers describe one model, batch shape, library stack, attention implementation, and NVIDIA L4.