A language model that can write a working parser, explain a legal contract and debug your CSS will tell you that "strawberry" contains two r's. The gap is jarring enough that it gets cited as proof that these systems do not really understand anything. It is proof of something much more specific and much less philosophical: the model was never shown the letters.
The short answer
Models do not read text. Before anything reaches the model, a tokeniser converts your text into numeric IDs representing chunks of words. "strawberry" becomes two or three numbers.
Asking how many r's it contains is asking about information that was discarded before the first layer of the network. The model is not reasoning badly — it is answering from memory about something it cannot see.
What the model actually receives
You type ten characters. The model receives something closer to this:
That last line is the whole article. Token 15717 is an arbitrary index into a vocabulary. It is not spelled anything. There is no "b", "e", "r", "r", "y" inside it to count — the model has a learned vector for that ID, and the vector encodes how the token behaves in language, not what letters compose it.
So when you ask for a letter count, the model cannot look. It has to recall facts about spelling that it absorbed indirectly — from text discussing spelling, from words split differently in other contexts — and then reason over that recollection. Sometimes that works. It is guessing from memory either way.
💡 A fair analogy
Imagine being shown a page of text as a set of word-shaped stamps rather than individual letters, then asked how many times a particular letter appears. You might know the answer for familiar words. You cannot check.
That is the model's position permanently. Its competence at spelling questions tracks how often the answer appeared in training text, not how carefully it thinks.
Why text is chunked at all
The obvious fix — feed the model individual characters — was tried and is worse. Tokenisation is a compromise between two failure modes.
| Approach | Vocabulary | Sequence length | Problem |
|---|---|---|---|
| Characters | ~100 | Very long | Wastes context; attention cost grows sharply |
| Whole words | Millions | Short | Every typo and rare word is unknown |
| Subwords | ~50k–200k | Moderate | Characters become invisible |
Character-level models spend their context window on the letters of common words, and attention cost scales badly with sequence length — you pay enormously for information that is almost never needed. Word-level models break the moment they meet a word they were not trained on, which is constantly.
Subword tokenisation, usually byte pair encoding, splits the difference. It starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary is full. Common words survive as single tokens; rare ones decompose into pieces; nothing is ever truly unknown.
It is a good trade for almost everything. The cost is paid entirely by tasks that need to see below the token boundary — and it is paid invisibly, which is why the failure feels so strange when you hit it.
What this actually breaks
Every one of these is the same root cause wearing a different hat:
| Task | Why it fails |
|---|---|
| Counting letters | Letters are not separately represented |
| Reversing a string | Reversing tokens is not reversing characters |
| Rhyming | Rhyme is about sound and ending letters |
| Syllable counting | Token boundaries are not syllable boundaries |
| Acrostics | Requires controlling one letter at a time |
| Spotting typos | A typo makes an entirely different token |
| Character ciphers | Operates strictly below the token |
| Long arithmetic | Digits do not group by place value |
The last one is the most consequential and the least discussed, because it produces confident wrong numbers rather than obviously silly answers.
Modern tokenisers often force digits into fixed groups specifically to reduce this, which helps and does not eliminate it. It remains the reason a model can explain long division correctly and then get one wrong.
⚠️ Non-English text pays twice
Tokeniser vocabularies are built from training corpora dominated by English, so English gets the efficient merges. The same sentence in a less-represented language can take two to four times as many tokens — and in some scripts, close to one token per character.
That is a direct cost multiplier on the API bill and a direct reduction in how much fits in the context window, for identical content. It is one of the more concrete inequities in how these systems are built.
Why newer models pass the test
Try the strawberry question on a current model and it will probably answer correctly. Tokenisation has not changed. Three other things did.
The example became famous. "How many r's in strawberry" appeared in enough articles, posts and benchmarks that the answer is now a memorised fact. The model is not counting; it is recalling. This is the least interesting reason and the largest contributor.
Reasoning models spell things out. Given room to think step by step, a model will often write the word one character at a time before counting — and that changes everything, because spaced characters tokenise separately.
Tool use bypasses it. A model that can run code writes three lines of Python and reports the result. The problem stops being a language problem.
🚨 Passing the famous test proves nothing
Because the fix is memorisation and technique rather than representation, the failure is still there — it has just moved to words nobody wrote articles about. Try a long unusual surname, an invented word, or a phrase in a language the tokeniser handles poorly, and the errors return.
If you are relying on character-level accuracy in production, do not conclude from a successful strawberry test that the class of problem is solved. Test the inputs you will actually see.
Getting it right anyway
In increasing order of reliability:
- Space the characters yourself. Passing
s-t-r-a-w-b-e-r-r-yputs every letter in view. Crude, effective, free. - Ask for step-by-step enumeration. "List each letter with its position, then count" forces the model to construct the visible representation before answering.
- Have it write code.
text.count('r')is exact. Any character-level task that must be correct should end up here. - Do it outside the model entirely. Counting, reversing and validating are things ordinary code has done perfectly for decades. A model is the wrong instrument for a task with a deterministic answer.
The general principle is worth more than the specific fixes: a language model is for tasks where language is the hard part. When the hard part is exact manipulation of symbols, use something exact. If you need to know how long a piece of text is, our word counter gives you characters, words and lines with no guessing involved.
The mental model to keep
Tokenisation explains a whole family of otherwise baffling behaviours, and once you can see it, you stop being surprised:
- Why the same content costs different amounts in different languages.
- Why a trailing space can change a completion.
- Why models are oddly bad at word games and oddly good at grammar.
- Why a typo sometimes derails a response far more than it should.
- Why your bill does not match your word count.
None of these are reasoning failures. They are all the same fact: the model's view of your text is chunked, and the chunks are not the units you think in.
Need an exact character or word count?
Count characters, words, sentences and lines in your browser — instant, exact, and nothing is uploaded.
Open Word Counter →Summary
- Models receive token IDs, not letters. Character information is gone before the first layer.
- The strawberry failure is a representation problem, not a reasoning one.
- Subword tokenisation is a deliberate trade — it beats characters on cost and words on coverage.
- Anything below the token boundary breaks: counting, reversing, rhyming, ciphers, typos.
- Digits group by frequency, not place value, which quietly degrades long arithmetic.
- Non-English text costs more tokens for identical content.
- Newer models pass by memorisation and spelling out, not because tokenisation changed.
- For exact character work, use code. The model is the wrong tool for a deterministic job.
Frequently Asked Questions
Why can't ChatGPT count the letters in a word?
Because it never receives the word as letters. Text is converted to tokens before the model sees it, so 'strawberry' arrives as two or three numeric IDs representing chunks of the word, not as ten separate characters. Asking how many r's it contains is asking about information that was discarded before the model's first layer.
Is the strawberry problem a sign that LLMs can't reason?
No, and it is a poor test of reasoning. It measures whether the model has access to character-level information, which it does not by default. It is closer to asking someone to count the pixels in a photograph they were shown as a summary — the failure is in what was supplied, not in the thinking.
Why do newer models get the strawberry question right?
Mostly for two reasons, neither of which is a fix to tokenisation. The example became famous enough to appear throughout training data, and reasoning models tend to spell words out one character at a time before counting, which converts the problem into one they can see. Ask about a rarer word and the failure often reappears.
What other tasks does tokenisation break?
Anything operating below the token boundary: counting characters, reversing strings, rhyming, syllable counting, spotting typos, acrostics, and character-level ciphers. Arithmetic on long numbers is affected too, because digits do not group into tokens the way place value requires.
How do I get a model to handle letters correctly?
Force the characters into view or move the work out of the model. Spelling the word with spaces or hyphens between letters makes each character its own token; asking the model to write code that does the counting removes the guesswork entirely. Code is the reliable option for anything that must be exactly right.