Every practical question about language models eventually becomes an arithmetic question. Will this document fit. What will this feature cost per user. How much history can the conversation keep. All three need a conversion between the units you have — words, characters, pages — and the unit the model bills in. The commonly quoted ratio works for one kind of text and fails badly for the rest.
The working numbers
For English prose: 1 token ≈ 0.75 words ≈ 4 characters. Multiply words by 1.3 to get tokens.
For code, JSON and non-English text those figures are optimistic by 30% to 200%. Use the table below rather than the general rule.
The conversion table
| Content type | Chars/token | Words → tokens | vs English |
|---|---|---|---|
| English prose | ~4.0 | × 1.3 | baseline |
| Technical English | ~3.6 | × 1.5 | +15% |
| Code | ~3.0 | × 1.8 | +35% |
| Minified JSON | ~2.5 | n/a | +60% |
| Western European | ~3.2 | × 1.6 | +25% |
| Cyrillic / Greek | ~2.0 | × 2.5 | +100% |
| CJK | ~1.5 | n/a | +150% |
| Random strings / UUIDs | ~2.0 | n/a | +100% |
The last row catches people out in data pipelines. A UUID is 36 characters of high-entropy text that no tokeniser has ever seen before, so it fragments into roughly a dozen tokens. Ten thousand rows carrying an ID the model will never reference is a five-figure token cost for nothing.
Bigger units
| Unit | Words | Tokens (English) |
|---|---|---|
| A tweet-length post | ~50 | ~65 |
| A page | ~500 | ~650 |
| A blog article | ~2,000 | ~2,600 |
| A 10-page report | ~5,000 | ~6,500 |
| An academic paper | ~8,000 | ~10,500 |
| A 300-page book | ~150,000 | ~200,000 |
These are the figures worth memorising, because they turn abstract context window sizes into something you can picture. A 200,000-token window is a book. A 32,000-token window is a long report and not much else once you have added a system prompt and room to answer.
Why the ratios move
Tokenisers are built by finding the most frequent character sequences in a training corpus and giving each one an ID. Frequency is the whole mechanism, and it explains every row above.
The corpus was predominantly English, so English got the efficient merges. Everything else pays a surcharge — which for some scripts approaches one token per character, meaning identical content costs two to three times as much and fits in a third of the window. That is a real and rarely acknowledged inequity in how these systems are priced. The mechanism behind it is covered in why models cannot count letters.
⚠️ Different models, different counts
There is no universal token. Each model family ships its own tokeniser with its own vocabulary, so the same paragraph produces different counts on different models — commonly a 10–20% spread, more for non-English text.
Estimates transfer between models; exact counts do not. If your logic depends on a precise number — truncating input, enforcing a limit, billing a customer — use the tokeniser for the model you are actually calling.
Estimating safely
For planning, characters are the better input. They are exact, trivially obtained, and do not depend on how you define a word.
Three sources of undercount to build headroom for:
- The system prompt, which is sent on every request and is easy to forget when totalling a conversation.
- Message formatting overhead — role markers and structural tokens the API adds around your content.
- The output, which shares the window with the input and is the part you have least control over.
🚨 The context window is not the input budget
A 128,000-token window does not mean you may send 128,000 tokens. Input, output and system prompt all share it, and hitting the ceiling mid-generation truncates the answer.
Reserve output space explicitly. If you need a 4,000-token response, your real input ceiling is the window minus 4,000 minus your system prompt minus a margin — and treating the headline number as your allowance is how conversations fail at the worst moment.
When you need the exact number
Estimation is for planning. Anything enforced needs a real count, and there are three ways to get one:
- Run the model's tokeniser locally. Most vendors publish theirs; this is exact and free.
- Use a token-counting endpoint where the provider offers one — exact, and no generation cost.
- Read the usage figures on the response. Exact, but only after you have paid for the call.
For the character and word counts that feed your estimate, our word counter gives exact figures in the browser — characters with and without spaces, words, lines and sentences, with nothing uploaded.
Cutting tokens without cutting content
Ordered by how much they typically save:
| Change | Typical saving |
|---|---|
| Send only the columns and fields you need | Up to 90% |
| CSV instead of JSON for tables | 50-60% |
| Drop unused IDs and UUIDs | 10-30% |
| Minify JSON you do send | 15-25% |
| Round numbers to needed precision | 5-15% |
| Trim the system prompt | Small, but on every call |
The first row dwarfs everything else and is the one people skip, because filtering data is more work than changing a format flag. Sending twelve relevant rows instead of the whole table is not a compression technique — it is the difference between a query and a data dump, and it improves accuracy as well as cost. The format-level detail is in token-efficient data formats.
Need exact word and character counts?
Count characters, words, sentences and lines instantly in your browser — nothing is uploaded.
Open Word Counter →Summary
- English prose: 1 token ≈ 0.75 words ≈ 4 characters. Words × 1.3 for tokens.
- Code costs about 35% more, minified JSON about 60% more, for the same characters.
- Non-English text pays a large surcharge — up to 150% for CJK scripts.
- UUIDs and random strings fragment badly. Strip identifiers the model will not use.
- A page ≈ 650 tokens; a book ≈ 200,000.
- Estimate from characters, then add 20% headroom.
- The window is shared between system prompt, input and output — reserve output space.
- Estimates transfer between models; exact counts do not.
Frequently Asked Questions
How many words is one token?
For ordinary English prose, roughly 0.75 words per token, or about 1.3 tokens per word. That ratio is a reasonable planning figure for articles and conversation, and it degrades quickly for code, structured data, technical vocabulary and any language other than English.
How many characters are in a token?
About four characters per token for English prose. This is often the more reliable rule because character counts are exact and easy to obtain, while word counts vary with how you define a word. For dense punctuation like JSON, expect closer to two or three characters per token.
How many tokens is a page of text?
A standard page of around 500 words is roughly 650 to 700 tokens. A typical business document of 10 pages lands near 7,000 tokens, and a 300-page book is in the region of 200,000 — which is why book-length context windows became a meaningful milestone.
Why does code use more tokens than prose?
Because code is dense in punctuation, indentation and identifiers that were never common enough to earn their own token. Braces, brackets, operators and camelCase names all split into pieces, so the same character count costs substantially more than prose — commonly a third more, and worse for minified or deeply nested code.
Should I estimate tokens or count them exactly?
Estimate for planning and count exactly for anything enforced. Rules of thumb are fine for deciding whether a document will fit or budgeting a feature, but if you are truncating input or billing a customer, run the actual tokeniser for the model you are using — different models tokenise the same text differently.