Tokenization basics: how LLMs count text and why it matters

What tokens are in plain language

A token is a small unit of text that a language model actually sees, for example a word, part of a word or sometimes just punctuation. Models do not work with letters or human words directly, they work with sequences of tokens that are mapped to numbers. Different models use different tokenizers, so the same sentence can become a different number of tokens. As a prompt engineer you rarely choose the tokenizer, but you do need a basic feel for how text breaks into tokens. This helps you reason about length limits, costs and why a model sometimes cuts off mid sentence.

How text is split into tokens

Most modern LLMs use subword tokenization. Frequent whole words like "you" or "house" may become a single token. Rare or longer words are split into several pieces, for example "unbelievable" can become "un", "believ" and "able". Emojis, line breaks and special symbols are tokens too. Importantly, spaces are usually encoded as part of tokens, not as separate ones. This explains why tiny edits, like adding a space or changing phrasing, can change token counts. You do not have to memorize rules, but you should understand that tokenization is mechanical and model specific.

Why token counts matter for prompt engineering

Every LLM call has a context window, which is the maximum number of tokens the model can consider in one go. Your input tokens plus the output tokens must fit inside this limit. More tokens usually mean higher cost and slightly slower responses. Long prompts with many examples or large pasted documents can silently eat most of the context window, so the model has little room left to answer. Understanding token counts helps you trade off detail against length. You can then decide what to keep, what to summarize and where to reduce repetition in your prompts.

How to estimate and control token usage

To work effectively, you need a rough sense of how many tokens your prompts use. A quick thumb rule is that 3 to 4 English words equal about 10 to 15 tokens, but this varies by language and content. For real work, use the tokenizer tools provided by your LLM platform to measure typical prompts and outputs. Then design your templates with a budget in mind. Shorten boilerplate, replace long repeated instructions with a single clear version and summarize long user inputs before sending them. This keeps you below limits and makes costs more predictable.

Common tokenization mistakes to avoid

Many beginners assume characters equal tokens and misjudge length badly. Others paste entire email threads, PDFs or logs and hit context limits without understanding why. A frequent issue is forgetting that invisible elements, such as extra spaces, markdown formatting and copied HTML, all increase token counts. Another trap is chaining prompts in a chat without pruning history, so older messages keep filling the context. To avoid these problems, regularly inspect token counts for real examples in your workflow. Also define simple rules, for example maximum input length or mandatory summarization, before sending content to the model.

Useful links

• OpenAI — Official information on models, pricing and token limits.

Keep learning