Every time you type a question into ChatGPT or Claude, the model does not read letters or words the way you do. It reads tokens.
AI tokens are the smallest units of data that a language model reads in and writes out
— usually a fragment of a word, sometimes a short whole word, sometimes just a punctuation mark. The context window is the memory limit that comes with them: the total number of tokens the model can hold in mind at any single moment.
Understanding these two units is the only reliable way to predict cost, control accuracy, and understand why AI sometimes "forgets" what you just told it.
What are AI tokens?
Tokens are the only "communication protocol" that artificial neural networks actually process. Rather than reading individual letters or whole sentences, the model sees the world of data as encoded fragments, each carrying a numeric ID. From that sequence of IDs, it learns linguistic patterns and predicts the most likely next fragment.
A token is typically a base unit: a character, a word, or a common phrase. Short, frequent words usually fit into a single token, while longer or rarer words are split. A few rules of thumb for English text:
- 1 token ≈ 4 characters
- 1 token ≈ ¾ of a word
- 100 tokens ≈ 75 words
This is not limited to text. Data is turned into tokens according to its modality: text becomes words and phrases, images become pixels or voxels, audio becomes spectrograms, and source code becomes syntactic segments. Whatever the data type, inference comes down to one job — predicting the next token in a sequence.
Tokens are also categorized by function. Input tokens and output tokens are the ordinary carriers of data — your prompt goes in, the model's response comes out. Alongside them are reasoning tokens, which appear in "long thinking" models. These models generate an intermediate chain of thought before answering, and that hidden work consumes enormous resources — potentially over 100× more than a standard inference pass.
This matters because the token count measures more than text length — it measures the workload a GPU must carry to produce a result. Counting tokens lets you predict both the complexity and the cost of a task.
How does tokenization turn text into machine-readable data?
Tokenization is the mandatory preprocessing step that turns unstructured language into the numerical format a neural network can digest. It happens before the model computes anything.
A tokenizer scans the text and cuts it along spaces and punctuation. The most common method is Byte Pair Encoding (BPE), favored because it balances flexibility and efficiency: rare words are split into common sub-word fragments, while frequent words stay intact as a single block. For example, "darkness" is split into "dark" + "ness." This lets the model recognize that "darkness" and "brightness" share the suffix "ness," and infer the meaning of words it has never seen from their component structure — no one has to teach it that "ness" denotes a state.
After each token receives its own numeric ID, the model converts them into vectors called embeddings. These vectors encode semantic relationships in a high-dimensional space, letting the model compute the "distance" between concepts rather than operating on raw characters. Input tokens are compressed into vectors for processing, then output tokens are decompressed back into the natural language you read.
Why do tokens matter for cost and performance?
In modern "AI factories," tokens are both a technical unit and a unit of currency. Providers such as OpenAI and Anthropic commercialize their models based on the number of tokens consumed, which makes token optimization central to resource management.
The room to optimize is large. Software optimization combined with newer hardware can cut the cost per token by up to 20×, and some real-world cases have recorded a 25× revenue increase in just four weeks from improved token-processing speed. Two metrics commonly measure the experience: Time to First Token — the latency before the AI begins responding — and inter-token latency — the speed at which subsequent tokens are produced.
There is always a trade-off between quality and speed. Deep-reasoning models produce "smarter" tokens but at higher latency and much higher cost. The job of the user — and of the engineer — is to balance processing cost against the real value each token delivers.
What is a context window and how does it work?
A context window is the model's "working memory": the maximum number of tokens — including the current prompt and the preceding conversation — that it can consider at once to produce a coherent response.
When the total tokens (both input and output) exceed the limit, the system applies FIFO (first in, first out) logic: the oldest tokens are pushed out to make room. This is exactly why an AI appears to "forget" instructions from the start of a conversation — they were evicted from the window before the model read your latest message.
Window sizes vary widely across architectures. A few classic illustrative figures:
| Model | Context window size (tokens) |
|---|---|
| BERT | 512 |
| GPT-3.5 | 4,000 (4K) |
| GPT-4 | 8,000 – 32,000 (8K–32K) |
| Claude | 100,000 (100K) |
These numbers reflect an earlier generation of models; the latest releases have pushed the limit into the hundreds of thousands, even millions of tokens. But the rule holds: the wider the window, the more an AI can summarize huge documents or handle complex research — at the cost of more noise, which demands smarter memory management so the model does not lose focus.
How do you use tokens efficiently?
Optimizing tokens is not only about saving money. It also raises accuracy by clearing out linguistic "noise" before the model has to attend to it.
- Write concise prompts. Focus on high-value keywords and cut filler and repetition to leave maximum room for the tokens that matter.
- Chunk long documents. Break large text into context-linked segments and process each one, rather than stuffing everything into a single call that overflows working memory.
- Manage priority. Filter out conversation history that is no longer needed, keeping only the information essential to the logic of the next response.
The essence of this is improving the signal-to-noise ratio. The fewer junk tokens loaded in, the less the model suffers attention drift, and the more accurate and direct its answers become.
What are the limitations and challenges today?
Tokenization is a mathematical approximation of language, not genuine understanding — and so it carries a few inherent errors.
- Semantic ambiguity. Words with multiple meanings, like "lie" or "play," can be assigned the wrong vector when context is too narrow, leading to logic errors in the response.
- Non-spaced languages. In Chinese or Japanese, determining token boundaries is extremely hard. The word for "hot dog" (热狗), if split wrongly into "hot" (热) and "dog" (狗), loses its original meaning entirely.
- Special cases. URLs, email addresses, source code, and phone numbers are often chopped into meaningless fragments, both wasting tokens and dragging accuracy down.
Remember that AI does not understand "words" — it operates on numeric probability patterns. Any error during tokenization can lead to "hallucination" or reasoning failures that users find hard to notice.
FAQ
Does one word always equal one token? No. Common words like "the" are usually a single token, but long or specialized terms (like "tokenization") are often split into several fragments — for example "token" + "ization."
Why does an AI start forgetting the beginning of a conversation? Because of the context window limit. When the token count exceeds the maximum, the oldest tokens are removed under FIFO logic to make room for new data.
Can tokens represent images? Yes. In multimodal systems, images are split into pixels or voxels and audio is converted into spectrograms — all processed as units equivalent to text tokens.