When you press Enter on an AI chatbot, you trigger a high-concurrency execution pipeline on a remote GPU cluster, and the pause that follows is not generic waiting.
It is the time the system needs to assemble a multi-part document, pass it through a safety cascade, and run a compute-bound "prefill" pass. Only after the model has read your entire input in one parallel sweep does it switch to the sequential "decode" phase that produces text.
The system treats your input as a fresh request because large language models (LLMs) are stateless. It remembers nothing between turns, so every interaction requires the backend to reconstruct your entire conversation history from scratch and push it through the transformer's attention mechanism. That initial pause is the hardware digesting your history to populate its memory, and the "typing" that follows is limited by how fast data moves through that hardware.
From a systems perspective, everything between input and first token is an optimization problem. Every stage, from context engineering through paged memory allocation, exists to maximize GPU throughput and minimize time to first token. Once you can see the pipeline, the strange behavior stops being strange: performance varies by language, by conversation depth, and by how many other people happen to be using the same machine.
The model never receives the sentence you typed
When you submit a query, the backend performs context engineering to wrap your raw message into a structured document. What arrives at the model is not your latest sentence but a bundle: a system prompt with behavioral rules, tool definitions, knowledge retrieved from external databases (RAG, retrieval-augmented generation), and the full history of the current session. Because LLMs are stateless by design, they have no inherent memory of previous turns, so engineers have to reintroduce the model to the entire conversation every time you speak.
Effective context engineering relies on precise formatting to help the model separate these inputs. Implementations use XML tagging (<background_information>, <instructions>) and Markdown headers to organize the document, which looks fussy for something only a machine will ever read. But that segmentation is what lets the model weigh the system instructions against your latest query in the right proportion.
This reconstruction leads to aggressive token compounding. A conversation might start at 1,100 tokens on turn one, and by turn twenty the system is reprocessing roughly 4,900 tokens for a single response. Because input volume grows while output stays roughly constant, most of the compute budget in a conversational product goes on re-reading history you have already seen, which is the least glamorous line item on any inference bill.
That growth eventually triggers context rot, where accuracy degrades as input length increases. It is a consequence of the transformer's architecture, which needs pairwise relationships between all tokens: as the sequence expands, the attention budget stretches thin and specific details buried in the middle of the document get harder to retrieve. A longer context window is not the same thing as a better memory.
The safety layer that runs before the model is called
Before your assembled document reaches the main model, a separate and much smaller classifier reads it and decides whether the input violates usage policy. Keeping that judgment in its own process lets engineers update safety rules or routing logic without retraining the main model's enormous parameter set.
Historically this check imposed a real latency tax. Early production classifiers added roughly 23.7% compute overhead, along with about a 0.38 percentage point rise in false refusals — harmless prompts blocked by mistake. At high volume that overhead was close to unsustainable.
The answer was a cascade, and it is one of the tidier trades in this whole pipeline. A very cheap, high-speed validator screens all incoming traffic, and only the prompts it flags get escalated to the expensive, accurate classifier. That cut compute overhead to roughly 1% and brought the false-refusal increase down to about 0.05 percentage points, which keeps the safety and drops most of the cost.
Why the same passage costs more tokens in some languages
Models process text as tokens, fragments of bytes rather than whole characters, produced by a tokenizer, most commonly Byte Pair Encoding. In English one token averages around three quarters of a word. But because tokenizers are trained on English-heavy datasets, they shatter other languages into much smaller, less efficient pieces.
The paper Language Model Tokenizers Introduce Unfairness Between Languages found that the same text translated across languages can vary in token count by as much as 15 times. That measurement was taken on a specific set of tokenizers at a specific moment, so it is not a universal constant for every model shipping today.
Vietnamese makes the gap concrete. Measured on that paper's own parallel corpus, the same body of text costs about 2.45 times as many tokens in Vietnamese as in English under cl100k_base, the tokenizer behind GPT-3.5 and GPT-4. The tokenizer has to be named for the number to mean anything, because older ones were far worse: the GPT-2 tokenizer charged 4.54 times.
The consequence survives the caveat. A speaker of a less-optimized language hits the context window limit sooner on the same amount of information, waits longer for the first token, and pays more per unit of meaning. The disparity appears at the tokenization stage, before the model is even invoked, which means no amount of clever prompting can undo it.
You are sharing a GPU with strangers
When you send a request, you join a queue and get batched with other users. LLM serving is memory-bandwidth bound: the GPU spends more time moving model parameters into its cores than doing arithmetic. To make serving affordable, providers load the parameters once and apply them to a batch of many users at once.
Static batching waits for the longest response in a group to finish before freeing any slot. Continuous batching — scheduling at the granularity of a single iteration — lets a new request take the slot of a finished one immediately. In Anyscale's benchmark that delivered up to 23 times the throughput of static batching.
Sharing introduces floating-point non-associativity, a hardware quirk where (a + b) + c does not always equal a + (b + c) because of rounding at finite precision. Your prompt is processed alongside a varying number of other requests, so the order of operations changes with server load. The result is run-to-run nondeterminism: the same prompt can produce different answers even at temperature zero, the setting meant to force the same answer every time.
That is not the model's reasoning wobbling; it is the underlying arithmetic changing based on how many strangers happen to share your GPU at that moment.
Prefill and decode: the pause and the typing
Response generation runs in two phases, and they behave like opposites.
Prefill is the pause. The model reads your entire prompt in a single parallel pass, using the GPU's parallel architecture to digest every input token at once and compute the internal states generation needs. This phase is compute-bound, and its latency is what "time to first token" measures.
Decode is the typing. The model produces exactly one token at a time, and each new token requires the hardware to read back the entire cached state of the conversation. Because that is limited by memory bandwidth rather than raw math capacity, the typing speed stays roughly constant, measured as time per output token. The two phases also fail differently, so a long wait before the first word and a slow crawl afterwards are not the same complaint.
The longer your conversation, the longer the initial pause grows as the GPU digests the history — but once text starts appearing, it appears at about the same speed as always.
Why the KV cache decides how many people one machine can serve
The real bottleneck in LLM serving is the KV cache, which stores the intermediate keys and values computed for every token in a conversation so decode does not have to recompute them. For a 13B parameter model that cache consumes roughly 800 KB of GPU memory per token. On an A100 40 GB, after about 26 GB goes to the model weights, what remains holds only around 15,000 tokens.
Traditional serving systems managed that memory badly, reserving one contiguous block sized for the maximum possible response length, which left 60% to 80% of expensive GPU memory allocated but unused. That fragmentation capped how many users a single machine could serve.
PagedAttention, pioneered by vLLM and modeled on operating-system virtual memory, splits the KV cache into fixed-size pages allocated on demand and stored in non-contiguous blocks. Over-provisioning disappears and memory waste drops below 4%, so the same hardware serves substantially more people. Borrowing an operating-system trick to fix a GPU memory problem is my favorite kind of engineering: no new math, just better bookkeeping.
Streaming: once a word is on screen it cannot be taken back
Streaming displays tokens as they are generated, which makes the system feel much faster even though the total time is unchanged, and it creates a real dilemma for safety.
If the output safety check runs only on the finished response, the retraction problem appears: the model may have already displayed something harmful before the system can flag it. You cannot un-say a word already on the user's screen. If the system instead waits for a full check before showing any text, the benefit of streaming disappears entirely.
So systems must moderate in parallel with generation, relying on very fast per-token guards, or read the model's internal states during generation to predict a violation before the token is ever rendered.
Tool calls send the whole pipeline back to the start
When a chatbot performs a tool call — a web search, a database query — the model does not perform the action. It generates text requesting the call. The surrounding application intercepts that text, executes the tool, and feeds the result back as new input.
This is not a straight line; it is a loop. Every time a tool returns data, the whole context (system prompt, original history, the tool request, and the tool output) is reassembled and pushed back through the prefill pause. A task that needs three tool calls runs the entire pipeline four separate times, which is exactly why tool-using responses take noticeably longer.
Costs compound sharply too. Across twenty tool calls, the earliest messages in the conversation are paid for twenty times over. To contain this, engineers use compaction — summarizing history to discard redundant data like raw tool logs while preserving key decisions — and structured note-taking, where the model keeps an external notes file pulled into context only when needed.
Where did those two seconds actually go?
The delay is a composite. A small slice goes to network travel and authentication, then the assembly of your context document. The safety cascade adds roughly 1% of compute time. Your request may then wait in a queue for a continuous batching slot if the server is busy.
The largest share, in any conversation that has been running a while, is the prefill pause: the GPU performing a massive parallel read of your entire history to populate the KV cache and compute attention scores. That is the price of statelessness — the model re-reads everything, every time.
There is a practical lever in that. Keep the stable parts of a prompt (the system instructions, fixed examples) at the top and put the changing content last, so prefix caching can hold the static computation in memory between turns. It cuts prefill time and the bill along with it.
References
- What Happens Inside an AI Chatbot Between Enter and the First Word? — ByteByteGo
- Effective context engineering for AI agents — Anthropic
- Next-generation Constitutional Classifiers — Anthropic
- Language Model Tokenizers Introduce Unfairness Between Languages
- Achieve 23x LLM Inference Throughput & Reduce p50 Latency — Anyscale
- Defeating Nondeterminism in LLM Inference — Thinking Machines Lab
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Mastering LLM Techniques: Inference Optimization — NVIDIA
- Prompt caching — Claude Platform Docs