What Is Context Evaluation? Measuring What You Feed an LLM

Context evaluation is the process of measuring the accuracy, relevance and completeness of the information retrieved and handed to a large language model (LLM) before it generates anything.

Instead of judging only the final answer, you measure the quality of the raw material going in — so the model has the facts it needs rather than a plausible gap to fill.

In practice it is noise control for a RAG (Retrieval-Augmented Generation) system. When the context is wrong, an LLM fills the gap with hallucination or with stale internal knowledge, and it does that in exactly the same confident register it uses when it is right, which is why you cannot catch the problem by reading answers alone. Measuring the context rigorously is what tells you whether your pipeline broke at the retrieval step or at the reasoning step.

Context evaluation is not a pass/fail test — it asks whether every token you spend on the prompt earned the space it took.

Diagram of measuring the quality of retrieved context before it is fed into a large language model

What is context evaluation?

Context engineering is an optimization problem against a finite attention budget. You are not trying to load a million tokens; you are trying to find the smallest possible set of high-signal tokens that maximize the likelihood of the outcome you want. Every additional token depletes the budget the model has to reason with.

Traditional benchmarks like Needle In A Haystack (NIAH) have reached their limits here, because NIAH measures one narrow capability: lexical retrieval, matching a string you already know is present. Real systems need semantic understanding to connect information scattered across sources, and that is where the benchmark stops being informative. A model finds the needle easily when the keyword matches exactly, then falls over when the question requires reasoning across equivalent concepts.

A stricter standard is the gold context, which is the minimum set of artifacts genuinely sufficient to resolve a specific request rather than a sample document set. Building one requires human experts to annotate by hand, followed by a verification pass in which a strong model must be able to solve the task when given only that context and nothing else. That is slow, expensive work, and I have yet to see a shortcut that survives contact with a real product, because the value of the set comes entirely from a human having decided what "sufficient" means here.

That is also why you separate what the system saw from what it used. Plenty of pipelines post high recall and still fail, because the context was noisy enough that the model never extracted the part that mattered, and a dashboard full of green retrieval numbers will happily hide that for months. Context evaluation exists to measure that gap.

Why a complete-looking context pipeline still feeds the model poor material

RAG failures usually start one layer below the retriever, in the data itself. Even when precision and recall score near-perfectly, the answer is still wrong if the underlying documents are stale or mutually inconsistent, and no amount of tuning above that layer will fix it. That is how a pipeline that looks complete keeps serving the model poor material: every component reports success while the index is quietly out of date.

The four data-quality dimensions governing documents before they enter the vector database: freshness, ownership, lineage and canonical alignment

Governing that raw layer before it reaches the vector database comes down to four dimensions. Freshness asks whether the content reflects the current state of the system. Ownership establishes who is accountable for a document being correct. Lineage tracks how data traveled from its source into the vector database and whether it was distorted along the way. And canonical alignment checks that a document matches how the organization officially defines things, because when each team uses its own vocabulary, the embedding model has no reason to place those documents near each other.

Duplicate and superseded documents are the sharpest version of this problem, and the easiest one to create by accident. Leave the 2022 and the 2024 edition of the same handbook in the index and the retriever will eventually surface both, so the model ends up holding two contradictory facts with no basis for choosing between them. Weak data governance permits errors, and worse than that, it automates their production at scale.

Retrieval failures and generation failures: where to measure each

To fix a broken RAG system you first have to know whether it failed at finding or at writing. So split the pipeline in two, retriever and generator, and give each half its own metrics and its own knobs, because a single end-to-end score cannot tell you which half moved.

A RAG pipeline split into retriever and generator halves, each with its own group of metrics

The retriever's metrics point directly at system parameters. Contextual recall measures whether the retrieved context contains all the facts needed for the ideal answer, and it is effectively a measure of your embedding model. Contextual precision measures whether relevant chunks are ranked above irrelevant ones, which makes it a measure of your reranker. Contextual relevancy measures how much of the retrieved context is pertinent at all, and it reflects how you configured chunk size and top-K.

The generator's metrics judge what the model did with what it received. Faithfulness measures whether the answer stays inside the retrieved context or invents beyond it. Answer relevancy measures whether the response actually addresses the question, and it reads directly on the quality of your prompt template.

The isolation rule falls out cleanly: bad context means you fix the retriever — chunk size, top-K, embedding model, reranker. Correct context with a bad answer means you fix the generator instead, which is the prompt template and the model. Treat overall quality as a product rather than a sum, because if either half fails outright, output quality goes to zero regardless of how good the other half is.

Context precision and context recall: right ranking or full coverage?

These two metrics sound alike and measure opposite things. Context precision measures whether relevant chunks sit at the top of the ranking, and it is the mean of precision@k (the share of the top k chunks that are relevant) across positions, weighted by a relevance indicator, so burying the critical chunk at position ten tanks the score even though the information was retrieved. Context recall measures coverage: the reference answer is broken into individual claims, and each claim is checked for whether the retrieved context supports it.

Comparison of context precision measuring correct ranking against context recall measuring full coverage

The distinction determines how you read a bad score. Low recall means the information was never retrieved at all, and no reranker on earth will recover it. Low precision means the information is present but buried under noise — a ranking problem, not a search problem. Those are two completely different repairs, which is why collapsing both into one "retrieval quality" number costs you the diagnosis.

Both metrics also have non-LLM variants: a string-similarity version that compares against reference_contexts, and an ID-based version that compares retrieved_context_ids against known relevant IDs. The ID variant is cheap and deterministic, which makes it the practical choice for a large system that needs to score continuously without paying for an LLM call per sample, and if you are scoring on every deploy it is the one I would reach for first.

python

from ragas.metrics.collections import ContextRecall
 
scorer = ContextRecall(llm=evaluator_llm)
 
result = await scorer.ascore(
    user_input="Where is the Eiffel Tower?",
    retrieved_contexts=["Paris is the capital of France."],
    reference="The Eiffel Tower is located in Paris.",
)

There is always a trade-off here, and it is not one you get to escape by tuning harder. Retrieving generously raises recall and protects you from missing a fact, but if precision drops far enough the signal thins out and the model starts losing its way among things that were never relevant.

Automated scoring with LLM-as-judge, and where not to trust it

The RAG triad is the fastest framework to put automated scoring in place, and it is three separate checks rather than one score. Context relevance asks whether the retrieved chunks are pertinent to the query. Groundedness decomposes the response into individual claims and searches the retrieved context for evidence supporting each one. Answer relevance asks whether the final response actually serves what the user asked. Each runs as a feedback function, meaning a carefully prompted LLM acting as judge, which is what lets you score at scale when you have little or no ground-truth data.

The three separate checks of the RAG triad: context relevance, groundedness and answer relevance

That mechanism is also the weakness, and it is worth being clear about how big that problem is. The judge is a language model, so it carries the same biases it is measuring: a fluent, confident, well-formatted answer scores better than a correct but blunt one. The number it returns is a signal rather than a verdict.

The healthy way to use it is as a filter rather than a court. Periodically compare machine scores against human labels on a small sample, and re-run that comparison every time you change the judge model or edit the scoring prompt, because those are the two changes that silently shift your entire scale without raising an error anywhere, and nothing in your pipeline will warn you when it happens.

Distractors, stale documents and context rot: when more context hurts

Context rot testing across 18 leading models — GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 among them — found that performance degrades unevenly as input length grows, even on tasks as simple as retrieval and text replication. A model does not handle its 10,000th token as reliably as its 100th, which quietly undercuts the way most teams talk about long context windows.

Model reliability falling as the context grows longer while distracting passages crowd in

Distractors are the main culprit, meaning passages topically related to the question that do not answer it. A single distractor is enough to reduce accuracy, and additional ones compound the loss. Semantic distance matters too: the lower the similarity between the question and the needle, the faster performance falls as context grows.

One finding still bothers me every time I reread it. Across all 18 models, performance was better on shuffled haystacks than on logically structured ones, which is the opposite of what anyone building a retrieval pipeline assumes. A coherent narrative creates semantic competition, because the surrounding ideas are similar enough to camouflage the one you actually need. The engineering conclusion is blunt — filter noise before you widen the window. Enlarging the context window does not remove the evidence-selection problem.

Why you still have to read the outputs yourself: error analysis and a failure taxonomy

Automated metrics are the surface. You cannot improve a system without reading real traces, meaning the full record of a run, covering both what the system retrieved and what it produced. It is tedious and it does not scale, and it is still the single most valuable activity in evaluation, because it decides which evals are worth writing in the first place instead of leaving you to borrow a generic metric suite.

The manual error-review process: reading traces, open coding, axial coding, then building a failure taxonomy

The process has three moves. First, read traces and do open coding: write open-ended notes in your own words about anything wrong, concentrating on the first failure in each trace, since later errors are usually downstream consequences. Then do axial coding: group similar notes into categories to build a failure taxonomy specific to your project, and count how often each category occurs. That grouping step is the most important one in the whole exercise.

You stop at saturation, which is when new traces stop producing new categories. As a rule of thumb, review at least 100 traces to start, and if roughly 20 in a row surface nothing new, you can stop. A working taxonomy tends to hold labels like Missing_Information (the data is not in the database), Outdated_Source (the wrong version was retrieved), Reasoning_Failure (the context was right and the model reasoned badly) and Extraction_Failure (the fact was present but lost in the noise). The point is that these labels must emerge from your own data rather than be copied from a list.

Where should you start measuring?

Do not tune the search engine or swap models while you still have no measurement of input context quality, because every change you make is then a guess dressed up as engineering. Pick the 50 anchor questions that matter most to your product, write the gold context for them by hand, and treat that set as a unit test for your data.

Its value is that it converts a vague change into a readable signal: if you switch chunking strategies or swap the embedding model and recall on those anchor questions drops, the system just regressed — and you know it before your users do, which is the whole point of building the set.