# What Is Context Evaluation? Measuring What You Feed an LLM

> Context evaluation measures retrieved context quality in an LLM pipeline, separating retriever from generator failures via precision, recall and error analysis.

Source: https://camnangai.com/en/engineering/what-is-context-evaluation  
Published: 2026-08-27

**Context evaluation** is the process of measuring the accuracy, relevance and completeness of the
  information retrieved and handed to a large language model (LLM) before it generates anything.

Instead of judging only the final answer, you measure the quality of the raw material going in — so
the model has the facts it needs rather than a plausible gap to fill.

In practice it is noise control for a
[RAG (Retrieval-Augmented Generation)](https://camnangai.com/en/knowledge/what-is-rag) system. When the context is wrong,
an LLM fills the gap with hallucination or with stale internal knowledge, and it does that in exactly
the same confident register it uses when it is right, which is why you cannot catch the problem by
reading answers alone. Measuring the context rigorously is what tells you whether your pipeline broke
at the retrieval step or at the reasoning step.


  Context evaluation is not a pass/fail test — it asks whether every token you spend on the prompt
  earned the space it took.


![Diagram of measuring the quality of retrieved context before it is fed into a large language model](https://camnangai.com/articles/context-evaluation-la-gi/01-infographic-en.webp)

## What is context evaluation?

[Context engineering](https://camnangai.com/en/for/developers/context-engineering) is an optimization problem against a
finite attention budget. You are not trying to load a million tokens; you are trying to find the
smallest possible set of high-signal tokens that maximize the likelihood of the outcome you want.
Every additional token depletes the budget the model has to reason with.

Traditional benchmarks like Needle In A Haystack (NIAH) have reached their limits here, because NIAH
measures one narrow capability: lexical retrieval, matching a string you already know is present.
Real systems need semantic understanding to connect information scattered across sources, and that is
where the benchmark stops being informative. A model finds the needle easily when the keyword matches
exactly, then falls over when the question requires reasoning across equivalent concepts.

A stricter standard is the **gold context**, which is the minimum set of artifacts genuinely
sufficient to resolve a specific request rather than a sample document set. Building one requires
human experts to annotate by hand, followed by a verification pass in which a strong model must be
able to solve the task when given only that context and nothing else. That is slow, expensive work,
and I have yet to see a shortcut that survives contact with a real product, because the value of the
set comes entirely from a human having decided what "sufficient" means here.

That is also why you separate what the system _saw_ from what it _used_. Plenty of pipelines post
high recall and still fail, because the context was noisy enough that the model never extracted the
part that mattered, and a dashboard full of green retrieval numbers will happily hide that for
months. Context evaluation exists to measure that gap.

## Why a complete-looking context pipeline still feeds the model poor material

RAG failures usually start one layer below the retriever, in the data itself. Even when precision
and recall score near-perfectly, the answer is still wrong if the underlying documents are stale or
mutually inconsistent, and no amount of tuning above that layer will fix it. That is how a pipeline
that looks complete keeps serving the model poor material: every component reports success while the
index is quietly out of date.

![The four data-quality dimensions governing documents before they enter the vector database: freshness, ownership, lineage and canonical alignment](https://camnangai.com/articles/context-evaluation-la-gi/02-en.webp)

Governing that raw layer before it reaches the vector database comes down to four dimensions.
**Freshness** asks whether the content reflects the current state of the system. **Ownership**
establishes who is accountable for a document being correct. **Lineage** tracks how data traveled
from its source into the vector database and whether it was distorted along the way. And **canonical
alignment** checks that a document matches how the organization officially defines things, because
when each team uses its own vocabulary, the embedding model has no reason to place those documents
near each other.

Duplicate and superseded documents are the sharpest version of this problem, and the easiest one to
create by accident. Leave the 2022 and the 2024 edition of the same handbook in the index and the
retriever will eventually surface both, so the model ends up holding two contradictory facts with no
basis for choosing between them. Weak data governance permits errors, and worse than that, it
automates their production at scale.

## Retrieval failures and generation failures: where to measure each

To fix a broken RAG system you first have to know whether it failed at finding or at writing. So
split the pipeline in two, retriever and generator, and give each half its own metrics and its own
knobs, because a single end-to-end score cannot tell you which half moved.

![A RAG pipeline split into retriever and generator halves, each with its own group of metrics](https://camnangai.com/articles/context-evaluation-la-gi/03-en.webp)

The retriever's metrics point directly at system parameters. **Contextual recall** measures whether
the retrieved context contains all the facts needed for the ideal answer, and it is effectively a
measure of your embedding model. **Contextual precision** measures whether relevant chunks are
ranked above irrelevant ones, which makes it a measure of your reranker. **Contextual relevancy**
measures how much of the retrieved context is pertinent at all, and it reflects how you configured
chunk size and top-K.

The generator's metrics judge what the model did with what it received. **Faithfulness** measures
whether the answer stays inside the retrieved context or invents beyond it. **Answer relevancy**
measures whether the response actually addresses the question, and it reads directly on the quality
of your prompt template.

The isolation rule falls out cleanly: bad context means you fix the retriever — chunk size, top-K,
embedding model, reranker. Correct context with a bad answer means you fix the generator instead,
which is the prompt template and the model. Treat overall quality as a product rather than a sum,
because if either half fails outright, output quality goes to zero regardless of how good the other
half is.

## Context precision and context recall: right ranking or full coverage?

These two metrics sound alike and measure opposite things. **Context precision** measures whether
relevant chunks sit at the top of the ranking, and it is the mean of precision@k (the share of the
top k chunks that are relevant) across positions, weighted by a relevance indicator, so burying the
critical chunk at position ten tanks the score even though the information was retrieved.
**Context recall** measures coverage: the reference answer is broken into individual claims, and
each claim is checked for whether the retrieved context supports it.

![Comparison of context precision measuring correct ranking against context recall measuring full coverage](https://camnangai.com/articles/context-evaluation-la-gi/04-en.webp)

The distinction determines how you read a bad score. Low recall means the information was never
retrieved at all, and no reranker on earth will recover it. Low precision means the information is
present but buried under noise — a ranking problem, not a search problem. Those are two completely
different repairs, which is why collapsing both into one "retrieval quality" number costs you the
diagnosis.

Both metrics also have non-LLM variants: a string-similarity version that compares against
`reference_contexts`, and an ID-based version that compares `retrieved_context_ids` against known
relevant IDs. The ID variant is cheap and deterministic, which makes it the practical choice for a
large system that needs to score continuously without paying for an LLM call per sample, and if you
are scoring on every deploy it is the one I would reach for first.

```python
from ragas.metrics.collections import ContextRecall

scorer = ContextRecall(llm=evaluator_llm)

result = await scorer.ascore(
    user_input="Where is the Eiffel Tower?",
    retrieved_contexts=["Paris is the capital of France."],
    reference="The Eiffel Tower is located in Paris.",
)
```

There is always a trade-off here, and it is not one you get to escape by tuning harder. Retrieving
generously raises recall and protects you from missing a fact, but if precision drops far enough the
signal thins out and the model starts losing its way among things that were never relevant.

## Automated scoring with LLM-as-judge, and where not to trust it

The RAG triad is the fastest framework to put automated scoring in place, and it is three separate
checks rather than one score. **Context relevance** asks whether the retrieved chunks are pertinent
to the query. **Groundedness** decomposes the response into individual claims and searches the
retrieved context for evidence supporting each one. **Answer relevance** asks whether the final
response actually serves what the user asked. Each runs as a feedback function, meaning a carefully
prompted LLM acting as judge, which is what lets you score at scale when you have little or no
ground-truth data.

![The three separate checks of the RAG triad: context relevance, groundedness and answer relevance](https://camnangai.com/articles/context-evaluation-la-gi/05-en.webp)

That mechanism is also the weakness, and it is worth being clear about how big that problem is. The
judge is a language model, so it carries the same biases it is measuring: a fluent, confident,
well-formatted answer scores better than a correct but blunt one. The number it returns is a signal
rather than a verdict.

The healthy way to use it is as a filter rather than a court. Periodically compare machine scores
against human labels on a small sample, and re-run that comparison every time you change the judge
model or edit the scoring prompt, because those are the two changes that silently shift your entire
scale without raising an error anywhere, and nothing in your pipeline will warn you when it happens.

## Distractors, stale documents and context rot: when more context hurts

Context rot testing across 18 leading models — GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 among them —
found that performance degrades unevenly as input length grows, even on tasks as simple as retrieval
and text replication. A model does not handle its 10,000th token as reliably as its 100th, which
quietly undercuts the way most teams talk about long context windows.

![Model reliability falling as the context grows longer while distracting passages crowd in](https://camnangai.com/articles/context-evaluation-la-gi/06-en.webp)

Distractors are the main culprit, meaning passages topically related to the question that do not
answer it. A single distractor is enough to reduce accuracy, and additional ones compound the loss.
Semantic distance matters too: the lower the similarity between the question and the needle, the
faster performance falls as context grows.

One finding still bothers me every time I reread it. Across all 18 models, performance was _better_
on shuffled haystacks than on logically structured ones, which is the opposite of what anyone
building a retrieval pipeline assumes. A coherent narrative creates semantic competition, because the
surrounding ideas are similar enough to camouflage the one you actually need. The engineering
conclusion is blunt — filter noise before you widen the window. Enlarging the
[context window](https://camnangai.com/en/knowledge/context-window) does not remove the evidence-selection problem.

## Why you still have to read the outputs yourself: error analysis and a failure taxonomy

Automated metrics are the surface. You cannot improve a system without reading real traces, meaning
the full record of a run, covering both what the system retrieved and what it produced. It is tedious
and it does not scale, and it is still the single most valuable activity in evaluation, because it
decides which evals are worth writing in the first place instead of leaving you to borrow a generic
metric suite.

![The manual error-review process: reading traces, open coding, axial coding, then building a failure taxonomy](https://camnangai.com/articles/context-evaluation-la-gi/07-en.webp)

The process has three moves. First, read traces and do **open coding**: write open-ended notes in
your own words about anything wrong, concentrating on the _first_ failure in each trace, since
later errors are usually downstream consequences. Then do **axial coding**: group similar notes into
categories to build a failure taxonomy specific to your project, and count how often each category
occurs. That grouping step is the most important one in the whole exercise.

You stop at saturation, which is when new traces stop producing new categories. As a rule of thumb,
review at least 100 traces to start, and if roughly 20 in a row surface nothing new, you can stop. A
working taxonomy tends to hold labels like `Missing_Information` (the data is not in the database),
`Outdated_Source` (the wrong version was retrieved), `Reasoning_Failure` (the context was right and
the model reasoned badly) and `Extraction_Failure` (the fact was present but lost in the noise). The
point is that these labels must emerge from your own data rather than be copied from a list.

## Where should you start measuring?

Do not tune the search engine or swap models while you still have no measurement of input context
quality, because every change you make is then a guess dressed up as engineering. Pick the 50 anchor
questions that matter most to your product, write the gold context for them by hand, and treat that
set as a unit test for your data.

Its value is that it converts a vague change into a readable signal: if you switch chunking
strategies or swap the embedding model and recall on those anchor questions drops, the system just
regressed — and you know it before your users do, which is the whole point of building the set.

## References

- [Effective Context Engineering for AI Agents — Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
- [Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma](https://www.trychroma.com/research/context-rot)
- [Context Precision — Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)
- [Context Recall — Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)
- [The RAG Triad — TruLens](https://www.trulens.org/getting_started/core_concepts/rag_triad/)
- [RAG Evaluation Metrics: Answer Relevancy, Faithfulness, Contextual Relevancy and More — Confident AI](https://www.confident-ai.com/blog/rag-evaluation-metrics-answer-relevancy-faithfulness-and-more)
- [Why Is Error Analysis So Important in LLM Evals? — Hamel Husain](https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html)
- [How to Evaluate RAG Systems — Atlan](https://atlan.com/know/how-to-evaluate-rag-systems-explained/)
- [ContextBench: A Benchmark for Context Retrieval in Coding Agents — arXiv](https://arxiv.org/abs/2602.05892)