AI evals are systematic measurements of quality across all components of an AI system, including models, prompts, retrieval pipelines, tools, and application code.
You build AI effectively by executing manual error analysis on real trace data, establishing domain-specific failure taxonomies, converting observed failures into binary pass/fail assertions, validating model judges against human expert labels, and integrating continuous checks into CI/CD pipelines and production monitoring.
Deploying AI applications without evals forces your team into reactive debugging loops. When you rely on informal testing, fixing one edge-case failure often introduces unseen regressions across prompts, model versions, or tool choices. Structured evaluation builds an objective feedback loop that replaces intuition with empirical software engineering practices.
Whether you are building conversational agents, automated coding tools, or retrieval-augmented generation (RAG) systems, implementing product-specific evaluations turns unpredictable text outputs into deterministic checks you can track in continuous integration. By continuously capturing, annotating, and testing against application traces, you ensure system improvements remain grounded in actual user requirements.

What are AI evals and why vibe checks fail
AI evals represent the systematic measurement of quality across every layer of your application stack. Rather than treating an AI system as a single black-box prompt, product-level evaluation systematically tests the interactions between your underlying models, prompt templates, retrieval mechanisms, tool execution pipelines, and glue code. Every evaluation subjects relevant execution examples to targeted logic, returning structured verdicts or binary scores that signal whether your product behavior aligns with operational goals.
To build effective evaluations, you must distinguish between public model benchmarks and product evals. Model benchmarks (such as GPQA Diamond for scientific reasoning, MMLU for general knowledge, or Terminal-Bench and SWE-bench for coding capability) measure general foundation model capabilities across standardized, public task sets. While help you select an initial base model, they cannot measure how your system handles internal business logic, custom tool APIs, or domain-specific user inputs. Product evals measure whether your complete software system solves your specific business problem by capturing actual failure modes encountered in your operational environment.
Relying on manual "vibe checks" inevitably fails when an AI product scales because large language models possess an infinite failure surface area.
Unstructured manual spot-checking suffers from criteria drift, where human evaluators shift their quality standards over time or become distracted by irrelevant formatting details.
Furthermore, manually tweaking a prompt to patch an isolated failure frequently introduces silent regressions across previously working edge cases.
Product evals address these limitations by applying the scientific method directly to software development: observing trace data, annotating failures, hypothesizing root causes, executing controlled experiments, and measuring outcomes against structured baselines.
def test_order_cancellation_execution(trace: dict) -> bool:
"""Asserts that the order cancellation tool executed correctly
and updated the backend state before notifying the user.
"""
tool_calls = trace.get("tool_calls", [])
system_state = trace.get("post_execution_state", {})
messages = trace.get("messages", [])
# Verify the cancel_order tool was invoked with a valid order_id
cancellation_calls = [
call for call in tool_calls if call.get("name") == "cancel_order"
]
if not cancellation_calls:
return False
target_order_id = cancellation_calls[0].get("args", {}).get("order_id")
if not target_order_id:
return False
# Assert backend state change succeeded prior to final assistant response
if system_state.get(f"order_{target_order_id}_status") != "CANCELLED":
return False
# Assert final message confirms cancellation to the user
final_response = messages[-1].get("content", "") if messages else ""
return "canceled" in final_response.lower() or "cancelled" in final_response.lower()The three levels of evaluation
Production evaluation architectures combine three distinct types of graders, balancing execution cost, speed, deterministic reproducibility, and subjective nuance. Code-based graders form the foundational layer of this architecture. These deterministic checks rely on string matches, regular expressions, structural JSON schema validation, linting, type-checking, and direct assertions on database state or API execution results. Code-based graders execute almost instantaneously at zero API cost, yielding perfectly reproducible binary outputs. They are ideal for evaluating objective criteria, such as verifying whether an agent invoked the correct tool parameters or outputted valid syntax.

Model-based graders—often implemented as LLM-as-a-Judge or specialized classifiers like Jev or BERT—address subjective evaluation criteria that code assertions cannot capture.
These evaluators grade properties such as tone, empathy, answer faithfulness, and instruction compliance. Model-based graders offer high flexibility and scale efficiently across complex natural language outputs. However, because can exhibit non-deterministic behavior, context bias, or self-enhancement bias, model-based graders cannot be trusted out of the box. They require prompt tuning and rigorous human calibration against labeled datasets to ensure their verdicts reliably mirror expert judgment.
Human graders provide the gold-standard reference for application quality. In most operational settings, human evaluation is best managed using a "benevolent dictator" model, where a single principal domain expert (such as a clinician for medical tools or a lawyer for legal assets) establishes definitive quality standards without getting bogged down in committee paralysis. While human review captures deep domain nuance and uncovers novel system failures, it remains expensive, slow, and unscalable for continuous real-time testing. Consequently, human graders are best deployed upstream to conduct qualitative error analysis and to periodically annotate calibration sets for validating automated model-based evaluators.
pass@k = 1 - (1 - p)^k [Probability of at least 1 success in k trials]
pass^k = p^k [Probability of all k trials succeeding]
When evaluating non-deterministic system trials over multiple runs ($k$), you must choose mathematical metrics that mirror your user experience requirements. The pass@k metric calculates the probability that an agent achieves at least one successful outcome across $k$ attempts ($1 - (1-p)^k$). As $k$ increases, pass@k rises, making it appropriate for offline code generation or ideation tools where presenting one valid solution out of several satisfies the user. Conversely, the pass^k metric measures the probability that all $k$ trials succeed ($p^k$). As $k$ increases, pass^k decreases exponentially, establishing a strict consistency bar essential for user-facing production agents where customers expect reliable, error-free execution on every interaction.
Error analysis: Looking at data before writing tests
Error analysis is the single highest-ROI activity in AI engineering, consuming 60% to 80% of total system development effort.
Building automated evaluators prematurely—a failure mode of naive "eval-driven development"—wastes engineering hours on synthetic metrics for imaginary failure modes that users never encounter. Instead of attempting to anticipate failures theoretically, high-performing teams systematically inspect execution traces (the complete record of system prompts, user inputs, retrieved contexts, tool calls, and final outputs) to let empirical application behavior dictate testing priorities.

A production-grade error analysis workflow follows a rigorous four-stage qualitative methodology:
- Dataset creation: Assemble an initial pool of roughly 100 diverse execution traces drawn from production logs or structured synthetic inputs.
- Open coding: A domain expert manually inspects at least 30 initial traces, recording open-ended qualitative notes on every user-facing defect while explicitly isolating the first upstream failure (such as a failed tool call or bad retrieval step) that caused downstream errors. Keep all relevant trace context on a single screen to preserve reviewer focus.
- Axial coding: Group raw qualitative notes into distinct, mutually exclusive categories to construct a structured "failure taxonomy" that quantifies the relative frequency of each defect type.
- Iterative refinement: Use human-agent active learning loops where an AI assistant clusters remaining traces and flags likely failure instances for expert verification, continuing until reaching theoretical saturation (the point at which analyzing new traces yields no new failure categories).
[100 Production Traces]
│
▼
[Stage 1: Open Coding (Manual Review of 30+ Traces)] ──► Annotate First Upstream Failure
│
▼
[Stage 2: Axial Coding (Group Notes into Taxonomy)] ──► Quantify Failure Frequencies
│
▼
[Stage 3: Iterative Refinement (Human-Agent Loop)] ──► Stop at Theoretical Saturation
During error analysis, you must enforce forced binary (Pass/Fail) labels rather than multi-point Likert scales (e.g., ratings from 1 to 5).
Likert scales introduce subjective ambiguity, cause criteria drift across annotators, require significantly larger sample sizes to establish statistical significance, and encourage reviewers to hide uncertainty by defaulting to middle scores. Forced binary grading forces clear definitions of acceptable quality, drastically increases labeling velocity, and yields stable datasets required to train and validate automated evaluators.
Beware of relying on off-the-shelf vendor metrics (such as generic "helpfulness," "coherence," or "quality" scores) as primary system performance indicators. These ready-to-use metrics measure abstract qualities detached from your operational rules and business constraints, creating false confidence when scores appear high. Generic vendor metrics should only ever be used as exploratory sampling signals to flag anomalous or low-scoring traces for manual human inspection—never as valid quality benchmarks for application success.
Three steps to building production-grade product evals
Translating manual error analysis into continuous automated evaluations requires a disciplined three-step execution pipeline that bridges human judgment and software infrastructure:

Step 1: Label Dataset ──► Step 2: Validate Evaluator ──► Step 3: Integrate CI/CD & Monitoring
(100-200 Labeled Traces) (Measure TPR / TNR Metrics) (Regression Tests & Production Traces)
Step 1: Label Sample Data
Assemble a curated evaluation dataset containing 100 to 200 execution traces split evenly between Pass and Fail outcomes across the failure categories identified during axial coding. Partition this data cleanly into three isolated subsets: a training set (10–20%) used to construct system prompts or few-shot examples; a development set (40–45%) used to iterate on evaluator prompts and tune heuristics; and an untouched held-out test set (40–45%) reserved strictly for final validation. Note that this specific 10–20% train split applies to zero-shot or few-shot LLM judges and lightweight classifiers like Jev; traditional supervised ML classifiers require a significantly larger proportion of to achieve convergence.
Step 2: Align & Validate LLM Evaluators
Develop targeted, single-purpose model-based evaluators configured as binary classifiers rather than multi-criteria judges. Validate your evaluator's predictions against human labels on the held-out test set by calculating True Positive Rate (TPR / Recall, measuring the proportion of actual system errors correctly flagged) and True Negative Rate (TNR, measuring the proportion of good outputs correctly passed). You must track both metrics to ensure the judge does not overfit to development data or generate excessive false alarms that waste engineering review time.
Step 3: Run Evals Continuously
Embed your validated evaluators directly into software deployment workflows. In CI/CD pipelines, run small, purpose-built test suites containing deterministic code assertions and regression examples to gate releases against known past bugs. In production monitoring, asynchronously sample live traffic traces to run validated LLM judges, tracking confidence intervals over time to detect capability drift and surface novel failure modes for future error analysis.
# Configuration schema for an automated coding agent evaluation task
task:
id: "fix-auth-bypass-001"
description: "Verify agent resolves null password authorization bypass vulnerability"
graders:
# Deterministic checks verify functional correctness at zero API cost
- type: "deterministic_tests"
required_files:
- "tests/test_auth_null_password.py"
- "tests/test_auth_standard_login.py"
# Static analysis enforces code safety and type constraints
- type: "static_analysis"
commands:
- "ruff check src/auth/"
- "mypy src/auth/"
# Model rubric evaluates subjective code quality and security style
- type: "llm_rubric"
model: "gpt-4o"
prompt_template: "prompts/rubrics/security_code_quality.md"
expected_verdict: "PASS"
# Tool call assertions ensure mandatory security verification steps occurred
- type: "tool_calls"
required_sequence:
- tool: "read_file"
params: { path: "src/auth/login.py" }
- tool: "edit_file"
- tool: "run_tests"
# Tracked metrics capture operational efficiency and token consumption alongside correctness
tracked_metrics:
transcript:
- "n_turns"
- "n_tool_calls"
- "n_total_tokens"
latency:
- "time_to_first_token"
- "total_execution_seconds"Evaluating RAG systems: The 3 variables and 6 core evals
Claims that " is dead" stem from a fundamental misunderstanding of retrieval-augmented generation architectures. While naive vector-only similarity search frequently fails when processing complex, highly structured, or domain-specific data, the core architecture of Retrieval-Augmented Generation remains essential. Modern production systems address complex information access by shifting away from simple vector lookup toward hybrid strategies combining keyword search, dense , and agentic, multi-hop retrieval pipelines. You can also explore specialized techniques for context evaluation to diagnose pipeline bottlenecks before feeding documents to models.

To evaluate RAG architectures effectively, you must decouple the system into two distinct performance layers:
- Retrieval evaluation treats retrieval purely as a search problem. It measures whether the index retrieves relevant context passages using standard Information Retrieval (IR) metrics, including Recall@k (fraction of relevant documents captured in top $k$), Precision@k (fraction of retrieved top $k$ documents that are relevant), and Mean Reciprocal Rank (MRR, position of the first relevant result).
- Generation evaluation measures how effectively the language model uses retrieved context to synthesize the final output based on the core variable triad: Question ($Q$), Context ($C$), and Answer ($A$).
Question (Q)
/ \
(C|Q) (A|Q)
/ \
Context (C) ─────── Answer (A)
(A|C)
The relationship between these three variables establishes the core triad of generation evaluation:
- Context Relevance ($C|Q$): Evaluates whether the retrieved context passages contain information directly pertinent to resolving the query, filtering out noisy or irrelevant text.
- Answer Faithfulness ($A|C$): Assesses whether every factual claim in the generated answer is strictly grounded in and supported by the retrieved context, flagging .
- Answer Relevance ($A|Q$): Measures whether the final response directly addresses the user's explicit question without omitting required details or adding unnecessary filler.
When processing large documents, chunk sizing must be tailored to the target task. For fixed-output tasks (such as extracting specific values or contract penalty terms), use larger chunks to provide maximum context in a single pass, avoiding context fragmentation while ensuring irrelevant text is pre-filtered. For expansive-output tasks (such as section-by-section summarization or exhaustive data extraction), use smaller, boundary-aligned chunks. Large text blocks introduce attention bottlenecks where models disproportionately attend to the beginning and end of long inputs while ignoring details in the middle; smaller chunks bound cognitive load, preserving local attention and ensuring output completeness.
Evaluating AI agents: Moving beyond single-turn tests
Evaluating multi-turn AI agents requires moving beyond single-turn prompt-response testing. Agents operate autonomously across multi-step execution loops: analyzing user requests, invoking internal tools, inspecting environmental feedback, and updating local state over extended trajectories. Because mistakes made early in an agentic loop compound exponentially over subsequent turns, evaluating an agent based solely on its final conversational output masks internal systemic failures and rapidly inflates operational token costs.

+-----------------------------------------------------------------------+
| Phase 1: End-to-End Success |
| Assert: Final Environment State matches expected Business Goal |
+-----------------------------------------------------------------------+
│
▼
+-----------------------------------------------------------------------+
| Phase 2: Step-Level Diagnostics |
| Inspect Trajectory: Tool Choice | Parameters | Recovery | Efficiency |
+-----------------------------------------------------------------------+
Agent evaluation requires a two-phase diagnostic framework:
- Phase 1: End-to-End Task Success. Treats the agent as a black box and measures whether the final environment state fulfills the intended operational goal. For example, in an e-commerce cancellation task, end-to-end success asserts that the backend database record reflects a "CANCELLED" status, independent of what the agent claimed in chat.
- Phase 2: Step-Level Diagnostics. Inspects the agent execution trace to evaluate intermediate steps. Diagnostics grade specific operational capabilities: tool choice accuracy, parameter extraction validity, API error recovery, prompt context retention over long trajectories, and step efficiency (total turns, token costs, and wall-clock latency).
To isolate failure hotspots across complex workflows, construct a Transition Failure Matrix. Map the agent's last verified successful execution state along the matrix rows and the state where the first failure occurred along the columns.
| Last Successful State \ Failure State | Plan Generation | Query Parsing | DB Execution | Result Formatting | Total Failures |
|---|---|---|---|---|---|
| User Request Received | 3 | 1 | 0 | 0 | 4 |
| Plan Generation | — | 14 | 2 | 0 | 16 |
| Query Parsing | — | — | 19 | 1 | 20 |
| DB Execution | — | — | — | 2 | 2 |
This matrix exposes where execution chains break. In the text-to-SQL agent workflow illustrated above, 19 failures occur during the transition from Query Parsing to DB Execution, identifying SQL syntax generation as the primary system bottleneck requiring immediate prompt engineering or tool schema refinement.
The boundary between evals and guardrails
Engineering scalable AI systems requires maintaining a strict operational distinction between inline guardrails and asynchronous evaluators. Conflating these two capabilities degrades application response times, increases operational API expenses, and introduces user-facing bugs.

Incoming Request ──► [Guardrail: Fast, Sync, Inline Schema/Regex] ──► LLM Output ──► User
│
▼ (Async Log)
[Evaluator: Heavy LLM Judge / Batch]
-
Guardrails operate as synchronous, inline safety assertions embedded directly within the real-time request and response path.
Guardrails run with strict latency budgets (typically executing in milliseconds) using fast, deterministic, high-precision mechanisms such as regex filters, JSON schema validators, type checkers, and lightweight heuristic classifiers.Their purpose is to intercept, redact, or block severe, unambiguous production risks—such as PII leaks, malformed syntax, SQL injections, or severe policy violations—before data reaches the end user.
-
Evaluators operate as asynchronous or offline processes that analyze trace data after the application has returned a response to the user. Evaluators execute heavy, compute-intensive analyses—such as multi-criteria LLM-as-a-Judge evaluations, groundedness assertions, and process efficiency metrics. Evaluators feed regression test suites, CI/CD pipelines, long-term monitoring dashboards, and model fine-tuning loops without impacting user-facing execution latency.
Deploying model-based LLM judges synchronously as inline guardrails introduces significant trade-offs. Using an LLM to check another LLM doubles or triples user latency, significantly increases token cost accumulation, and introduces non-deterministic false-positive risks where valid responses are incorrectly blocked. Synchronous guardrails must remain fast, simple, and conservative, while subjective quality assessment belongs strictly within asynchronous evaluation pipelines.
Grounded takeaway: Turning evals into a durable moat
Foundation models, orchestration framework wrappers, and third-party observability vendors are rapidly commoditizing. Relying solely on external tooling or generic, off-the-shelf vendor metrics like "helpfulness" or "coherence" provides no lasting competitive differentiation and creates a false sense of security. A durable product moat is built by establishing an internal engineering practice grounded in continuous manual error analysis, domain-specific failure taxonomies, and custom-built evaluation suites validated directly against domain expert judgment.
Prioritize direct observation of trace data over superficial dashboard metrics. Have your team inspect production logs, maintain high-quality labeled datasets, and continuously iterate on purpose-built test cases. Systematically translating real-world failures into deterministic code assertions and calibrated model judges creates an engineering feedback loop that continuously drives down system defect rates and secures your product's operational advantage.
References
- AI Evals: Everything You Need to Know — Hamel Husain
- Your AI Product Needs Evals — Hamel Husain
- Task-Specific LLM Evals that Do & Don't Work — Eugene Yan
- Demystifying Evals for AI Agents — Anthropic
- An LLM-as-Judge Won't Save The Product—Fixing Your Process Will — Eugene Yan
- There Are Only 6 RAG Evals — Jason Liu
- Evaluation Best Practices — OpenAI