# LLM Observability and Evaluation: The Complete Engineering Guide

> Master LLM observability and evaluation across production environments with tracing architectures, dual-speed evals, and custom failure taxonomies.

Source: https://camnangai.com/en/engineering/llm-observability-and-evaluation  
Published: 2026-10-31

![Comprehensive LLM Observability and Evaluation Architecture](https://camnangai.com/articles/llm-observability-va-evaluation-la-gi/01-infographic-en.webp)

Production **LLM observability and evaluation** entails the systematic instrumentation, execution tracing, and multi-tier testing of non-deterministic Large Language Model applications across both offline CI/CD development pipelines and online production runtimes. Standard software observability frameworks assume deterministic state transitions, where a given input leads through a static execution path to a predictable output. Generative AI systems break this model by introducing probabilistic reasoning, multi-step tool orchestration, and unconstrained natural language outputs that require novel tracing architectures and evaluation workflows.

Unlike classic microservices where failure modes manifest as unhandled exceptions or network timeouts, LLM application failures are predominantly semantic. A system can execute without throwing a single runtime error while delivering factually incorrect data, hallucinating API arguments, or breaching business policy. To maintain system reliability, you must continuously log execution graphs—capturing raw prompt templates, model hyperparameters, retrieved context, tool calls, and complete text outputs—while running structured evaluation pipelines composed of deterministic code assertions, validated LLM judges, and human domain-expert rubrics.


  Implementing rigorous LLM observability and evaluation ensures that every model update, prompt
  refinement, or retrieval parameter tweak is verified against concrete failure taxonomies before
  deployment, replacing speculative prompt engineering with empirical data.


## The Fundamental Shift: Why Traditional APM Fails for LLMs

![Traditional APM vs. LLM Observability](https://camnangai.com/articles/llm-observability-va-evaluation-la-gi/02-en.webp)

Traditional Application Performance Monitoring (APM) platforms were architected around deterministic microservice architectures. In standard web applications, observability relies on tracking HTTP response status codes, CPU and memory saturation metrics, database query durations, and static call stack traces. When a classical microservice fails, it typically surfaces an explicit failure code—(such as an HTTP 500 error, a NULL pointer exception, or a database connection timeout). The root cause is directly tied to a specific line of code or infrastructure constraint that can be isolated using static call graphs.

In Generative AI systems, standard APM tools fail because a successful HTTP 200 OK status code from an LLM API endpoint frequently conceals total application failure. Standard network monitors view an LLM call as a simple black-box HTTP POST request; they cannot evaluate whether the generated response contains catastrophic factual hallucinations, subtle prompt injection vulnerability exploits, invalid tool parameter extractions, severe policy breaches, context rot, or inappropriate tone mismatches. Because the underlying model operates probabilistically, syntactically valid outputs can be functionally destructive to the business without generating a single system-level error flag.

Consequently, the fundamental unit of work in observability shifts from monitoring isolated microservice HTTP requests to tracking probabilistic execution chains. Debugging an AI application requires full visibility into the execution graph: the precise prompt context, system instructions, model hyperparameters (such as temperature and top-p), token consumption metrics, intermediate reasoning steps, retrieved document chunks, and multi-step tool call payloads. Traditional APM tools lack the data models necessary to capture these hierarchical, multi-turn relationships or to evaluate semantic correctness across non-deterministic state transitions.

On top of that, operational performance metrics diverge radically from classical software benchmarks. Traditional web APIs operate against sub-millisecond API response latency targets, where response latency spikes trigger immediate infrastructure alerts. In contrast, LLM applications operate across multi-second or multi-minute execution windows. Engineering teams must measure and optimize novel performance vectors, including Time-To-First-Token (TTFT), output token generation velocity (tokens/second), context window usage, and per-request token consumption costs. Managing these trade-offs requires dedicated telemetry protocols optimized for high-latency, probabilistic execution graphs.

## Instrumentation Architecture: OpenTelemetry and GenAI Semantic Conventions

Instrumenting production AI applications requires a telemetry pipeline capable of capturing deep execution context without introducing user-facing latency. To achieve this, instrumentation libraries use asynchronous, non-blocking telemetry pipelines. OpenTelemetry (OTel) collectors and vendor SDKs queue trace events in local memory buffers, executing background batch flushing over HTTP/2 or gRPC transport layers. This decoupled architecture guarantees that telemetry collection adds zero overhead to the critical path of the user request, preserving the application's responsiveness even under heavy logging loads.

Under OpenTelemetry GenAI semantic conventions, AI application traces are structured around standardized span attributes that capture the full execution environment. Every LLM invocation span must record core metadata, including the target model name (`gen_ai.request.model`), vendor provider (`gen_ai.system`), hyperparameter settings (`gen_ai.request.temperature`, `gen_ai.request.max_tokens`), full input prompt messages, and generated completion responses. At the same time, spans must track granular token usage metrics—separating input/prompt tokens (`gen_ai.usage.input_tokens`) from output/completion tokens (`gen_ai.usage.output_tokens`)—to enable precise per-trace financial cost accounting and latency profiling (such as TTFT and output generation speed).

The structured schema below demonstrates an OpenTelemetry-compliant JSON payload for a GenAI invocation span, capturing standard execution metadata, usage metrics, and parameters:

```json
{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "5c37492a3e0f081d",
  "name": "gen_ai.chat.completions",
  "kind": "SPAN_KIND_CLIENT",
  "start_time_unix_nano": 1771234567000000000,
  "end_time_unix_nano": 1771234569500000000,
  "attributes": {
    "gen_ai.system": "openai",
    "gen_ai.request.model": "gpt-4o",
    "gen_ai.request.temperature": 0.2,
    "gen_ai.request.max_tokens": 1000,
    "gen_ai.usage.input_tokens": 452,
    "gen_ai.usage.output_tokens": 128,
    "gen_ai.usage.total_tokens": 580,
    "gen_ai.client.token_cost": 0.00348,
    "gen_ai.latency.time_to_first_token_ms": 320,
    "gen_ai.latency.tokens_per_second": 56.4,
    "gen_ai.prompt.0.role": "system",
    "gen_ai.prompt.0.content": "You are a customer support agent.",
    "gen_ai.prompt.1.role": "user",
    "gen_ai.prompt.1.content": "Cancel order #94812",
    "gen_ai.completion.0.role": "assistant",
    "gen_ai.completion.0.content": "I have processed the cancellation for order #94812."
  },
  "status": {
    "code": "STATUS_CODE_OK"
  }
}
```

Capturing these standardized spans across every service boundary provides the structural foundation for deep execution graph analysis. By maintaining consistency with open telemetry standards, your platform can route execution metadata smoothly into downstream evaluation, alerting, and cost-attribution pipelines without vendor lock-in.

## Tracing Complex AI Systems: Traces, Spans, and Execution Graphs

![Application Tracing and Execution Graph Architecture](https://camnangai.com/articles/llm-observability-va-evaluation-la-gi/03-en.webp)

Understanding complex AI architectures requires a formal domain model for execution hierarchies. A **trace** represents the complete end-to-end user session across all application boundaries, identified by a unified `trace_id`. Within a trace, a **span** represents an individual, contiguous unit of work, such as a single LLM API call, a vector database semantic search, or an external tool invocation. Spans maintain parent-child relationships, assembling a directed acyclic graph (DAG) that accurately models the execution path of multi-agent and Retrieval-Augmented Generation (RAG) systems.

```
[Trace: User Query "Cancel Order #94812"]
 ├── [Span 1: Agent Controller - Route Request]
 ├── [Span 2: Vector DB Retrieval - Query Policy] (32ms)
 ├── [Span 3: LLM Call - Determine Tool & Parameters] (850ms)
 ├── [Span 4: Tool Execution - cancel_order(id=94812)] (120ms)
 └── [Span 5: LLM Call - Format User Response] (410ms)
```

In multi-turn, multi-agent systems, tracing topologies must record complex state transitions across temporal boundaries. Every session trace records state modifications, environment feedback loops, sub-agent handoffs, and multi-hop retrieval queries. When an autonomous agent attempts a complex task, it may execute tens of intermediate turns—querying databases, parsing JSON payloads, handling API exceptions, and refining its internal reasoning path. Capturing every interaction within structured trace hierarchies enables platform engineers to audit the exact environmental conditions and agent history that led to a specific outcome.

When debugging multi-turn conversation or agent step failures, you must follow a strict diagnostic protocol: identify, isolate, and annotate the **first upstream failure** in the trace before analyzing downstream errors. Because LLM pipelines are sequential execution chains, an error on turn 1, such as retrieving an irrelevant context chunk or misparsing a tool parameter, cascades into compound failures on subsequent turns. Annotating or debugging downstream symptoms (such as an improper final response tone on turn 4) wastes engineering effort, as the root cause almost always resides in the initial upstream state divergence.

To make complex multi-turn debugging tractable, adopt trace simplification protocols. When a multi-turn agent fails on turn N, reproduce the bug by extracting the exact conversation prefix up to turn N-1 from the production trace log. Construct a single-turn unit test using this isolated state prefix (an "N-1 test") to evaluate whether the agent can successfully generate the correct step N response given the isolated upstream context. Simplifying multi-turn trajectories into reproducible, single-turn integration tests eliminates non-deterministic execution noise and accelerates root-cause resolution.

## The Two Speeds of LLM Evaluation: Offline CI/CD vs. Online Production Monitoring

A mature AI engineering architecture operates evaluation loops across two distinct speeds: offline pre-deployment testing in CI/CD pipelines, and asynchronous, online monitoring in production.

```
+-----------------------------------------------------------------------+
| OFFLINE EVALUATION (CI/CD) |
| |
| [Golden Dataset] ---> [App/Agent] ---> [Evaluators] ---> Pass/Fail |
| (100+ Examples) (Code/Judges) (Gatekeeper)|
+-----------------------------------------------------------------------+
 ^
 | Feed Failure Traces
 v
+-----------------------------------------------------------------------+
| ONLINE PRODUCTION MONITORING |
| |
| [User Input] ---> [Live App] ---> [Production Traces] ---> Async Eval|
| (Drift/CI)
+-----------------------------------------------------------------------+
```

### Offline CI/CD Evals

Offline evaluation acts as the primary release gatekeeper during application development. Prior to merging code or updating prompts, the application is executed against a curated "golden dataset"—a purpose-built test suite containing 100 or more representative examples covering core functionality, edge cases, and historical production failures. In automated CI pipelines (such as GitHub Actions via integrations like `langfuse/experiment-action`), these evaluation runs execute assertions against application outputs. If a proposed change causes metrics to drop below predefined quality thresholds, the pipeline raises a `RegressionError` and blocks deployment.

Evaluating non-deterministic agent workflows in CI requires statistical metrics that account for run-to-run variance across multiple trial attempts (k). Two key metrics define performance expectations:

- **`pass@k`**: Measures the probability that an agent generates at least one successful solution across $k$ attempts. This metric is ideal for optimization workflows (such as code generation or creative synthesis) where generating multiple candidate outputs is acceptable as long as one satisfies criteria.
- **`pass^k`**: Measures the probability that an agent succeeds across _all_ $k$ trials concurrently. This metric enforces strict behavioral consistency, making it essential for customer-facing production applications (such as automated support agents) where failure on a single attempt causes user dissatisfaction. Because `pass^k` requires success across every trial, performance degrades exponentially as $k$ increases. For instance, an agent with a $75\%$ per-trial success rate achieves a `pass^3` of `(0.75)^3 ≈ 42%`, demonstrating how even moderate per-trial unreliability compounds into severe multi-trial operational risk.

### Online Production Monitoring

Online evaluation operates asynchronously over live production traffic. Because production inputs lack reference ground-truth answers, online evaluators rely on reference-free grading models, code-based guardrails, and user feedback signals (e.g., explicit thumbs-up/down or implicit retry behaviors). Production monitoring platforms continuously sample live traces, track moving confidence intervals, and alert engineers to statistical score drift. If an online quality score drops significantly below baseline targets, alerts trigger immediate human review and investigation.

The offline and online speeds form a closed-loop engineering framework. When production monitoring flags anomalous traces or new semantic failure patterns through online evaluators, engineers extract these live traces, redact sensitive data, assign expert labels, and merge them directly into the offline CI/CD golden regression dataset. This continuously expands test coverage based on real-world system usage.

## Evaluation Methodologies: Deterministic Checks, LLM-as-a-Judge, and Human Rubrics

Designing a reliable evaluation suite requires combining three primary grading methodologies based on speed, cost, and necessary semantic nuance.

```
+-------------------------------------------------------------------+
| EVALUATION METHODOLOGIES |
+-------------------------------------------------------------------+
| 1. Deterministic Checks (Code-Based) |
| - String matching, JSON schema, regex, unit/execution tests |
| - Fast, cheap, objective, reproducible |
+-------------------------------------------------------------------+
| 2. Model-Based Graders (LLM-as-a-Judge) |
| - Binary Pass/Fail classification on subjective criteria |
| - Validated against human labels (TPR / TNR tracking) |
+-------------------------------------------------------------------+
| 3. Human Domain Expert Rubrics |
| - "Benevolent Dictator" model setting canonical ground truth |
| - Used for error discovery, calibration, and edge cases |
+-------------------------------------------------------------------+
```

### Deterministic Code-Based Graders

Deterministic graders execute programmatic assertions directly against task outcomes or trace step outputs. Common implementations include string matching, regular expression filtering, JSON schema validation, static analysis tools (e.g., linters, type checkers), and unit test execution environments (e.g., verifying that generated code passes `pytest`). Code-based checks are extremely fast, virtually free to compute, 100% reproducible, and objective. Platform engineers should use deterministic checks as the first line of defense for all structural, safety, and syntactic criteria.

### Model-Based Graders (LLM-as-a-Judge)

When evaluation criteria require subjective judgment, semantic understanding, or domain knowledge, such as evaluating answer completeness, tone appropriateness, or policy adherence, use model-based graders. When configuring LLM judges, implement binary (Pass/Fail) classification tasks rather than multi-point Likert scales (e.g., 1 to 5 ratings). Likert scales introduce severe subjective variance between runs, suffer from center-rating bias (where judges default to 3s to avoid hard decisions), and require larger statistical sample sizes to detect meaningful quality shifts. Binary scoring forces strict evaluation criteria, speeds up human labeling, and yields actionable Pass/Fail metrics.

### LLM Judge Calibration & Validation

An automated evaluator cannot be trusted until it is empirically validated against human domain expert labels. To calibrate an LLM judge, curate a dataset of 100 to 200 task outputs labeled as Pass or Fail by domain experts. Split this data into held-out datasets:

- **Train set (10–20%)**: Used to construct the judge's prompt and zero-shot instructions.
- **Dev set (40–45%)**: Used to iteratively refine prompts and analyze judge-vs-human disagreements.
- **Test set (40–45%)**: Reserved strictly for final, untouched validation to prevent prompt overfitting.

Measure alignment using two core classification metrics:
`TPR (True Positive Rate / Recall) = (Failures Correctly Flagged by Judge) / (Total Actual Failures Labeled by Humans)`

`TNR (True Negative Rate) = (Good Outputs Correctly Passed by Judge) / (Total Actual Good Outputs Labeled by Humans)`

Iterate on the judge prompt until both TPR and TNR meet your target thresholds, ensuring high failure detection with minimal false alarms.

### Human Domain Expert Rubrics

Human evaluation remains the gold standard for defining quality. However, relying on democratic consensus across multiple annotators often leads to deadlock, inconsistent rubrics, and high organizational overhead. Instead, adopt the "Benevolent Dictator" alignment model: appoint a single domain expert (e.g., a lead legal counsel for contract analysis apps, or a lead product manager for user support) as the definitive authority on quality standards. This individual sets canonical criteria, resolves edge-case ambiguities, and validates automated judge predictions.

The YAML schema below illustrates a comprehensive multi-grader task definition, combining deterministic code checks, state checks, and an LLM rubric:

```yaml
task_id: "eval_order_cancellation_v3"
description: "Verify agent correctly processes order cancellation and updates DB"
inputs:
 user_message: "Please cancel my order #88312 immediately."
 user_id: "usr_9921"

graders:
 - id: "check_json_schema"
 type: "deterministic_code"
 name: "JSON Syntax Check"
 check: "response.format == 'json' and response.payload.has_key('status')"

 - id: "verify_db_state"
 type: "state_check"
 name: "Database Cancellation Check"
 target_system: "postgres_orders"
 expected_state:
 order_id: 88312
 status: "CANCELLED"

 - id: "llm_empathy_and_policy"
 type: "llm_as_a_judge"
 model: "gpt-4o"
 prompt_template: |
 You are an evaluation grader. Analyze the conversation transcript below.
 Determine if the assistant maintained a professional tone, verified order details,
 and communicated the cancellation clearly without violating refund limits.
 Return ONLY a JSON object: {"verdict": "PASS" | "FAIL", "reason": "<explanation>"}
 input_context:
 transcript: "{{trace.messages}}"
 expected_outcome: "PASS"
```

## Core Metrics That Matter: Grounding, Relevance, and Agent Trajectories

Evaluating specialized AI architectures requires domain-specific metrics aligned with application workflows.

### RAG System Evaluations

Retrieval-Augmented Generation systems split into two operational components: Retrieval (Information Retrieval) and Generation (LLM Synthesis). Evaluating RAG architectures requires isolating failure modes across both phases:

```
RAG System Evaluation Pipeline:
 [User Question (Q)]
 │
 ├─── Retrieval Phase ────────► [Retrieved Context (C)]
 │ - Recall@k, Precision@k, MRR
 │ - Context Relevance (C|Q)
 │
 └─── Generation Phase ────────► [Generated Answer (A)]
 - Faithfulness / Groundedness (A|C)
 - Answer Relevance (A|Q)
```

1. **Retrieval Metrics (Traditional IR)**:

- **Recall@k**: The proportion of all known relevant reference documents successfully fetched within the top $k$ retrieved chunks.
- **Precision@k**: The proportion of retrieved chunks within top $k$ that are genuinely relevant to the query.
- **Mean Reciprocal Rank (MRR)**: Measures how high up in the returned result list the first relevant document chunk appears.

2. **Generation Metrics (Triad RAG Evals)**:

- **Context Relevance (C|Q)**: Evaluates whether the retrieved context chunks (C) contain information directly relevant to the user query (Q), filtering out context noise.
- **Faithfulness / Groundedness (A|C)**: Evaluates whether every claim made in the generated answer (A) is strictly supported by the retrieved context (C), identifying factual hallucinations.
- **Answer Relevance (A|Q)**: Evaluates whether the final generated answer (A) directly answers the user's initial question (Q), ignoring unhelpful or tangential responses.

### Agentic Metrics

Agentic workflows operate autonomously over variable execution paths. Metrics must evaluate both execution outcome and step-level diagnostic efficiency:

- **Tool Selection Accuracy**: Evaluates whether the agent selected the optimal tool for the current step.
- **Parameter Extraction Validity**: Verifies that tool arguments extracted from context match required types, ranges, and JSON schemas.
- **Precondition & Authorization Verification**: Validates that required security checks or user permissions were confirmed before executing irreversible state actions (e.g., executing a bank transfer).
- **Execution Efficiency**: Tracks total turn count, total latency, and cumulative token consumption per task attempt.

### Transition Failure Matrices

To diagnose systemic bottlenecks in complex agent loops, construct a State Transition Failure Matrix. This matrix maps the last successful state against the target state where the first failure occurred:

| Last Successful State \ Failure State | Parse Query | Retrieve Context | Generate Plan | Tool Execution | Final Response |
| :------------------------------------ | :---------- | :--------------- | :------------ | :------------- | :------------- |
| **Parse Query**                       | —           | 1                | 4             | 0              | 0              |
| **Retrieve Context**                  | 0           | —                | 12            | 2              | 0              |
| **Generate Plan**                     | 0           | 0                | —             | 18             | 3              |
| **Tool Execution**                    | 0           | 0                | 1             | —              | 5              |

To calculate actionable failure rates from raw matrix counts, divide the count in each failure cell by the total outgoing state transitions initiated from that row's starting state:

`Transition Failure Rate (Start -> Fail) = (Count of Failures from Start to Fail) / (Total Transitions Originating from Start)`

For instance, if **Generate Plan** initiates 25 total state transitions, the 18 failures transitioning into **Tool Execution** yield an $18 / 25 = 72\%$ state failure rate. Computing percentage distributions isolates actual system bottlenecks from volume bias, directing prompt engineering and tool refinement precisely where execution chains breakdown.

### The Danger of Generic Metrics

Avoid off-the-shelf, generic evaluation metrics such as vague "helpfulness", "coherence", or "quality" scores provided by generic vendors. Generic metrics measure abstract qualities that rarely correlate with business requirements or domain-specific failure modes. Relying on generic scores creates false confidence—a system can achieve a 95% "helpfulness" rating while consistently generating invalid database queries or breaching regulatory constraints. Build custom evaluators driven by empirical failure analysis on your actual application traces.

## The Virtuous Feedback Loop: Turning Production Traces into Golden Datasets

The core driver of continuous AI system improvement is turning live production traces into high-value offline evaluation datasets through structured error analysis.

```
PRODUCTION RUNTIME ERROR ANALYSIS & CURATION
+-------------------+ +-----------------------+
| Live User Traffic| | 1. Trace Sampling |
+---------+---------+ +-----------+-----------+
 | |
 v v
+-------------------+ +-----------------------+
| Trace Collector +------------------->| 2. Open Coding |
+-------------------+ | (Journal Notes) |
 +-----------+-----------+
 |
 v
CI/CD REGRESSION SUITE +-----------------------+
+-------------------+ | 3. Axial Coding |
| Golden Dataset |<-------------------+ (Build Taxonomy) |
| (100+ Examples) | +-----------------------+
+-------------------+
```

### The Error Analysis Pipeline

Error analysis is a systematic four-step qualitative methodology adapted from grounded research techniques:

1. **Dataset Creation**: Collect a representative pool of 50 to 100 diverse production traces covering diverse workflows.
2. **Open Coding**: Manually read raw traces without predefined rubrics. Write open-ended annotations journaling every issue—including bad prompt context, bad formatting, slow responses, user confusion, or missing UI affordances. Annotate at least 30 traces manually before using automated assistants.
3. **Axial Coding**: Group open-ended notes into a structured failure taxonomy (e.g., categorizing notes like "casual tone" and "slang used" under a unified `Persona-Tone Mismatch` taxonomy).
4. **Iterative Refinement**: Use AI assistants or active learning algorithms to search remaining production traces for instances matching your emerging failure categories. Continue annotating until you reach **theoretical saturation**—the point at which reviewing new traces yields no new failure modes or taxonomy adjustments.

### Trace Sampling Strategies

Reviewing production data efficiently requires targeted trace sampling strategies:

- **Random Sampling**: Establishes an unbiased baseline of system performance. Maintain a small random sample in every review batch to discover unanticipated failure patterns.
- **Clustering**: Groups traces semantically based on input embedding proximity or user persona, enabling reviewers to evaluate performance across diverse topic distributions.
- **Outlier Sorting**: Filters traces based on extreme tail metrics(such as high latency, high token counts, high step counts, or high retry frequencies).
- **User Feedback Sorting**: Isolates traces flagged with explicit negative user feedback (e.g., thumbs-down or ticket escalations).
- **Active Learning / Classification**: Uses lightweight classifiers or automated evaluators to surface traces flagged with high model-uncertainty scores for human labeling.

### Synthetic Data Generation

When launching new features or testing rare edge cases with sparse production traffic, generate synthetic test datasets using a structured multi-dimensional tuple approach:

1. **Define Important Dimensions**: Identify key axes of variation, such as Query Type (`simple`, `complex`, `malicious`), User Mood (`neutral`, `frustrated`), and Domain Constraints (`standard_policy`, `edge_case`).
2. **Generate Structured Tuples**: Combine dimensions into distinct test vectors (e.g., `[Complex, Frustrated, Edge_Case]`).
3. **Two-Step Query Generation**: Pass structured tuples to an LLM to generate natural language queries, separating tuple generation from text synthesis to prevent repetitive phrasing.

Synthetic data generation is unreliable in specialized technical domains (e.g., complex legal filings or specialized medical diagnostics), low-resource languages, or high-stakes edge cases where LLMs lack underlying training distribution depth. In these scenarios, real production traces curated by domain experts are mandatory.

## Practical Implementation: Tooling Ecosystem and Production Pitfalls

The modern AI observability landscape offers solid open-source and commercial platforms designed for trace logging, dataset curation, and evaluation management. Open-source solutions like Langfuse and Arize Phoenix provide self-hosted telemetry architectures with native OpenTelemetry support, detailed trace visualizations, and SDK integration. Commercial platforms such as Braintrust offer end-to-end experiment tracking and automated evaluation environments, while infrastructure observability tools like Honeycomb deliver high-cardinality distributed tracing across multi-agent microservices. In addition, specialized evaluation runners such as Harbor offer containerized environments for executing standardized benchmark evaluations across agent fleets.

Before adopting heavy SaaS platforms or engineering complex automated evaluation pipelines, conduct manual error analysis on your first 50 to 100 production traces. Build simple, lightweight annotation UI interfaces close to your domain data using AI coding tools or notebooks, version your prompts directly in Git alongside application source code, and allow empirical failure taxonomies discovered during manual trace reviews to dictate where automated evaluators should be constructed.

## References

- [Your AI Product Needs Evals – Hamel's Blog](https://hamel.dev/blog/posts/evals/)
- [AI Evals: Everything You Need to Know – Hamel's Blog](https://hamel.dev/blog/posts/evals-faq/)
- [Demystifying evals for AI agents \ Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [Generative AI Semantic Conventions | OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
- [Evaluation of LLM Applications | Langfuse](https://langfuse.com/docs/evaluation/overview)
- [LLM Observability & Application Tracing | Langfuse](https://langfuse.com/docs/tracing)
- [Arize Phoenix Documentation | AI Observability & Evaluation Platform](https://docs.arize.com/phoenix)
- [LLM Observability in Practice | Honeycomb](https://www.honeycomb.io/blog/llm-observability)