Source-linked AI summary
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Mukund Pandey
TL;DR
Production agentic systems can develop gradual failures that standard evaluation frameworks do not address. The paper proposes PAEF for continuous monitoring and tests metric blind spots; its experiments find that standard accuracy-based metrics provide no signal or a lagged, sub-threshold signal across the reported failure modes.
Problem
Standard evaluations do not address gradual production failures, continuous distribution shifts, or the real consequences of agent decisions.
Source: §1.1 · p. 1, §1.2 · p. 2Method
The paper proposes PAEF, a continuous evaluation framework that monitors tool reliability, distribution health, and explanation validity.
Source: §5.3 · p. 6, §5.4 · p. 7, §5.5 · p. 7Results
Across four experiments, standard accuracy-based metrics provided no signal for FM-1 and FM-5 or a sub-threshold lagged signal for FM-2 and FM-3, while PAEF detected FM-1 and FM-5 immediately.
Source: §6.6 · p. 10Takeaways & Limitations
PAEF is presented as a continuous production-evaluation approach to the gap between easy-to-measure metrics and production reliability concerns.
Source: §8 · pp. 11–12Experiments use public benchmark datasets, and PAEF’s cascade-uncertainty and explanation-validity dimensions require intermediate outputs and model inputs unavailable from black-box agents.
Source: §7.3 · p. 11
Abstract
from arXiv · showhide
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of ground truth for long-horizon tasks. This paper makes three contributions. First, we present a taxonomy of seven failure modes unique to production agentic systems, each grounded in observations from systems operating at billion-event scale. Second, we demonstrate empirically where standard metrics -- ROUGE, BERTScore, accuracy/AUC, and the agentic benchmarks above -- fail to detect each failure mode. Third, we propose PAEF (Production Agentic Evaluation Framework), a five-dimension evaluation framework with an open-source reference implementation, designed for continuous evaluation on production traffic rather than episodic benchmark runs. Our analysis shows that standard metrics fail to detect four of the seven failure modes entirely and detect three others only after a lag of multiple evaluation cycles.
1 Introduction
Existing LLM evaluations measure capability in controlled settings, while production agents face gradual, continuous failures with real consequences. The paper motivates production-focused evaluation and presents a taxonomy and metric-gap analysis.
1.1 The Evaluation Gap
- Production deployment creates gradual, systematic failures that standard evaluation frameworks were not designed to catch.Systems may appear to perform well on aggregate metrics while failing for a specific user cohort or silently losing decision quality.Source: §1.1 · p. 1
1.2 What Production Changes
- Production agents make chained decisions, operate continuously as distributions shift, and affect users with real consequences.Episodic scores cannot observe error propagation; changing models, prompts, and data require continuous monitoring, while correctness alone is insufficient.Source: §1.2 · pp. 1–2
1.3 Contributions
- The paper contributes a seven-mode taxonomy with production-grounded examples and analysis of why existing metrics miss failures.It also analyzes which standard metrics detect each failure mode and where their blind spots lie.Source: §1.3 · p. 2
3. PAEF: Production Agentic Evaluation Framework
The paper presents PAEF as a five-dimension framework for continuous production monitoring and reports experimental validation on public benchmarks.
Framework
- PAEF is a five-dimension framework designed for continuous production monitoring.The framework has an open-source reference implementation.Source: §3 · p. 2
Reference implementation
- An open-source reference implementation accompanies PAEF.The implementation is identified as the llmeval-toolkit.Source: §3 · p. 2
Experimental validation
- Experiments on public benchmarks demonstrate standard-metric detection gaps and PAEF coverage.The passage characterizes the experiments as validation of those detection gaps and framework coverage.Source: §3 · p. 2
2 Background and Related Work
Prior evaluation frameworks cover model capability and agent task completion, while production systems raise challenges involving continuous operation, drift, explanations, and alignment. The paper relates these gaps to existing work and proposes production-grounded detection approaches.
2.1 LLM Evaluation Frameworks
- HELM, BIG-bench, and MT-Bench broaden capability and conversation evaluation but do not capture continuous production behavior and its constraints.HELM is static and single-turn; BIG-bench is episodic, while MT-Bench does not model production constraints or output-distribution health over time.Source: §2.1 · p. 2
2.2 Agentic Evaluation Frameworks
- AgentBench and other agentic benchmarks evaluate task completion in controlled or bounded settings, not sustained reliability and alignment in production.AgentBench assumes guaranteed tool availability and episodic execution; WebArena and SWE-bench have bounded or single-episode scope.Source: §2.2 · p. 2
2.3 Drift Detection in ML Systems
- Traditional drift tools target structured inputs and scalar outputs, whereas agentic systems produce sequential decisions and natural-language outputs.The paper extends drift-detection concepts to agentic output sequences, building on ROUGE-based drift descriptors.Source: §2.3 · pp. 2–3
2.4 Explainability and Attribution
- Post-hoc attribution can become unreliable under correlated features and feature crosses, and a correct decision with a wrong explanation can mislead audits and debugging.The paper names this Explanation-Decision Decoupling (FM-5) and proposes a perturbation consistency check.Source: §2.4 · p. 3
2.5 AI Alignment and Reward Specification
- The paper frames Proxy Goal Convergence (FM-7) as a production-grounded instance of Goodhart’s Law, reward hacking [1], and specification gaming.Its proposed detection uses multi-objective monitoring and counterfactual evaluation.Source: §2.5 · p. 3
3 Taxonomy of Production Agentic Failure Modes
The taxonomy describes seven production failure modes and explains why standard evaluation metrics can miss them. The examples show how failures may remain hidden despite locally acceptable outputs or metrics.
FM-1 — Cascading Decision Error
- An incorrect early identity resolution can propagate through later pipeline steps into a coherent but wrong decision for a reproducible input cohort.Individual steps passed validation, and downstream distribution spikes took several days to reveal the failure.Source: §3 · p. 3
FM-2 — Silent Degradation via Availability-Truth Decoupling
- Schema-valid cached or default tool responses can conceal missing data, allowing downstream logic to make incorrect approvals without logged failures.Detecting this degradation requires correlating tool health with decision quality.Source: §3 · pp. 3–4
FM-3 — Distribution Collapse Under Metric Optimisation
- Output diversity can collapse while aggregate metrics remain stable, with AUC and precision@k failing to capture accumulating session-level effects.A diversity audit revealed repetitive recommendations after a detection lag of approximately 1–2 weeks.Source: §3 · p. 4
FM-4 — Consistency Collapse Across Entry Points
- Semantically identical requests can receive different decisions across API and UI surfaces, a mismatch revealed by paired-request tests and cross-channel replay audits.The individual channels performed acceptably in isolation, and existing benchmarks test inputs on a single surface.Source: §3 · p. 4
FM-5 — Explanation-Decision Decoupling
- A correct decision can be paired with an attribution that highlights a benign proxy rather than the true behavioral signal.Accuracy and AUC do not measure explanation validity; perturbing the claimed feature can expose negligible prediction change.Source: §3 · p. 4
FM-6 — Silent Correctness Erosion Under Latency Pressure
- Latency-triggered fallback paths can skip enrichment or use partial inputs while latency and error-rate metrics remain acceptable.The production example required correlating p95 latency spikes with downstream quality measures.Source: §3 · pp. 4–5
FM-7 — Proxy Goal Convergence
- An engagement system can improve CTR while long-term signals such as retention, session quality, and user trust decline.The proxy-goal feedback loop was detected through lagging indicators, distribution audits, and counterfactual evaluation, with a lag of several weeks.Source: §3 · p. 5
4 Where Standard Metrics Fail
Table 1 compares standard metrics’ detection coverage across production agentic failure modes, showing that their limitations follow a consistent pattern: they are episodic, aggregate, surface-level, and single-signal.
4 Where Standard Metrics Fail
- Standard metrics fail because they are point-in-time, aggregate, surface-level, and single-signal rather than continuous, distribution-aware, causally grounded, and cross-signal.PAEF is designed to address all four limitations.Source: §4 · p. 5
- Table 1 marks reliable detection within one evaluation cycle as a check, conditional or delayed detection as partial, and no signal as a cross.Partial detection may occur under specific conditions or with significant lag.Source: §4 · p. 5
- The table’s coverage pattern indicates that standard metrics do not reliably surface all production failure modes within one evaluation cycle.A detection mark means a failure would reliably surface within one cycle under normal monitoring conditions.Source: §4 · p. 5
5 PAEF: Production Agentic Evaluation Framework
PAEF is a five-dimension framework for continuous evaluation on live or shadow production traffic. It targets seven production failure modes with distribution-aware, cross-signal monitoring and dimension-specific detection signals.
5 PAEF: Production Agentic Evaluation Framework
- PAEF is a five-dimension evaluation framework designed for continuous monitoring on live or shadow production traffic, not only offline benchmarks.Each dimension targets one or more of the seven failure modes identified in Section 3.Source: §5 · p. 5
5.1 Design Principles
- PAEF detects all seven failure modes, while no standard metric detects more than two or any reliably within one evaluation cycle.Its design principles prioritize continuous operation, output distributions, and correlations across signals; dimension scores can also trigger deployment blocks, alerts, or human review.Source: §5.1 · p. 5, Table 1 · p. 6
5.2 Dimension 1: Cascade Uncertainty (FM-1)
- Cascade Uncertainty tracks confidence through pipeline steps and flags a low-confidence non-terminal step that passes its output downstream without backtracking.High internal consistency with low external correctness signals the coherence illusion (FM-1).Source: §5.2 · p. 6
5.3 Dimension 2: Tool Reliability (FM-2, FM-6)
- Tool Reliability tracks SUCCESS, PARTIAL, and FAILED calls; rising partial responses with stable accuracy signal FM-2, while latency-quality correlation signals FM-6.PARTIAL means schema-valid but incomplete; the framework also checks whether rising p95 tool latency accompanies degraded decision quality.Source: §5.3 · p. 6
5.4 Dimension 3: Distribution Health (FM-3, FM-7)
- Entropy collapse precedes accuracy degradation by an average of 1–2 evaluation cycles, providing early warning for FM-3 and FM-7.Distribution Health tracks output entropy, intra-session diversity, and repeat rate; delayed reward signals can be integrated for proxy goal convergence.Source: §5.4 · p. 7
5.5 Dimension 4: Explanation Validity (FM-5)
- Explanation Validity perturbs top-attributed features and flags decoupling when their impact is too low to support the claimed explanation.The attribution consistency score compares claimed attribution weights with measured perturbation impacts; configurable thresholds include θACS = 0.5 and δmin = 0.05.Source: §5.5 · p. 7
5.6 Dimension 5: Cross-Surface Consistency (FM-4)
- Cross-Surface Consistency measures whether semantically equivalent requests receive the same decision across different surface representations.It extends pairwise cosine similarity of sentence embeddings with decision-level agreement tracking; a below-threshold agreement rate triggers a consistency flag.Source: §5.6 · p. 8
5.7 Framework Architecture
- PAEF’s modular architecture runs standalone metric modules through an LLMEvaluator and returns a unified EvalReport for observability-stack ingestion.The report includes per-dimension scores, confidence estimates, latency measurements, and structured metadata.Source: §5.7 · p. 8
6 Empirical Evaluation
Across four controlled experiments, PAEF detects failures that accuracy-based metrics miss or detect only after a lag. The experiments use synthetic traces shaped by production observations, so they test four of seven failure modes directly.
6.1 Experimental Setup
- The experiments use synthetic traces matching production-observed confidence distributions, tool-call states, output categories, and attribution rankings, with a fixed random seed.They directly test FM-1, FM-2, FM-3, and FM-5; FM-4 is validated qualitatively, while FM-6 and FM-7 receive empirical support through FM-2 and FM-3 experiments.Source: §6.1 · p. 8
6.2 Experiment 1 — FM-3: Distribution Collapse Detection
- With accuracy held at 0.86–0.88, diversity falls from 0.200 (W1) to 0.030 (W5), a 6.5× reduction, while repeat rate reaches 1.000 at W5.Entropy drops from 0.979 to 0.867 earlier than diversity; Table 2 indicates PAEF diversity metrics surface the collapse at W3.Source: §6.2 · p. 8, Table 2 · p. 9
6.3 Experiment 2 — FM-2: Availability-Truth Decoupling
- At Stage 2, PAEF detects silent degradation when partial responses exceed 0.20, while accuracy remains 0.86 and changes only −0.01 from Stage 1.By Stage 4 the PAEF score is 0.110, whereas accuracy moves only −0.03 across all four stages, unlikely to exceed an alert threshold.Source: §6.3 · p. 9, Table 3 · p. 10
6.4 Experiment 3 — FM-1: Cascade Uncertainty and the Coherence Illusion
- In the low-confidence-step-1 scenario, four downstream steps average 0.875 confidence despite step 1 being only 31% confident; ground-truth correctness is 0.41.The internal-external divergence is 0.35, and individual downstream-step accuracy checks would pass while the cascade metric surfaces propagation failure.Source: §6.4 · pp. 9–10, Table 4 · p. 10
6.5 Experiment 4 — FM-5: Explanation-Decision Decoupling
- When the proxy feature ranks first, PAEF’s perturbation consistency score drops from 0.907 to 0.357, although accuracy is identical across the three attribution orderings.The proxy feature’s perturbation impact is 0.036, below the 0.05 decoupling threshold; zeroing transaction velocity moves the prediction by 45.1%.Source: §6.5 · p. 10
6.6 Summary
- Across four experiments, accuracy-based metrics provide no signal for FM-1 and FM-5 or only lagged, sub-threshold signals for FM-2 and FM-3; PAEF detects each earlier.PAEF detects FM-1 and FM-5 immediately, FM-2 when partial responses exceed 20%, and FM-3 two weekly windows before accuracy.Source: §6.6 · p. 10
7 Discussion
Across the discussed failure modes, proxy metrics can remain healthy while true quality degrades, motivating monitoring of the gap between them. PAEF addresses some cross-team monitoring needs, but organizational ownership and several deployment constraints remain unresolved.
7.1 A Unified Pattern
- Six of seven failure modes share a healthy proxy metric alongside degrading quality, suggesting evaluation should monitor the gap between proxy metrics and true objectives.Examples include stable accuracy as output diversity collapses and apparently correct attribution as explanation validity fails.Source: §7.1 · p. 10
- Accuracy can remain identical across FM-5 cases while PAEF’s perturbation consistency score distinguishes correct from incorrect attributions.Table 6 summarizes PAEF detection against standard metrics across four experiments.Source: §7.1 · p. 10, Table 5 · p. 11, Table 6 · p. 11
7.2 The Cross-Team Problem
- Detecting FM-3 and FM-7 can require combining ML and product signals, but teams may use separate dashboards and sampling frequencies without shared escalation paths.PAEF partially addresses this by co-locating distribution-health and model-quality monitoring; shared organizational ownership remains beyond its scope.Source: §7.2 · p. 11
7.3 Limitations
- All experiments use public benchmark datasets, while production failure examples lack specific metrics from proprietary systems.The examples are described at O(109) scale.Source: §7.3 · p. 11
- For black-box agents, PAEF’s cascade-uncertainty and explanation-validity dimensions require intermediate outputs and model inputs; distribution health and cross-surface consistency remain applicable.Dimensions 1 and 4 require additional instrumentation for black-box agents.Source: §7.3 · p. 11
- PAEF thresholds are heuristic and need domain-specific calibration against domain-specific ground truth.The reference implementation provides defaults based on observed failure signatures.Source: §7.3 · p. 11
8 Conclusion
The paper argues that production agentic systems fail in ways lab benchmarks are not designed to catch, and presents PAEF as a continuous, distribution-aware, cross-signal framework. It concludes that the framework may provide a practical foundation for deployment teams and broader work on production evaluation standards.
8 Conclusion
- The paper identifies seven production-specific failure modes and describes a gap between easily measured proxies and production objectives.The modes include cascading decision errors, silent tool degradation, distribution collapse, cross-surface inconsistency, explanation decoupling, latency-driven correctness erosion, and proxy goal convergence.Source: §8 · p. 11
- PAEF is a continuous, distribution-aware, cross-signal framework intended to address the gap between measured proxies and production objectives.The conclusion contrasts it with episodic, aggregate, single-metric evaluation.Source: §8 · p. 11
- The authors hope PAEF will serve as a practical foundation for teams deploying agentic systems at scale and as a starting point for production evaluation standards.Source: §8 · p. 12