Source-linked AI summary
Towards a Science of AI Agent Reliability
Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan
TL;DR
AI agents increasingly perform consequential tasks, yet single-run accuracy and success rates leave important reliability properties unmeasured. The paper introduces a safety-critical framework with twelve metrics across four dimensions and finds that rapid capability gains produced only small reliability improvements. It concludes that reliability should be evaluated alongside capability, with safety treated separately because tail risks can be obscured by aggregation.
Problem
Single success metrics do not reveal consistency, robustness to perturbations, predictable failure recognition, or bounded error severity in deployed agents.
Method
The paper proposes twelve reliability metrics organized into consistency, robustness, predictability, and safety, independently of raw task success.
Results
24 months of rapid capability gains produced only small improvements in reliability, while more accurate models remained inconsistent, brittle, and poor at recognizing likely failure.
Takeaways & Limitations
Reliability should become a key evaluation concern alongside capability, using multidimensional measures to assess how agents perform, degrade, and fail.
Takeaways & Limitations
The metric decomposition and normalization-based aggregation reflect subjective design choices, and safety is reported separately because overall reliability does not capture the full picture.
Abstract
from arXiv · showhide
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.
1. Introduction
AI agents are increasingly used for consequential tasks, but benchmark accuracy and mean success rates do not capture whether their behavior is reliable in deployment. The paper proposes a safety-critical evaluation framework that decomposes reliability into four dimensions and reports persistent gaps between capability and reliability.
- Agents now perform consequential tasks, making failures costly despite their potential to automate routine work.
- Real-world incidents expose a gap between benchmark capability and deployment outcomes, motivating a systematic definition and evaluation of agent reliability.
- Mean task success rates obscure predictable versus random failures, perturbation sensitivity, failure recognition, and the differing severity of errors.
- The paper adapts safety-critical engineering by defining consistency, robustness, predictability, and safety as four reliability dimensions.
- Twelve computable metrics evaluate reliability independently of raw task success, and the framework profiles where modern agents succeed and fail.
- The framework narrowly treats safety as bounded operational severity and excludes adversarial attacks and broader sociotechnical concepts such as value alignment.
2. A Cross-Domain Perspective of Reliability
Safety-critical engineering treats reliability as a multidimensional, consequence-aware property rather than an average success rate. The paper synthesizes this perspective into four dimensions that organize otherwise scattered reliability concerns in machine learning.
- Cross-domain safety-critical practices identify four recurring reliability dimensions: consistency, robustness, predictability, and safety.
- Consistency: Consistency concerns whether repeated runs under identical conditions produce stable outcomes, because high variance makes otherwise acceptable average performance unpredictable.
- Robustness: Robustness concerns graceful degradation when operating conditions deviate from nominal, including infrastructure, environment, and input changes.
- Predictability: Predictability concerns whether systems recognize when they are likely to fail, supporting safe modes or other responses when uncertainty is high.
- Safety: Safety concerns the severity of consequences when failures occur, distinguishing operational risk from compliance with harmful-request policies.
- The proposed decomposition unifies these dimensions and treats them as independent of raw capability, so capability and reliability can diverge.
3. Operationalizing Reliability for AI Agents
The paper operationalizes reliability as computable behavioral measures spanning consistency, robustness, predictability, and safety. It aggregates related metrics while preserving safety separately to avoid obscuring tail risks.
- The framework provides a working operationalization that makes reliability concerns observable and comparable without claiming to redefine reliability.
- Consistency: Consistency metrics measure repeatability of outcomes, trajectories, and computational or monetary resources across repeated attempts.
- Robustness: Robustness metrics assess resilience to infrastructure faults, semantic-preserving environment changes, and equivalent prompt reformulations.
- Predictability: Predictability metrics evaluate whether confidence matches empirical success, separates successes from failures, and jointly reflects both properties.
- Safety: Safety metrics separately measure constraint compliance and the severity of violations, distinguishing how often violations occur from how harmful they are.
- Aggregation: Reliability metrics are aggregated within dimensions and across pillars, while safety is excluded from the overall average because averaging can obscure tail risks.
- Aggregation: Practitioners may use alternative weightings and should inspect application-relevant individual metrics rather than relying only on aggregation.
4. Experiments
The evaluation tests 15 models on GAIA and τ-bench using reliability metrics spanning consistency, robustness, predictability, and safety. Reliability gains remain uneven: consistency, prompt robustness, calibration, discrimination, and safety reveal distinct strengths and persistent weaknesses across benchmarks.
- Evaluation setup: 15 models are evaluated on GAIA and τ-bench, which provide complementary challenges for assessing agent reliability.GAIA covers browsing, file manipulation, and multi-step reasoning, while τ-bench tests multi-turn customer-service interactions and consequential actions.
- Main results: Reliability improves only modestly over 24 months, with moderate gains on τ-bench but barely any improvement on GAIA.The benchmark difference may reflect τ-bench’s structured setting versus GAIA’s open-ended failure modes, though other factors may contribute.
- Consistency: Outcome consistency remains low, while token and compute usage vary substantially across runs, especially on GAIA.Agents often select similar action types but vary in execution order, producing a “what but not when” pattern.
- Robustness: Prompt robustness varies substantially across models, whereas fault and environment robustness show ceiling effects across most models.Agents generally handle technical failures gracefully but remain vulnerable to surface-level instruction paraphrases.
- Predictability: Calibration improves in recent models, but discrimination trends diverge: it generally improves on τ-bench while remaining flat or worsening on GAIA.Measuring both sub-metrics matters because better calibration alone does not ensure reliable identification of likely failures.
- Difficulty effects: On harder GAIA tasks, resource consistency generally degrades, predictability declines modestly, and robustness shows no systematic relationship with difficulty.Consistency typically changes monotonically with task difficulty rather than following the expected U-shape.
- Safety: Recent frontier models show lower violation rates, yet incorrect financial charges or refunds remain the most common τ-bench failure mode.Most violations are low-to-moderate severity, but infrequent high-severity events can carry outsized costs.
- Benchmark quality: Correcting τ-bench benchmark errors improves predictability and safety for most agents, especially calibration, but not consistently consistency or robustness.The evaluation uses the verified 26-task subset because 24 of the original 50 airline tasks contain grading errors.
5. Recommendations
The paper recommends evaluating, designing, governing, and deploying agents around reliability rather than single-run capability, with requirements scaled to autonomy. It emphasizes dynamic testing and reliability metrics that expose uneven progress across dimensions.
- Recommendation 1: Reliability evaluation should use dynamic benchmarks that move beyond single-run accuracy and fixed environments.Generative, parameterized tests can vary inputs, rename fields, reorder responses, inject faults, and support temporal re-evaluation.
- Recommendation 2: Agent architectures should optimize reliability alongside capability because calibration and safety have improved more than consistency and discrimination.Systematic evaluation makes uneven progress visible and identifies dimensions needing improvement.
- Recommendation 3: Reliability metrics and incident analyses should inform deployment decisions, change management, compliance, and continuous improvement.Organizations could require minimum consistency and safety thresholds before moving agents from sandboxed pilots to production.
- Recommendation 4: Reliability requirements should scale with agent autonomy.Human review can provide a reliability backstop in augmentation settings, whereas autonomous systems translate unreliability directly into failures.
6. Limitations
The evaluation covers a narrow task and scaffold range, relies on LLM-based safety judging, and reflects subjective metric and aggregation choices. Temperature-zero evaluation may overestimate reliability achievable when nonzero temperatures improve accuracy.
- Scope: The analysis covers only two complementary benchmarks, representing a narrow slice of tasks agents face in practice.
- Scope: Each benchmark uses a single well-performing scaffold, so other scaffolds could produce qualitatively different reliability profiles.
- Evaluation: LLM-based safety judging introduces reliability concerns into the safety evaluation itself.
- Method: The metric set and normalization-based separation of reliability from capability reflect subjective design choices, and alternate decompositions are possible.Safety is reported separately to avoid masking tail risks, so R does not capture the full picture.
- Evaluation: Temperature zero may overestimate achievable reliability when nonzero temperatures maximize accuracy.
- Scope: The framework is a starting point that should complement, not replace, human oversight, sandboxed testing, and monitoring.
7. Conclusion
The paper decomposes agent reliability into measurable dimensions and evaluates 15 models across two benchmarks. Despite rapid capability gains, reliability improves only slightly, motivating evaluation of how agents fail, not only how often they succeed.
- Conclusion: 24 months of rapid capability gains produced only small improvements in reliability across evaluated agents.
- Conclusion: More accurate models remain inconsistent across runs, brittle to prompt rephrasings, and often unable to recognize when they are likely to succeed.
- Conclusion: The proposed framework uses a four-dimensional decomposition with twelve distinct sub-metrics grounded in safety-critical engineering.
Impact Statement
Making reliability measurable can support deployment decisions, agent development, governance, and more informed comparisons. The paper also warns that reliability metrics are insufficient on their own and require standardized, transparent evaluation practices.
- Implications: Reliability metrics can provide concrete criteria for comparing agents and judging suitability for particular deployment contexts.For example, customer-service agents could face consistency and predictability thresholds before production deployment.
- Risks: Optimizing reliability scores without addressing underlying failure modes can create a false sense of security.
- Risks: High reliability scores do not guarantee safe or beneficial behavior when an agent remains misaligned with user intent.
- Risks: Reliability evaluation should complement, not replace, human oversight, sandboxed testing, incident monitoring, and ongoing assessment.
- Future needs: Realizing broader benefits requires standardized benchmarks, shared evaluation protocols, and transparent reporting norms.
A. Extended Recommendations
The paper argues that reliability must be evaluated independently of mean task success and recommends evolving benchmarks, agent architectures, governance, and requirements according to autonomy.
- Motivation: Reliability is a multidimensional progress axis that mean task success alone cannot capture.The framework treats reliability as distinct from raw capability and identifies specific dimensions for evaluation and optimization.
- Recommendation 1: Dynamic benchmarks should replace single-run evaluations in fixed environments.Such benchmarks should test repeated success, responses to rephrased instructions, and behavior under changing conditions.
- Recommendation 2: Agent architectures should optimize reliability alongside capability because reliability dimensions improve unevenly across model generations.Calibration and safety improved noticeably, whereas consistency and discrimination improved little.
- Recommendation 3: Reliability metrics and incident analyses should inform deployment governance, change management, and regulatory compliance.Organizations could require minimum consistency and safety thresholds before moving agents from sandboxed pilots into production.
- Recommendation 4: Reliability requirements should scale with agent autonomy because human oversight can compensate for shortcomings in augmentation settings but not automation.In automation, unpredictable failures directly affect real-world outcomes, making reliability a deployment prerequisite.
B. Extended Limitations
The paper identifies scope, benchmark, scaffold, metric, aggregation, disentanglement, and experimental-setting limitations, while treating reliability as context-dependent rather than universally optimized.
- Empirical scope: The empirical analysis covers only τ-bench and GAIA, a narrow slice of real-world agent tasks.Although complementary in structure and scope, the two benchmarks do not represent the full diversity of deployment tasks.
- Scaffold diversity: Each benchmark is evaluated with a single scaffold, so other scaffolds could produce different reliability profiles.The authors plan to extend evaluation to additional agentic scaffolds.
- Safety judging: Safety evaluation relies on LLM-based judging, which introduces reliability concerns of its own.Judge-free and human-validated safety metrics remain future directions.
- Metric choices: Metric selection and capability disentanglement involve subjective or non-unique methodological choices.Alternate decompositions are possible, and normalization or conditioning may vary in adequacy across domains.
- Safety aggregation: Safety is reported separately rather than aggregated, so the overall reliability score does not capture the full reliability picture.This avoids masking tail risks through averaging but leaves integration for future work.
- Experimental setting: Setting temperature to zero limits stochasticity and may overestimate achievable reliability when nonzero temperature is needed for accuracy.The authors note that maximizing accuracy may require a nonzero temperature.
- Context dependence: Reliability dimensions are application-dependent: trajectory diversity may be desirable, and practitioners can exclude or reinterpret dimensions.The framework allows dimensions to be weighted, excluded, or reinterpreted relative to deployment context.
C. Extended Metric Details
The appendix operationalizes reliability through deployment-oriented metrics covering consistency, robustness, predictability, compliance, and harm severity, with examples showing how each captures distinct operational risks.
- Metric design: The appendix connects each metric to a deployment scenario and the practical consequences of poor performance.These examples make reliability concerns observable in concrete operational settings.
- Consistency: Outcome consistency measures whether identical requests produce the same outcomes across runs, independently of overall accuracy.It normalizes stochastic unreliability by the maximum possible variance at a given accuracy level.
- Consistency: Trajectory consistency distinguishes whether agents use similar action types and whether they execute them in the same order.Distributional and sequential trajectory consistency capture complementary forms of path variability.
- Consistency: Trajectory diversity can improve flexibility, so trajectory consistency may be treated diagnostically rather than as an aggregation requirement.Practitioners can exclude it from the aggregate score when multiple solution paths are valuable.
- Consistency: Resource consistency measures variability in costs across successful runs, capturing operational unpredictability even when tasks succeed.Large swings in tokens or tool calls complicate budgeting, latency control, and rate-limit management.
- Robustness: Robustness metrics test graceful behavior under transient faults, environmental changes, and semantically equivalent prompt variations.Examples include retrying failed APIs, handling reordered fields or changed date formats, and interpreting paraphrased requests.
- Predictability: Calibration measures whether confidence matches empirical success, while discrimination measures whether confidence ranks successes above failures.Together they distinguish trustworthy confidence levels from useful confidence ordering.
- Predictability: The Brier score jointly evaluates calibration and discrimination as a holistic predictive-quality measure.It rewards confidence estimates that are both well-calibrated and well-ranked.
D.1. Real-World Agent Failures
Real-world incidents demonstrate that benchmark capability can coexist with unreliable, high-consequence behavior, motivating a measurable framework adapted from safety-critical engineering while acknowledging important scope boundaries.
- Incident evidence: High-profile deployments reveal systematic gaps between average benchmark performance and reliable real-world operation.The appendix presents incidents as symptoms of evaluation gaps rather than isolated anomalies.
- Incident evidence: The Air Canada chatbot case combined incorrect, confident information with absent uncertainty signaling and direct financial and legal consequences.The tribunal held the airline responsible for chatbot information and ordered damages.
- Evaluation implication: Calibration and risk-coverage evaluation could expose failures where agents cannot recognize when to abstain or escalate.These predictability metrics target confidence misalignment and failure-awareness gaps illustrated by the chatbot incident.
- Incident evidence: The Bing Chat case shows that reliability can degrade within extended sessions as small instabilities compound into larger behavioral deviations.Conversation-length limits implicitly acknowledged degradation over long interactions.
- Classical reliability: Safety-critical industries contribute recurring reliability dimensions, but classical reliability constructs do not transfer directly because agent failures, operating profiles, and conditions remain ill-defined.Applying classical failure-probability formulas would require assumptions not currently established for agents.
- Operationalization: The proposed metrics are a working operationalization that makes reliability observable while formal definitions and operating envelopes mature.The authors position measurement as an initial step rather than a replacement for eventual formalization.
- Future directions: Future work includes session reliability, multi-agent failure propagation, reliability optimization, and monitoring architectures whose failures are independent of supervised agents.Longer-horizon evaluation requires persistent environments, interruptions, and graceful-degradation measures.
E.6. Specification and Verification
The section outlines open questions for specifying, testing, verifying, communicating, and governing AI-agent reliability. It emphasizes distributional testing, verified wrappers, adaptive human control, and continuous organizational evaluation.
- Behavioral specification: Reliability specifications may combine expressive natural-language intent with precise formal safety constraints.
- Testing methodologies: Agent testing should define pass criteria distributionally and explore prompt variations, tool configurations, and environments systematically.
- Partial verification: Verified wrappers could check constraints and outputs at runtime, providing partial guarantees without verifying the entire agent.
- Shared control design: Reliability profiles can guide delegation: high-consistency, safe tasks may be delegated, while low-predictability tasks need human checkpoints.
- Continuous evaluation and change management: Organizations should integrate automated reliability regression tests and reliability impact assessments into deployment change processes.
- Trust calibration: Self-assessment is the practically accessible confidence-estimation method for users of frontier models through standard interfaces.
F.3.5. SAFETY EVALUATION PROTOCOL
The safety protocol evaluates compliance and harm severity from agent execution traces using predefined, benchmark-specific constraints and an LLM judge. The broader evaluation argues that systematic reliability testing could have identified reported deployment vulnerabilities before release.
- Compliance analysis: Compliance analysis checks execution traces against predefined constraints and records whether each constraint was violated.
- Severity interpretation: Compliance severity categories range from low technical concerns to critical violations with immediate harm potential.
- Context-sensitive checks: The protocol distinguishes contextual exceptions, such as requested data disclosure and explicitly confirmed operations, from actual violations.
- Constraint sets: Benchmark-specific constraints cover privacy, factual accuracy, code execution, destructive operations, financial accuracy, authentication, and policy circumvention.
- Error severity analysis: An LLM judge assigns execution errors a numeric severity score from 0–10 and maps scores to low, medium, or high categories.
- Pre-deployment value: Systematic reliability evaluation could have identified each of three reported deployment vulnerabilities before release.
G.5. Full robustness results
Robustness varies most with prompt phrasing, while fault and environment robustness are largely saturated under the tested perturbations. Structured τ-bench tasks generally support stronger reliability than open-ended GAIA tasks, but difficulty and model type still shape consistency and resource use.
- Fault and environment robustness: Fault and environment robustness largely saturate, with most models recovering reliably from tool errors, timeouts, and malformed API responses.The perturbation suite omits schema migrations, API version changes, and shifting document layouts.
- Prompt robustness: Prompt robustness varies widely: naturalistic rephrasings cause minimal losses for most frontier models on τ-bench but pronounced drops for several models on GAIA.GAIA’s open-ended search paths can compound small interpretation differences across subsequent steps.
- Consistency: Outcome consistency remains the hardest consistency metric, while agents select similar action types but vary in execution order.
- Predictability: Most models show modest selective-prediction ability on GAIA, whereas τ-bench curves are generally indistinguishable from the random baseline.Claude Opus 4.5 and 4.7 are exceptions on τ-bench.
- Model type: Reasoning models are generally more reliable than non-reasoning models, although reliability improves more slowly than accuracy.
- Task difficulty: On harder GAIA tasks, Claude and Gemini models invest substantially more actions, while robustness metrics other than prompt robustness remain relatively stable.