PaperlayerView exact revision on arXiv

Source-linked AI summary

ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions

Aayush Gupta

arXiv:2601.06112v1cs.AI

TL;DR

Existing benchmarks often measure tool-using agents with single-run success rates, leaving production-relevant reliability under repeated execution, varied instructions, and tool failures insufficiently characterized. ReliabilityBench evaluates these dimensions through a unified surface, action metamorphic relations, and configurable fault injection across multiple models, architectures, and domains. Its headline findings include an 8.8% perturbation-induced drop, higher surface volume for ReAct than Reflexion, and comparable reliability from Gemini 2.0 Flash at far lower cost than GPT-4o.

  • Problem

    Existing agent benchmarks emphasize single-run success, while production agents must remain reliable across repeated trials, semantically varied instructions, and infrastructure failures.

  • Method

    ReliabilityBench evaluates agent reliability with R(k, ε, λ), end-state-based Action Metamorphic Relations, and configurable fault injection across multiple domains, models, and architectures.

  • Results

    8.8% perturbation-induced reliability drop; ReAct has 2.5% higher surface volume than Reflexion, while GPT-4o costs 82× more than Gemini with comparable reliability.

  • Takeaways & Limitations

    ReliabilityBench provides a systematic framework for assessing LLM-agent deployment readiness under production-like stress conditions.

  • Takeaways & Limitations

    The simulated domains lack the full complexity of production systems, including external APIs, authentication, and real-time data dependencies.

Abstract

from arXiv · show

Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce \textbf{ReliabilityBench}, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using $\mathrm{pass}^k$, (ii) robustness to semantically equivalent task perturbations at intensity $ε$, and (iii) fault tolerance under controlled tool/API failures at intensity $λ$. ReliabilityBench contributes a unified reliability surface $R(k,ε,λ)$, \textit{action metamorphic relations} that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at $ε=0$ to 88.1% at $ε=0.2$. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.

1 Introduction

ReliabilityBench addresses the mismatch between single-run benchmark scores and production reliability by evaluating consistency, robustness to varied instructions, and fault tolerance together. It introduces a unified framework and evaluates these properties across realistic domains, models, and agent architectures.

  • Motivation: Production reliability requires repeated execution, varied phrasings, and resilience to intermittent API failures, not just single-run task success.Existing benchmarks can overestimate reliability under idealized conditions.
  • Reliability Dimensions: Three reliability dimensions are evaluated: consistency across repeated runs, robustness to semantically equivalent instructions, and fault tolerance under infrastructure failures.These dimensions interact, so evaluating them separately gives an incomplete picture.
  • Contributions: ReliabilityBench introduces R(k, ε, λ), a unified surface for comparing agents across consistency, robustness, and fault-tolerance conditions.The framework is intended to support systematic comparison across production-like conditions.
  • Contributions: Action Metamorphic Relations define correctness through equivalent end states rather than textual similarity, while configurable fault profiles simulate production failures.The benchmark includes timeout, rate-limit, partial-response, and schema-change scenarios through its fault-injection design.
  • Contributions: The benchmark covers scheduling, travel booking, customer support, and e-commerce using 25+ domain-specific tools and state-based verification oracles.It evaluates multiple models and agent architectures across realistic task domains.
  • Empirical Findings: Empirical analysis finds that pass@1 overestimates reliability by 20-40%, simpler architectures perform better under stress, and fault tolerance degrades more steeply than robustness.These findings motivate evaluating reliability as a multidimensional production property.

2 Related Work

Prior agent benchmarks largely measure tool use or general capability through single-run success under idealized conditions. ReliabilityBench extends this work with end-state-based metamorphic evaluation and configurable fault injection across all three reliability dimensions.

  • Existing Benchmarks: ToolBench, API-Bank, ToolAlpaca, and AgentBench evaluate tool selection, API accuracy, unseen-tool generalization, or broad agent capability.Their scopes span large API collections, argument extraction, simulated responses, and multiple environments.
  • Existing Benchmarks: Existing benchmarks generally report single-run success under idealized conditions, while StableToolBench addresses API stability through caching without measuring consistency or controlled faults.This leaves repeated execution and failure recovery insufficiently evaluated.
  • Consistency Evaluation: τ-bench introduces passk and shows that consistency metrics reveal limitations obscured by pass@1, but it does not include perturbations or fault injection.ReliabilityBench extends this consistency-focused insight to additional reliability dimensions.
  • Metamorphic Evaluation: Metamorphic testing verifies relationships between transformed inputs and outputs, but agent tasks require semantic preservation and goal-state correctness rather than discrete classification labels.This distinction separates agent evaluation from text-classification robustness testing.
  • Metamorphic Evaluation: Action Metamorphic Relations treat task descriptions as equivalent when they produce the same final system state, regardless of intermediate actions or response format.For example, semantically equivalent requests may both yield a booked meeting.
  • Fault Injection: ReliabilityBench applies chaos-engineering principles to LLM agents through configurable fault profiles that simulate production failure modes.This positioning complements prior work on chaos engineering and classical fault injection.
  • Positioning: ReliabilityBench uniquely combines consistency, robustness, and fault tolerance with multi-domain evaluation.Its positioning differs from prior benchmarks that focus on narrower capability or reliability dimensions.

3 Formal Framework

ReliabilityBench formalizes agent reliability across repeated execution, semantic perturbations, and injected tool faults. Its framework evaluates correctness through task-goal and end-state conditions while modeling reliability as a three-dimensional surface.

  • 3.1 Task and Agent Formalization: An agent is a policy that maps a task description and current state to a sequence of tool calls.An agentic task includes a natural-language description and an available tool set.
  • 3.1 Task and Agent Formalization: Execution succeeds when the verifier evaluates the final state against the initial state as v(Sf, S0) = 1.
  • 3.2 Reliability Metrics: Reliability is measured through consistency under repeated execution, robustness to task perturbations, and tolerance of tool-call faults.The framework introduces these as three reliability dimensions.
  • 3.2.1 Consistency: passk: pass_k measures repeated execution success; under independence, pass_k = (pass_1)^k, although stochastic coupling can cause deviations.Each success_i denotes success on the i-th independent run.
  • 3.2.2 Robustness: Perturbation Levels: Perturbation intensity ε ranges from baseline ε = 0 to heavy ε = 0.3, adding synonym substitution, reordering, distractors, paraphrase, and corrections.The ε = 0.1, 0.2, and 0.3 levels are labeled light, medium, and heavy, respectively.
  • 3.2.3 Fault Tolerance: Fault Intensity: Fault intensity λ specifies per-tool-call fault injection probability, with profiles from λ = 0 baseline through λ = 0.3 heavy faults.The heavy profile represents 25-30% failures, while the fault distribution is profile-specific.
  • 3.3 The Reliability Surface: The reliability surface R(k, ε, λ) captures how reliability degrades across repeated execution, perturbation, and fault dimensions.A critical threshold is the point where R drops below an acceptable level θ.
  • 3.4 Action Metamorphic Relations: Action metamorphic relations require semantically equivalent task descriptions to produce equivalent end states, even when intermediate actions differ.This replaces textual output equivalence with end-state equivalence for agent tasks.

4 Benchmark Implementation

ReliabilityBench implements realistic multi-domain tasks with deterministic state-based verification and configurable agent architectures. Its execution layer supports fault injection that modifies tool responses according to predefined fault profiles.

  • 4.1 Domains: The benchmark covers scheduling, travel, customer support, and e-commerce tasks with domain-specific multi-step tools.Each domain supports operations such as booking, ticket escalation, product ordering, or returns.
  • 4.2 State-Based Verification: Each domain maintains a modifiable state dictionary, and success is verified by matching the final state against task-specific predicates.
  • 4.2 State-Based Verification: ReliabilityBench uses deterministic state-based oracles instead of LLM judges or text matching.The verifier takes initial state S0, final state Sf, and verifier v, and outputs a success boolean.
  • 4.3 Task Generation and Agents: The benchmark includes controlled task complexity levels and evaluates ReAct alongside Reflexion using function calling with structured tool outputs.ReAct uses thought-action-observation loops, while Reflexion adds self-reflection and trajectory refinement after failures.
  • 4.4 Fault Injection: The fault injector wraps tool execution and can return errors or modified tool responses under a configured fault profile.Fault-injected execution receives a tool, arguments, and fault profile as inputs.
  • 4.4 Fault Injection: Preset fault profiles increase failure rates from 0.0 at baseline to 0.275 at intensity 0.3 while adding rate limits, partial responses, cascading faults, and schema drift.The 0.1 profile uses transient timeouts and high latency; higher profiles add progressively broader fault types.

5 Experiments

ReliabilityBench evaluates two models and two agent architectures across multiple domains and stress conditions, revealing substantial degradation from perturbations and faults. Gemini 2.0 Flash and ReAct show favorable reliability-cost and robustness results in the reported experiments.

  • Experimental Setup: The evaluation compares Gemini 2.0 Flash and GPT-4o with ReAct and Reflexion across scheduling, travel, support, and e-commerce tasks.The experiments include main, ablation, and cost analyses over 1,280 episodes.
  • Perturbation Robustness: 8.8% degradation occurs as perturbation intensity rises from ε = 0 (96.88%) to ε = 0.2 (88.12%).The result indicates brittleness to semantically equivalent task paraphrasing.
  • Model Comparison: Gemini 2.0 Flash achieves +0.62% higher overall pass rate than GPT-4o at 1/82nd the cost.The reported cost comparison uses December 2024 model prices.
  • Architecture Comparison: ReAct achieves 2.5% higher reliability-surface volume than Reflexion, with a steeper Reflexion fault-degradation slope of −0.50 per 0.1 λ versus −0.38 for ReAct.The architecture gap widens under perturbation stress, and ReAct is reported to show superior fault recovery.
  • Domain Findings: Scheduling tasks achieve perfect consistency, whereas travel tasks show higher variance because flight selection introduces more decision points.Qualitative failures also include missing payment information during booking confirmation and inconsistent support-ticket escalation.
  • Fault Ablation: Rate limiting causes the largest fault-ablation degradation, 2.5% below the mixed baseline, while transient timeouts reach a 98.75% pass rate.The ablation isolates fault types at λ = 0.2; rate-limit errors can cause abandonment rather than retry.

6 Discussion

ReliabilityBench suggests that production readiness depends on stress testing beyond clean, single-run accuracy. Its results highlight degradation from perturbations and faults, while the evaluation remains bounded by limited perturbation depth, model coverage, synthetic tasks, and trial scale.

  • 90% benchmark accuracy may correspond to only 70–80% production reliability after accounting for consistency and faults.
  • Simpler architectures may be more robust under realistic conditions because added complexity introduces failure modes that outweigh its benefits.
  • Robust retry mechanisms are critical because agents without them show significant degradation under transient failures.
  • The 1,280-episode scale may be insufficient for tight confidence intervals on rare failure modes, which may require 10,000+ episodes.
  • 8–9% reliability drops occur from baseline to medium perturbation, with the steepest decline at ε = 0.1.
  • 93.75% degradation makes rate limiting the most damaging fault, whereas transient timeouts are best handled at 98.75%.
  • The study evaluates ε values only through 0.2, uses two models and k = 2 trials, and relies on synthetic workflows lacking full production complexity.
  • The fault-injection framework is intended for authorized pre-deployment evaluation rather than unauthorized vulnerability discovery in production.

7 Conclusion

ReliabilityBench presents a unified evaluation framework for LLM-agent reliability under production-like conditions. Across its framework and empirical findings, perturbations reduce reliability, ReAct performs better under stress, and Gemini offers comparable reliability at much lower cost.

  • ReliabilityBench evaluates consistency, robustness, and fault tolerance through the unified surface R(k, ε, λ).
  • Action Metamorphic Relations use synonym substitution, distractor injection, and paraphrasing to test end-state equivalence rather than textual similarity.
  • Rate limiting causes the largest reliability impact in the fault-injection ablation, producing 2.5% degradation.
  • Perturbations cause an 8.8% reliability drop, while ReAct achieves 2.5% higher surface volume than Reflexion.
  • GPT-4o costs 82× more than Gemini while providing comparable reliability.
  • The framework is intended to support deployment-readiness decisions as LLM agents move from research prototypes toward production systems.

A Domain Tool Specifications

This appendix specifies the tool interfaces used across scheduling, travel, customer support, and e-commerce tasks, along with configurable fault profiles. The tools cover calendar operations, flight booking, ticket workflows, product search and ordering, and injected failure conditions.

  • Scheduling: Scheduling tools support booking, checking, canceling, and listing meetings across dates and time ranges.
  • Travel: Travel tools search available flights, place temporary holds, and confirm bookings with passenger and payment information.
  • Customer support: Customer-support tools create, search, list, update, close, and escalate tickets, including knowledge-base search.
  • E-commerce: E-commerce tools search products, check inventory, create orders, inspect order status, process returns, and apply coupons.
  • Fault profiles: Fault profiles range from a baseline with zero failure rate to light and heavier configurations with increasing failure rates and weighted fault types.

C.1 Scheduling Task

The scheduling example defines a meeting-booking request and its verified calendar end state. It provides a concrete task instance for evaluating whether the agent produces the intended calendar state.

  • The task asks the agent to book a meeting about “Review” on 2026-01-01 at 09:00.
  • The verified end state records “Review” in the calendar at 09:00 on 2026-01-01.

C.2 Travel Task

The travel task requires booking the cheapest flight from LON to PAR on 2026-01-05 for Bob. Correctness is verified by confirming AA-500 is reserved for Bob with confirmed status.

  • The task asks the agent to book the cheapest flight from LON to PAR on 2026-01-05 for Bob.The listed options are BA-200 at price 500 and AA-500 at price 300.
  • AA-500 is the cheapest listed option, priced at 300 versus 500 for BA-200.
  • The verifier requires reservations["AA-500"].passenger == "Bob" and status == "confirmed".
Loading 2601.06112v1…