Towards a Science of AI Agent Reliability

• Recent deployment incidents—including database deletion, an unauthorized purchase, and illegal business advice—show that agents judged capable internally can still fail unreliably in practice.These incidents motivate the paper’s central question of how agent reliability should be defined and evaluated (Pan et al., 2025).Source: §1 · p. 1
• Mean task success rates obscure whether failures are predictable or random, whether errors are benign or catastrophic, and how agents respond to perturbations or recognize likely failure.Standard benchmarks primarily track capability while leaving reliability unmeasured.Source: §1 · pp. 1–2
• Across 15 models and two benchmarks, reliability gains lag noticeably behind capability progress, with overall reliability showing minimal improvement despite 24 months of model releases.Improving accuracy alone does not guarantee reliability gains on complex real-world tasks, and frontier providers cluster similarly, indicating an industry-wide plateau.Source: Figure 1 · p. 1, §1 · p. 2
• The paper decomposes reliability into consistency, robustness, predictability, and safety, proposing twelve computable metrics independent of raw accuracy.These dimensions capture repeatability, stability under perturbations, calibrated confidence, and bounded failure severity.Source: §1 · p. 2
• The evaluation targets empirically measurable behavior under natural variation and incidental faults, excluding adversarial attacks, broader value alignment, and safety beyond bounded operational severity.The paper’s reliability profile identifies consistency and predictability as areas requiring immediate research focus.Source: §1 · p. 2

Cross-domain synthesis

• Four dimensions recur across aviation, nuclear power, automotive, and process control: consistency, robustness, predictability, and safety.Their cross-domain recurrence suggests they capture fundamental reliability aspects rather than domain-specific concerns.Source: §2 · p. 2, Table 1 · p. 3

Reliability dimensions

• Reliability requires consistent behavior across identical runs, graceful degradation under perturbations, predictable failure recognition, and consequence-aware assessment of failure severity.These dimensions correspond to deterministic operation, graceful rather than abrupt failure, knowing when failure is likely, and distinguishing failures by their consequences.Source: §2 · p. 2

Cross-domain synthesis

• Existing ML research addresses related issues such as prompt sensitivity, input variation, prompt injection, calibration, and selective prediction, but usually treats them as disconnected phenomena.The safety-critical framing unifies these scattered research efforts into a broader reliability framework.Source: §2 · pp. 2–3

Safety

• Safety-critical reliability evaluates operational consequences, whereas ML safety work has largely emphasized harmful-request compliance or sycophancy.The distinction is between consequence-aware failures on legitimate tasks and behavioral compliance concerns.Source: §2 · pp. 2–3

Synthesis

• Reliability is independent of raw capability: highly capable systems can be unreliable, less capable systems can be reliable within their operating envelope, and capability gains do not guarantee reliability.Independent reliability evaluation is therefore necessary.Source: §2 · p. 3

3. Operationalizing Reliability for AI Agents

• Reliability is operationalized through formal metrics spanning consistency, robustness, predictability, and safety.Table 2 provides formal definitions for the full metric suite.Source: §3 · p. 3, Table 2 · p. 4

3.1. Consistency (RCon)

• Consistency measures repeated outcome, trajectory, and computational or monetary resource variability under identical conditions.Its metrics capture success or failure consistency, action-frequency and ordering consistency, and cost or latency variation.Source: §3.1 · p. 3

3.2. Robustness (RRob)

• Robustness measures performance preservation under infrastructure failures, semantic-preserving environment changes, and equivalent prompt reformulations.The framework emphasizes retries or fallbacks for tool errors and invariance to changes such as reordered fields, date formats, parameter names, rephrasings, or translations.Source: §3.2 · p. 3

3.3. Predictability (RPred)

• Predictability evaluates whether confidence reflects actual performance through calibration, discrimination, and the Brier score.These metrics support decisions about whether users should act, verify, or defer.Source: §3.3 · pp. 3–4

3.4. Safety (RSaf)

• Safety separately quantifies constraint compliance and the severity of harmful violations caused by external interactions or irreversible side effects.Harm severity is conditioned on violating tasks, distinguishing how bad violations are from how often they occur.Source: §3.4 · p. 4

3.5. Aggregation

• The aggregation framework averages reliability pillars by default, while excluding safety from the overall score and treating safety deterioration as a hard constraint.It equally weights consistency categories, can omit trajectory consistency for creative workflows, and permits context-specific weights or metric selection.Source: §3.5 · p. 5

4.1. Setup

• The evaluation covers GAIA’s 165 validation tasks and verified 26-task τ-bench subset across 15 models, using repeated runs, paraphrased prompts, injected faults, environment changes, confidence assessment, and safety analysis.GAIA tests browsing, file manipulation, and multi-step reasoning across three difficulty levels; τ-bench tests multi-turn customer-service actions, and the clean subset addresses 24 flawed original tasks.Source: §4.1 · pp. 5–6

4.2. Main Results

• Reliability improves only modestly over 24 months: τ-bench shows moderate gains, while GAIA shows barely any improvement, even among the latest models.These results indicate that improving raw task performance may not be sufficient for dependable agents and may require targeted reliability improvements beyond capability scaling.Source: §4.2 · p. 6
• Consistency remains low across models: agents often solve tasks inconsistently, vary execution order despite similar action choices, and allocate resources unpredictably, especially on GAIA.The gap between pass@k and pass∧k reflects low outcome consistency, while higher distribution than sequence consistency suggests a “what but not when” pattern.Source: §4.2 · p. 6, Figure 2 · p. 6
• Robustness is asymmetric: fault and environment robustness show ceiling effects, whereas sensitivity to superficial prompt paraphrases varies substantially across models.Thus, models may handle genuine technical failures more gracefully than surface-level changes in task specifications, although the evaluated perturbations cover only part of real-world shifts.Source: §4.2 · p. 6, Figure 3 · p. 7
• Predictability improves unevenly: calibration has noticeably improved, especially for Claude models, while discrimination improves on τ-bench but not GAIA and worsens in some recent models.Calibration and discrimination must both be measured because better confidence alignment alone does not ensure that agents identify likely failures.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
• Safety violations are lower in recent frontier models, but high-severity failures remain consequential despite generally high harm-severity scores.On τ-bench, financial-accuracy violations are most prevalent, while unauthorized data exposure and other rare severe failures remain critical deployment blockers.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

Recommendation 1

• Evaluate reliability with dynamic benchmarks that combine repeated runs, systematic perturbations, generative test sets, and temporal re-evaluation.These protocols expose variance, adaptation to changed inputs and infrastructure, shortcut behavior, and degradation as environments drift.Source: §5 · p. 9

Recommendation 2

• Optimize agent architectures for reliability dimensions, because calibration and safety have advanced noticeably while consistency and discrimination have improved little.Systematic evaluation reveals uneven progress and identifies missing dimensions as optimization targets beyond capability-oriented evaluation.Source: §5 · p. 9

Recommendation 3

• Use reliability metrics and incident analyses to guide deployment decisions, change management, compliance, and continuous improvement.Organizations could require minimum consistency and safety thresholds before moving agents from sandboxed pilots to production, alongside incident reporting and post-mortem analysis.Source: §5 · p. 9

Recommendation 4

• Scale reliability requirements with agent autonomy, since human review can buffer imperfect assistants but autonomous outputs directly become failures.An agent succeeding on 90% of tasks may be useful with human approval yet unacceptable when operating autonomously and failing unpredictably.Source: §5 · p. 9

• The evaluation covers only two benchmarks, τ-bench and GAIA, and uses one well-performing scaffold per benchmark, limiting generalizability across tasks and agent designs.Other scaffolds could produce qualitatively different reliability profiles.Source: §6 · p. 9
• Safety evaluation relies on LLM-based judging, which introduces additional reliability concerns.Source: §6 · p. 9
• Metric selection and normalization-based aggregation involve subjective design choices, alternate decompositions are possible, and R does not capture the full picture because safety is reported separately.Separate safety reporting is intended to avoid masking tail risks.Source: §6 · p. 9
• Using temperature zero may overestimate achievable reliability when nonzero temperatures maximize accuracy.Source: §6 · p. 9
• The framework is presented as a starting point, with reliability evaluation complementing rather than replacing human oversight, sandboxed testing, and monitoring.The authors also address objections that reliability is redundant with sufficient capability or that its dimensions are not universally desirable in Appendix B.Source: §6 · p. 9

Conclusion

• Across 15 models and two benchmarks, 24 months of capability gains yielded only small reliability improvements, with more accurate models remaining inconsistent, brittle, and poor at judging when they will succeed.These limitations motivate treating reliability as a key evaluation concern for high-stakes agent deployment.Source: §7 · p. 9
• The proposed safety-critical framework shifts evaluation from success frequency to predictable, consistent, robust, and safe behavior through four dimensions and twelve sub-metrics.The decomposition is presented as one possible framing, while the broader shift in evaluation perspective is emphasized as the central contribution.Source: §7 · p. 9

Impact Statement

• Reliability metrics can guide deployment decisions, agent development, governance frameworks, and principled comparisons of deployment suitability.Organizations could require consistency and predictability thresholds for customer-service agents and stricter safety requirements for code-generation agents performing destructive operations.Source: Impact Statement · p. 10
• Reliability scores can be gamed, creating false security if optimization does not address underlying failure modes.The warning applies even when scores appear improved through metric-focused optimization.Source: Impact Statement · p. 10
• High reliability does not guarantee safe or beneficial behavior because an agent may remain misaligned with user intent.Consistency, robustness, and calibration alone are insufficient to establish alignment.Source: Impact Statement · p. 10
• Reliability evaluation should complement, not replace, human oversight, sandboxed testing, incident monitoring, and ongoing assessment.Treating reliability as sufficient could justify premature deployment.Source: Impact Statement · p. 10
• Reporting reliability profiles alongside accuracy can help downstream users make informed deployment decisions and give regulators concrete, auditable readiness criteria.This broader reporting goal depends on treating reliability as a standard evaluation dimension.Source: Impact Statement · p. 10
• Realizing these benefits requires community effort toward standardized benchmarks, shared evaluation protocols, and transparent reporting norms.These infrastructure and reporting practices support reliable adoption across developers, users, and regulators.Source: Impact Statement · p. 10

Recommendation 1

• Benchmarks should use repeated, perturbed, generative, parameterized, and temporally refreshed tests rather than single-run accuracy in fixed environments.Such designs assess variance, adaptation to distribution shifts, infrastructure changes, and genuine problem solving while reducing shortcut behavior (Kapoor et al., 2025).Source: A. Extended Recommendations · p. 16

Recommendation 2

• Agent architectures should optimize explicit reliability dimensions alongside capability, because calibration and safety have improved more than consistency and discrimination across model generations.Measurable dimensions reveal uneven progress and provide targets for systematic optimization, even when current training objectives are not transparent.Source: A. Extended Recommendations · p. 16

Recommendation 3

• Reliability metrics and incident analyses should guide deployment decisions, change management, regulatory compliance, and continuous improvement through thresholds, reporting, and post-mortems.The framework also supports contributions spanning metrics, benchmarks, algorithms, interfaces, and governance.Source: A. Extended Recommendations · p. 16

Recommendation 4

• Reliability requirements should scale with autonomy: moderate reliability may suffice with human oversight, but unpredictable failures make high reliability a prerequisite for automation.In augmentation settings, humans review outputs; in automation settings, the agent’s output becomes the final action without a human buffer.Source: A. Extended Recommendations · pp. 16–17

• The evaluation covers only two benchmarks and one strong scaffold per benchmark, so other tasks and scaffolds may produce different reliability profiles.The authors plan to extend evaluation to scaffolds such as Claude Code and OpenAI Codex.Source: B. Extended Limitations · p. 17
• Safety evaluation relies on LLM judges, while reporting safety separately preserves tail-risk visibility but leaves aggregate R incomplete.The authors propose future judge-free or human-validated safety metrics and principled safety integration.Source: B. Extended Limitations · p. 17
• Metric selection and capability disentanglement involve subjective, nonunique choices whose adequacy may vary across applications and task domains.Practitioners may reasonably disagree about suitable metrics and decomposition strategies.Source: B. Extended Limitations · p. 17
• Setting temperature to zero limits stochasticity and may overestimate achievable reliability when nonzero temperature is needed for accuracy.This limitation is especially relevant when deployment requires balancing reliability against sampling-driven performance.Source: B. Extended Limitations · p. 17
• Reliability is not redundant with capability: models with equal accuracy can differ in predictability and robustness, including fixed versus unpredictable failures and sensitivity to minor input changes.The redundancy objection holds only at perfect accuracy, a regime current models have not reached.Source: B. Extended Limitations · pp. 17–18
• Reliability dimensions are deployment-dependent rather than universally maximized, so practitioners can weight, exclude, or reinterpret them according to application goals.Consistency may suit CI/CD but conflict with brainstorming, creativity, exploration, or adaptive solution paths.Source: B. Extended Limitations · p. 18

C. Extended Metric Details

• The appendix extends Section 3 with metric design principles, notation, derivations, interpretations, and implementation guidance.Source: C. Extended Metric Details · p. 18

C.1. Why Each Metric Matters: Intuitive Examples

• Concrete deployment scenarios explain the reliability concern captured by each metric and the practical consequences of poor performance.Source: §C.1 · p. 18

C.1.1. CONSISTENCY

• Consistency metrics distinguish repeatable outcomes, action distributions and sequences, and resource use, while trajectory diversity may be treated diagnostically when flexibility is valuable.Cout separates stochastic unreliability from capability; Cd_traj measures action-type similarity, Cs_traj action-order similarity, and Cres cost variability across successful runs.Source: §C.1.1 · pp. 18–19

C.1.2. ROBUSTNESS

• Robustness metrics assess whether agents preserve accuracy under transient faults, semantic-preserving environment changes, and equivalent prompt paraphrases.Rfault covers retries, alternatives, and transparent reporting; Renv targets structural and schema changes; Rprompt prevents dependence on users discovering magic wording.Source: §C.1.2 · p. 19

C.1.3. PREDICTABILITY

• Predictability requires both confidence accuracy and confidence ranking: Pcal measures calibration, Pauroc discrimination, and Pbrier their joint quality.High discrimination can support selective automation even with poor calibration, whereas Brier rewards confidence that is both calibrated and correctly ranked.Source: §C.1.3 · pp. 19–20

C.1.4. SAFETY

• Safety metrics distinguish rule adherence from consequences, with Scomp tracking compliance and Sharm measuring the severity of failures beyond task accuracy.Compliance captures violations even without immediate harm, while harm severity differentiates benign errors from irreversible data loss or other serious damage.Source: §C.1.4 · p. 20

D. Extended Background

• The extended background adds detailed case studies of real-world agent failures and a survey of reliability practices in safety-critical engineering domains.Source: D. Extended Background · p. 20

D.1. Real-World Agent Failures

• Real-world incidents show that agents can provide confident misinformation without uncertainty or escalation mechanisms, causing direct financial and legal consequences, while long conversations can produce hallucinations and persona drift.The Air Canada tribunal held the airline responsible for chatbot information, and Microsoft imposed conversation-length limits after Bing Chat reliability degraded over extended interactions.Source: §D.1 · p. 21

D.2. Reliability in Safety-Critical Domains

• Safety-critical engineering motivates four reliability dimensions: consistency, robustness, predictability, and safety, covering repeatability, perturbation stability, characterized failures, and consequence-aware harm.Suggested measures include outcome variance, trajectory similarity, input sensitivity, calibration, selective operation, cost structure, tail risk, and catastrophe avoidance.Source: §D.2 · pp. 21–22

D.3. Relationship to Classical Reliability Engineering

• Classical reliability assumptions do not directly transfer to stochastic, open-ended agents, so the framework offers computable behavioral measures as a first step toward formalization.The framework retains the term reliability, focuses on agent behavior rather than infrastructure availability or maintainability, and treats measurement and formalization as parallel efforts.Source: §D.3 · p. 23

E. Extended Research Agenda

• The extended agenda calls for reliability evaluation to become standard in model and scaffold releases, with interdisciplinary collaboration and online signals that enable proactive intervention.It also proposes extended-session modeling, multi-agent extensions, and direct optimization for reliability rather than capability alone.Source: E. Extended Research Agenda · p. 23

E.1. Defining and Operationalizing Reliability

• Future work should standardize reliability metrics and failure taxonomies while determining whether reliability emerges with scale or requires explicit architectural support.The agenda distinguishes transient from persistent errors, maps error types to reliability dimensions, and calls for studies controlling model size, training data, and scaffolding.Source: §E.1 · p. 24

E.2. Long-Horizon and Stateful Reliability

• Long-horizon research should model error accumulation, state drift, and session reliability over multi-hour or multi-day operation, including persistence, interruptions, partial progress, and recovery.Proposed infrastructure includes metrics for state divergence and mechanisms for checkpointing, rollback, and reliable resumption.Source: §E.2 · p. 24

E.3. Robustness to Distribution Shift and Adversaries

• Robustness research should evaluate realistic distribution shifts and agent-specific adversarial threats across tools, memory, and multi-turn interaction, while measuring defensive capability–robustness tradeoffs.Candidate defenses include input filtering, sandboxing, redundant verification, and compositional combinations of lightweight mechanisms.Source: §E.3 · pp. 24–25

E.4. Multi-Agent Reliability

• Multi-agent reliability research should trace error propagation, design aggregation robust to correlated failures, and develop theory for when collective systems outperform or underperform their components.The agenda highlights hallucination amplification, ensemble diversity, voting, debate, arbitration, sequential pipelines, and hierarchical delegation.Source: §E.4 · p. 25
• Multi-agent reliability requires causal methods to attribute incorrect or harmful outputs across agent boundaries for debugging and governance.Source: §E.4 · p. 25

E.5. Online Monitoring and Intervention

• Reliable online monitoring depends on external predictive signals, including uncertainty, anomaly scores, tool-error patterns, action entropy, and context utilization, rather than agent self-reports alone.Source: §E.5 · p. 25
• Monitor architectures must balance coverage and guarantees while avoiding correlated failures, since meta-agents inherit agent limitations whereas rule systems may miss novel failures.Source: §E.5 · p. 25
• Intervention policies must trade off false alarms against missed detections, with adaptive thresholds potentially incorporating task criticality, action reversibility, and accumulated session risk.Source: §E.5 · p. 25
• Structured incident databases could convert post-mortem failures into cross-organization learning by standardizing metadata on dimensions, severity, root causes, and environmental conditions.Source: §E.5 · p. 25

E.6. Specification and Verification

• Specification research must balance expressive but ambiguous natural-language constraints against precise but difficult formal specifications, with hybrid intent-and-safety constraints proposed as a practical middle ground.Testing should use distributional pass criteria and systematic prompt and tool variation, while verified wrappers may provide runtime guarantees around larger agents.Source: §E.6 · pp. 25–26

E.7. Human–Agent Interaction

• Human–agent interaction research should calibrate trust and communicate uncertainty using empirically derived reliability estimates, interpretable signals, and adaptive delegation policies.Agents with high consistency and safety may be delegated more fully, whereas low-predictability tasks may require human checkpoints.Source: §E.7 · p. 26

E.8. Lifecycle Reliability and Governance

• Lifecycle governance should integrate reliability regression testing, structured change-control assessments, standardized incident reporting, and domain-specific reliability thresholds.These processes address model, prompt, and scaffold updates, support root-cause analysis and cross-organization benchmarking, and connect reliability research with regulated deployment.Source: §E.8 · pp. 26–27

F. Extended Experimental Details

• The appendix documents benchmark, agent, and protocol details, including consistent cloud execution, network access for GAIA, isolated τ-bench environments, and trace logging for cost analysis.GAIA uses network-enabled browsing, whereas τ-bench runs in simulated isolated environments; Weave logs API calls for cost and trace inspection.Source: F. Extended Experimental Details · p. 27, §F.2 · p. 30

F.1.1. τ-BENCH

• τ-bench evaluates multi-turn customer-service agents using deterministic database-state matching, policy constraints, and realistic failures such as incorrect information or incomplete transactions.It covers airline and retail domains with user personas, instructions, database states, and policy rules. (Yao et al., 2024)Source: §F.1.1 · pp. 27–28

F.1.2. GAIA

• GAIA benchmarks AI assistants on tasks requiring multi-step reasoning, tool use, and information integration across multiple sources, evaluating answers by exact match against ground-truth responses.Its Level 3 tasks require 6+ steps, complex tool chains, sophisticated reasoning, and planning.Source: §F.1.2 · p. 28

F.2. Implementation

• The study evaluates 15 models from three providers with standardized HAL execution, automatic cost tracking, reproducible protocols, and benchmark-specific scaffolding.τ-bench uses structured tool calling, while GAIA uses a ReAct-style loop with web, code, and file tools; deterministic generation settings isolate capability from sampling variance. (Kapoor et al., 2025)Source: §F.2 · p. 29

F.3.1. PROMPT PERTURBATION PROTOCOL

• Prompt robustness is tested with semantic-preserving paraphrases at four perturbation strengths, generated once per benchmark and cached for reproducibility.The protocol targets whether agents understand task intent rather than memorize phrasing, and default results use naturalistic prompt variation. (Rprompt)Source: §F.3.1 · p. 30, Table 4 · p. 30
• Prompt perturbation preserves task information while introducing realistic conversational variation, including casual phrasing, self-corrections, hedging, asides, typos, missing articles, and fragments.For simulated-user benchmarks such as τ-bench, style directives are injected into the simulator’s system prompt so follow-up messages maintain the perturbation style.Source: §F.3.1 · pp. 30–31

F.3.2. FAULT INJECTION PROTOCOL

• Fault robustness injects seven production-like API or tool failure types probabilistically, then permits up to 3 recovery attempts with increasing recovery probability and exponential backoff.Each wrapped call has a default 20% fault probability; recovery probability follows precover(i) = 0.3 + 0.2i for i ∈ {0, 1, 2}.Source: §F.3.2 · pp. 31–32

F.3.3. ENVIRONMENT ROBUSTNESS PROTOCOL

• Environment robustness measures agent sensitivity to semantically preserving changes in data formats and tool interfaces, evaluated with benchmark-specific perturbations at mild, medium, and severe intensity levels.GAIA perturbations modify question text and tool outputs, whereas TauBench perturbations target tool responses and parameter definitions.Source: §F.3.3 · pp. 32–33

F.3.4. CONFIDENCE ESTIMATION PROTOCOL

• After completing a task, an agent assesses confidence in answer correctness using its full conversation history to reflect on execution, with numeric responses parsed and normalized to [0, 1].For typical frontier-model users, post-hoc self-assessment is practical because standard interfaces do not expose logits or internal representations, while alternatives require labeled calibration data or add latency and cost.Source: §F.3.4 · pp. 33–34

F.3.5. SAFETY EVALUATION PROTOCOL

• Safety evaluation uses gpt-4o to assess constraint compliance and classify execution errors by context, evidence, and severity.Compliance checks predefined, benchmark-specific constraints, while harm analysis assigns numeric severity scores and maps them to categorical levels.Source: §F.3.5 · pp. 34–35

G.1. Connection to Real-World Failures

• Systematic evaluation with the proposed reliability metrics could have identified each vulnerability in the reviewed real-world failures before deployment.The analysis maps failures from the Replit agent, OpenAI Operator, and NYC chatbot to reliability dimensions and early-warning metrics.Source: §G.1 · p. 37, §F.3.5 · p. 37

G.2. Reliability trends over time and accuracy

• Many reliability metrics improve only marginally with model release date and accuracy, except predictability and safety on τ-bench and robustness on GAIA.τ-bench’s structured environment shows stronger capability-reliability alignment, whereas GAIA’s open-ended web interaction produces weaker correlations and slower progress.Source: §G.2 · p. 37

G.3. Full consistency results

• Outcome consistency is hardest to achieve, while models show higher trajectory-distribution than sequence consistency, indicating similar action types but varying execution order.τ-bench generally has higher consistency than GAIA, and GAIA has lower resource consistency because web content and external tools vary.Source: §G.3 · p. 37

G.4. Full predictability results

• Calibration has improved across generations, especially for Claude models, whose confidence estimates increasingly track actual success rates on τ-bench and GAIA.Earlier models were frequently overconfident, whereas newer models better temper aggregate confidence through post-training alignment.Source: §G.4 · p. 38
• On τ-bench, newer models improve somewhat at discriminating successes from failures, but GAIA discrimination stagnates or degrades despite better aggregate calibration.Structured task features make τ-bench outcomes more predictable, while open-ended web interactions make individual failures harder to anticipate.Source: §G.4 · p. 38
• Calibration and discrimination must be tracked independently because accurate average confidence does not guarantee useful per-task warnings about likely failures.The results show that progress in one predictability component does not entail progress in the other.Source: §G.4 · p. 38

G.5. Full robustness results

• Fault and environment robustness largely saturate: most models recover reliably from tool errors, timeouts, and malformed API responses, with scores near 1.0 on τ-bench.The perturbation suite omits schema migrations, API version changes, and shifting document layouts, limiting coverage of deployment disruptions.Source: §G.5 · p. 38
• Prompt robustness varies widely: rephrasings cause minimal losses for most frontier models on τ-bench but pronounced drops for several models on GAIA.τ-bench constrains valid tool-call sequences, whereas GAIA permits misinterpretations to compound through divergent web-search strategies.Source: §G.5 · pp. 38–39, Figure 17 · p. 49

G.6. Level-stratified analysis on GAIA

• On GAIA, accuracy degrades with difficulty, and the gap between stronger and weaker models widens substantially at L3 Hard.Gemini and Claude models use dramatically more actions on medium and hard tasks, suggesting greater compute investment on difficult problems.Source: §G.6 · p. 39, Figure 20 · p. 52
• Resource consistency becomes less predictable on complex GAIA tasks, while outcome consistency varies because some models repeat accuracy-dependent failure modes.Calibration and discrimination also degrade modestly on harder tasks on average, with considerable model-specific variation.Source: §G.6 · p. 39
• Fault and environment robustness remain relatively stable across GAIA difficulty levels, indicating robustness is largely orthogonal to task complexity.Models that handle perturbations well on easy tasks generally do so on hard tasks as well, suggesting architecture or training effects.Source: §G.6 · p. 39, Figure 20 · p. 52
• Consistency analyses show agents usually choose similar actions but vary in execution order, while outcome consistency remains the most challenging metric across benchmarks.Resource consistency also varies noticeably in compute usage, particularly on GAIA.Source: Figure 11 · p. 43
• Selective prediction is modest on GAIA, especially for Gemini 2.5 Flash, GPT-5.2, and Claude Opus 4.5, but largely fails on τ-bench against the random baseline.Only Claude Opus 4.5 and 4.7 show modest selective prediction ability on τ-bench.Source: Figure 15 · p. 47, Figure 16 · p. 48