PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability Engineering for AI Systems: Challenges, Methods, and Directions

Rong Pan, Yili Hong, Min Xie

arXiv:2609.35316v1cs.AIstat.AP

TL;DR

Capability scores alone do not establish whether increasingly autonomous AI systems operate dependably and safely across their intended conditions. The paper adapts reliability engineering into a program for defining, diagnosing, testing, and monitoring AI systems. Its case evidence shows exposure-adjusted disengagement trends differ across manufacturers, while the analysis also identifies boundaries on what those records establish.

  • Problem

    Capability measures do not establish dependable, safe performance across repeated operational demands, including retrieval, memory, tools, permissions, oversight, and interactions.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The paper adapts reliability engineering methods into a program using failure definitions, FMEA, matched metrics, TEVV, monitoring, and FRACAS.

    Source: §1 · p. 2

  • Results

    Exposure-adjusted disengagement trends differ: Cruise and Pony AI remain negative, Zoox remains flat, and Waymo’s β = −0.021 (SEQ = 0.016) is not distinguishable from zero.

    Source: §6 · pp. 12–13

  • Takeaways & Limitations

    Self-updates should be treated as versioned repairs with independent TEVV and a FRACAS comparison before and after the change.

    Source: §7 · p. 15

    Disengagement is a human-gated surrogate that misses hazardous behavior without takeover.

    Source: §6 · p. 13

Abstract

from arXiv · show

AI reliability concerns whether an AI system performs its intended function dependably over a stated period and under stated operating conditions, with stated evidence. As these systems become more autonomous, that function includes more than a correct output. Retrieval, memory, tool use, permissions, human oversight, and interactions among systems must operate consistently and safely, and, for generative systems, so must the reasoning process that produces the output. Average benchmark accuracy measures capability; it does not quantify this broader reliability claim. This paper adapts established reliability engineering methods, from failure definitions and operational envelopes to FMEA, accelerated testing, field monitoring, and reliability growth, to AI systems. A four-level diagnostic framework classifies failures as component, operational-loop, agentic-conduct, or network and governance failures. Test, evaluation, verification, and validation (TEVV), sequential monitoring, and FRACAS create and refresh evidence. SMART provides statistical guidance for measurement, analysis, assessment, and test planning; the NIST AI Risk Management Framework provides organizational guidance for governance, evaluation, monitoring, and mitigation. Three cases illustrate the program: adversarial testing of a convolutional neural network, perception-error propagation, and autonomous-vehicle disengagements. Established reliability engineering provides a usable foundation; new measurements and safety guardrails are still needed as these systems are self-evolving.

1. Introduction

The paper frames AI reliability as a claim about an operational system performing dependably within a stated envelope, not simply a model producing correct outputs. It proposes one engineering program to define, diagnose, test, monitor, and improve these systems.

1. Introduction

  • A deployed AI system includes data, software, tools, memory, human oversight, and interactions with other systems, not just a machine-learning model.As autonomy grows, reliability assessment also covers the operational loop, persistent state, permissions, and actions.Source: §1 · p. 1
  • Capability measures alone cannot establish whether an increasingly autonomous system will perform consistently and safely over time within its operating envelope.Self-evolving systems can change both task performance and the process by which later improvements are generated.Source: §1 · p. 1
  • Reported failures span unsupported generation, inaccurate live-service advice, and failures of containment when agents act.The paper treats these as operational-system failures, not only model mistakes.Source: §1 · p. 2
  • The proposed program defines systems, envelopes, failure modes, and severity, then uses FMEA, matched metrics, TEVV, sequential monitoring, FRACAS, and reliability growth.It uses SMART and the NIST AI Risk Management Framework as starting points, extending them for autonomous and self-evolving systems.Source: §1 · p. 2

2. Defining the Reliability Program

The section defines reliability around a deployed system’s function, operating envelope, and evidence, then maps established statistical and organizational frameworks onto AI. It also distinguishes reliability of generative reasoning from final-answer accuracy.

2.1. Engineering meaning of AI reliability

  • A reliability claim requires a defined system failure, a specified operating envelope, and documented evidence that remains valid while those conditions hold.Failure definitions include unacceptable processes as well as incorrect outcomes; the envelope includes users, inputs, toolchain assumptions, drift, and adversarial conditions.Source: §2.1 · p. 2
  • Capability metrics do not measure dependable performance without unacceptable side effects across repeated operational demands; all-of-k and stateful workflow evaluations better capture that quantity than single-run scores.For generative systems, reliability also concerns whether reasoning traces are sound, grounded, complete, and stable—not only whether the answer is correct.Source: §2.1 · pp. 3–4

2.2. Organizing frameworks: SMART and NIST AI RMF

  • SMART organizes statistical work around structure, metrics, failure-cause analysis, reliability assessment, and test planning, while the NIST AI RMF organizes governance, mapping, measurement, and management.Evidence must match the exact deployed configuration; changes to prompts, policies, models, tools, skills, or workflows require documentation and affected-part retesting.Source: §2.2 · p. 4
  • Established methods such as FMEA, accelerated testing, sequential monitoring, recurrent-event models, reliability growth, and FRACAS still apply, but AI changes how stress, repair, and exposure are interpreted.Examples include input perturbations or corpus change as stress, retraining or gate tightening as repair, and miles, task-hours, or tool calls as exposure.Source: §2.2 · p. 4

3. Diagnosing AI Failures: Levels, FMEA, and Multi-Agent Modes

The section diagnoses AI failures at four complementary levels, uses AI-FMEA to connect failure modes with causes and controls, and examines how multi-agent structures propagate information loss and errors.

3.1. A four-level diagnostic framework

  • AI failures span components, operational loops, agent conduct, and network or governance arrangements, and a severe incident may cross several levels.The loop can fail through tool selection, stale memory, skipped validation, or stopping failures; conduct concerns whether resulting actions stay within the authorized envelope.Source: §3.1 · pp. 5–6

3.2. AI-FMEA: failure modes, causes, signals, and controls

  • AI-FMEA connects failure modes with their causes, observable detection signals, and engineering controls across technical and organizational conditions.The listed conditions include data, models, tools, operating environments, and human or organizational processes.Source: §3.2 · pp. 6–7

3.3. Multi-agent information loss and collective failure

  • Adding agents does not automatically improve reliability: stages that add no decision-relevant information can reprocess shared evidence and lose details during handoffs.The design question is whether stages add independent evidence, preserve sources, perform meaningful checks, or direct review to uncertain or severe cases.Source: §3.3 · p. 7
  • Chains can degrade evidence, hubs create single points of failure, and dense networks can reinforce false consensus; field failure-rate estimates require deployment records and exposure.Suggested controls preserve independent judgments, provenance, and uncertainty, while evaluation can track propagation depth, cascade radius, error amplification, and recovery.Source: §3.3 · p. 8

4. Measuring Reliability and Building Evidence

Reliability assessment must measure performance across reasoning processes, task trajectories, and operating conditions, using evidence suited to the event and data type. TEVV, field monitoring, action controls, and FRACAS keep that evidence current and connect failures to corrective action.

4.1. Reliability metrics and statistical models

  • Generative-system reliability measures process soundness, grounding, and stability alongside outcome accuracy, with predictability scored against process correctness as well as the final answer.Faithfulness checks and LLM judges can screen reasoning, but neither makes the reasoning trace a certificate of validity.Source: §4.1 · p. 8
  • Assess trajectories and severity, not only final answers: a correct answer may follow unsafe tool calls, while safe abstention or escalation can count as reliable behavior.Predictability requires both discrimination and calibration when confidence thresholds govern acting, abstaining, or escalating.Source: §4.1 · p. 9
  • Reliability measurement should match the evidence type and define the event, exposure, and sampling process before analysis, rather than rely on mean task success alone.Binary and count data support GLM-style rates; continuous scores support robustness surfaces; time-to-event and recurrent-event data support NHPP and software-reliability growth models.Source: §4.1 · p. 9

4.2. TEVV and accelerated testing

  • Pre-deployment TEVV tests representative conditions and operational boundaries, then field monitoring checks whether that evidence holds in actual use.Incidents and detected changes inform corrective action and renewed testing.Source: §4.2 · p. 9
  • Agent TEVV should exercise complete task trajectories, including ambiguous instructions, tool failures, malicious content, high-risk actions, and persistent-memory or multi-agent failure modes.Tests record permissions, information validity, reasoning quality, abstention, and side effects; production-like perturbations and adversarial tests probe different sensitivities.Source: §4.2 · pp. 9–10
  • Accelerated testing concentrates evidence in informative regions through increased task cycles, perturbed inputs, or adverse environments, then checks degradation, detection, containment, and recovery.Space-filling and mixture designs can vary interacting factors efficiently; agent tests may inject trap content, corrupted memory, unauthorized targets, or faults.Source: §4.2 · p. 10

4.3. Field monitoring, FRACAS, and reliability growth

  • Field monitoring combines input and output checks with action monitoring before external effects, using independent enforcement and severity-scaled responses.Measures include missed high-severity actions, false alarms, detection and containment time, and unsafe actions intercepted before external effects.Source: §4.3 · p. 10
  • FRACAS models interruptive failures as recurrent events with system covariates, linking incidents and corrective actions to reliability growth.For persistent agents, the loop also covers state validation, deletion, audit, and recovery.Source: §4.3 · p. 10

5. Controls and Logging for Agents

Adversarial information can influence agent reasoning, memory, and actions, so the paper translates these exposures into enforceable operating principles. Independent controls and reconstructable event records make failures more detectable, containable, and assessable.

5.1. From adversarial exposure to enforceable principles

  • Adversarial informational exposures—including content injection, semantic manipulation, and coordinated steering—can affect reasoning, memory, and action, so they belong in TEVV factors rather than field-rate estimates.Some effects may persist after the original content disappears.Source: §5.1 · p. 11
  • Information quality, memory governance, and action discipline require verified information, controllable persistent state, and constrained external effects; authorization must be enforced externally.The specified controls include least privilege, step-up verification, reversible transactions, and explicit stop conditions.Source: §5.1 · p. 11
  • Cybersecurity test actions that reach real systems depart from the test envelope, and behavioral instructions alone may not ensure dependable containment.The paper identifies isolated sandboxes, default-deny network access, allowlisted targets, and kill switches as external controls.Source: §5.1 · p. 11

5.2. Control and event-record requirements

  • An independent action mediator checks proposed external effects, while trajectory logs preserve enough task, configuration, exposure, action, and recovery lineage to analyze failures and repairs.Such records support exposure-adjusted failure rates, recurrent-event analysis, and assessment of whether a repair improved reliability.Source: §5.2 · p. 11

6. Case studies

Three cases illustrate how established reliability models can analyze AI failures across public-road testing, laboratory attacks, and simulated perception-error propagation. The cases also identify records needed to extend the program to tool-using agents.

Cross-case implications

  • The three illustrations apply recurrent-event analysis, accelerated-test logic, and dependent-failure modeling; extending the program to agents requires exposure-linked event logs and relevant covariates.Proposed records include task-hours or tool calls and covariates for version, corpus, memory, and policy gates.Source: §6 · pp. 12, 14, Table 8 · p. 14

Case A (field recurrent events)

  • The California analysis covers 479 disengagements from four manufacturers during December 2017–November 2019, using monthly mileage as exposure.Disengagements are exits from autonomous mode rather than crashes.Source: §6 · p. 12
  • Cruise (β̂ = −0.072, SEQ = 0.012) and Pony AI (β̂ = −0.146, SEQ = 0.042) show measurable improvement per mile; Waymo (β̂ = −0.021, SEQ = 0.016) does not.Zoox remains flat, and cumulative counts versus exposure-adjusted intensity produce different manufacturer rankings.Source: §6 · pp. 12–13, Table 7 · p. 13, Figure 3 · p. 13
  • Disengagements are a human-gated surrogate that misses hazardous behavior without takeover, and collision events require an explicit severity model before pooling.The analysis aggregates vehicle heterogeneity; Zoox’s short exposure makes its flat slope weakly identified.Source: §6 · p. 13

Case B (laboratory ALT)

  • Faddi et al. (2025) use designed FGSM and PGD attacks on a CIFAR-10 CNN to track misclassification and accuracy over adaptive retraining, rather than a single robustness score.Learning rate, attack mix, and ε improve defect-discovery tracking, while accuracy can fall and recover.Source: §6 · p. 13

Case C (virtual module tests)

  • Pan et al. (2024) inject detection and localization errors in simulation and model recurrent module errors, with triggering intensity representing propagation between dependent modules.Weather and fault-injection schedule are designed factors; the study is a virtual accelerated test with ground truth for injected errors.Source: §6 · pp. 13–14

7. Conclusion and research directions

Established reliability engineering can be adapted to AI by defining system-specific failures and operating conditions, then building evidence through testing, monitoring, and corrective action. Research directions center on governing self-updates and collecting detailed field records to support reliability estimates.

7. Conclusion and research directions

  • Reliability practice must adapt failure definitions, operating conditions, FMEA, TEVV, monitoring, and FRACAS to each AI system’s function and autonomy.Stress can include input perturbations or malicious content, repair can include retraining or policy revision, and exposure can be measured in miles, task-hours, or tool calls.Source: §7 · p. 14
  • Before increasing autonomy, engineers should write a reliability case, map failures with AI-FMEA, and test beyond nominal operation.Early field use should require human confirmation, with irreversible actions gated until controls have been shown to hold in the field.Source: §7 · pp. 14–15
  • For self-improving systems, treat each update as a repair with a versioned baseline, independent TEVV, and a before-and-after FRACAS comparison.Retained updates can degrade earlier capabilities, and higher task scores may reflect evaluator exploitation rather than genuine improvement.Source: §7 · p. 15
  • Field failure-rate estimates require validating mechanisms from simulated tasks and constructed faults against field observations.A useful field repository would link failures to exposure and operating context, including versions, tools, memory, policies, agent connections, and reasoning traces.Source: §7 · p. 15
  • Reliable assessment needs event records linked to exposure, system version, trajectory, reasoning trace, severity, and corrective action.Current benchmarks and incident registries lack longitudinal exposure, consistent reporting, and change-controlled field data.Source: §7 · p. 15
  • Detailed field-test data remain difficult to obtain because they are often held by industry.The paper identifies finer-grained industry data sharing as valuable to academic research beyond currently available sources such as California DMV data.Source: §7 · p. 15

Data availability

The California AV disengagement and mileage files distributed with DR-AIR supplied the data used for the reported manufacturer-month totals and quasi-Poisson fits.

Data availability

  • Manufacturer-month totals and quasi-Poisson fits in Table 7 were computed from the California AV disengagement and mileage files distributed with DR-AIR.The files are attributed to Zheng et al. (2026).Source: Data availability · p. 15

Related papers

Loading 2609.35316v1…