Source-linked AI summary
How Do Hardware Faults Manifest Differently in LLM Agents?
Aiden Rubey, Zachary Coalson, Cong Chen, Jean-Philippe Monteuuis, Jonathan Petit, Sanghyun Hong
TL;DR
Prior hardware-fault studies largely assess LLM outputs, leaving unclear how corruption affects longer agent executions. This study evaluates random two-bit weight faults across agent benchmarks and finds that closed-loop execution can produce recovery, added cost, subtle semantic errors, and security- or alignment-relevant changes.
Problem
Prior studies largely treat a fault-corrupted LLM output as the final outcome, leaving its effects within longer agent executions unclear.
Source: §1 · p. 1Method
Results
Agent-level effects differ from model-level corruption: agents may recover from faulty outputs, incur higher costs, produce coherent but semantically wrong actions, or show model- and scenario-dependent security and alignment changes.
Source: §1 · p. 2Takeaways & Limitations
The findings suggest that model-level evaluations alone may not fully capture hardware-fault resilience in agents, whose outcomes also depend on subsequent actions and environment interactions.
Source: §5 · p. 5The study focuses on persistent two-bit weight faults across a limited set of models and benchmarks; generalization to other fault conditions and agentic environments remains open.
Source: §5 · p. 5
Abstract
from arXiv · showhide
We present the first systematic study of the resilience of large language model (LLM) agents to random hardware-induced faults. Prior work on hardware fault resilience in LLMs focuses on model-level outcomes, such as incorrect outputs or generation failures. Yet LLM agents repeatedly interact with tools and environments, creating loops that can change how hardware faults manifest. We conduct an extensive evaluation across three agent benchmarks, comprising 6,814 agent executions: 2,400+ clean runs and 4,300+ fault-injected runs with independently sampled random two-bit weight faults. We trace how these faults propagate through agent execution. Unlike prior studies, we find that agentic execution transforms model-level corruption into distinct system-level outcomes: (1) agents can recover from faulty model outputs and still complete tasks correctly; (2) this recovery increases computational and execution costs; (3) a few faults cause subtle but semantically meaningful errors while the agent otherwise behaves normally; and (4) for some models and scenarios, faults can alter security and alignment behavior.
1 Introduction
Prior hardware-fault studies largely stop at corrupted LLM outputs, leaving unclear how corruption affects longer agent executions. This study traces persistent weight faults through closed-loop action and observation, finding that agent-level outcomes can differ from model-level corruption.
1 Introduction
- Prior studies report generation failures, degraded output quality, and subtle errors in plausible LLM outputs, but largely treat the generated output as the fault’s endpoint.The paper frames agent executions as a setting where corrupted outputs instead become intermediate steps.Source: §1 · p. 1
- Agents repeatedly act, observe results, and decide what to do next, so corrupted outputs can affect later execution and environment state.The study asks what happens when a fault-corrupted model output becomes an intermediate step in an agent trajectory.Source: §1 · p. 1
- The study injects randomly sampled two-bit faults into LLM weights and evaluates complete agent executions across three settings.The supplied method passage ends before specifying further evaluation details.Source: §1 · p. 1
- The study focuses on model weights because their corruption persists across successive LLM invocations.This lets the authors examine how persistent model-state corruption propagates through closed-loop agent execution.Source: §1 · p. 2
- Faulty outputs do not always cause task failure: agents can recover, but faulty executions can increase mean output tokens per execution by 38.0%.Other observed outcomes include coherent but semantically incorrect actions and model- or scenario-dependent security and alignment changes.Source: §1 · p. 2
2 Background and Related Work
Prior work examines failures caused by perturbing agent-stack components and hardware faults within model inference. This paper instead studies random hardware faults across complete, closed-loop agent executions.
LLM Agents and Failure Modes
- Perturbations to tools, API responses, and external inputs can cause incorrect results or inconsistent inputs, followed by replanning, recovery, or cascading failures.The paper distinguishes its setting by placing persistent corruption in the LLM model state itself.Source: §2 · p. 2
Hardware Fault Resilience of LLMs
- LLM hardware-fault studies include inference-quality analysis by Sun et al. [2025] and fault-propagation analysis by LLMFI [Huang et al., 2026].These studies largely treat generated model output as the endpoint; this work examines reliability over complete, closed-loop agent executions.Source: §2 · p. 2
3 Our Fault Injection Framework (Figure 1)
The framework evaluates random two-bit weight corruption across three agent workloads, retaining each benchmark’s metrics to assess task outcomes, execution costs, and consequential behavior.
Fault Model and Injection
- 6,814 executions cover 218 tasks across three benchmarks, with 8–13 clean runs and 15–25 fault-injected runs per task.Each faulted execution flips two bits in one randomly selected weight; the corrupted weight persists, while the runtime, tools, and environment remain fault-free.Source: §3 · p. 3
Agent Workloads and Models
- Three workloads capture complementary agent settings: structured action generation, security-sensitive tool use, and consequential autonomous decisions.Qwen3.6-27B is the primary model; Qwen3.8-27B and Granite4.1-30B are included to examine model dependence.Source: §3 · p. 3
Metrics
- The evaluation retains benchmark-specific metrics across task outcome, execution cost, and consequential behavior.Cost includes additional computation such as generated input and output tokens; consequential behavior uses benchmark-specific security- and alignment-relevant metrics.Source: §3 · p. 3
4 Results
Across four results, agent loops can repair some faulty outputs, but faults also raise execution costs and produce coherent yet semantically wrong actions. Security and alignment effects vary across faults, workloads, and models.
4.1 Closed-Loop Execution Can Recover from Faulty Outputs
- Agents recover from 3.3–25.9% of malformed calls, but never recover from token storms, which contain no valid action.Recovery occurs when agents detect that an intended environment change failed or identify a schema mismatch and reformulate the call.Source: §4.1 · p. 3, Table 1 · p. 3
4.2 Fault Impact Extends Beyond Task Correctness
- On HomeBench, mean output tokens rise 51.3%, from 1,656 to 2,505 per execution, while exact-match accuracy falls 4.2 percentage points, from 75.6% to 71.4%.Pooled across benchmarks, token storms account for 96.0% of the additional output tokens, which increase by 38.0%.Source: §4.2 · p. 4, Table 2 · p. 4
4.3 Some Faults Remain Coherent but Semantically Wrong
- Some faults produce structurally valid but semantically incorrect actions, including wrong numerical interpretations, unjustified transaction updates, and misdirected escalations.Because the actions remain structurally valid, generation validity alone can miss these errors.Source: §4.3 · p. 4, Figure 2 · p. 4
4.4 Security and Alignment Effects Are Model- and Scenario-Dependent
- A replayed Qwen3.6-27B fault raises AgentDojo attack success from 31.0% to 43.0% while preserving task utility, and security and alignment effects vary by model and scenario.On the Slack suite, attack success rises from 57.0% to 92.0%; the same harmful-behavior trend does not hold consistently for Qwen3.8-27B or Granite4.1-30B.Source: §4.4 · p. 4, Table 3 · p. 4
5 Conclusion
This study finds that hardware-fault effects in LLM agents depend on how actions and environment interactions unfold after a corrupted model output. The authors identify limited scope and statistical analysis as open issues for follow-up work.
5 Conclusion
- The study examines random hardware-induced faults in LLM agents beyond model-level outputs.Its conclusion says fault consequences depend on subsequent actions and environment interactions.Source: §5 · p. 5
- The authors argue that resilience may not be fully captured by model-level evaluations alone.They call for system-level evaluation of end-to-end agent execution.Source: §5 · p. 5
- The study is limited to persistent two-bit weight faults across a limited set of models and benchmarks.The authors identify generalization to other fault conditions and agentic environments, and more rigorous statistical analyses, as open directions.Source: §5 · p. 5
A Experimental Setup in Detail
The experiments use a Python and PyTorch fault-injection framework, Inspect AI for agent execution, and two multi-GPU compute systems.
A Experimental Setup in Detail
- The fault-injection framework uses Python 3.11.15, PyTorch 2.11.0, and CUDA 13.0.Agents run using Inspect AI v0.3.123.Source: A Experimental Setup in Detail · p. 7
- AgentDojo and Agentic Misalignment use Inspect Evals2 v0.10.0 implementations, while HomeBench uses its official repository implementation.The passage identifies these as the implementations used for the benchmarks.Source: A Experimental Setup in Detail · p. 7
- Experiments run on systems with eight NVIDIA A40 GPUs and eight NVIDIA H100 GPUs, respectively.The systems have 48 or 112 Intel Xeon CPU cores and 768 GB or 2 TB of memory, respectively.Source: A Experimental Setup in Detail · p. 7
B AgentDojo Utility Results
AgentDojo utility changes under random faults vary across models, with faulted execution producing both decreases and increases relative to clean execution.
B AgentDojo Utility Results
- AgentDojo utility measures resolution of the user’s original task, regardless of whether prompt injection succeeds.The reported changes indicate noticeable but inconsistent effects on agent performance.Source: B AgentDojo Utility Results · p. 8
- The utility changes span both negative and positive values, rather than a consistent direction across models.The reported range is -3.7 to +6.0 percentage points.Source: B AgentDojo Utility Results · p. 8
- -3.7 to +6.0 percentage points: AgentDojo utility changes across models under random faults.Table 4 reports utility under clean and faulted execution.Source: Table 4 · p. 8, B AgentDojo Utility Results · p. 8