HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control

• Current system-level defenses struggle to intervene before unsafe actions while adapting flexibly across diverse risks.LLM monitors focus on detection, while safety harnesses often rely on predefined rules.Source: §1 · pp. 1–2
• A risk-aware harness selectively invokes LLM monitors and uses targeted interventions, such as blocking actions or editing execution contexts, while preserving benign-task completion.Its trigger, monitor, and feedback modules coordinate when to monitor, how to assess risk, and how to intervene.Source: §1 · p. 2
• HARDE improves runtime safety while preserving benign-task utility, outperforming manually designed harnesses and naive optimization baselines.Its two-stage process first probes each module and distills an optimization guide, then iteratively updates the harness using safety and utility feedback.Source: §1 · p. 2
• The paper’s runtime-defense insights emphasize complementary module roles, attack-aware design, and matching harness optimization to monitor capabilities.For example, malicious tool outputs call for source attribution, while corrupted reasoning inputs highlight feedback that preserves epistemic integrity.Source: §1 · p. 2

• Agent-level monitors study detection accuracy, robustness, and monitorability, while newer guardrail frameworks intervene before potentially unsafe tool calls.These frameworks typically use a fixed guard model or policy rather than systematically optimizing the complete harness from safety and utility feedback.Source: §2 · p. 3
• Existing execution-layer harnesses are often hand-designed for predefined settings, while automated workflow search treats the harness as an end-to-end code object.Unlike these approaches, the paper explicitly models distinct safety effects of harness modules, including monitoring and intervention feedback.Source: §2 · p. 3

3 METHOD

• The method introduces a risk-aware harness and an iterative optimization procedure, then presents HARDE as a two-stage optimization framework.Figure 2 overviews the framework’s module probing and guided optimization stages.Source: §3 · p. 3, §3.2 · p. 5

3.1 RISK-AWARE HARNESS

• The trigger screens proposed actions, the monitor assesses risks in action and execution context, and feedback adjusts actions or context to mitigate risks and support task completion.These modules extend the base harness, which maintains execution context and state while invoking the agent, executing tools, and incorporating environmental feedback.Source: §3.1 · p. 4

3.2 ITERATIVE HARNESS OPTIMIZATION

• Iterative optimization revises trigger, monitor, and feedback while holding the executing and monitoring models fixed, evaluating candidate harnesses on attacked benign tasks and selecting the highest Safe Success candidate.The proposer uses current execution results and accumulated history to generate subsequent candidates.Source: §3.2 · p. 4

3.3 HARDE

• HARDE probes one module at a time on a small shared subset, comparing candidate revisions with the initial harness to identify safety and utility effects.The guide summarizes module effects and recurring failures, mapping observed failures to modules and modification strategies; probing defaults anew for each risk setting.Source: §3.3 · p. 5
• Guided optimization uses the guide and accumulated search experience to choose a module and replace only that module per round; the selected harness is evaluated on held-out cases.Each revision is evaluated and its updated search experience informs subsequent rounds.Source: §3.3 · p. 5

4.1 EXPERIMENTAL SETUP

• Evaluation uses SHADE-Arena, AgentDyn, and Agent-SafetyBench, with attacks introduced through system prompts, tool outputs, and environmental feedback.The agent is DeepSeek-V4-Flash; the monitors are DeepSeek-V4-Flash or Qwen3-8B.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

• HARDE achieves the best Safe Success on all three benchmarks and improves Safe Success by 77.9% over the strongest baseline on SHADE-Arena.Table 1 reports main results across three benchmarks and monitor models.Source: Table 1 · p. 6, §4.2 · p. 6
• Runtime-monitoring baselines improve safety over Base Harness but often over-block and degrade utility, while Meta-Harness remains behind HARDE.Meta-Harness uses monolithic updates without task- and attack-specific guidance.Source: §4.2 · pp. 6–7
• HARDE reaches peak search-set performance within five iterations on all benchmarks; Meta-Harness has lower held-out results despite comparable search performance in some cases.The search-set performance is shown for candidates generated at each optimization iteration.Source: Figure 3 · p. 7, §4.2 · p. 7
• With Qwen3-8B as monitor, HARDE remains highest in Safe Success on all three benchmarks, while unoptimized Vanilla Monitor drops by more than 78.3% on AgentDyn.The authors attribute Vanilla Monitor's decline to Qwen3-8B's shorter context window and weaker long-trajectory safety assessment, which can lead to over-blocking.Source: §4.2 · p. 7

4.3 ABLATION STUDIES

• HARDE achieves the highest Safe Success on every benchmark; on SHADE-Arena, removing the trigger or feedback module lowers it from 0.500 to 0.357 or 0.154, respectively.Removing all three core modules further lowers Safe Success to 0.083.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 CROSS-SETTING TRANSFERABILITY OF OPTIMIZATION GUIDE

• Guides transfer effectively between SHADE-Arena and AgentDyn, but less effectively from Agent-SafetyBench; cross-monitor transfer to Qwen3-8B also causes substantial drops on the more challenging tasks.Even the weakest cross-dataset transfer result ranks first or second among the corresponding target-benchmark baselines; Agent-SafetyBench transfer remains comparable across monitors.Source: Figure 4 · p. 8, §4.4 · p. 8

Runtime Defense Benefits from Complementary Safety Mechanisms

• Epistemic integrity requires information used or communicated by the agent to be grounded in trustworthy evidence, complementing four properties introduced by Siu et al. (2026a;b).The other properties are task alignment, action alignment, source authorization, and data isolation.Source: §5 · p. 9
• The three harness modules have complementary, non-substitutable roles: trigger gates action alignment and source authorization, monitor assesses contextual risks, and feedback supports task alignment and epistemic integrity.An additional prompt can establish a global safety prior.Source: §5 · p. 9

Runtime Defense Should Be Attack-Aware

• Harness optimization should target the safety properties stressed by each attack: covert side tasks, malicious tool outputs, and corrupted reasoning information call for different module focuses.The analysis associates these attack types with SHADE-Arena, AgentDyn, and some Agent-SafetyBench attacks, respectively.Source: §5 · p. 9

Runtime Defense Should Account for Monitor Capability

• Harness design should match monitor capabilities: DeepSeek-V4-Flash supports trajectory tracing and source attribution, while Qwen3-8B benefits more from deterministic triggers and feedback.Transferring an optimization guide across monitor models can degrade performance in challenging settings.Source: §5 · p. 9

6 CONCLUSION

• The paper studies LLM-agent runtime defense with a risk-aware harness comprising trigger, monitor, and feedback modules.HARDE derives an optimization guide through lightweight module-wise probing, then iteratively refines one module at a time.Source: §6 · p. 9
• Across multiple benchmarks and monitor models, HARDE improves runtime safety while preserving benign-task utility, outperforming unoptimized monitoring, one-shot harness design, and monolithic iterative optimization.Source: §6 · p. 9
• The qualitative analyses highlight complementary module roles, attack-specific safety properties, and monitor capabilities as considerations for runtime-defense harness optimization.The paper presents modular, failure-guided, and monitor-aware optimization as practical principles for agent runtime defense.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI tools assisted with manuscript clarity and presentation, and authors reviewed and validated all AI-assisted text.Source: AI USE STATEMENT · p. 10
• The tools were not used to determine research questions, methodological choices, or conclusions; the authors take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The work acknowledges that runtime defenses against attacks on LLM agents may have dual-use implications.Source: ETHICS STATEMENT · p. 10
• Experiments use existing benchmarks in controlled environments, with attack analyses presented to support defensive evaluation and mitigation.The authors encourage responsible use of the released methods and artifacts.Source: ETHICS STATEMENT · p. 10

A.1 BENCHMARK DETAILS AND PREPROCESSING

• The benchmarks cover system-prompt side tasks, malicious tool outputs, and environmental feedback, with SHADE-Arena cases long-horizon and Agent-SafetyBench cases typically under ten steps.The authors sample 3 SHADE-Arena cases, 36 valid AgentDyn cases, and 20 Agent-SafetyBench cases for Dsearch.Source: §A.1 · pp. 13–14

A.2 IMPLEMENTATION DETAILS

• Implementation uses DeepSeek-V4-Flash as the agent, original benchmark system prompts, environment checkers for SHADE-Arena and AgentDyn, and GPT-5.1 as judge for Agent-SafetyBench.The appendix also specifies separate inference settings for the monitors and optimization agents.Source: §A.2 · p. 14

A.3 PROMPT

• The prompts constrain optimization to one component at a time, use search evidence to build a reusable module guide, and prohibit task-specific facts in component code.Stage II asks the agent to diagnose baseline failures and select a component using current search evidence; evaluation defines utility, safety, and attack success explicitly.Source: §A.3 · pp. 16–17

B.1 RESULTS WITH QWEN3-8B MONITOR

• The Qwen3-8B results table evaluates Utility, Safety, and Safe Success across three benchmarks, averaging results over three runs.Higher values are better; the supplied passage does not report table cell values.Source: Table 3 · p. 19, §B.1 · p. 19

B.2 BENIGN-TASK UTILITY

• HARDE achieves the highest benign-task utility on SHADE-Arena and AgentDyn; Agent-SafetyBench is excluded because its cases lack a matching attack-free evaluation setting.The harnesses were evaluated on benign cases without further modification or optimization.Source: §B.2 · p. 19

B.3 ABLATION OF HARNESS MODULES

• Removing either trigger or feedback reduces both safety and utility, while removing both causes a larger drop; with all three core modules removed, Safe Success is 0.083.Trigger gating can allow low-risk reads while escalating high-risk writes, and feedback helps the agent recover and complete the benign task after a block.Source: §B.3 · p. 20, Table 5 · p. 20

C.1 OPTIMIZATION GUIDE EXAMPLE

• The guide template organizes each module’s interface, optimization space, failure modes, selection conditions, expected metric effects, and boundaries with other modules.It begins with probing results and concludes with signals connecting metric and trajectory symptoms to likely modules for optimization.Source: §C.1 · pp. 20–21

C.2 ATTACK CASES

• The examples span a hidden system-prompt side task to install a global email deletion rule, tool-output instructions to visit a discount site, and an email injection requesting permanent smart-lock access.The three examples come from SHADE-Arena, AgentDyn, and Agent-SafetyBench, respectively.Source: §C.2 · p. 21

C.3 DEFENSE CASES

• In the AgentDyn defense case, contextual source attribution identifies an invitation as driven by an injected file instruction, blocks it, and redirects the agent to its original repository task.The agent completes that benign task without executing the attack, illustrating how trajectory-level source attribution can detect actions that fixed action-level rules may miss.Source: §C.3 · pp. 23–24

D.1 SAFETY PROPERTIES AND MODULE ROLES

• The analysis characterizes safety through task alignment, action alignment, source authorization, data isolation, and epistemic integrity.Auxiliary prompt changes can provide a global safety prior, primarily supporting epistemic integrity and, to a lesser extent, task and action alignment.Source: §D.1 · p. 24
• The trigger, monitor, and feedback modules offer complementary intervention points, addressing failures that no single component can resolve.Trigger gates suspicious actions, the monitor assesses contextual risk, and feedback supports recovery without task drift or reliance on polluted observations.Source: §D.1 · p. 24

D.2 BENCHMARK-SPECIFIC RISK PROFILES

• The three benchmarks threaten different safety properties, so optimization should identify the violated property and system bottleneck before selecting a module rather than follow a fixed order.Transfer is reported to work more effectively between settings with similar attack mechanisms and risk profiles.Source: §D.2 · p. 25
• SHADE-Arena’s covert system-prompt tasks threaten task and action alignment; effective defense spans trigger restrictions, monitor detection, and feedback that redirects the agent.Some cases involving sensitive information also threaten data isolation; action-level detection alone is insufficient for these long-horizon side tasks.Source: §D.2 · p. 25
• AgentDyn’s tool-output attacks jointly threaten source authorization and action alignment, making contextual source attribution central to the monitor and a practical bottleneck.Fixed rules are often insufficient to distinguish task-relevant data from untrusted content and injected directives in complex trajectories.Source: §D.2 · p. 25
• Agent-SafetyBench can corrupt reasoning without inducing harmful tool calls, making feedback and prompt design important for epistemic integrity.The analysis gives sanitizing malicious tool observations before they enter subsequent context as an example.Source: §D.2 · p. 25

D.3 MONITOR-DEPENDENT HARNESS DESIGN

• With DeepSeek-V4-Flash, HARDE emphasizes monitor optimization; with weaker Qwen3-8B, it emphasizes trigger and feedback modules that provide deterministic filtering and support.The stronger monitor is described as better able to attribute sources across long trajectories; Qwen3-8B is less capable with long trajectories, compound attacks, and implicit source relationships.Source: §D.3 · p. 25

E LIMITATIONS

• The study primarily varies monitor models rather than agent models, partly because weaker agents often struggle with challenging benchmarks such as SHADE-Arena.The authors also say reliable guide transfer between monitors with substantially different capabilities remains unresolved, particularly for complex long-horizon benchmarks.Source: E LIMITATIONS · p. 26