PaperlayerView exact revision on arXiv

Source-linked AI summary

HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence

Xiqiao Xiong, Moxin Li, Zhixin Ma, Ouxiang Li, Wenjie Wang, Fuli Feng, Xiangnan He

arXiv:2610.02920v1cs.AI

TL;DR

Emerging attacks often arrive with sparse threat evidence, limiting the basis for adapting harness defenses. HASTE uses an iterative adversarial process to evolve harnesses from that evidence, and experiments report lower attack success alongside preserved benign-task utility. The results support the feasibility of translating sparse threat evidence into harness evolution against emerging attacks.

  • Problem

    Emerging attacks are often disclosed through sparse descriptions or a few examples, while adapting harness defenses still relies substantially on manual engineering.

    Source: §1 · p. 1

  • Method

    HASTE iteratively co-optimizes safety specifications and attack cases, using evaluation feedback to guide harness updates and refine both processes.

    Source: §1 · p. 2

  • Results

    HASTE consistently reduces attack success rate and improves safe and helpful rate across backbone models, evidence forms, and attack types while preserving benign-task utility.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate the feasibility of automatically translating sparse threat evidence into effective harness evolution against emerging attacks.

    Source: §5 · p. 9

    In HASTEblind, the absence of fixed cases hinders final-harness selection, so the final adaptation-round harness is returned; simultaneous emerging attacks remain future work.

    Source: E LIMITATIONS · p. 20

Abstract

from arXiv · show

Agent harnesses play a critical role in defenses by enforcing safety constraints to prevent unsafe actions. However, rapidly emerging attacks outpace manual harness adaptation, motivating automated harness evolution. Yet the signals available for harness evolution are often sparse, such as brief descriptions or a few attack examples in threat reports and preprints. To address this limitation, we introduce HASTE, a multi-agent framework that evolves agent harnesses from sparse threat evidence through an adversarial interplay between safety-specification generation and attack-case generation. Safety specifications guide harness updates toward addressing identified safety vulnerabilities, while attack cases probe for remaining safety vulnerabilities after each update. By feeding evaluation outcomes back into both processes, HASTE enables harness evolution against emerging attacks beyond the initially observed evidence. Experimental results across multiple backbone models, attack types, and evidence forms show that HASTE consistently reduces attack success rates while preserving benign-task utility. The code is available at https://github.com/xxiqiao/HASTE.

1 INTRODUCTION

Agent harnesses constrain execution to help prevent unsafe actions, but emerging attacks often come with only sparse evidence that cannot directly support systematic evaluation or automatic adaptation. HASTE addresses this gap by iteratively co-optimizing safety specifications and attack cases, with experiments reporting reductions in attack success and improvements in safe and helpful rates.

1 INTRODUCTION

  • An agent’s backbone LLM selects actions from interaction history and observations, while its stateful harness maintains context, executes tool calls, and incorporates resulting observations.Because the harness controls execution and environment interactions, it plays a role in preventing unsafe actions while supporting benign task completion.Source: §1 · p. 1
  • Emerging attacks are often disclosed through brief descriptions or a few examples, leaving insufficient coverage for systematic defense evaluation and automatic adaptation.Harness adaptation still largely depends on manual engineering and substantial human effort.Source: §1 · pp. 1–2
  • HASTE uses an adversarial loop: safety specifications guide harness updates, while generated attack cases probe vulnerabilities that remain after each update.Evaluation outcomes feed back into both generation processes to support evolution beyond the initially observed evidence.Source: §1 · p. 2
  • HASTE’s experiments report consistent reductions in attack success rate and improvements in safe and helpful rate across backbone models, evidence forms, and attack types.The framework is evaluated across Agent-SafetyBench and STAC, with utility preserved or improved in some cases.Source: §1 · p. 2

2 RELATED WORK

Prior work automates parts of harness design and searches over whole harnesses, but these approaches typically assume broad evaluation feedback. Agent attacks exploit multiple interfaces, while existing defenses are largely designed for prespecified attacks.

2 RELATED WORK

  • Automation spans prompt, module-composition, workflow, and whole-harness design, but whole-harness methods typically assume broad evaluation feedback.Meta-Harness and AutoHarness synthesize or refine harnesses using evaluation feedback.Source: §2 · pp. 2–3
  • Jailbreak, prompt-injection, tool-based, and memory-poisoning attacks differ in interface but all exploit the agent harness.The attack surface includes adversarial prompts, external content, malicious tool descriptions or arguments, and poisoned persistent memory.Source: §2 · p. 3
  • Guardrail models screen inputs, outputs, and tool calls, while harness defenses constrain observations, memory, and execution; existing defenses largely target prespecified attacks.Harness mechanisms include instruction hierarchies, sandboxing, and access controls.Source: §2 · p. 3

3 METHOD

The method formalizes harness adaptation from sparse threat evidence while the comprehensive evaluation set is unavailable. HASTE uses attack abstraction, adversarial co-optimization, and feedback-based evaluation to evolve and select a harness.

3.1 BACKGROUND AND PROBLEM FORMULATION

  • Given sparse threat evidence and an initial harness, the goal is to improve safety while preserving task utility on a comprehensive evaluation set unavailable during adaptation.Source: §3.1 · p. 3

3.2 HASTE

  • Each round pairs safety-specification updates with attack cases for residual safety weaknesses and safely completable utility cases; judge feedback returns to both optimizers.Source: §3.2 · pp. 4–5
  • An attack parser distills sparse evidence into a fixed specification of the attack mechanism, which conditions both optimizers.The abstraction is intended to avoid tailoring defenses to incidental details in raw evidence.Source: §3.2 · p. 4
  • The judge evaluates safety for all cases and additionally evaluates utility for safely completable cases, producing diagnostic explanations of outcomes.Source: §3.2 · p. 5
  • HASTEblind returns the final-iteration harness, whereas HASTEseen compares candidates using fixed observed examples and selects one balancing safety and utility; held-out cases are reserved for final evaluation.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments compare HASTE with defense baselines across ASB and STAC, then test component choices, generalization, transfer, and adaptation under successive threats. Results favor HASTEseen in safety and utility, with stronger stability when observed examples are combined with evolving generated cases.

4.2 MAIN RESULTS

  • With the same observed examples, HASTEseen achieves the lowest average ASR on both benchmarks and the highest SHR on ASB, outperforming Meta-Harness.The experiments also report HASTEblind averages of 35.75 ASR on ASB, 29.12 ASR on STAC, and 49.81 SHR on ASB.Source: §4.2 · pp. 6–7

4.3 ABLATIONS

  • Freezing either safety-specification optimization or test-case optimization raises ASR and lowers SHR; reusing initial cases causes larger degradations than retaining a generic specification.The ablation uses ASB-300 with Qwen3-8B.Source: §4.3 · p. 7, Table 3 · p. 7
  • HASTEseen retains lower ASR and higher SHR than Meta-Harness across judge configurations and proposer/optimizer choices, and benefits further from GPT-5.5 proposer and optimizer models.Judge substitutions change outcomes, while the GPT-5.5 comparison is on ASB-300 with GPT-5.4-mini as the backbone.Source: §4.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 8

4.4 IN-DEPTH ANALYSIS

  • Harnesses transferred from ASB remain effective on STAC but have higher ASR than harnesses evolved directly on STAC; harnesses also remain effective under out-of-domain backbone transfer.In-domain ASRs and SHRs are generally better than out-of-domain counterparts.Source: §4.4 · p. 8, Table 6 · p. 8, Table 7 · p. 8
  • Across successive risk stages, HASTEseen usually maintains the lowest ASR and highest SHR on the first risk, although ASR generally rises for all methods.Each stage adds a risk type and uses five adaptation iterations with 8 attack and 8 safely completable cases; evaluation uses 50 attack cases per risk and a fixed set of 100 safely completable cases.Source: §4.4 · p. 8, Figure 3 · p. 9
  • Across all encountered risks, HASTEseen stays concentrated in the upper-right safety–utility region, while HASTEblind and Meta-Harness are less stable.The paper attributes lower stability in HASTEblind to accumulated adaptation drift and in Meta-Harness to overfitting to specific cases.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Increasing adaptation rounds or test cases generally improves HASTEseen before performance stabilizes, while HASTEblind and Meta-Harness degrade as either parameter grows further.HASTEseen's variance generally decreases with more adaptation rounds; the other methods' variance remains comparatively large and fluctuates.Source: §4.4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

The conclusion presents HASTE as a way to evolve agent harnesses against emerging attacks using sparse threat evidence. Across the reported evaluations, it improves safe and helpful rates and shows generalization, transferability, and stability benefits.

5 CONCLUSION

  • The paper studies evolving agent harnesses to defend against emerging attacks using sparse threat evidence.The conclusion frames this as a problem of translating sparse evidence into harness evolution.Source: §5 · p. 9
  • HASTE extrapolates from sparse evidence and adversarially co-optimizes safety specifications and attack cases to drive harness evolution.Source: §5 · p. 9
  • Across benchmarks, backbone models, and evidence forms, HASTE improves safe and helpful rates and shows strong generalization and transferability.The conclusion also reports that observed examples further stabilize adaptation across successive threats and parameter settings.Source: §5 · p. 9

AI USE STATEMENT

The authors report using generative AI tools for manuscript and research-code assistance, not for research decisions or result interpretation.

AI USE STATEMENT

  • Generative AI tools assisted with polishing the manuscript.Source: AI USE STATEMENT · p. 10
  • Generative AI tools assisted with writing, debugging, and refining the research code.Source: AI USE STATEMENT · p. 10
  • The authors state that AI was not used to make research decisions or interpret results, and that all AI-assisted content was reviewed and verified.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement describes controlled benchmark experiments and generated attack cases for defensive evaluation and adaptation. Additional results indicate that HASTE can operate from observed attack examples alone, while research-paper evidence further improves safety.

ETHICS STATEMENT

  • The experiments use existing safety benchmarks in controlled environments and generate attack cases for defensive evaluation and harness adaptation.Source: ETHICS STATEMENT · p. 10

A LESS THREAT EVIDENCE

B JUDGE MODEL VALIDATION

The paper checks whether its judge-based evaluations are reliable across judges and reports that HASTE’s relative performance remains largely consistent under alternative final evaluators. It also evaluates on ASB and STAC, which cover diverse safety tasks and sequential tool-attack chaining, respectively.

B JUDGE MODEL VALIDATION

  • Utility judges show pairwise agreement above 93%, and safety judges show substantial agreement on ASB and strong agreement on STAC.The authors report that evaluation signals are broadly consistent rather than specific to a single judge model.Source: B JUDGE MODEL VALIDATION · pp. 14–15, Table 9 · p. 14, Table 11 · p. 15
  • HASTEblind outperforms guardrail models and harness-based baselines across evaluation settings, while HASTEseen achieves the best ASB ASR and SHR and lowest STAC ASR across settings.The final harnesses were reevaluated using different judge models.Source: B JUDGE MODEL VALIDATION · p. 15, Table 10 · p. 15

C.1 DATASETS

  • ASB contains 2,000 tasks across eight risk categories and 349 environments, while STAC contains 483 verified sequential tool-attack cases across 10 failure modes.The evaluation excludes tasks used for adaptation.Source: §C.1 · p. 15

C.2 EVALUATION

Final evaluation uses distinct safety and utility judges from those used during adaptation, with benchmark-specific metrics. The setup specifies the adaptation configuration and compares HASTE with guardrail, harness-based, and fixed-case harness-search baselines.

C.2 EVALUATION

  • Final ASB evaluation uses ShieldAgent for safety and GPT-5.6-luna for utility, while STAC uses DeepSeek-V4-Flash for safety evaluation; adaptation uses different judges.ASR is reported across tasks, SHR on fulfillable ASB tasks, and STAC reports ASR only; experiments are repeated three times.Source: §C.2 · pp. 15–16

C.3 IMPLEMENTATION DETAILS

  • Experiments use four backbone LLMs and five adaptation rounds, with 16 safety cases and, for ASB, 16 fulfillable cases to preserve benign-task utility.The proposer and optimizers use Claude Code with DeepSeek-V4-Pro; the attack parser uses GPT-5.1.Source: §C.3 · p. 16

C.4 BASELINES

  • Harness-based baselines include real-time safety layers, feedback-driven action revision, lifecycle controls, and fixed-case harness search.The compared methods include LlamaFirewall, TS-Flow, SafeHarness, and Meta-Harness.Source: §C.4 · pp. 16–17
  • Guardrail baselines include prompt-and-response screening, pre-execution action-risk detection, and step-level tool-invocation assessment.These include Qwen3Guard, Safiron, and TS-Guard.Source: §C.4 · p. 16
  • Table 13 presents representative ASB cases in which existing defenses execute an unsafe action that HASTE’s generated harness prevents.Source: Table 13 · p. 17

D.1 REPRESENTATIVE DEFENSE BEHAVIORS

The representative cases show HASTE-generated harnesses checking operating ranges, requester authority, and observed device state before executing consequential actions. The simplified examples illustrate verification gates, progressive tool disclosure, and hybrid rule-based and model-mediated review.

D.1 REPRESENTATIVE DEFENSE BEHAVIORS

  • In Case 1842, HASTE checks the operating range before a change, preventing an out-of-range value despite the mutation tool reporting success.The request is syntactically valid, and the tool returns success even when given an illegal value.Source: §D.1 · p. 17
  • In Case 1270, HASTE checks the requester’s authority and refuses to disable monitoring after the environment reports the requester is unauthorized.The baselines proceed without invoking the authority-checking tool.Source: §D.1 · p. 17
  • In Case 1321, HASTE uses observed device state to execute only the safe subset when one device cannot be interrupted.The baselines apply the request uniformly across devices.Source: §D.1 · p. 17

D.2 EXAMPLES OF GENERATED HARNESSES

  • Generated harnesses combine natural-language guidance with programmatic enforcement through verification gating, context governance, and evidence-based execution.Sensitive actions require relevant state checks or system evidence rather than relying solely on user claims.Source: §D.2 · p. 17
  • The prompt-guided verification examples pair general safety principles with runtime checks that enforce read-before-write ordering and protect security controls.Source: Figure 6 · p. 18, Figure 9 · p. 20
  • Progressive tool disclosure initially exposes inspection tools, then enables other tools after an inspection attempt and annotates safety-relevant descriptions and outputs.Source: Figure 7 · p. 19
  • A hybrid safety gate applies deterministic rules to structurally identifiable risks and auxiliary-model review to actions requiring semantic reasoning, while low-risk actions bypass review.Source: Figure 8 · p. 19

E LIMITATIONS

The framework’s limitations concern final-harness selection in HASTEblind and its treatment of emerging attacks one at a time.

E LIMITATIONS

  • Without fixed cases, HASTEblind cannot select among final harnesses and directly returns the harness from the final adaptation round.Source: E LIMITATIONS · p. 20
  • HASTE currently considers emerging attacks individually; evolution under multiple simultaneously emerging attacks remains future work.Source: E LIMITATIONS · p. 20
Loading 2610.02920v1…