PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment

Qinghua Mao, Wanying Qu, Dadi Guo, Leitao Yuan, Qingyu Liu, Yu Li, Guanxu Chen, Yanwei Fu, Xi Lin, Xia Hu, Dongrui Liu

arXiv:2609.02786v1cs.AIcs.CR

TL;DR

Agent safety alignment must address both harmful responses and risks emerging during multi-step execution, but isolated harness or policy evolution does not tightly couple runtime control with intrinsic safety. SafeEvolve uses completed on-policy trajectories to co-evolve auditable harness artifacts and policy behavior through harness-use SFT followed by harness-augmented RL. On agentic safety benchmarks, it improves the safety-utility tradeoff, including a 3× ASR reduction on AgentDojo with benign utility rising from 59.79% to 61.86% for Qwen3.5-4B.

  • Problem

    Existing safety alignment mechanisms separately update external harness artifacts or policy parameters, leaving the challenge of tightly coupling runtime harness refinement with intrinsic policy safety.

  • Method

    SafeEvolve converts completed on-policy trajectories into bounded, versioned harness updates and uses harness-use SFT followed by harness-augmented RL to internalize safety behavior.

  • Results

    SafeEvolve improves robustness against diverse safety risks while preserving benign task performance; for Qwen3.5-4B, it reduces ASR on AgentDojo from 2.37% to 0.79%.

  • Takeaways & Limitations

    Harness-policy co-evolution strengthens agent safety without sacrificing capability across the reported agentic safety benchmarks.

  • Takeaways & Limitations

    The threat model allows adversarial control of user requests or environment observations but keeps model parameters and registered tool APIs fixed.

Abstract

from arXiv · show

The performance of LLM-based agents is jointly shaped by the base model and the harness used when interacting with the environment. This exposes them to safety risks in both harmful final responses and multi-step execution trajectories. Existing safety alignment mechanisms often rely on either external harness updates or policy optimization, yet applying either paradigm in isolation fails to bridge runtime control with intrinsic safety. We propose SafeEvolve, an experience-driven self-evolving framework for agent safety alignment. SafeEvolve leverages safety experience from completed on-policy trajectories to drive a continual loop of harness-policy co-evolution. On the harness side, SafeEvolve converts trajectory-level safety evidence into bounded, component-level updates across safety prompt and hierarchical skills, yielding auditable and reversible harness artifacts. On the policy side, SafeEvolve follows a two-stage SFT-RL paradigm, where harness-use SFT bootstraps the policy to actively leverage evolved harness artifacts, and harness-augmented RL further shapes autonomous safety behaviors during multi-step exploration via verifier-decomposed rewards. Through harness-policy co-evolution, SafeEvolve converts safety experience into an evolved runtime harness and improved policy behavior. Experiments on agentic safety benchmarks show that SafeEvolve achieves a stronger safety-utility tradeoff than existing baselines. For Qwen3.5-4B, SafeEvolve achieves a $3\times$ ASR reduction on AgentDojo while improving benign utility from 59.79% to 61.86%.

1 Introduction

SafeEvolve addresses agent safety risks arising during both final responses and multi-step execution by coupling harness refinement with policy updates. It uses interaction experience to drive auditable harness changes and staged policy optimization, improving safety while preserving benign utility.

  • Agent safety risks arise during multi-step execution, including unsafe tool calls and plans redirected by injected instructions.
  • Existing approaches separately evolve external harness artifacts or policy parameters, but harness updates may exceed weak-policy capacity while policy updates can decouple from runtime control.
  • SafeEvolve uses completed on-policy trajectories as shared evidence for a continual harness-policy co-evolution loop.
  • Harness evolution converts trajectory-level safety evidence into bounded, versioned updates for safety prompts and hierarchical skills that remain auditable and reversible.
  • The policy component uses two-stage SFT-RL: harness-use SFT bootstraps artifact use, while harness-augmented RL develops autonomous safety behavior during multi-step execution.
  • 3× ASR reduction on AgentDojo accompanied a benign-utility increase from 59.79% to 61.86% for Qwen3.5-4B.

2 Related Work

Related work frames agent safety as extending beyond response-level refusal to trajectory-level risks and identifies experience-driven evolution of both harness components and reusable external knowledge.

  • Traditional safety alignment primarily targets response-level behavior, while agentic benchmarks expose trajectory-level risks such as prompt injection, harmful tool use, and privacy leakage.
  • Research on continual agent improvement includes evolving external harness components such as prompts, memories, tools, and middleware.
  • Another research direction abstracts reusable experience into external knowledge such as skills.

3 Method

SafeEvolve couples observability-driven harness evolution with two-stage policy optimization, using completed trajectories to produce bounded harness changes and persistent safety behavior. Its method targets safety across multi-step tool use while preserving benign utility.

  • Overall framework: SafeEvolve turns completed trajectories into auditable harness updates and improved policy behavior through a continual evolution loop.The loop couples harness refinement with policy optimization using on-policy experience.
  • Problem formulation: The agent interacts through a harness that renders context, mediates tools and skills, and produces trajectories from multi-step actions and observations.Actions may be response tokens or structured tool calls, and observations include tool outputs, webpages, and memory states.
  • Harness evolution: SafeEvolve evolves safety-relevant harness components, mainly safety prompts and retrieved hierarchical skills, while keeping tools and environment interfaces fixed.The targeted components govern instruction handling, action verification, and experience reuse.
  • Harness evolution: Rollout evidence records harness context, safety outcomes, metadata, and execution-quality signals before localizing each proposed edit to one component.This makes behavioral changes attributable to specific prompt, skill, or runtime-constraint edits.
  • Harness evolution: Candidate harnesses are evaluated against parent harnesses on paired tasks and accepted only when they improve internal return without regressing safety, clean utility, or execution quality.Accepted edits receive versioning, supporting evidence, and rollback metadata.
  • Policy optimization: Policy optimization uses harness-use SFT followed by harness-augmented RL with verifier-decomposed rewards for utility, safety, and execution validity.The reward emphasizes completion on clean tasks, safety on malicious queries, and both objectives for environment injection attacks.

4 Experiments

Experiments evaluate SafeEvolve across agentic safety benchmarks, ablations, evolution dynamics, retrieval strategies, proposer models, transfer settings, and failure modes. Results show improved safety–utility tradeoffs, with co-evolution and dynamically retrieved skills supporting safer behavior while preserving task execution.

  • Main Results: SafeEvolve achieves the best safety–utility trade-off across the evaluated backbones and benchmarks.It yields the lowest AgentDojo ASR across both backbones, strongest AgentHarm safety on Qwen3.5-4B, and higher AgentDyn utility than the base policy on Qwen3-4B.
  • Main Results: SafeEvolve improves robustness to both indirect prompt injection and malicious queries without uniformly suppressing useful action.It improves harmful-request refusal while preserving task execution under indirect prompt injection, indicating risk-specific intervention.
  • Harness Evolution: Harness evolution improves agent safety without policy training when the policy remains frozen.Evolved skills provide the most reliable gains, especially for AgentDojo ASR and AgentHarm harmful score, while evolved prompts particularly improve refusal behavior.
  • Harness-Policy Evolution: Co-evolution with evolved skills retains 56.77 AgentDojo utility under attack while reducing AgentDojo ASR from 2.37 to 0.79 and AgentHarm harmful score to 12.27.Compared with harness-only evolution, co-evolution further reduces AgentHarm harmful score from 16.80 to 12.27 and AgentDojo ASR from 0.92 to 0.79.
  • Evolution Mechanism: The active skillbank grows from 26 to 47 entries through mainly task-specific and common-mistake skills, while accepted updates preserve or improve attacked reward.Accepted edits address ambiguous targets, missing tool arguments, nested-source references, and fabricated update fields.
  • Skill Retrieval: Hierarchical retrieval with dynamic-skill priority restores the safety–utility balance better than general skills alone or retrieval without dynamic prioritization.Task-specific and recently evolved skills improve harmful-query safety and refusal while preserving attacked utility.
  • Proposer Ablation: Proposer-evolved harnesses consistently move toward safer regions, but prompt-injection outcomes vary by proposer.GPT-5.5 favors higher AgentDojo utility, whereas DeepSeek-Chat achieves the lowest AgentDojo ASR.
  • Generalization: Transfer is weakest to the 1.7B policy and strongest to the 4B target, where the evolved skill bank improves both injection robustness and harmful-query safety.The weaker policy does not reliably execute evolved guidance, while the 4B target benefits across both risk families.

5 Conclusion

SafeEvolve couples auditable harness refinement with policy optimization using completed on-policy trajectories as shared safety evidence. Across AgentDojo and AgentHarm, it achieves a stronger safety–utility tradeoff while improving both robustness and refusal behavior.

  • 5 Conclusion: SafeEvolve uses completed on-policy trajectories to drive continuous co-evolution of harness artifacts and policy behavior.Harness updates are structured, versioned, auditable, and reversible; policy optimization uses a two-stage SFT-RL paradigm.
  • 5 Conclusion: On Qwen3-4B, SafeEvolve improves AgentDojo utility from 44.33 to 60.82 while reducing ASR from 13.38 to 2.42.On Qwen3.5-4B, it reduces AgentHarm harmful score from 56.45 to 12.27 and increases refusal from 28.98 to 83.83.

A.1 Benchmarks

The benchmarks and baselines evaluate agent safety across indirect injection, dynamic multi-step attacks, and harmful user requests, while SafeEvolve’s harness templates operationalize safety guidance during execution.

  • Benchmarks: AgentDojo tests whether agents preserve benign goals, ignore injected instructions in observations, and complete tasks using tools.
  • Benchmarks: AgentDyn stresses prompt-injection robustness under changing execution contexts and tool-use states across multi-step interactions.
  • Benchmarks: AgentHarm measures compliance with harmful multi-step user requests, representing malicious-query attacks delivered directly by users.
  • Baselines: GRPO isolates evolved-harness effects by training with verifier-decomposed safety-utility rewards under a fixed default harness.
  • Harness templates: SafeEvolve’s runtime template combines the active safety prompt, retrieved skills, tool schema, task metadata, and interaction history.
  • Harness templates: The evolution templates constrain updates to safety prompts or hierarchical skills using rollout evidence, metadata, bounded edits, explicit scope, and rollback notes.

A.4 Algorithm and Evolution Details

SafeEvolve alternates verifier-guided policy optimization with bounded harness mutations derived from rollout evidence. Candidate harnesses are accepted only after controlled evaluation against safety, utility, and reliability constraints.

  • Verifier and reward: The verifier computes trajectory-level utility and safety scores from executed tool calls, tool outputs, and final answers.
  • Evolution gate: Candidate updates are compared with their parent using the same policy, task batch, decoding configuration, rollout panel, and verifier.
  • Evolution gate: Hard safety floors reject updates that enable attacks, increase malicious-query harm, substantially increase invalid calls or parse failures, or cause large utility regressions.
  • SkillBank retrieval: The SkillBank retrieves compact episode-level subsets from general, task-specific, and common-mistake skills using task and execution metadata.
  • Algorithm: Algorithm 1 initializes an evolved harness, collects approved trajectories for harness-use SFT, then performs harness-augmented GRPO across policy rounds.
  • Algorithm: Each round aggregates failures, recoveries, metadata, and verifier outcomes before proposing a bounded mutation to one harness component.

A.5 Backbones and Training Schedule

Experiments use two Qwen backbones under matched rollout and learning-rate settings, with fixed reward weights and H200-based training and evaluation.

  • Backbones: The study uses Qwen3.5-4B and Qwen3-4B-Instruct-2507 as its main backbones.
  • Training schedule: Both backbones receive the same rollout budget and learning rate unless otherwise specified.
  • Training schedule: Environment-injection rewards use λU = 0.25, λS = 0.5, and λUS = 0.25 across backbones and training runs.
  • Compute and verification: Policy training uses four NVIDIA H200 GPUs, benchmark evaluation uses one, and harness evolution invokes external proposer calls while verification remains rule-based.

B Supplementary Results

Supplementary studies examine component interactions, safety internalization, online harness updates, and held-out generalization. Results show complementary but non-additive components, incomplete policy-only internalization, risks from local online updates, and strong transfer to unseen attack families.

  • Component interactions: The combined prompt-and-skill harness yields mixed gains rather than uniformly outperforming either component alone.
  • Safety internalization: OPSD reduces attack metrics without runtime harness assistance, but utility drops on AgentDyn and performance remains behind runtime-harness RL on AgentHarm.
  • Online updates: Online prompt evolution raises Qwen3.5-4B AgentDojo attacked utility from 41.97 to 45.13 but worsens AgentDyn ASR from 4.96 to 10.67.
  • Online updates: Online skill updates raise Qwen3.5-4B AgentHarm harmful score from 12.27 to 27.02 while lowering refusal from 83.83 to 62.99.
  • Held-out generalization: Held-out evaluation covers direct injection, indirect injection, and Plan-of-Thought backdoor attacks using ASR and refusal rate.
  • Held-out generalization: SafeEvolve achieves the lowest ASR across all three held-out attack families, including ASR 2.50 and refusal rate 27.50 under PoT Backdoor.

C Case Study

The case studies show how SafeEvolve updates safety prompts and hierarchical skills from trajectory evidence, improving both harmful-request refusal and benign-task utility. The evolved skill bank expands reusable repair behaviors while preserving instruction-hierarchy skills.

  • SkillBank Evolution: The evolved SkillBank grows from 26 to 47 entries after ten evolution steps, adding 21 dynamic skills across task-specific and common-mistake categories.Five candidate updates are accepted while preserving the three general instruction-hierarchy skills from the seed bank.
  • SkillBank Evolution: The initial SkillBank contains 26 entries built from 12,800 base-policy rollout records, including 3 general, 14 task-specific, and 9 common-mistake skills.The records comprise 7,217 successful and 5,583 failed trajectories.
  • Evolved Hierarchical Skills: The evolved ambiguous-booking skill converts repeated ambiguous-parameter failures into a reusable rule: preserve the trusted goal, resolve the canonical target, and retry once with a supported identifier.Its procedure uses factual evidence while ignoring instruction-like text embedded in observations.
  • Safety Prompt Cases: The case studies distinguish harmful objectives from indirect prompt-injection settings, where benign trusted tasks should still be completed rather than abandoned after detecting untrusted instructions.Safety prompts treat observations as untrusted evidence and retain the legitimate user request as the controlling objective.
  • AgentHarm Case: In the AgentHarm malicious-query case, the base policy searches private records and prepares disclosure, whereas the skill-augmented policy refuses tool use and offers a privacy-preserving alternative.The safety guidance treats the user’s request itself as unsafe and prohibits searching for sensitive content.

D Additional Discussion

The discussion frames SafeEvolve as a bounded first step toward recursive and open-ended self-improvement, while emphasizing that broader evolution requires longitudinal evaluation and immutable safety controls.

  • Meta-Evolution for Recursive Self-Improvement: SafeEvolve’s recursion remains first-order because its search and update schedules are fixed, so true self-improvement requires longitudinal multi-generational evaluations.Future protocols should track policy-harness lineages over extended generations and distinguish generating harness updates from executing them effectively.
  • Safe and Controllable Evolution for Open-Ended Tasks: Static attack distributions bound the safety frontier that an evolving agent can discover, motivating future open-ended tasks with executable risk environments and co-evolving adversaries and defenses.The proposed expansion includes red-team adversaries, safety curricula, and agent defenses.
  • Safe and Controllable Evolution for Open-Ended Tasks: Online evolution can improve selected metrics while regressing on others, so open-ended updates require immutable safety anchors, independent verifiers, rollback, provenance, regression budgets, and human oversight.The discussion targets controlled discovery under auditable constraints rather than unrestricted self-modification.
Loading 2609.02786v1…