PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment

Bo Yin, Qi Li, Xinchao Wang

arXiv:2605.11882v1cs.AI

TL;DR

Agent safety requires trajectory-level supervision that balances security and utility, rather than relying mainly on response-level or off-policy signals. FATE uses same-policy repair proposals filtered by verifiers and Pareto optimization, yielding consistent safety improvements across benchmarks while preserving useful behavior, including a 33.5% reduction in attack success rate.

  • Problem

    Existing agent-safety signals often operate at response level or off-policy, leaving a need for dense trajectory-level supervision that balances safety and utility.

  • Method

    FATE uses the current policy to generate repair candidates for its verifier-scored failures, then applies Pareto-filtered supervision and PFPO across security, utility, refusal, and trajectory-control objectives.

  • Results

    FATE consistently improves safety–utility trade-offs across models, scales, rounds, and benchmarks, including a 33.5% reduction in attack success rate versus strong baselines.

  • Takeaways & Limitations

    Failed trajectories can serve as structured repair supervision for safer self-evolving agents while preserving useful task behavior.

  • Takeaways & Limitations

    FATE depends on verifier quality and policy-generated repairs, and its evaluation covers important but limited agent-safety settings.

Abstract

from arXiv · show

Tool-using LLM agents fail through trajectories rather than only final responses, as they may execute unsafe tool calls, follow injected instructions, comply with harmful requests, or over-refuse benign tasks despite producing a seemingly safe answer. Existing safety-alignment signals are largely response-level or off-policy, and often incur a safety-utility trade-off: improving agent safety comes at the cost of degraded task performance. Such sparse and single-objective rewards severely limit real-world usability. To bridge this gap, we propose FATE, an on-policy self-evolving framework that transforms verifier-scored failures into repair supervision without expert demonstrations. For each failure, the same policy proposes repair candidates, which are then re-scored by verifiers and filtered across security, utility, over-refusal control, and trajectory validity. This dense trajectory-level information is then used as a supervision signal for agent self-evolution. During this process, we further introduce Pareto-Front Policy Optimization (PFPO), combining supervised warmup with Pareto-aware policy optimization to preserve safety-utility trade-offs. Experiments on AgentDojo, AgentHarm, and ATBench show that FATE improves safety across different models and scales while preserving useful behavior. Compared with strong baselines, FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%. These results suggest that failed trajectories can provide structured repair supervision for safer self-evolving agents.

1 Introduction

Agentic safety must evaluate complete tool-use trajectories because agents can execute unsafe actions before harmless responses or preserve safety through broad refusal that harms benign utility. FATE addresses this gap by converting verifier-scored failures into on-policy, Pareto-filtered repair supervision and optimizing multiple safety-utility objectives with PFPO.

  • Motivation: Agentic safety requires evaluating observations, tool calls, state-changing actions, and final environment states rather than responses alone.Agents may appear harmless after unsafe tool calls, while trajectory-level reasoning captures the full interaction process.
  • Motivation: Response-level or off-policy supervision and scalar safety rewards can encourage broad refusal, sacrificing benign task performance.Existing signals include human labels over single replies and expert demonstrations that rarely cover an agent’s own trajectory-level failures.
  • Motivation: FATE uses failed on-policy trajectories to generate dense repair supervision while preserving multiple safety-utility objectives and updating the underlying policy.The framework treats failures as raw material for supervision aligned with the policy’s own failure distribution rather than relying on reactive external defenses.
  • Method: FATE generates multiple repair candidates from each verifier-scored failure and applies Pareto filtering with PFPO across security, utility, over-refusal, and trajectory-control objectives.Repair candidates are produced by the same policy that generated the failure, conditioned on the task, failed trajectory, and verifier feedback, without extra expert repair demonstrations.
  • Results: 33.5% reduction in attack success rate and 26.0% improvement on task success rate under attack are reported for FATE on AgentDojo versus the strongest baselines.The experiments span AgentDojo, AgentHarm, and ATBench across different model families, scales, evolution rounds, and baselines.

2 Related Work

Recent work evaluates safety risks in tool-using agents through trajectory-level failures rather than isolated chatbot responses. AgentDojo, AgentHarm, and ATBench expose diverse unsafe or low-utility trajectories and provide verifier signals, but are primarily designed for evaluation.

  • Agent safety evaluation and defenses: Tool-using agents face trajectory-level risks distinct from those of isolated chatbots.These risks arise when language models act as agents rather than producing standalone responses.
  • Agent safety evaluation and defenses: AgentDojo, AgentHarm, and ATBench expose failures involving prompt injection, harmful agentic requests, and fine-grained trajectory diagnosis.Together, the benchmarks cover multiple categories of agent safety failure.
  • Agent safety evaluation and defenses: These benchmarks provide verifier signals for identifying unsafe or low-utility trajectories but are primarily designed for evaluation.Their verifier signals support trajectory assessment rather than directly supplying a self-evolution procedure.

FATE: Failure Trajectory Evolution

FATE turns verifier-scored agent failures into same-policy repair supervision, filters repairs across a Pareto frontier, and internalizes selected repairs through SFT and PFPO for iterative self-evolution. It addresses a gap in prior defenses and refinement methods, which typically diagnose failures or revise behavior without converting failed trajectories into corrected policy-training targets.

  • FATE framework: FATE mines the current policy’s failures, proposes same-policy repairs, verifier-filters them into Pareto repair supervision, and internalizes selected repairs with SFT and PFPO.The resulting policy enters another self-evolution round.
  • Motivation: FATE transforms verifier-scored failures into repair supervision for updating the policy itself, unlike defenses that only target specific failure modes.Runtime defenses and guard models can reduce failures such as indirect prompt injection or harmful compliance, but typically do not create corrected training targets.
  • Related work: Prior refinement methods improve behavior through previous-trial feedback, structured reasoning-action interaction, stored reflections, or model-generated revisions at inference time.Examples include ReAct, Reflexion, and self-refinement methods.

3 FATE: Failure-Trajectory Evolution

FATE is an on-policy self-evolving framework that turns verifier-scored failure trajectories into repair supervision without expert demonstrations. It combines same-policy repair proposals, verifier filtering, Pareto-aware selection, and supervised warmup followed by PFPO.

  • Failure-Trajectory Evolution: FATE rolls out the current policy to collect failures, then uses that same policy to propose repairs for those failures.This makes repair candidates local to the policy’s own failure distribution, while leaving them subject to safety, validity, and conservativeness errors.
  • Verifier Re-scoring: Verifier re-scoring separates repair generation from supervision construction, ensuring that proposed trajectories are evaluated before becoming training targets.Executable candidates are rerun from the same initial state with state-based verifiers; non-executable settings use compatible rule checks or diagnostic labels.
  • Pareto-Aware Supervision: FATE filters candidates for utility, over-refusal control, and trajectory validity, then retains non-dominated repairs on the Pareto front.This prevents scalar safety ranking from selecting degenerate refusal-only repairs and avoids choosing candidates that excel on one objective while failing badly on another.
  • Pareto-Aware Supervision: The resulting supervision distribution is a constrained projection from same-policy proposals to verifier-filtered repair targets, rather than globally ranking all candidates with a scalar score.Scalar tie-breaking is applied only after feasibility filtering and Pareto-front projection.
  • Policy Refinement: Policy refinement uses supervised repair warmup followed by Pareto-Front Policy Optimization, which sharpens preferences among newly sampled repairs.SFT internalizes accepted repairs induced by the policy’s own failures, while PFPO samples completion groups and computes group-relative advantages after re-scoring.

4 Experiments

FATE is evaluated under strict held-out protocols across task types, backbone families, model scales, iterative rounds, external trajectory-safety diagnosis, and baseline comparisons. Across these settings, the reported findings indicate improved safety–utility trade-offs and transfer of trajectory-level safety supervision beyond self-evolution environments.

  • Evaluation protocol: Self-evolution mines failures only on Bdev, while in-domain results use held-out Btest excluded from repair generation, replay construction, and policy updates.At each round, FATE samples same-policy repairs, verifier-scores them, builds Pareto-front replay, and updates with SFT followed by PFPO.
  • Results across different backbone families: FATE consistently improves safety–utility trade-offs across five open-weight backbone families on AgentDojo and AgentHarm.AgentDojo measures ASR, TSR, and BRR; AgentHarm measures HCR, VRR, and an overall safety score.
  • Scaling with backbone capacity: FATE improves all six Qwen3 scales under the same self-evolution protocol, although gains are not purely monotonic with parameter count.The controlled study spans Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen3-32B.
  • Effect of iterative self-evolution: Across additional self-evolution rounds, FATE progressively reduces unsafe behavior while preserving useful behavior on held-out AgentDojo and AgentHarm tasks.The round-wise evaluation tracks ASR and TSR on AgentDojo, and HCR and VRR on AgentHarm.
  • External trajectory-safety generalization: FATE’s refined Qwen3-8B policy improves both ATBench-C and ATBench-F, indicating transfer of trajectory-level safety cues to external diagnosis.ATBench is used only for external safe/unsafe classification and fine-grained diagnosis, with no trajectories entering failure mining, replay construction, or policy updates.

5 Conclusion … D Formal Analysis of FATE Supervision Construction

FATE converts on-policy verifier-scored failure trajectories into repair supervision through same-policy proposals, feasibility filtering, Pareto-front selection, SFT, and PFPO. Its supervision distribution assigns probability only to feasible, non-dominated repairs and is characterized as a KL-regularized projection of the on-policy proposal, while experiments report improved safety with useful behavior preserved.

  • 5 Conclusion; A Algorithmic Details; C.6 SFT and PFPO: FATE mines current-policy failures, samples same-policy repair candidates, verifier-scores and filters them, selects Pareto-front targets, then updates through SFT and PFPO.The procedure repeats for self-evolution rounds, producing a refined policy.
  • A Algorithmic Details; C.2 Repair Proposal and Supervision Distribution: The current policy induces the repair proposal distribution, but verifier re-scoring, feasibility filtering, and Pareto-front selection define the actual supervision distribution.This separates on-policy proposal generation from supervision construction.
  • B Prompt Templates; B.2 Verifier-Compatible Diagnosis Prompt: FATE’s repair instructions target security, useful benign-task completion, over-refusal control, valid tool use, safe state changes, and sensitive-information protection.Verifier-compatible diagnosis scores security, utility, over-refusal control, and trajectory control.
  • C Mathematical Details; C.1 On-Policy Failure Set: On-policy failures are trajectories generated by the current policy whose verifier objective vector violates at least one objective.The failure set is defined at each evolution round from rollouts on the development split.
  • C.3 Feasibility Filtering; D.1 Setup: Feasibility filtering removes refusal-only or invalid repairs and retains candidates meeting utility, over-refusal, and trajectory-control thresholds.The feasible set is subsequently used for Pareto-front construction.
  • C.4 Pareto-Front Projection; C.5 Front-Only Tie-Breaking; D.1 Setup: The Pareto front contains feasible candidates that are not strictly dominated by another feasible candidate, enabling balanced candidate selection.Front-only tie-breaking defines the verifier-filtered supervision distribution.
  • D Formal Analysis of FATE Supervision Construction; D.2 Support Guarantees: The supervision distribution assigns positive probability only to feasible Pareto-front repairs and zero probability to strictly dominated candidates.This follows because the Pareto front is defined over the feasible set and dominated candidates are excluded.
  • D Formal Analysis of FATE Supervision Construction; D.3 KL-Projection View: Under the stated non-empty-front and positive-proposal assumptions, FATE’s distribution is the unique KL-regularized projection of the on-policy proposal onto Pareto-front support, rather than an unconstrained scalar ranking.The analysis is conditional on fixed verifier scores and does not claim global convergence or real-world safety guarantees.

E Benchmark and Metric Details · E.1 Verifier Details · F Implementation Details

The evaluation uses executable safety benchmarks, an external trajectory-diagnosis benchmark, and metrics covering attack resistance, task utility, refusal behavior, harmful compliance, and diagnosis. Verifiers rely on deterministic environment predicates for executable benchmarks, while ATBench uses diagnostic labels for trajectory classification.

  • E Benchmark and Metric Details: AgentDojo and AgentHarm provide executable agent-safety environments, while ATBench evaluates external trajectory diagnosis.These benchmarks form the evaluation suite summarized in Table 6.
  • E Benchmark and Metric Details: Tasks are grouped as benign, attacked-but-legitimate, or harmful-request modes.Benign tasks should be completed, attacked-but-legitimate tasks contain potentially injected instructions, and harmful requests should be safely refused.
  • E Benchmark and Metric Details: AgentDojo reports ASR, TSR under attack, and BRR, respectively measuring injected-instruction following, legitimate-task completion, and unnecessary refusal.Lower ASR and BRR are better, while higher TSR is better.
  • E Benchmark and Metric Details: AgentHarm reports HCR, VRR, and SafeScore, measuring harmful compliance, appropriate harmful-request refusal, and trajectory-level safety behavior.Lower HCR is better, while higher VRR and SafeScore are better.
  • E Benchmark and Metric Details: ATBench-C evaluates coarse safe/unsafe classification with accuracy, precision, recall, and F1, while ATBench-F measures fine-grained unsafe-trajectory diagnosis accuracy.ATBench-F includes risk source, failure mode, and real-world harm categories.
  • E.1 Verifier Details: Verifier scores use deterministic environment predicates whenever available for executable benchmarks.This instantiates verifier scoring through benchmark environment conditions.
  • E.1 Verifier Details: Diagnostic labels are used only for ATBench-style trajectory classification.The verifier implementation therefore distinguishes executable benchmark predicates from diagnostic classification labels.

F.1 Backbones and Data Splits · F.2 Pareto Weights and Feasibility Thresholds

FATE self-evolves on Bdev, reports in-domain results on held-out Btest, and reserves ATBench for external evaluation. Feasibility thresholds filter degenerate candidates, while objective weights break ties only after Pareto-front projection.

  • F.1 Backbones and Data Splits: Self-evolution is performed exclusively on the development split Bdev.
  • F.1 Backbones and Data Splits: All in-domain results are reported on the held-out test split Btest.
  • F.1 Backbones and Data Splits: ATBench is reserved solely for external evaluation.It is never used for repair generation or policy updates.
  • F.1 Backbones and Data Splits: The protocol uses strict split separation between self-evolution and in-domain testing.Self-evolution uses Bdev, whereas in-domain results use held-out Btest.
  • F.2 Pareto Weights and Feasibility Thresholds: For each task mode τ, FATE assigns feasibility thresholds κτ.These thresholds remove degenerate candidates.
  • F.2 Pareto Weights and Feasibility Thresholds: For each task mode τ, FATE assigns objective weights wm(τ).The weights are used only for front-only tie-breaking after Pareto-front projection.
  • F.2 Pareto Weights and Feasibility Thresholds: Feasibility thresholds and objective weights serve distinct roles in candidate selection.Thresholds remove degenerate candidates, while weights break ties only after Pareto-front projection.

G Per-Benchmark and Per-Category Results … H.1 Baseline Descriptions

The paper supplements aggregate results with per-category analyses testing whether FATE improves safety without sacrificing benign utility or inducing unnecessary refusal. It also compares FATE against alternative repair-selection, optimization, training, and teacher-generation baselines.

  • G Per-Benchmark and Per-Category Results: Per-category breakdowns test whether FATE’s average safety gains avoid sacrificing specific task groups.The analysis examines whether improvements overfit to only a subset of categories.
  • G.1 AgentDojo Per-Category Results: AgentDojo evaluates attacked-but-legitimate tool-use scenarios using ASR for resistance to injected instructions and TSR for preserving the original user goal.Lower ASR and higher TSR are desirable.
  • G.2 AgentHarm Per-Category Results: AgentHarm reports harmful-request outcomes across harm categories, assessing reduced harmful compliance alongside valid refusal behavior.The intended outcome is safer handling without undermining appropriate refusals.
  • G.3 Benign Utility Breakdown: Benign utility analysis measures task success and broad refusal behavior to detect whether safety refinement collapses into unnecessary refusal.These results complement the safety-category breakdowns by testing preservation of benign task completion.
  • H Additional Baselines: Additional comparisons distinguish FATE from direct self-training, safety-only optimization, and external repair generation.These baselines and variants were omitted from the main paper because of space constraints.
  • H.1 Baseline Descriptions: Direct SFT on failed trajectories tests whether reusing failures as demonstrations is sufficient, despite potentially reinforcing unsafe or low-utility behavior.The baseline fine-tunes directly on failed trajectories.
  • H.1 Baseline Descriptions: Self-repair SFT without verifier filtering tests whether same-policy repair candidates can serve as labels without re-scoring or Pareto-front selection.Candidates are fine-tuned directly after sampling from the current policy.
  • H.1 Baseline Descriptions: The remaining baselines isolate selection and training factors through security-only, random, external-teacher, and longer-training repairs.They test utility collapse, verifier-based selection, the necessity of on-policy proposals, and whether gains arise merely from additional training.

H.2 Additional Baseline Results

The baseline comparison focuses on improving safety without sacrificing utility. FATE addresses failure modes in direct SFT, unfiltered self-repair, and top-security-only selection through on-policy repair and verifier-filtered Pareto supervision.

  • Baseline comparison: The central baseline criterion is improving safety without sacrificing utility.This frames the comparison around the safety–utility trade-off.
  • Baseline comparison: Direct SFT on failures and unfiltered self-repair can inherit unsafe or invalid behavior.These approaches may reproduce problematic failure behavior rather than reliably repairing it.
  • Baseline comparison: Top-security-only selection may reduce unsafe actions but increase refusal or reduce task completion.Optimizing only security can produce degenerate solutions that harm utility.
  • FATE: FATE separates on-policy repair proposal from verifier-filtered Pareto supervision to avoid these degenerate solutions.The framework distinguishes candidate generation from supervision filtering.

I Additional Ablations and Sensitivity Analyses … K Limitations

The supplementary analyses test FATE’s sensitivity to candidate generation, Pareto weighting, feasibility thresholds, and verifier budgets, while sanitized examples illustrate balanced repair selection. The limitations identify dependence on verifier quality, policy repair capability, and computational cost.

  • I Additional Ablations and Sensitivity Analyses: FATE’s ablations examine repair-candidate count, evolution rounds, Pareto weights, feasibility thresholds, and verifier calls.These analyses assess the framework’s sensitivity across candidate generation, optimization, filtering, and evaluation costs.
  • I.1 Number of Repair Candidates: Larger K increases repair-candidate diversity and verifier-call cost, with gains expected to saturate once candidate sets are sufficiently diverse.The intended trade-off is between finding balanced repairs and increasing computation.
  • I.2 Pareto Weight Sensitivity: Pareto-weight sensitivity tests whether FATE remains robust across different safety–utility trade-offs.Security-heavy weights may reduce unsafe behavior but lower utility, whereas utility-heavy weights may preserve task success while leaving safety failures.
  • I.3 Feasibility Threshold Sensitivity: Feasibility-threshold sensitivity tests dependence on the protected-objective thresholds κτ used before Pareto-front projection.Loose thresholds may retain noisy repairs, while overly strict thresholds may reduce replay diversity.
  • I.4 Verifier Call Budget: Higher verifier budgets improve filtering quality and candidate selection, but marginal benefits may decrease after most high-quality repairs are identified.Verifier calls are described as a major cost, motivating budget sensitivity analysis with the backbone and benchmark fixed.
  • J Qualitative Examples from Sanitized Trajectories: Sanitized qualitative examples preserve failure patterns, verifier signals, and repair decisions while omitting raw private fields and operationally harmful details.The examples are constructed from benchmark-style instances and self-evolution rollout patterns.
  • J.3 Sanitized Over-Refusal Failure: A sanitized over-refusal example shows FATE selecting a repair that preserves benign utility while maintaining safe tool use.The repair avoids unrelated account-security content and illustrates optimization of over-refusal control rather than a safety-only objective.
  • J Qualitative Examples from Sanitized Trajectories: FATE converts failures into balanced repair targets by combining task context, error localization, verifier feedback, and Pareto-filtered replay rather than imitating failed trajectories.This process uses the failed rollout to localize errors and verifier feedback to identify violated objectives.
Loading 2605.11882v1…