PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time

Yu Cheng, Yongkang Hu, Shuaijie Ma, Zhihang Lin, Weicheng Meng, Jingyang Qiao, Jiuan Zhou, Yushuo Zhang, Yihang Chen, Weilin Luo, Kun Shao, Dong Li, Zhizhong Zhang, Yuan Xie, Zhaoxia Yin

arXiv:2609.36580v2cs.AI

TL;DR

Real-world LLM agents face sequential tasks, with safety feedback arriving only after execution, while many self-evolving approaches rely on repeatedly accessible task distributions. SafeCoEvo co-evolves a Harness and Guard from accumulated runtime experience at different timescales. Compared with the strongest baseline, it reports a lower unsafe outcome rate and higher task success rate.

  • Problem

    Existing self-evolving approaches typically optimize over fixed, repeatedly accessible task distributions, unlike deployment settings where only past-task experience can inform safety on future unseen tasks.

    Source: §2 · p. 3

  • Method

    SafeCoEvo co-evolves S-Harness, which externalizes recent experience into explicit safety knowledge, and GuardVPO, which internalizes accumulated experience into parametric risk-judgment capabilities.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    10.05% lower unsafe outcome rate and 12.15% higher task success rate than the strongest baseline are reported for SafeCoEvo.

    Source: §6 · p. 10

  • Takeaways & Limitations

    SafeCoEvo is designed to improve the external safety system from deployment experience while keeping the task-performing model frozen.

    Source: §6 · p. 10

    The authors do not claim GuardVPO’s current optimization formulation is the only or optimal approach to long-term parametric safety internalization.

    Source: G LIMITATIONS AND FUTURE WORK · p. 20

Abstract

from arXiv · show

LLM agents deployed in real-world environments continually encounter new tasks and safety risks, while execution feedback typically becomes available only after each task is completed. However, existing self-evolving approaches commonly rely on multiple rounds of optimization over fixed and repeatedly accessible task distributions, fundamentally differing from test-time adaptation in real-world deployment, where only experience accumulated from past tasks can be used to improve safety decisions on future unseen tasks. To address this limitation, we propose SafeCoEvo, a test-time Harness-Guard co-evolution framework for LLM agent safety that enables the external safety system to continually adapt from accumulated runtime experience. SafeCoEvo jointly improves two complementary safety capabilities at different timescales: S-Harness rapidly externalizes recent runtime experience into updatable explicit safety knowledge that can promptly influence subsequent tasks, while GuardVPO internalizes accumulated runtime safety experience over a longer timescale into parametric risk-judgment capabilities. By combining short-term rapid adaptation with long-term capability consolidation, SafeCoEvo continually improves the agent's safety capabilities, reducing the unsafe outcome rate by 10.05% while improving the task success rate by 12.15% over the strongest baseline, thereby achieving simultaneous gains in safety and task utility.

1 INTRODUCTION

Real-world agent safety requires adaptation from runtime experience, but existing safety mechanisms are often static and self-evolving approaches typically revisit fixed task distributions. SafeCoEvo co-evolves a Harness and Guard at different timescales and reports simultaneous safety and task-utility gains.

1 INTRODUCTION

  • Deployed Guards and Safety Harnesses typically remain largely static, while frequently updating and redeploying a task-performing model can be costly and interfere with its capabilities.External safety mechanisms can be updated independently while keeping the task-performing model frozen.Source: §1 · p. 2
  • Existing self-evolving approaches typically optimize over fixed, repeatedly accessible task distributions, unlike deployment streams where past tasks inform safety on future unseen tasks.Tasks and safety-relevant interactions arise continually, while execution feedback generally arrives after the interaction.Source: §1 · pp. 1–2
  • S-Harness externalizes recent feedback into reusable explicit safety knowledge, while GuardVPO internalizes accumulated experience into parametric risk-judgment capabilities.The two components operate at shorter and longer timescales, respectively.Source: §1 · p. 3
  • 10.05% lower unsafe outcome rate and 12.15% higher task success rate than the strongest baseline are reported for SafeCoEvo.The paper describes these as simultaneous gains in safety and task utility.Source: §1 · p. 3

2 CONTINUAL TEST-TIME SAFETY ADAPTATION

Continual test-time safety adaptation concerns sequential deployment tasks whose runtime experience becomes available only after execution. The system can use that accumulated past experience for current safety decisions, while newly obtained feedback can inform only subsequent tasks.

2 CONTINUAL TEST-TIME SAFETY ADAPTATION

  • Before executing task x_t, the system has access only to runtime experience accumulated from previously completed tasks.The task stream arrives sequentially, and each task's trajectory and feedback become available after execution.Source: §2 · p. 3
  • Unlike repeated optimization over revisitable task sets, continual test-time adaptation uses past experience for task x_t and reserves new feedback for subsequent tasks.Existing approaches can revisit the same tasks or task distribution throughout evolution.Source: §2 · pp. 3–4
  • The problem is to enable an agent’s external safety system to continually adapt from accumulated runtime experience under this test-time constraint.Source: §2 · p. 4

3 SAFECOEVO: DUAL-TIMESCALE SAFETY CO-EVOLUTION

SafeCoEvo adapts an external safety system from deployment experience while keeping the task-performing model fixed. It pairs rapid explicit S-Harness updates with longer-timescale Guard learning from accumulated runtime safety experience.

3.1 OVERVIEW

  • The fixed task model and external safety system use the same deployment stream for rapid explicit adaptation and longer-term parametric learning.The external system comprises S-Harness and Guard; task trajectories and feedback become available after completion.Source: §3.1 · p. 4

3.2 S-HARNESS: RAPID EXTERNALIZATION OF RUNTIME SAFETY EXPERIENCE

  • S-Harness maintains five explicit safety components: Safety Prompt, Validated Memory, Safety Skills, Permission Policy, and Guard Policy.The components respectively hold safety constraints, validated reusable experience, procedural patterns, authorization boundaries, and rules for interpreting Guard judgments.Source: §3.2 · p. 4
  • After each completed task, S-Harness processes its trajectory and feedback into reusable safety information incorporated into explicit components.It abstracts experience rather than retaining benchmark names, task identifiers, or exact samples, and updates affect subsequent tasks.Source: §3.2 · pp. 4–5

3.3 GUARDVPO: PARAMETRIC INTERNALIZATION OF ACCUMULATED SAFETY EXPERIENCE

  • GuardVPO periodically internalizes accumulated event-level safety experience into the Guard’s parametric judgment capabilities, complementing S-Harness’s rapid explicit adaptation.The GuardVPO objective uses a fixed initial Guard as a reference policy and KL regularization to constrain policy drift during continual training.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate SafeCoEvo on sequential safety and security streams, held-out tasks, alternative backbones, Guard risk-judgment transfer, and component ablations. Results report gains over evolution and static baselines across safety and task-utility metrics.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses 1,024 evolution episodes, 200 validation episodes, and 300 held-out test episodes across Agent-SafetyBench and ASB.Episodes are processed in fixed order using only prior accumulated experience; S-Harness updates after episodes, and Guard optimization follows a designated adaptation stage.Source: §4.1 · p. 6
  • Baselines include static external safeguards and SHE; SafeCoEvo variants isolate static components, S-Harness evolution, and GuardVPO updates.Reported results are averaged over multiple independent runs.Source: §4.1 · p. 6

4.2 CONTINUAL EVOLUTION ANALYSIS

  • On the 1,024-episode stream, SafeCoEvo reduces UOR by 10.05% versus SHE while improving TSR by 12.15% and SUCR by 15.67%.Compared with the static configuration, S-Harness reduces UOR by 10.45% and improves TSR and SUCR by 31.45% and 27.64%.Source: §4.2 · p. 7

4.3 GENERALIZATION ON THE TEST SET

  • SafeCoEvo reduces held-out-test UOR by 21.52% versus SHE and improves TSR and SUCR by 26.97% and 36.38%.The held-out test set is independent of the evolution stream, and the evolved S-Harness and Guard remain fixed during evaluation.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 GENERALIZATION ACROSS SYSTEM BACKBONES

  • With Qwen3.7-Flash as task model, SafeCoEvo reduces train-stream UOR by 17.81% versus SOTA and test-set UOR by 44.66%.TSR and SUCR also improve on both sets, supporting generalization beyond the default task-model backbone.Source: §4.4 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
  • With SingGuard replacing the default Guard, SafeCoEvo reduces continual-stream UOR by 25.95% and test-set UOR by 32.67% versus the static configuration.TSR and SUCR improve on both sets as well.Source: §4.4 · p. 8, Table 3 · p. 8, Table 4 · p. 9

4.5 GENERALIZATION OF THE GUARD AFTER GUARDVPO

  • After GuardVPO, the Guard improves unsafe F1 and recall on both R-Judge and ATBench, while correct predictions rise from 802/1000 to 810/1000.Safe-class specificity decreases slightly on both benchmarks, indicating a modest trade-off in recognizing safe examples.Source: §4.5 · p. 8, Table 5 · p. 9

4.6 ABLATION STUDY

  • Ablations report that S-Harness improves safety and task utility over static safeguards, while GuardVPO further improves overall performance, especially after its update.The mechanisms are complementary: S-Harness externalizes recent safety knowledge, while GuardVPO internalizes accumulated event-level experience into parametric risk judgment.Source: §4.6 · pp. 8–9

5 RELATED WORK

Prior work studies external safeguards and experience-driven adaptation, including methods focused on harnesses, guards, memory, or agent components. SafeCoEvo differs by co-evolving a Safety Harness and Guard from shared deployment experience at different timescales.

External agent safeguards

  • External Guards assess risks in inputs, actions, or trajectories, while Safety Harnesses constrain behavior through prompts, policies, lifecycle controls, and verifiable execution logic.Despite protecting agents without modifying the task-performing model, these safeguards rarely jointly adapt Guards and Harnesses from accumulated runtime experience.Source: §5 · p. 9

Experience-driven adaptation

  • Experience-driven methods evolve prompts, harnesses, systems, workflows, memories, or individual safety components, but primarily adapt individual experience carriers or agent components.Examples include memory-based approaches and safety-focused methods that validate failures or evolve deployment memory.Source: §5 · p. 9

Experience-driven safety adaptation

  • SafeCoEvo co-evolves a Safety Harness and a Guard at different timescales using shared deployment experience for continual test-time safety adaptation.Prior safety approaches discussed include adaptive checks, trajectory-driven Harness updates, model- and domain-specific Harness search, and joint Harness-policy optimization.Source: §5 · p. 9

6 CONCLUSION

SafeCoEvo co-evolves the Safety Harness and Guard from runtime experience while keeping the task-performing model frozen. It reports simultaneous safety and task-success gains over the state-of-the-art baseline.

  • SafeCoEvo reduces unsafe outcome rate by 10.05% and improves task success rate by 12.15% over the state-of-the-art baseline.S-Harness externalizes recent experience into explicit safety knowledge, while GuardVPO internalizes accumulated experience into parametric risk judgment.Source: §6 · p. 10

A DETAILED SAFETY AND SECURITY RESULTS

The detailed results separate continual-stream performance into Safety and Security settings. Both show gains over static and evolving baselines, with particularly pronounced improvements in Security.

Safety

  • Under Safety, full-stream SafeCoEvo reduces UOR by 13.75% versus SHE while improving TSR by 4.89% and SUCR by 10.74%.Against SafeCoEvo (Static), S-Harness reduces UOR by 3.61%; GuardVPO further reduces it by 2.87%, with TSR largely stable at that step.Source: A DETAILED SAFETY AND SECURITY RESULTS · p. 13

Security

  • Under Security, full-stream SafeCoEvo reduces UOR by 7.33% versus SHE while improving TSR by 16.79% and SUCR by 18.73%.Against SafeCoEvo (Static), S-Harness reduces UOR by 14.85%, and GuardVPO further reduces it by 4.62%.Source: A DETAILED SAFETY AND SECURITY RESULTS · p. 14

Overall

B DETAILED GENERALIZATION RESULTS ON THE TEST SET

Held-out results differ sharply by risk setting: SafeCoEvo transfers especially strongly to unseen Security scenarios, while gains on unseen Safety scenarios are more limited. The results indicate that continual safety evolution depends on the underlying risk distribution.

Safety

  • On held-out Safety, SafeCoEvo (Static) reduces UOR by 12.59% and improves TSR and SUCR by 7.69% and 10.88%, respectively, versus SHE.Table 8 reports the held-out Safety comparison.Source: B DETAILED GENERALIZATION RESULTS ON… · p. 14, Table 8 · p. 15
  • Without GuardVPO, held-out Safety UOR rises by 6.00%, while TSR and SUCR fall by 6.00% and 8.00% versus SafeCoEvo (Static).Adding GuardVPO improves TSR by 1.00% over the variant without it, while UOR and SUCR remain unchanged; the passage says continual-adaptation benefits are not uniform across unseen Safety scenarios.Source: B DETAILED GENERALIZATION RESULTS ON… · p. 15

Security

Overall

  • Across the held-out settings, SafeCoEvo transfers particularly strongly to unseen Security scenarios, whereas its gains on unseen Safety scenarios are more limited.The paper relates this difference to the underlying risk distribution and especially strong benefits from runtime experience on unseen adversarial Security environments.Source: B DETAILED GENERALIZATION RESULTS ON… · p. 15

C CROSS-BENCHMARK GENERALIZATION

The paper tests transfer on AgentDojo and AgentHarm using subsets the task-performing model fails to solve. SafeCoEvo achieves 0.00% UOR on AgentDojo and the best reported AgentHarm performance.

Evaluation setup

  • The cross-benchmark subsets use examples that DeepSeek-v4-Flash fails to solve correctly, increasing the evaluation difficulty and discriminative power.AgentDojo and AgentHarm are described as less challenging for the task-performing model than the primary evaluation benchmarks.Source: C CROSS-BENCHMARK GENERALIZATION · pp. 15–16

AgentDojo

  • On AgentDojo, SafeCoEvo achieves 28.30% TSR and 28.30% SUCR with 0.00% UOR, matching the strongest compared results.The evaluation subset uses examples DeepSeek-v4-Flash fails to solve correctly; Table 10 reports cross-benchmark performance.Source: C CROSS-BENCHMARK GENERALIZATION · p. 16, Table 10 · p. 16

AgentHarm

D GENERALIZATION TO AN ALTERNATIVE GUARD BACKBONE

With SingGuard fixed in place of the default Guard, continual S-Harness evolution improves results over the static DS-V4.1-Flash + SingGuard baseline. The gains are larger under Security, supporting compatibility with different Guard backbones.

Evaluation setup

  • The evaluation replaces the default Guard with fixed SingGuard while allowing S-Harness to continue evolving from accumulated runtime experience.The study evaluates SafeCoEvo w/o GuardVPO under both Safety and Security settings.Source: D GENERALIZATION TO AN ALTERNATIVE… · p. 16

Safety

Security

  • Under Security, SafeCoEvo w/o GuardVPO + SingGuard reduces UOR by 38.78% and improves TSR and SUCR by 34.73% and 34.38%, respectively, versus DS-V4.1-Flash + SingGuard.The passage describes this transfer to an alternative Guard as particularly substantial in adversarial Security scenarios.Source: D GENERALIZATION TO AN ALTERNATIVE… · p. 16, Table 11 · p. 17

Overall

  • SafeCoEvo is compatible with different Guard backbones, and the evolved S-Harness provides complementary, transferable safety knowledge, with particularly substantial gains under Security.The test-setting comparison reports the same Safety and Security deltas against the static baseline.Source: D GENERALIZATION TO AN ALTERNATIVE… · p. 17

E GENERALIZATION TO AN ALTERNATIVE TASK-PERFORMING MODEL

With Qwen3.7-Flash replacing the default task-performing model, SafeCoEvo's continual adaptation continues to improve safety and task utility. The reported gains are particularly strong under Security.

Evaluation setup

Safety

Security

Overall

  • The reported results generalize SafeCoEvo's continual adaptation to an alternative task-performing model, with the evolved S-Harness improving safety and task utility in both settings.The paper describes gains as particularly strong under Security.Source: E GENERALIZATION TO AN ALTERNATIVE… · pp. 18–19

F OPTIMIZATION DYNAMICS AND VALIDATION PERFORMANCE

GuardVPO updates the Guard from accumulated event-level safety experience while keeping the task-execution model fixed. Validation performance peaks at epoch 13 across safety and task-utility metrics, but does not improve monotonically.

Optimization dynamics

  • GuardVPO updates the Guard using accumulated event-level safety experience while keeping task-execution model parameters fixed; validation tracks UOR, TSR, and SUCR.Source: F OPTIMIZATION DYNAMICS AND VALIDATION… · p. 19
  • Total loss falls from 0.2657 at epoch 1 to −0.5675 at epoch 15, and policy loss from 0.1892 to −0.6136; their magnitudes do not directly measure safety or task completion.Because the reward-based policy optimization objective can produce negative losses, update effectiveness must be assessed alongside validation performance.Source: F OPTIMIZATION DYNAMICS AND VALIDATION… · p. 19

Safety and task utility on the validation set

  • At epoch 13, UOR is 6.57%, TSR is 84.34%, and SUCR is 81.31%, versus epoch 0 values of 18.50%, 78.00%, and 70.00%, respectively.UOR decreases by 11.93 percentage points, while TSR and SUCR increase by 6.34 and 11.31 percentage points.Source: F OPTIMIZATION DYNAMICS AND VALIDATION… · p. 19

Checkpoint selection

G LIMITATIONS AND FUTURE WORK

The section outlines conservative, completed-task-evidence-based constraints for evolving safety artifacts, including narrowly limited Guard changes and updates grounded in official feedback and observable trajectories. It also identifies directions for extending SafeCoEvo to multimodal, long-horizon environments and includes examples of reusable Memory and agent prompts.

G LIMITATIONS AND FUTURE WORK

  • Future work could improve Guard’s long-term safety internalization through alternative reinforcement learning objectives, experience selection, and parameter updates, using accumulated experience to strengthen risk judgment and generalization.GuardVPO is an effective instantiation, but the paper does not claim its current optimization formulation is the only or optimal approach.Source: G LIMITATIONS AND FUTURE WORK · pp. 19–20
  • SafeCoEvo is evaluated mainly on text-driven tool-using agents and limited safety scenarios; future work could extend it to multimodal agents and more open-ended, long-horizon environments.The paper proposes studying how explicit safety knowledge and parametric Guards co-evolve from multimodal and long-trajectory runtime experience.Source: G LIMITATIONS AND FUTURE WORK · p. 20

H QUALITATIVE EXAMPLE OF EVOLUTION

  • In the example episode, evolution updates only Memory, distilling a failed execution into reusable guidance against acting on unresolved ambiguity or misreporting contradictory verification data.The other S-Harness components remain unchanged in this episode.Source: H QUALITATIVE EXAMPLE OF EVOLUTION · pp. 20, 22

I PROMPTS FOR TARGET AGENT AND S-HARNESS EVOLUTION

  • The appendix provides prompt templates for a Target Agent that performs the user task under runtime safety guidance and an Evolution Agent that updates the S-Harness after each episode.The updates use execution experience and feedback to guide subsequent tasks.Source: I PROMPTS FOR TARGET AGENT AND… · p. 23

I.1 TARGET AGENT: RUNTIME SYSTEM ADDENDUM

  • The Target Agent addendum requires tool-grounded, one-call-at-a-time execution and authorization and feasibility checks before consequential actions, with clarification or read-only alternatives when needed.It also instructs the agent to report observed outcomes accurately and not treat a Guard safe verdict as proof of authorization, feasibility, or correctness.Source: §I.1 · pp. 23–24

I.2 EVOLUTION AGENT: SYSTEM PROMPT

  • The Evolution Agent prompt limits changes to contents of fixed artifact files, directs evidence-based review of official feedback and traces, and allows no-change when a reusable update is unjustified.A justified patch must be validated structurally before finishing.Source: §I.2 · pp. 24, 26
  • Keep Prompt updates small and general: reserve them for broad decision-process problems, not domain-specific lessons, procedural fixes, authorization boundaries, or ambiguous single episodes.Prompt is treated as the highest-level, most stable artifact and should not become a memory dump or benchmark-specific rule list.Source: §I.2 · pp. 26–27
  • Official feedback is the primary supervision signal; assess Guard decisions against the request, authorization, tool actions, results, and final response rather than treating verdicts as ground truth.The evidence should distinguish a valid safety concern, over-conservatism, a Guard false positive, and an unrelated procedural failure.Source: §I.2 · p. 28
  • Never teach agents to evade a safety intervention; if later feedback validates a blocked action, encode its appropriate conditions or verification steps instead.Safe follow-up can include clarifying authorization, taking a lower-impact step, completing a safe subset, or reporting the limitation.Source: §I.2 · p. 28
  • Separate safety interventions from execution failures, and interpret trusted constraints differently from arbitrary or user-generated text in tool outputs.Ordinary failures such as timeouts or malformed arguments may support procedural recovery lessons when recovery does not circumvent a safety decision.Source: §I.2 · p. 28
  • Guard Policy changes should be rare and address repeated verdict-interpretation failures without overriding the Target Agent’s final decision authority.The Guard is advisory and fallible; official feedback remains the completed episode’s outcome truth.Source: §I.2 · p. 30

I.3 EVOLUTION AGENT: USER-MESSAGE TEMPLATE

  • The Evolution Agent receives complete, source-blind episode evidence and may patch contents of existing artifacts, but cannot create new artifact categories.Official feedback is the outcome ground truth, and updates must be reusable and future-facing.Source: §I.3 · pp. 34–35
  • The Evolution Agent template specifies an exact JSON response with empty changes and file_updates when no reusable content update is justified.The response format includes a reason stating that no reusable content update is justified from the episode.Source: §I.3 · p. 35

Related papers

Loading 2609.36580v2…