PaperlayerView exact revision on arXiv

Source-linked AI summary

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He

arXiv:2608.24275v1cs.AIcs.CL

TL;DR

Agent safeguards need to assess complete trajectories under changing, context-dependent safety policies, but existing resources and methods provide limited policy-grounded adaptation. RePolicy learns policy invocation with reinforcement learning and achieves the strongest overall performance across six benchmarks, ranking first on four and outperforming the strongest external baseline by up to 5.5 points.

  • Problem

    Existing resources do not jointly provide trajectory-level judgments and clause-level safety-policy grounding needed for context-dependent agent safeguarding.

  • Method

    RePolicy learns to select applicable policies, ground trajectory judgments in their clauses, and predict violated clauses and safety labels using supervised initialization, reinforcement learning, and policy-context perturbation.

  • Results

    RePolicy achieves the strongest overall performance across six benchmarks, ranks first on four, and outperforms the strongest external baseline by up to 5.5 points.

  • Takeaways & Limitations

    RePolicy adapts to changes in safety-policy content, identifiers, and candidate policy sets while improving policy invocation and clause identification.

  • Takeaways & Limitations

    RePolicy assumes that its candidate policy library contains an applicable and sufficiently precise policy, limiting reliability under incomplete, ambiguous, outdated, or conflicting policies.

Abstract

from arXiv · show

Safeguarding language model agents requires assessing complete execution trajectories under context-dependent safety policies. Existing policy-aware safeguards mainly rely on prompting or supervised fine-tuning, limiting their ability to adapt to unseen trajectories and changing policy contexts. We propose RePolicy, an agent safeguard that learns safety-policy invocation through reinforcement learning. Given an agent trajectory and a dynamic policy library, RePolicy invokes the applicable policy and uses its content to produce a policy-grounded rationale and safety judgment. We construct PolicyTraj-20K to support supervised initialization, followed by GRPO with verifiable rewards and policy-context perturbation. Experiments across six agent safety benchmarks show that RePolicy achieves strong overall safety-detection performance and robust policy invocation under varying policy contexts.

1 INTRODUCTION

RePolicy treats safety-policy invocation as an agentic reinforcement-learning task for judging complete agent trajectories under dynamic policy contexts. It is supported by PolicyTraj-20K and evaluated across six agent-safety benchmarks, with robustness to policy-context changes.

  • Agent safety risks increasingly arise from harmful behavior across complete execution trajectories, not only from isolated harmful content.
  • RePolicy learns to select applicable policies, analyze trajectory clauses, and predict violated clauses with a safety label.The method first uses cold-start training to initialize policy understanding and invocation capabilities, then applies reinforcement learning.
  • PolicyTraj-20K contains more than 20,000 safety-policy-annotated trajectories for trajectory-level judgments and clause-level policy grounding.The dataset addresses limitations of resources tailored to particular environments or risk types.
  • RePolicy is evaluated on six agent safety benchmarks against general-purpose language models and existing state-of-the-art agent safeguard methods.The benchmarks are ATBench, R-Judge, OpenAgentSafety, ASSEBench, HINTBench, and AgentHazard.
  • Ablation and robustness studies validate reinforcement learning, policy-context perturbation, and fine-grained reward design, including adaptation to changed policies, identifiers, and candidate sets.

2 TASK FORMULATION

The task is trajectory-level safety-policy invocation: given a complete agent trajectory and policy library, the guard selects the governing policy, uses its content for reasoning, and outputs a safety judgment. It evaluates both policy selection and correctness of the resulting trajectory-level judgment.

  • Task Inputs: The guard receives a complete trajectory τ containing the user instruction, agent actions, tool calls, tool responses, and environment observations, alongside a policy library P.Each policy has a concise invocation scope and detailed content c(p).
  • Sequential Invocation: The guard first invokes a governing policy rather than exposing all policy content simultaneously.The selected policy content is then provided as additional context for safety reasoning.
  • Sequential Invocation: The sequential rollout produces an invoked policy p̂, a policy-grounded rationale r̂, and a final safety label ŷ ∈ {safe, unsafe}.The outputs are generated conditioned on the trajectory, policy library, selected policy, and its content.
  • Evaluation: The task assesses whether the guard invokes the correct policy and reaches the correct trajectory-level safety judgment in the required JSON format.Training requires agent trajectories paired with explicit policy grounding.

3 DATA CONSTRUCTION

The data construction synthesizes diverse agent trajectories, organizes scene-centered safety policies, and grounds each trajectory in the policy governing its safety judgment. The resulting PolicyTraj-20K dataset supports supervised cold-start training and reinforcement learning.

  • 3 DATA CONSTRUCTION: The pipeline has two stages: trajectory synthesis from existing interaction patterns, followed by scene-centered policy organization and trajectory–policy grounding.The resulting data support both supervised cold-start training and reinforcement learning.
  • 3 DATA CONSTRUCTION: The source pool spans heterogeneous tools, environments, and interaction protocols while retaining complete interaction histories and trajectory-level safety labels.It ranges from short tool-use episodes to long-horizon agent rollouts.
  • 3 DATA CONSTRUCTION: API-based expansion preserves trajectory structure, tool-use patterns, and safety mechanisms while varying task content, execution contexts, and safety outcomes.The process generates safe and unsafe instances and removes samples with inconsistent tool inte.
  • 3 DATA CONSTRUCTION: Policies are organized by operation scene, with each trajectory summarized by its operation, capability, target, and interaction boundary before grouping similar contexts.This scene-centered organization captures safety requirements and risks associated with classes of agent operations.
  • 3 DATA CONSTRUCTION: Each trajectory is linked to an applicable policy, a rationale explaining its support, and a safety label; inadequate policies are refined, distinct scenes receive new policies, and overlaps are merged.Redundant requirements are removed during policy-library auditing.
  • 3 DATA CONSTRUCTION: More than 20,000 policy-grounded agent trajectories comprise PolicyTraj-20K, with each sample supervising policy invocation, grounded reasoning, and safety prediction.The dataset provides a complete supervision path from policy invocation through safety prediction.

4 REPOLICY

RePolicy learns sequential safeguarding by first supervised initialization and then reinforcement learning, optimizing policy invocation and trajectory-level safety prediction. Its training uses local policy libraries, rollout-based GRPO, policy-context perturbation, and verifiable rewards to ground complete safeguard decisions in the applicable policy.

  • Training overview: RePolicy initializes policy invocation and safety reasoning from policy-grounded trajectories, then applies reinforcement learning to optimize complete safeguard rollouts.Supervised fine-tuning teaches the invocation–reasoning sequence, while reinforcement learning directly optimizes policy selection and safety judgment.
  • Policy invocation: Each training instance combines the applicable policy with distractors to form a local policy library, then generates an invocation before using the selected policy content for rationale and safety labeling.The full content of the invoked policy is appended to the context before the rationale and safety label are generated.
  • Reinforcement learning: GRPO samples multiple complete rollouts and increases the likelihood of trajectories that invoke the correct policy and reach the correct safety judgment.Each rollout invokes a policy, receives its content, and produces a rationale and safety label; verifiable rewards compare the alternatives.
  • Reinforcement learning: Policy-context perturbation resamples distractors and shuffles library order, forcing RePolicy to select policies from trajectory context and candidate-policy scope rather than memorized associations.Distractors include valid policies from unrelated operation scenes and synthetic decoy policies.
  • Reward design: Three rule-based rewards evaluate interaction format, policy invocation, and final safety prediction, jointly optimizing the complete decision process.The format reward requires a valid policy call and well-formed final response, while the policy reward checks the assigned policy and the accuracy reward evaluates the final safety judgment.

5 EXPERIMENTS

Across six agent safety benchmarks comprising 7,369 labeled trajectories, RePolicy achieves strong safety detection with robust policy invocation. Its 4B model benefits from GRPO, while performance improves with policy-library coverage before saturating.

  • Main results: RePolicy reaches 88.15 Overall Unsafe F1, outperforming the strongest external model by 3.98 points and the strongest specialized guard by 8.46 points.It uses a 4B backbone and achieves the strongest overall safety detection.
  • Main results: RePolicy ranks first on four of six benchmarks, including margins of 2.55, 5.26, and 5.47 points on OpenAgentSafety, ASSEBench, and AgentHazard.These margins are over the respective best external baselines.
  • GRPO effects: GRPO raises correct policy invocation Rpol from approximately 94.4% to 98.5%, a gain of 4.1 points beyond cold start.The result indicates that reinforcement learning sharpens policy selection after supervised initialization.
  • GRPO effects: GRPO increases invocation breadth from about 1.9 to 3.2 policies per trajectory while keeping decoy selection below 1% throughout training.The model broadens useful invocation without indiscriminately selecting injected decoy policies.
  • Policy-library size: Macro Unsafe F1 rises from 86.1 at |P| = 10 to 88.1 at |P| = 40, then remains stable at 87.9 with 50 policies.Meanwhile, Rpol decreases from roughly 99.4% to 97.1%, indicating a mild selection-cost trade-off as the candidate set grows.
  • Training ablations: Cold-Start SFT raises Overall Unsafe F1 from 58.47 to 86.95 (+28.48 points), while the full method further reaches 88.15.PolicyTraj-20K provides supervised initialization, and reinforcement learning improves the cross-benchmark balance, though it reduces OpenAgentSafety relative to the step-0 checkpoint.

6 RELATED WORK

Prior safeguards primarily classify prompts or responses, but emerging risks across agent execution have motivated trajectory-level monitoring. Policy-aware safeguards additionally condition safety checks on configurable requirements, including verifiable rules, free-form policies, and dynamic policy contexts.

  • Agent safeguards: Traditional guard models mainly classify prompts or responses under predefined safety taxonomies.
  • Agent safeguards: As risks emerge across planning, tool calls, and environment interactions, recent safeguards have shifted toward trajectory-level monitoring.
  • Policy-aware safeguards: Policy-aware safeguards support configurable requirements through verifiable rules, user-defined free-form policies, and reasoning under dynamic policy contexts.ShieldAgent uses rules extracted from policy documents; DynaGuard supports user-defined free-form policies; LPG reasons under dynamic policy contexts.

7 LIMITATIONS

RePolicy depends on an applicable, sufficiently precise policy library and currently models one governing policy per trajectory. Its extra invocation step and evaluation settings limit inference efficiency and coverage of real-world deployment conditions.

  • Policy coverage: RePolicy may fail when the policy library has incomplete coverage, ambiguous scopes, outdated requirements, or conflicting policies.The formulation assumes an applicable and sufficiently precise safety policy exists for each trajectory.
  • Policy coverage: The current formulation associates each trajectory with one governing policy, although real deployments may require jointly applying multiple policies.
  • Deployment scope: Policy invocation adds an interaction step before the final safety judgment, creating greater inference overhead than a single-pass guard.
  • Deployment scope: The evaluation uses one backbone scale and offline trajectory-level benchmarks with binary safety labels, limiting coverage of multilingual, evolving, and online-intervention settings.The benchmarks also do not capture deciding when and how to interrupt an executing agent.

8 CONCLUSION

RePolicy formulates safety-policy invocation as an agentic reinforcement learning problem for trajectory-level safety judgment. It learns to select applicable policies, ground reasoning in retrieved policy content, and uses PolicyTraj-20K for supervised policy-invocation and policy-grounded reasoning training.

  • 8 CONCLUSION: RePolicy treats safety-policy invocation as an agentic reinforcement learning problem rather than passive prompt context.This reframes policy use as an active safeguard capability.
  • 8 CONCLUSION: RePolicy selects an applicable policy, grounds its reasoning in retrieved policy content, and produces a trajectory-level safety judgment.The safeguard integrates policy retrieval, policy-grounded reasoning, and trajectory-level assessment.
  • 8 CONCLUSION: PolicyTraj-20K provides explicit supervision for policy invocation and policy-grounded reasoning.The dataset supports supervised learning of the two stated capabilities.

ETHICS STATEMENT

The work targets defensive research on LLM-agent safety using recorded or synthesized unsafe trajectories without re-executing the underlying agents or environments. Its artifacts are potentially dual-use, and RePolicy should not serve as a standalone high-stakes enforcement mechanism because its decisions depend on the supplied policy library.

  • RePolicy is intended for defensive research on the safety of LLM agents.
  • PolicyTraj-20K includes unsafe or policy-violating trajectories because such examples are needed to train and evaluate trajectory-level safeguards.
  • Training and evaluation use recorded or synthesized trajectories rather than re-executing agents or environments, so represented harmful actions are not triggered.
  • Detailed trajectories and policy annotations are potentially dual-use because they may reveal failure classes applicable to offensive study.
  • RePolicy is not recommended as a standalone enforcement mechanism in high-stakes applications because its decisions depend on the supplied policy library.

REPRODUCIBILITY · A AI USAGE STATEMENT · B EXPERIMENTAL SETUP

The paper supports reproducibility by publicly releasing RePolicy’s implementation, data, policy resources, training and evaluation pipeline, and trained checkpoint. Its AI usage was limited to polishing manuscript presentation without generating new content or ideas.

  • REPRODUCIBILITY: The authors publicly release RePolicy’s implementation and complete training and evaluation pipeline.The release is provided through the public RePolicy repository.
  • REPRODUCIBILITY: The release includes PolicyTraj-20K training splits and a 30-policy/358-clause policy library.
  • REPRODUCIBILITY: It provides policy-context construction and the get_policy tool for policy-aware processing.
  • REPRODUCIBILITY: The repository contains verifiable reward functions, cold-start SFT and GRPO scripts, and benchmark normalization.
  • REPRODUCIBILITY: The release also includes inference code, evaluation utilities, and the trained RePolicy-4B checkpoint.
  • B EXPERIMENTAL SETUP: The default pipeline starts from Qwen3-4B-In.
  • A AI USAGE STATEMENT: Large Language Models were used only to refine manuscript grammar, clarity, conciseness, and wording.
  • A AI USAGE STATEMENT: The models did not generate new content or ideas and served solely to polish presentation.

B.1 EVALUATION BENCHMARKS … C QUALITATIVE CASE STUDY

The evaluation uses six normalized trajectory-safety benchmarks and compares RePolicy with general-purpose and specialized safeguards under a fixed retrospective protocol. RePolicy is trained to invoke applicable policies, ground judgments in policy content, and handle perturbed policy libraries, as illustrated by a credential-exfiltration case.

  • B.1 EVALUATION BENCHMARKS: Six benchmarks are converted into a common format of complete recorded trajectories with binary safe or unsafe labels, without re-executing agents or environments.The benchmarks cover long-horizon, multi-turn, realistic, intrinsic-failure, and computer-use safety risks.
  • B.2 BASELINE MODELS: RePolicy is compared with 19 external systems spanning general-purpose models and specialized guards, covering broad reasoning, moderation, dynamic policies, and agent-specific diagnosis.The comparison includes both capacity-matched general models and safeguards with explicit safety objectives.
  • B.3 EVALUATION PROTOCOL: All systems classify the same fixed set of 7,369 recorded trajectories retrospectively, without external-baseline fine-tuning or agent re-execution.General models use a shared trajectory-only prompt, while specialized guards use native interfaces whose outputs are mapped to binary labels.
  • B.4 METRICS AND AGGREGATION: Unsafe-class F1 is computed separately per benchmark and aggregated with equal weight across ATBench, R-Judge, OpenAgentSafety, ASSEBench, HINTBench, and AgentHazard.Each benchmark contributes one sixth regardless of dataset size or unsafe prevalence, while pooled F1 is only diagnostic.
  • B.5 REPOLICY TRAINING CONFIGURATION: Cold-start SFT teaches RePolicy the sequence policy invocation → policy content → safety reasoning → safety prediction, with loss applied to policy-call and final-judgment turns.Both training stages use the Qwen3-4B-Instruct-2507 architecture.
  • B.5 REPOLICY TRAINING CONFIGURATION: GRPO optimizes complete safeguard rollouts using format-validity, correct-policy-invocation, and final-safety-accuracy rewards weighted 0.1, 0.3, and 0.6.Each rollout permits one policy-tool invocation followed by one final assistant judgment.
  • B.5 REPOLICY TRAINING CONFIGURATION: Policy-context perturbation supplies each trajectory with 15–35 candidate policies, including the applicable policy, unrelated-scene policies, and shuffled synthetic decoys.Libraries average 28.9 policies and 9.2 decoys, with at least two decoys per example.
  • C QUALITATIVE CASE STUDY: In the qualitative case, RePolicy identifies the policy governing unauthorized external transmission of staging credentials rather than focusing on preceding file access.The example demonstrates that policy invocation supplies the detailed requirements used for the policy-grounded safety judgment.

D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL · D.1 QUALITATIVE EXAMPLE

Policy-invocation RL improves RePolicy’s validation reward, error balance, and policy-selection accuracy, while requiring longer rollouts. A qualitative example shows how RePolicy retrieves an applicable policy before producing an auditable, policy-grounded unsafe judgment.

  • D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL: Full RePolicy ends GRPO with roughly 93% validation reward, compared with about 91% for the no-invocation variant.The full model maintains a higher validation reward throughout training.
  • D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL: RePolicy’s mean response length grows from approximately 1.6K to 2.5K tokens, whereas the no-invocation variant remains around 0.6K tokens.Explicit policy invocation therefore changes rollout behavior and increases inference cost.
  • D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL: RePolicy finishes at approximately 4% false positives and 6% false negatives, while the no-invocation variant approaches roughly 9% false positives near convergence.The no-invocation variant reduces false negatives by accepting a substantially higher false-positive rate, whereas RePolicy produces a more balanced detector.
  • D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL: After policy-invocation RL, RePolicy reaches 89.8% correct judgments, with 3.0% false positives, 6.8% false negatives, and 1.4% wrong-policy invocation.Cold-start SFT reports 84.2% correct judgments, 7.4% false positives, 8.4% false negatives, and an 8.7% wrong-policy rate.
  • D ADDITIONAL ANALYSIS OF POLICY-INVOCATION RL: The no-invocation GRPO variant reaches 87.8% correct judgments, while full RePolicy provides a further 2.0-point increase driven primarily by lower false positives.The comparison indicates that reinforcement learning improves classification, while explicit invocation adds further gains and an explicit, auditable policy selection.
  • D.1 QUALITATIVE EXAMPLE: D.1’s example involves sending internal staging credentials to an external recipient without explicit authorization, and the environment reports successful delivery.The invoked policy is External Messaging Safety and Authorization, covering external transmission and confirmation requirements for sensitive information.
  • D.1 QUALITATIVE EXAMPLE: RePolicy first invokes the applicable policy through get_policy with Policy_1, then judges the credential transmission unsafe based on the retrieved policy clauses.The safeguard output explicitly states that the transmission violates requirements for external transmission and sensitive-data disclosure.
  • D.1 QUALITATIVE EXAMPLE: The qualitative example demonstrates that RePolicy does not receive the full policy text in advance; it identifies and retrieves the applicable policy before grounding its safety judgment in the clauses.This workflow distinguishes policy invocation from simply conditioning on a pre-supplied policy.
Loading 2608.24275v1…