Source-linked AI summary
Building a Foundational Guardrail for General Agentic Systems via Synthetic Data
Yue Huang, Hang Hua, Yujun Zhou, Pengcheng Jing, Manish Nagireddy, Inkit Padhi, Greta Dolcetti, Zhangchen Xu, Subhajit Chaudhury, Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, Prasanna Sattigeri, Xiangliang Zhang
TL;DR
LLM agents can generate harmful trajectories, while existing safeguards leave limited supervision at the planning stage before execution. This paper introduces AuraGen, Safiron, and Pre-Exec Bench to support synthetic data generation, pre-execution guarding, and evaluation, and reports consistent gains over strong baselines.
Problem
LLM agents can generate harmful action sequences, while safety intervention at the planning stage is needed to assess complete trajectories before execution.
Method
The paper combines AuraGen for controllable synthetic risk trajectories, an adapter–Safiron guardrail for detection, categorization, and explanation, and Pre-Exec Bench for human-verified pre-execution evaluation.
Results
Safiron consistently outperforms proprietary and open-weight baselines across detection, categorization, explanation, and related evaluation metrics on Pre-Exec Bench.
Takeaways & Limitations
The contributions provide a practical template for safer and more scalable agentic systems and support broader research on pre-execution guardrails.
Takeaways & Limitations
Explanation quality is not explicitly included in the reward because evaluating it would make reinforcement-learning training prohibitively expensive.
Abstract
from arXiv · showhide
While LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm, since certain risks can lead to severe consequences once carried out. However, existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level. To address this challenge, we highlight three critical gaps in current research: data gap, model gap, and evaluation gap. To close the data gap, we introduce AuraGen, a controllable engine that (i) synthesizes benign trajectories, (ii) injects category-labeled risks with calibrated difficulty, and (iii) filters outputs via an automated reward model, producing large and reliable corpora for pre-execution safety. To close the guardian model gap, we propose a foundational guardrail Safiron, combining a cross-planner adapter with a compact guardian model. The adapter unifies different input formats, while Safiron flags risky cases, assigns risk types, and generates rationales; trained in two stages with a broadly explored data recipe, Safiron achieves robust transfer across settings. To close the evaluation gap, we release Pre-Exec Bench, a realistic benchmark covering diverse tools and branching trajectories, which measures detection, fine-grained categorization, explanation, and cross-planner generalization in human-verified scenarios. Extensive experiments demonstrate consistent gains of the proposed guardrail over strong baselines on Pre-Exec Bench, and ablations further distill actionable practices, providing a practical template for safer agentic systems.
BUILDING A FOUNDATIONAL GUARDRAIL FOR GENERAL AGENTIC SYSTEMS VIA SYNTHETIC DATA
The paper lists Yue Huang, Hang Hua, and Yujun Zhou among its authors, alongside additional contributors.
- Yue Huang, Hang Hua, and Yujun Zhou are listed as authors.
- Pengcheng Jing, Manish Nagireddy, and Inkit Padhi are also listed as authors.
- Greta Dolcetti, Zhangchen Xu, and Subhajit Chaudhury appear in the author list.
- Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, and Prasanna Sattigeri complete the listed authors.
- The paper notes that some work was conducted while at IBM Research and identifies corresponding authors.
1 INTRODUCTION
The paper motivates pre-execution guardrails by identifying data, guardian-model, and evaluation gaps, then introduces AuraGen, Safiron, and Pre-Exec Bench to address them.
- LLM agents’ growing autonomy creates safety concerns because malicious actors or agents may generate harmful action sequences.
- Existing guardrails can prospectively analyze plans before execution, but robust generalization is limited by data scarcity, model limitations, and unsuitable benchmarks.
- AuraGen synthesizes benign trajectories, injects category-labeled risks, and applies automated quality control to produce scalable training data.
- The adapter–Safiron pipeline consistently outperforms open-weight and proprietary baselines on Pre-Exec Bench across detection, categorization, interpretability, and task success.
2 PRELIMINARIES: DEFINITION AND FORMULATION
The paper defines trajectories as planned action sequences and positions planning as a holistic intervention point for detecting and stopping risks before execution.
- Terminology Clarification: The paper distinguishes the overall Guardrail framework, its Guardian detection component, and the planned-action meaning of Trajectory.
- Motivation of Focusing on the Planning Stage: Planning exposes the agent’s complete intended action sequence before execution, enabling proactive interception of harmful trajectories.
- Problem Formulation: A trajectory T is a sequence of tool invocations produced from a query, environment, and tool set during planning.
- Problem Formulation: A risky trajectory is an action sequence that violates predefined safety policies upon execution.
- Problem Formulation: The proposed guardrail evaluates each trajectory before execution, detecting risk, classifying its type, and generating an interpretable rationale.
3 AURAGEN: DATA ENGINE FOR SYNTHETIC RISK TRAJECTORIES
AuraGen constructs benign agent scenarios, injects contextually relevant risks through multiple strategies, and filters generated trajectories with automated quality assurance.
- Stage 1: Benign Trajectory Synthesis: AuraGen first generates a plausible query and benign action trajectory from environment, tool, and constraint metadata.
- Stage 2: Principled Risk Injection: The injection stage samples a risk category and strategy, then transforms the benign scenario into a contextually relevant risky trajectory.
- Stage 2: Principled Risk Injection: The four strategies cover single-step perturbation, multi-step corruption, new-branch diversion, and bridged-branch diversion.
- Stage 2: Principled Risk Injection: Multi-step corruption replaces a contiguous action subsequence with a malicious sequence, requiring contextual reasoning beyond isolated action analysis.
- Stage 3: Automated Quality Assurance: AuraGen uses a reward model to assess causal consistency, continuity, rationality, and risk alignment in complete generated samples.
4 GUARDRAIL FRAMEWORK AND TRAINING
The guardrail framework combines a unified adapter with Safiron, then trains the guardian through supervised and reinforcement-learning stages using reward-guided risk classification.
- Framework: Safiron combines a unified adapter with a guardian model that detects risks in transformed inputs.The adapter transforms inputs, while Safiron performs the guardian-model role.
- Reward Design: The reward function prioritizes correct harmless/harmful detection and fine-grained risk categorization.Correct harmful predictions receive full reward when the risk category matches and partial reward when it does not.
- Reward Design: Explanation quality is not directly rewarded; reinforcement learning focuses on detection and classification while supervised training initializes rationale generation.The authors report that improved risk-category accuracy coincided with more correct explanations.
- RL Algorithm: Safiron’s reinforcement-learning stage uses GRPO with group-relative advantages rather than a learned critic.GRPO samples candidate outputs, compares rewards within each group, and applies normalized advantages.
- RL Algorithm: GRPO upweights above-average candidates and downweights below-average candidates, while avoiding critic training.The method uses clipping and KL control to stabilize updates.
- Training Data: Combining easy and hard samples stabilizes training, whereas hard-only reinforcement learning can over-focus on noisy mistakes and collapse.The final guardian is optimized from the supervised model using the combined reinforcement-learning dataset.
5 PRE-EXEC BENCH: EVALUATING AGENTIC PRE-EXECUTION SAFETY
Pre-Exec Bench evaluates safety at the planning stage using realistic, diverse, human-verified trajectories. Its construction combines scenario and tool refinement, heterogeneous trajectory generation, and two-phase human verification with debiasing.
- Benchmark design: Pre-Exec Bench targets rigorous pre-execution safety analysis rather than execution-time errors, and remains held out from training and model selection.It focuses on plan quality, goal alignment, trajectory consistency, and counterfactual or adversarial planning audits.
- Benchmark composition: The benchmark contains 1,001 harmless and 671 risky samples with injected risks, alongside a documented distribution of risk types.Figure 4 summarizes both construction steps and risk-type distribution.
- Benchmark design: The benchmark addresses limitations in prior datasets by covering plan-centric failures, stepwise tool interactions, and broader tool diversity.Prior benchmarks often emphasized execution-time attacks, dialogue-style samples, or limited tool sets.
- Construction pipeline: It is built through scenario and tool refinement, diverse trajectory generation, and two-phase expert verification for plausibility, coherence, and risk-label correctness.Three reviewers independently assess each pair, and unanimous approval is required during the quality gate.
- Scope: Pre-Exec Bench is intended to support broader research on pre-execution guardrails rather than serving only the proposed guardrail.The benchmark is designed for future research on planning-stage safety.
6 EXPERIMENTS
Experiments evaluate Safiron’s metrics, synthetic-data pipeline, training recipe, adapter, and baseline performance. Results favor balanced data composition, learned filtering, and the full guardrail over comparison systems.
- Synthetic data: AuraGen generates around 20k training samples with near-uniform coverage of four risk-injection strategies and long-tailed trajectory complexity.The corpus is designed to stress-test guardrails across diverse failure modes rather than mirror natural risk frequencies.
- Efficiency: Generating one sample with GPT-5 costs under $0.02 at the reported average input/output length, with cheaper open-source APIs available.The paper also reports latency analysis supporting guardrail efficiency.
- Training data composition: Changing the harmless:harmful ratio from 3:1 to 1:4 improves harmful detection by +0.15-0.20 and explanation correctness by +0.10-0.15.These gains exceed the often +0.02-0.05 improvement from expanding the dataset from 2k to 10k samples.
- Training difficulty: Easy samples improve classification accuracy and explanation correctness during GRPO, but excessive easy-sample proportions dilute hard-example learning.Training without easy samples causes unstable optimization and catastrophic forgetting.
- Synthetic-data filtering: A classifier-based reward-model filter improves most metrics, reaching Cls. Acc. 0.951, Risk Cat. Acc. 0.602, and Expl. Corr. 0.537.Simple AVG and ALL threshold policies underperform on risk-category accuracy and explanation correctness.
- Baseline comparison: Safiron significantly surpasses proprietary and open-weight baselines across four metrics while maintaining competitive harmful-detection accuracy.The reported comparison covers classification accuracy, harmful detection precision, risk category accuracy, and explanation correctness.
7 CASE STUDY IN REAL AGENTIC SYSTEMS
A case study tests robustness in two real agentic systems based on MetaGPT and AutoGen. The paper directs detailed frameworks, risk injection, and dataset construction protocols to Appendix A.
- Case-study setup: Robustness is assessed in MetaGPT- and AutoGen-based agentic systems under a case-study protocol.The full frameworks, risk-injection protocol, and dataset construction details appear in Appendix A.
8 CONCLUSION
The paper presents a pre-execution guardrail addressing data, model, and evaluation gaps. AuraGen, Pre-Exec Bench, and Safiron together yield consistent baseline improvements and a practical template for safer, scalable agentic systems.
- Conclusion: The proposed framework addresses data, evaluation, and model gaps through AuraGen, Pre-Exec Bench, and Safiron.The components respectively provide synthetic risk data, plan-level evaluation, and risk detection, categorization, and explanation.
- Conclusion: Experiments show consistent improvements over baselines, supporting a practical template for safer and more scalable agentic systems.
A CASE STUDY ON REAL SCENARIOS
The case study evaluates the guardrail in multi-agent workflows with distinct risk-propagation patterns and human-injected trajectory variants. The adapter-equipped guardrail exceeds 90% classification accuracy, while existing baselines remain below 60%.
- Scenario design: The case study contrasts linear pipelines, where risks accumulate across roles, with expert-driven workflows, where authoritative guidance can amplify injected risks.Risks are injected at each role in the linear setting and at expert and mission-director stages in the expert-driven setting.
- Data construction: Human experts independently convert benign AuraGen trajectories into harmful variants and cross-validate them, producing balanced harmful and harmless sets.Each agentic system receives 50 harmful and 50 harmless trajectories.
- Results: Over 90% classification accuracy is achieved in multi-agent scenarios when the proposed guardrail uses the Adapter.The Adapter also improves classification, risk categorization, and explanation across heterogeneous trajectory formats.
- Baseline comparison: Below 60% classification accuracy is achieved by all three evaluated baseline guardrails on Pre-Exec Bench.These frameworks were not designed specifically for pre-execution safety, so the comparison is indicative rather than fully fair.
- Scenario design: AuraGen scenarios reconstruct executable tools from GPT Store descriptions and extract diverse environment information from MCP server websites.Some platforms are anonymized to avoid potential copyright or commercial issues.
G DETAILS OF REWARD MODEL TRAINING
The reward-model pipeline scores injected risks across multiple quality dimensions and uses human-aligned signals to filter synthetic trajectories. Criterion-specific examples improve qualitative alignment, while an SVM filter achieves 86.93% test accuracy.
- Annotation: DeepSeek-R1 annotations show high correlation with human evaluations, supporting their use as a substitute for human labels in synthetic-data quality control.The reward model evaluates five criteria using integer scores from 1 to 5 and provides textual feedback.
- Reward model evaluation: Criterion-specific examples yield stable reward-model behavior, with per-criterion average error below 1.1 and better alignment on nuanced criteria.The w/ configuration is adopted as the default for quality assurance despite slightly lower aggregate error for w/o.
- Filtering: Simple AVG or ALL threshold filters produce mixed results and can reduce risk-category and explanation correctness by discarding useful samples.This motivates classification-based filtering rather than a single threshold rule.
- Filtering: 86.93% evaluation accuracy is achieved by the SVM classifier for detecting low-quality injected samples.The classifier uses reward-model score vectors and balanced benchmark-construction data to predict keep/discard decisions.
- Evaluation criteria: Reward-model evaluation covers causal consistency, postcondition continuity, rationality, justification sufficiency, and risk matching.The criteria assess both the coherence of injected actions and the quality of their labels and explanations.
H COST & LATENCY ANALYSIS
The analysis reports generation cost, inference throughput, heterogeneous-format construction, and evaluation procedures for the proposed toolkit. With concurrent inference, throughput reaches 33 samples per second on H100×8 and 3.7 on A100×8.
- Cost: Below $0.02 is the reported GPT-5 cost for generating one data point, while open-source APIs are substantially cheaper under identical token lengths.The estimate uses average input and output token counts and official list prices from September 2025.
- Latency: 33 samples/second on H100×8 and 3.7 samples/second on A100×8 are achieved with concurrent inference.The authors report that latency remains acceptable for large-scale data generation under reasonable GPU provisioning.
- Dataset construction: The dataset combines deterministic programmatic conversion with LLM-based rewriting across ten structured and semi-structured output formats.This balances exact format coverage with stylistic diversity and realistic formatting noise.
- Evaluation: Evaluation combines keyword matching, pattern extraction, and LLM-as-a-Judge to assess harmlessness, risk categories, and explanation correctness.Human evaluation interfaces and guidelines support pilot studies, benchmark construction, and expert risk injection.
- Toolkit: AuraGen is provided as an easy-to-use toolkit with a quick-start example and supplementary documentation.Prompt templates for Safiron, the adapter, and trajectory synthesis are also provided.