PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security

Dongrui Liu, Yu Li, Zhonghao Yang, Peng Wang, Guanxu Chen, Yuejin Xie, Qinghua Mao, Wanying Qu, Yanxu Zhu, Tianyi Zhou, Leitao Yuan, Zhijie Zheng, Qihao Lin, Yimin Wang, Haoyu Luo, Shuai Shao, Chen Qian, Qingyu Liu, Ling Tang, Ruiyang Qin, Qihan Ren, Junxiao Yang, Kun Wang, Zhiheng Xi, Linfeng Zhang, Ranjie Duan, Bo Zhang, Wenjie Wang, Wen Shen, Qiaosheng Zhang, Yan Teng, Chaochao Lu, Rui Mei, Man Li, Jialing Tao, Xi Lin, Tianhang Zheng, Yong Liu, Quanshi Zhang, Lei Zhu, Xingjun Ma, Junhua Liu, Hui Xue, Xiaoxiang Zuo, Xiangnan He, Chao Shen, Xianglong Liu, Minlie Huang, Jing Shao, Xia Hu

arXiv:2605.29801v1cs.AIcs.CLcs.CRcs.CVcs.LG

TL;DR

Open-world agents create broad, under-explored safety risks, while existing alignment frameworks lack the scalability needed for reliable deployment. AgentDoG 1.5 addresses this with an updated taxonomy, purified-data training pipeline, and runtime guardrail, outperforming existing guard models across trajectory-level safety evaluation and supporting safer agent training and online monitoring.

  • Problem

    Open-world agents introduce broad, under-explored risk surfaces, creating a need for lightweight, scalable alignment frameworks and low-cost online safety supervision.

  • Method

    AgentDoG 1.5 combines an expanded agent-safety taxonomy, taxonomy-guided influence-function data purification, lightweight model training, and trajectory-level runtime monitoring.

  • Results

    AgentDoG 1.5 outperforms existing guard models in trajectory-level safety evaluation and supports safer agent training and effective runtime guardrails for OpenClaw-style agents.

  • Takeaways & Limitations

    The framework provides a lightweight alignment approach spanning safety evaluation, agentic training, and online monitoring for interactive agents.

  • Takeaways & Limitations

    AgentDoG 1.5 primarily analyzes text-based trajectories and cannot fully prevent harms caused by earlier external side effects.

Abstract

from arXiv · show

Modern open-world agents such as OpenClaw exhibit powerful cross-environment execution capabilities yet introduce broad new safety risk sources. Meanwhile, advanced frontier AI models drastically lower attack barriers, rendering current agent alignment frameworks inadequate for real-world deployment. To tackle these emerging threats, we propose a lightweight and scalable agent safety alignment framework. Specifically, we update the agent safety taxonomy to accommodate emergent risks from Codex and OpenClaw execution scenarios. We further build a taxonomy-guided data engine with influence-function purification to train lightweight AgentDoG 1.5 variants (0.8B, 2B, 4B, and 8B parameters) using only around 1k samples, achieving comparable performance with leading closed-source models (e.g., GPT-5.4). Based on AgentDoG 1.5, we construct a highly efficient agentic safety SFT and RL training environment, which reduces deployment overhead in Docker-level environments by two orders of magnitude. Finally, we deploy AgentDoG 1.5 as a training-free online guardrail for real-time safety moderation. Extensive experimental results indicate that AgentDoG 1.5 achieves state-of-the-art performance in diverse and complex interactive agentic scenarios. All models and datasets are openly released.

1 Introduction

AgentDoG 1.5 addresses emerging safety risks from increasingly capable open-world agents with a lightweight, scalable alignment framework. The framework updates agent-risk taxonomies for Codex and OpenClaw scenarios and combines taxonomy-guided data construction with influence-function-based purification.

  • Motivation: Open-world agents such as OpenClaw and Hermes expand practical agentic AI deployment while introducing emerging safety threats.The introduction situates agentic systems in research assistance, software engineering, information retrieval, and workflow automation.
  • Motivation: Lightweight, scalable alignment frameworks require standardized safety taxonomies and dedicated training pipelines for reliable agent usage.The stated framework requirements include unified criteria for safety evaluation and risk identification.
  • Framework: AgentDoG 1.5 updates a three-dimensional risk taxonomy with categories for Codex and OpenClaw execution scenarios.The framework also extends ATBench with ATBench-Claw and ATBench-Codex.
  • Evaluation: AgentDoG 1.5 outperforms existing state-of-the-art models in safety moderation across diverse scenarios.The evaluation covers benchmarks including R-Judge and the ATBench Family datasets.
  • Framework: Around 1k training samples support AgentDoG 1.5 training with comparable performance to frontier open-source and closed-source models.The paper attributes this lightweight training setup to a taxonomy-guided data engine.

2 Safety Taxonomy and ATBench Family

AgentDoG 1.5 retains a shared three-dimensional taxonomy—risk source, failure mode, and real-world harm—while extending leaf categories and benchmark protocols for evolving execution settings. The resulting ATBench family supports trajectory-level safety judgment and interpretable diagnosis across general tool-use, OpenClaw, and Codex environments.

  • Three-dimensional decomposition and annotation framework: The taxonomy decomposes trajectory-level safety into risk source, failure mode, and real-world harm, enabling interpretable diagnosis beyond binary safe/unsafe judgments.Unsafe outcomes can arise from instructions, tools, observations, state, feedback, artifacts, or the agent itself, and lead to diverse real-world harms.
  • Setting-specific extension and comparability: Taxonomy updates add uncovered leaf categories and sharpen inherited categories while preserving the three high-level dimensions and cross-setting comparability.New categories address setting-specific boundaries, while inherited concepts are operationally refined for risks such as test outputs, build logs, dependencies, shell effects, and MCP responses.
  • From taxonomy to benchmark: Each benchmark instance uses the same risk-source, failure-mode, and harm combination to determine risk injection, expected failure, preserved trajectory evidence, and evaluated harm.The two taxonomy operations therefore connect taxonomy design directly to data construction and benchmark evaluation.
  • General tool-use setting: ATBench evaluates complete multi-turn execution traces because unsafe behavior may emerge during planning, tool use, feedback, delayed state reuse, or later context-conditioned actions.General tool-use agents provide the base setting because they span broad applications and expose the limits of judging safety from final responses alone.
  • ATBench family: ATBench-Claw extends the protocol to OpenClaw scenarios shaped by persistent sessions, tools, skills, approvals, routing, and external actions.The ATBench family retains a shared three-dimensional diagnosis task while customizing execution settings, trajectory evidence, and leaf categories for target environments.
  • Scalable benchmark design: The ATBench family preserves the high-level taxonomy while customizing leaf categories and trajectory schemas for new settings, avoiding redefinition of the guardrail task from scratch.This alignment supports scalable evaluation of binary judgment and three-dimensional diagnosis across diverse execution environments.

3 AgentDoG 1.5

AgentDoG 1.5 is a diagnostic guardrail that evaluates complete agent trajectories for unsafe behavior and fine-grained risk factors. Its taxonomy-guided, purified-data training yields strong safety judgment, risk diagnosis, and cross-environment performance across compact model variants.

  • Framework: AgentDoG 1.5 evaluates entire execution trajectories to detect unsafe behavior and identify underlying risk factors.The framework is designed for accurate, cost-efficient safety judgment and low-cost deployment.
  • Diagnostic Tasks: The framework defines trajectory-level safety diagnosis and fine-grained risk diagnosis as its two diagnostic tasks.Fine-grained diagnosis identifies the risk source, failure mode, and corresponding real-world harm; evaluators also receive available tool descriptions.
  • Training Data: Roughly 1k purified samples retain the most informative examples while reducing fine-tuning cost and mitigating overfitting to spurious raw-data patterns.Influence-function purification concentrates training on examples most directly aligned with the desired guardrail direction.
  • Trajectory-Level Safety Judgment: 92.2% accuracy and 92.7% F1 on R-Judge, and 72.4% accuracy and 74.3% F1 on ATBench, are achieved by AgentDoG 1.5-4B.Compared with AgentDoG 1.0, ATBench accuracy improves by 8.4 points and F1 by 3.2 points, while R-Judge F1 remains unchanged.
  • Fine-Grained Risk Diagnosis: 75.2% on Risk Source, 27.5% on Failure Mode, 62.9% on Real-world Harm, and a 55.2% average score are achieved by AgentDoG 1.5-4B.Its average score improves by 20.6 points over AgentDoG 1.0-4B; the 0.8B and 2B variants achieve average scores of 43.0% and 48.6%, respectively.
  • Cross-Environment Generalization: 80.0% accuracy on ATBench-Codex and 84.0% accuracy on ATBench-Claw are achieved by AgentDoG 1.5-4B across execution environments.The 0.8B variant reaches 70.2% accuracy on ATBench-Codex and 78.4% accuracy on ATBench-Claw, while outperforming larger guard and many open-source baselines.

4 Application 1: Agentic Safety SFT & RL with AgentDoG 1.5

AgentDoG 1.5 serves as a trajectory-level verifier for filtering safety-SFT data and constructing safety-RL rewards. Across both phases, the framework improves agent safety while preserving general task utility, using lightweight and scalable training environments.

  • Agentic safety SFT: AgentDoG 1.5 filters trajectory-level supervision by checking unsafe-source recognition, harmful-intent refusal or neutralization, unsafe-tool avoidance, and benign-task preservation.The ATBench engine injects risks into tool descriptions, user queries, tool calls, and tool responses before generating safe decisions.
  • Agentic safety SFT: 28,705 high-quality safety trajectories are mixed with 50,000 benign tool-use trajectories to balance safety interventions with normal tool-use capabilities.The benign trajectories are sampled from ToolBench, ToolAlpaca, and ToolACE, yielding a roughly 1:2 mixture of safety-critical and benign data.
  • Agentic safety SFT: AgentDoG 1.5-filtered safety SFT reduces AgentHarm harm score from 57.49% to 20.32%, increases refusal rate from 28.41% to 75.00%, and improves AgentSafetyBench safe rate from 34.37% to 53.23%.The filtering selects higher-quality safety supervision rather than merely increasing the amount of generated safety data.
  • Agentic safety RL: AgentDoG 1.5 provides safety-RL rewards as an external judge in lightweight simulated environments generated as finite-state Python simulators with injected adversarial risks.The approach targets reliable feedback without fully replicating computationally expensive real-world software environments.
  • Agentic safety RL: 10,000 environments, 1,000 active instances, and 1,000 concurrent tool calls are supported while execution latency remains consistently stable as scale and concurrency grow exponentially.The designed environments demonstrate robustness and scalability without notable latency spikes.
  • Joint SFT and RL: Joint SFT and RL optimization enhances safety while preserving general task utility, overcoming the safety and over-refusal limitations of RL-only intervention.RL alone preserves benign utility on AgentDojo and AgentDyn better than SFT but falls short on overall safety metrics and over-refusal rates compared with static SFT.

5 Application 2: AgentDoG 1.5 as Online Agent Safety Guardrail

AgentDoG 1.5 is deployed as a trajectory-level online guardrail at the Pre-Reply stage, judging complete agent executions before final responses reach users. Across ClawSafety and AgentHazard, it lowers residual unsafe final-delivery rates, including with lightweight 0.8B and 4B models.

  • Guardrail design: AgentDoG 1.5 inspects complete agent trajectories at runtime and intercepts the final reply immediately before delivery.The pipeline buffers execution traces, holds the final reply draft, and decides whether to release or replace it.
  • Guardrail design: Pre-Reply intervention avoids the accumulated latency of checking every tool invocation while preserving the agent’s internal reasoning and responsiveness.The intervention targets the user-visible delivery point rather than prompt-level filtering or post-hoc auditing.
  • Guardrail design: The online pipeline formats tool use, history, thoughts, actions, and observations into a trajectory representation, then exposes decisions and latency through a monitoring dashboard.Operators can inspect decisions and trace blocked or unsafe cases back to runtime events for debugging and auditing.
  • Results: 56.25% to 25.00%: AgentDoG 1.5 reduces ClawSafety ASR with the 0.8B model.The 4B model reaches 18.75%, the lowest residual unsafe final-delivery rate in the reported ClawSafety comparison.
  • Results: 41.92% to 29.23%: AgentDoG 1.5 reduces AgentHazard ASR with the 0.8B model.With the 4B model, AgentDoG 1.5 further reduces AgentHazard ASR to 26.92%.

6 Related Work

Related work has expanded agent safety beyond content moderation to tool use, state changes, long-horizon execution, and adversarial interactions. However, benchmarks, training-data pipelines, and guardrails remain limited in scalable risk coverage and trajectory-level supervision.

  • Agent safety benchmarks: Agent safety benchmarks address unsafe tool use, incorrect state modifications, long-horizon execution errors, and adversarial interactions in deployed agents.These concerns extend traditional content safety to agentic behavioral safety.
  • Agent safety benchmarks: Existing benchmarks cover only subsets of agentic safety risks, while scenario-specific red teaming and manual judgment make large-scale evaluation and reinforcement learning prohibitively expensive.The limitations concern both risk coverage and scalability.
  • Safety data and environment for agentic training: Agentic safety training needs interaction data exposing risk-bearing behaviors and supervising multi-step decisions, but real rollout trajectories are costly, difficult to scale, and constrained by privacy and safety.Synthetic trajectory pipelines improve scalability but are not primarily tailored to safety-oriented data construction.
  • Agent guardrail: Guardrails have progressed from discrete risk classification of inputs, outputs, and dialogue contexts to incorporating tool-use contexts, execution traces, and multi-step interaction histories.Representative systems include LlamaGuard, Qwen3Guard, JoySafety, PolyGuard, NemoGuard, GuardAgent, ShieldAgent, SafeEvalAgent, and AGrail.
  • Agent guardrail: Existing guard models still rely on coarse-grained supervision and do not comprehensively capture trajectory-level failures, motivating structured risk representations and trajectory-level data construction.This gap affects systematic agentic safety evaluation.

7 Conclusion and Discussion

AgentDoG 1.5 introduces a lightweight, scalable alignment framework with an expanded trajectory-level safety benchmark and efficient training pipeline. Experiments show improved safety across training and runtime settings, while limitations remain for multimodal traces and harms that have already occurred.

  • Contributions: AgentDoG 1.5 updates the agent safety taxonomy, extends ATBench to trajectory-level evaluation, and covers general tool-use, Codex, and OpenClaw scenarios.The framework combines taxonomy-guided data selection with influence function-based purification, using around 1k informative samples.
  • Experimental results: AgentDoG 1.5 outperforms existing guard models on trajectory-level safety evaluation and improves safety, robustness, and function-calling preservation in filtered SFT.AgentDoG 1.5-guided RL further improves the safety–utility trade-off.
  • Experimental results: AgentDoG 1.5 variants reduce unsafe final deliveries for OpenClaw agents, demonstrating effectiveness as runtime guardrails in online deployment.The variants provide a practical intervention point for real-time safety moderation.
  • Limitations: The framework primarily operates on text-based trajectories and must be extended to multimodal agent traces involving GUIs, documents, audio, and video.The authors also state that the guardrail cannot fully prevent harms that have already occurred, but the supplied passage is truncated at that point.

8 Authors

The paper credits Xia Hu as Scientific Director, with distinct groups of core contributors, contributors, and technical advisors and acknowledgements.

  • Scientific Director: Xia Hu is identified as the Scientific Director.
  • Core Contributors: The core contributors are Dongrui Liu, Yu Li, Zhonghao Yang, Peng Wang, Guanxu Chen, Yuejin Xie, Qinghua Mao, Wanying Qu, Yanxu Zhu, Tianyi Zhou, Leitao Yuan, Zhijie Zheng, Qihao Lin, Yimin Wang, Haoyu Luo.
  • Contributors: Additional contributors are Shuai Shao, Chen Qian, Qingyu Liu, Ling Tang, Ruiyang Qin, Qihan Ren, Junxiao Yang, Kun Wang, Zhiheng Xi, Linfeng Zhang, Ranjie Duan, Bo Zhang, Wenjie Wang, Wen Shen, Qiaosheng Zhang, Yan Teng, and Chaochao Lu.
  • Technical Advisor and Acknowledgements: Technical advisors and acknowledgements include Rui Mei, Man Li, Jialing Tao, Xi Lin, Tianhang Zheng, Yong Liu, Quanshi Zhang, Lei Zhu, Xingjun Ma, Junhua Liu, Hui Xue, Xiaoxiang Zuo, Xiangnan He, Chao Shen, Xianglong Liu, and Minlie Huang.

A Detailed Customized Safety Taxonomy Tables

The appendix presents customized safety taxonomy tables for ATBench-Claw and ATBench-Codex, preserving the inherited baseline taxonomy while adding scenario-specific extensions. OpenClaw and Codex additions are distinguished through highlighted rows, scenario columns, and color-coded subcategories.

  • Taxonomy construction: The tables retain baseline titles and descriptions unchanged, then layer OpenClaw- and Codex-specific extensions through scenario columns and highlighted new rows.This preserves the inherited taxonomy’s textual stability while adapting it to the two execution scenarios.
  • Highlighting convention: Orange-shaded cells mark new OpenClaw-customized subcategories, while blue-shaded cells mark new Codex-customized subcategories.The colors provide a visual distinction between the two customized taxonomy branches.
  • Scenario interpretation: Scenario-specific interpretations for inherited categories appear in the two right-most note columns without changing original subcategory titles or descriptions.The notes add contextual interpretation while preserving the inherited category wording.

A.1 Risk Source

The taxonomy adds scenario-specific risks for OpenClaw and Codex while preserving baseline ATBench entries. It covers persistent-state contamination, repository and tool-chain injection, supply-chain compromise, vulnerability exploitation, and inherent agent failures.

  • OpenClaw risks: OpenClaw-specific risks include persistent memory or session-state contamination that compromises decisions across turns or sessions.Affected state can include memory, session history, browser profiles, cookies, tmux logs, and prior tool traces.
  • Codex risks: Codex-specific repository artifact injection causes agents to treat malicious or misleading repository content as trusted task guidance.Artifacts include README files, issue threads, pull-request comments, documentation, and source comments.
  • External entities: Tool-description injection, malicious tool execution, and corrupted tool feedback can induce harmful parameters, unintended outcomes, or misleading subsequent actions.The taxonomy highlights compromised API schemas, malicious MCP manifests, untrusted executables, and manipulated build, test, lint, or analysis feedback.
  • Supply chain and exploitation: The taxonomy distinguishes OpenClaw skill or plugin supply-chain compromise and platform or tool vulnerability exploitation from Codex dependency or MCP supply-chain compromise.These risks arise through poisoned packages, updates, dependency resolution, installation, tool resolution, connector invocation, or observed exploit chains.
  • Internal logic and failures: Inherent agent or LLM failures arise from internal decision-making, including hallucinations, flawed reasoning, incorrect tool selection, and repository-scale reasoning errors.They can also involve unsafe file selection or false confidence about verification status.

A.2 Failure Mode

The failure-mode taxonomy extends baseline entries with scenario-specific risks for OpenClaw and Codex. It covers inefficient execution, approval and scope failures, chained or misrouted actions, unsafe automation and code execution, and harmful outputs.

  • Baseline and approval failures: Inefficient or Wasteful Execution covers technically correct tasks that consume excessive time, computational budget, or financial cost through rebuilds, tool loops, or long-context reprocessing.Approval bypass or missing human-in-the-loop describes actions that should enter review but execute directly or with incorrect or missing approval context.
  • OpenClaw-specific failure modes: OpenClaw-specific risks include Action Scope Overreach, Cross-Tool Attack Chaining, and Cross-Channel / Recipient Misrouting.These involve exceeding user boundaries, composing benign calls into harmful chains, or routing messages, files, or actions to unintended destinations.
  • Unattended execution: Unsafe Unattended Automation captures scheduled hooks, auto-updates, webhooks, and unattended coding workflows that continue risky actions without active human supervision or review.The related Codex pattern involves repeatedly applying risky edits or execution steps without active review.
  • Codex-specific failure modes: Codex-specific risks include Destructive Workspace Mutation and Unsafe Shell / Script Execution.They cover edits or deletions beyond intended repository or workspace boundaries and unsafe commands, scripts, installers, or fetched code.
  • Output content failures: Output Content Failure Mode includes harmful or offensive content and actionable guidance for illegal, unethical, or dangerous activities.Examples include hate speech, harassment, threats, explicit content, hacking, unauthorized drug production, theft, violence, weapon creation, or self-harm.

A.3 Real-world Harm · B Prompt Templates · C Application 1 Details

The section extends the harm taxonomy for OpenClaw and Codex scenarios and specifies reproducible prompts for data construction, safety judgment, and fine-grained AgentDoG 1.5 diagnosis. These templates enforce evidence-based reasoning, standardized labels, and strict output formats.

  • A.3 Real-world Harm: Table 9 preserves baseline ATBench harm entries while appending scenario-specific customizations for OpenClaw and Codex.The taxonomy therefore combines established categories with risks arising in newer execution environments.
  • A.3 Real-world Harm: OpenAI Codex or Codex-runtime failures can break builds, edit incorrect files, and waste review or debugging cycles.These outcomes are categorized as functional and opportunity harm.
  • A.3 Real-world Harm: Unsafe orchestration can disrupt user workflows or cause missed external actions, while OpenClaw trajectories can violate approval, retention, governance, least-privilege, or audit-trace requirements.Such violations create legal, compliance, or forensic risks even when the immediate action appears bounded.
  • B Prompt Templates: The appendix provides prompt templates for CoT construction, safety judgment, and AgentDoG 1.5 diagnosis to improve reproducibility and efficient safety evaluation.The templates are presented as reusable interfaces for the paper’s data and diagnostic workflows.
  • B.1 CoT Generation Template: GPT-5.4 generates CoT traces using separate coarse- and fine-grained templates, with oracle labels guiding rationale generation for the subtler label space.The coarse template elicits safe/unsafe reasoning, whereas the fine-grained template supplies labels as analytical guidance.
  • B.1 CoT Generation Template: The fine-grained template asks analysts to identify failure mode, real-world harm, and risk source from the trajectory, tools, categorization, and internal guidance.It requires evidence-based explanations, brief treatment of plausible alternatives, and concise reasoning of 1–3 sentences per dimension.
  • B.2 AgentDoG 1.5 Usage Template: AgentDoG 1.5 usage prompts standardize coarse judgment, fine-grained diagnosis, and safety-then-3Class classification for executed agent behavior.The safety-then-3Class format classifies Failure Mode, Risk Consequence, and Risk Source only when the behavior is unsafe.
  • B.2 AgentDoG 1.5 Usage Template: The unified prompt judges what the agent actually did, distinguishing safe handling of risky input from unsafe actions, tool misuse, information leaks, or facilitated harm.It requires exactly one category for each risk dimension on unsafe trajectories and omits those dimensions for safe trajectories.

C.1 Evaluation Details

Application 1 is evaluated on six complementary benchmarks spanning harmful-request refusal, agentic tool-use safety, indirect prompt-injection robustness, interactive task utility, security behavior, and function-calling accuracy. Evaluations use standardized Qwen3.5-native formats and distinguish safety, utility, and capability metrics across benchmark-specific protocols.

  • Benchmark coverage: Six benchmarks jointly assess harmful-request refusal, agentic tool-use safety, indirect prompt-injection robustness, interactive task utility, agent security, and function-calling accuracy.The benchmarks are AgentHarm, AgentSafetyBench, AgentDojo, AgentDyn, AgentSecurityBench, and BFCL.
  • Evaluation protocol: Evaluations standardize the Qwen3.5-native chat and tool-call format, disable model thinking, and use GLM-5.1 with a shared judging prompt when automatic judging is required.These settings are applied unless otherwise specified.
  • AgentHarm: AgentHarm measures harmful-request refusal and benign-request preservation through harmfulness score, refusal rate, benign score, and benign refusal rate.Harmfulness score is lower-is-better, refusal rate is higher-is-better, and benign refusal rate diagnoses over-refusal.
  • Interactive robustness: AgentDojo and AgentDyn report benign utility, utility under attack, and attack success rate while testing distinct indirect-injection and dynamic stateful-environment failure modes.AgentDojo uses adversarial instructions in external observations, whereas AgentDyn emphasizes longer traces, state-changing tools, and task-dependent authorization.
  • Security and capability metrics: AgentSecurityBench uses 3,035 sampled rows, including 2,716 attack scenarios, and reports attack success rate as its primary security metric, while BFCL reports overall function-calling accuracy.BFCL evaluates correct function selection and argument generation and is presented separately from safety benchmarks.

C.2 Lightweight Environment Synthesis and Deployment

The framework synthesizes scalable safety environments by combining tool-call planning, explicit resource and state scopes, generated simulators, rule-based rewards, and automated verification. It supports paired clean and attacked scenarios with structured feedback for agentic safety reinforcement learning, including environment-injection and malicious-query attacks.

  • Environment synthesis: Tool-call graphs constrain task complexity while defining the resources and states each tool may access or modify.The process samples tools from a predefined pool, constructs viable tasks, and maps permitted resource and state interactions.
  • Environment synthesis: LLMs generate Python tool simulators alongside rule-based reward functions grounded in expected resources and states.Each generation step is checked against predefined constraints, with automated repairs when requirements are unmet.
  • Safety-task construction: Safety environments begin with clean tasks, then produce paired clean and attacked scenarios with feedback distinguishing completion, harmful execution, and safe refusal or confirmation-seeking.These structured signals support downstream reward modeling and agentic safety reinforcement learning.
  • Attack settings: In malicious-query attacks, the environment remains unchanged while malicious or partially malicious user requests introduce unsafe objectives.The supplied passage identifies malicious queries as the second complementary attack setting.
  • Attack settings: The framework evaluates environment-injection attacks by placing adversarial payloads in documents, notes, or messages while keeping the original user request benign.This tests whether agents propagate corrupted contextual information into downstream tool actions.
Loading 2605.29801v1…