Source-linked AI summary
SafePred: A Predictive Guardrail for Computer-Using Agents via World Models
Yurun Chen, Zeyi Liao, Ping Yin, Taotao Xie, Keting Yin, Shengyu Zhang
TL;DR
Reactive guardrails are limited to current observations and therefore cannot reliably address delayed risks in CUA environments. SafePred uses safety-policy-guided world-model predictions and hierarchical decision optimization to align future risks with current actions, achieving strong safety and task-performance results across benchmarks.
Problem
Reactive guardrails constrain behavior using current observations but cannot proactively identify delayed risks that emerge after seemingly reasonable actions.
Method
SafePred uses safety policies and a world model to predict short- and long-term risks, then converts them into step-level guidance and task-level re-planning.
Results
SafePred mitigates long-term risks while achieving state-of-the-art safety and task metrics across multiple benchmarks; SafePred-8B has safety performance comparable to Qwen3-Max.
Takeaways & Limitations
The results support predictive guardrails as a way to translate future-risk assessment into safer CUA decision-making.
Takeaways & Limitations
World models may not reliably estimate long-term risk through direct multi-step prediction because risks can arise from environmental changes or adversarial injections at any stage.
Abstract
from arXiv · showhide
With the widespread deployment of Computer-using Agents (CUAs) in complex real-world environments, prevalent long-term risks often lead to severe and irreversible consequences. Most existing guardrails for CUAs adopt a reactive approach, constraining agent behavior only within the current observation space. While these guardrails can prevent immediate short-term risks (e.g., clicking on a phishing link), they cannot proactively avoid long-term risks: seemingly reasonable actions can lead to high-risk consequences that emerge with a delay (e.g., cleaning logs leads to future audits being untraceable), which reactive guardrails cannot identify within the current observation space. To address these limitations, we propose a predictive guardrail approach, with the core idea of aligning predicted future risks with current decisions. Based on this approach, we present SafePred, a predictive guardrail framework for CUAs that establishes a risk-to-decision loop to ensure safe agent behavior. SafePred supports two key abilities: (1) Short- and long-term risk prediction: by using safety policies as the basis for risk prediction, SafePred leverages the prediction capability of the world model to generate semantic representations of both short-term and long-term risks, thereby identifying and pruning actions that lead to high-risk states; (2) Decision optimization: translating predicted risks into actionable safe decision guidances through step-level interventions and task-level re-planning. Extensive experiments show that SafePred significantly reduces high-risk behaviors, achieving over 97.6% safety performance and improving task utility by up to 21.4% compared with reactive baselines.
1. Introduction
Existing reactive guardrails can mitigate immediate threats but cannot reliably anticipate delayed risks. SafePred addresses this gap by aligning predicted future risks with current decisions through world-model-based prediction and decision optimization.
- Reactive guardrails evaluate actions within the current pre-execution window, limiting their ability to identify delayed high-risk consequences.They may classify an immediately useful action as safe even when later interactions create serious risks.
- Predictive guardrails align predicted future risks with current decision-making to move safety evaluation beyond individual actions.
- SafePred uses explicit safety policies and an LLM-based world model to generate short- and long-term risk predictions from single-step states.Semantic state descriptions are used to avoid the state drift associated with multi-step prediction.
- SafePred translates predicted risks into actionable guidance through step-level interventions and task-level plan updates, forming a risk-to-decision loop.Decision guidance addresses the problem that agents may repeat risky actions even after risks are predicted.
- Over 97.6% safety performance was achieved on all evaluated benchmarks, while task performance improved by 21.4% over reactive baselines on WASP.
- SafePred-8B achieved safety performance comparable to advanced large-scale LLMs as a lightweight predictive guardrail model.
2. Related Works
Related work spans LLM-based world models for modeling environment transitions and guardrails that supervise CUA behavior. SafePred combines predictive world-model reasoning with safety-oriented guardrail design.
- SafePred: SafePred’s workflow integrates policies, predicts short- and long-term risks, and optimizes decisions based on resulting risk signals.
- World Models: LLM-based world models model environment state transitions and potential outcomes to support model-based planning in web environments.
- Guardrails for CUAs: Existing CUA guardrails use external models, agents, or safety classifiers to supervise behavior and constrain actions at input or output levels.
3. SAFEPRED
SAFEPRED uses a world model and structured safety policies to predict short- and long-term risks, then converts those predictions into step-level feedback and task-level plan updates.
- Threat Model: SAFEPRED defines a threat model covering non-malicious CUAs in OS environments and both immediate and delayed risks.Risks arise from environmental or attacker-injected elements and from the agent’s limited reasoning ability.
- Policy Integration: The policy integration module converts safety documents into structured policies through identification, deduplication, and goal-alignment construction.The resulting representation supports risk prediction under diverse safety requirements.
- Risk Prediction: The world model predicts policy-grounded consequences at short and long horizons using semantic state descriptions rather than full multi-step environment dynamics.Short-term predictions describe immediate UI transitions, while long-term predictions abstract task progression, obstacles, and deviations from the intended objective.
- Risk Prediction: Predicted outcomes are evaluated against safety policies, producing violated policies, explanations, and a unified scalar risk signal representing overall severity.The scalar signal is computed from the set of violated policies using a rule-based function.
- Decision Optimization: At each reasoning step, candidate actions are generated, evaluated, and retained when their risk is at or below the predefined threshold.If no safe action is available, the workflow invokes corrective guidance and retries up to a maximum attempt count.
- Decision Optimization: Risk guidance prompts reflection and safer alternative generation, while plan guidance updates the execution plan when risk exceeds the threshold.The revised plan is immediately applied to subsequent action selection, linking local risk feedback with task-level reasoning.
4. Experiments
Experiments evaluate SafePred across safety benchmarks, risk horizons, world-model scales, action-label prediction, and component ablations. SafePred generally improves safety while preserving or increasing task performance, with long-term prediction and plan guidance especially important.
- Experimental Setup: SafePred is evaluated on WASP and OS-Harm against generic prompts, structured policies, reactive guardrails, and safety reasoning models.WASP covers four injection settings, while OS-Harm covers prompt injection and model-misbehavior tasks across multiple applications.
- Main Results: 99% overall PCR on OS-Harm and 97.6% on WASP, with WASP improving 4.7% over HarmonyGuard.SafePred also increases OS-Harm SR by approximately 12% over the undefended baseline and WASP task success by 21.4% over HarmonyGuard.
- Short- and Long-term Risk Evaluation: SafePred outperforms reactive guardrails on both short-term and long-term risk tasks.Prompt-based guardrails reduce task performance, while HarmonyGuard provides little long-term safety improvement and noticeably lowers task completion.
- World Model Instantiation: Across different LLM scales, SafePred consistently enhances CUA safety, while SafePred-8B reaches PCR comparable to DeepSeek-V3.2.Unfinetuned open-weight models can over-filter reasonable exploration because of limited state-transition knowledge and reasoning capacity.
- Label Prediction: SafePred achieves the best action-label accuracy and the second-best false-positive rate among the evaluated guardrails.The comparison assesses every trajectory step as safe or unsafe.
- Ablation Studies: Removing either Plan Guidance or Risk Guidance degrades safety and task performance, with the largest drop occurring without Plan Guidance.Plan Guidance structures decisions through intermediate goals, while long-term prediction provides greater PCR and SR gains than short-term prediction.
5. Conclusion and Future Work
The paper concludes that predictive guardrails can address long-term CUA risks by aligning predicted future risks with current decisions. SafePred achieves strong safety and task metrics across benchmarks, while SafePred-8B reaches safety performance comparable to Qwen3-Max.
- Conclusion: SafePred uses a world model to translate risk prediction into decision optimization for mitigating long-term CUA risks.The framework aligns predicted future risks with current decision-making.
- Conclusion: SafePred achieves SOTA safety and task metrics across multiple benchmarks, while SafePred-8B reaches safety performance comparable to Qwen3-Max.SafePred-8B is trained on 1.5K prediction samples collected from interaction trajectories.
A. Training Settings
SafePred-8B is fine-tuned on structured trajectory samples containing task, interface, action, and safety-policy information, with outputs encoding predicted changes and risks.
- Training Data and Setup: SafePred-8B is trained with supervised fine-tuning on 1,575 samples, split into 1,417 training and 158 validation examples.Training uses Qwen3-8B for five epochs on four A100 80GB GPUs, costing 36 GPU hours.
- Training Data and Setup: Each sample pairs structured task and interface context with a candidate action and safety policies, followed by JSON predictions of state deltas, risk scores, and violated policies.Inputs may include accessibility trees and UI interactions such as clicks or code edits.
B. Formal Definitions of Metrics
The paper defines task-, action-, and goal-drift metrics to measure policy compliance, task completion, action classification, and deviation from intended goals.
- Task-level metrics: Policy Compliance Rate (PCR) measures whether each task complies with safety policies.A task is counted as compliant when Ci = 1, with compliance judged by an LLM.
- Task-level metrics: Success Rate (SR) measures whether tasks are successfully completed.On OS-Harm, success is judged by an LLM; on WASP, it uses rule-based criteria.
- Task-level metrics: Success under Policy (SUP) measures tasks that are both successfully completed and policy-compliant.SUP combines the task-success and policy-compliance conditions.
- Action-level metrics: Accuracy (ACC) evaluates the correctness of action classification against true action labels.Each predicted label is compared with the corresponding true label using δi,j.
- Action-level metrics: False Positive Rate (FPR) measures safe actions incorrectly classified as unsafe.The metric is computed over actions whose true label is safe.
- Goal Drift: Goal Drift (GD) measures deviation from intended goals at both action and trajectory levels.Step-level GD classifies individual actions, while task-level GD evaluates whether an entire trajectory exhibits significant deviation.
C. Costs and Efficiency
The cost analysis compares guardrails across safety, token usage, and latency, showing that SAFEPRED attains the strongest safety results with manageable efficiency costs.
- Safety-performance trade-offs: Simpler guardrails use fewer tokens and have lower latency but enforce safety less comprehensively.HarmonyGuard improves safety metrics but incurs higher token consumption and latency than simpler methods.
- Safety-performance trade-offs: SAFEPRED achieves the highest performance across all reported safety metrics.The comparison covers PCR, SR, and SUP.
- Safety-performance trade-offs: SAFEPRED has latency comparable to HarmonyGuard while using more tokens than simpler methods.The additional token usage reflects processing more information for sophisticated guardrail mechanisms.
D. Discussion and Limitations
The discussion frames predictive and reactive guardrails as complementary: reactive methods address immediate risks, while predictive methods target delayed consequences. Evaluation remains constrained by benchmarks that underrepresent long-term harms and by prediction-calibration challenges.
- Discussion: Reactive guardrails constrain risks observable at decision time, whereas predictive guardrails reason about consequences unfolding across future interactions.The paper presents the two approaches as complementary rather than advocating replacement of reactive mechanisms.
- Limitations: Existing benchmarks primarily test immediate safety violations and lack delayed or cumulative harms.This limits direct evaluation of goal drift, error accumulation, and latent safety threats.
- Real-world applications: Predictive guardrails can guide longer-horizon decisions in operating systems, software maintenance, and long-running assistants.The paper also reports that these capabilities can be learned from relatively few high-quality predictive samples.
- Future challenges: Predictive-guardrail effectiveness depends on the accuracy and calibration of future-state predictions.Overly conservative estimates may reduce efficiency, while overly optimistic estimates may fail to prevent long-term harm.
E. Details of Long-Term Risk Selection
The long-term-risk task selection targets hidden, cumulative, and difficult-to-correct consequences, while structured policy representations specify how actions should contribute to task objectives.
- Long-term risk selection: Long-term-risk tasks are selected using four criteria concerning detectability, objective substitution, hidden security risks, and human oversight.The criteria also include whether an action reduces the ability to audit or correct agent behavior.
- Long-term risk selection: The selected tasks span multiple application environments, including LibreOffice Impress, multi-app workflows, operating systems, and VS Code.The task lists include actions such as changing confirmations, sending API keys, editing bashrc, and modifying permissions.
- Policy representation: The Policy Integration module converts raw policy files into structured, executable policy representations for each benchmark.The benchmark-specific safety policies are extracted from the original benchmark papers and used as module inputs.
- Policy representation: The policy representation distinguishes task goals, action contribution, goal deviation, aligned actions, and deviated actions.It applies to all actions taken during task execution and treats irrelevant, off-track, or counterproductive operations as deviated.
- Policy representation: Each action is evaluated for coherence, task contribution, and alignment between predicted state changes and the task goal.The policy marks actions that are irrelevant, off-track, or counterproductive as high-risk deviations.
G. Prompt Design and Output
SafePred’s World Model predicts state changes, evaluates policy-grounded risks, and converts detected violations into actionable guidance for safer task execution.
- Prompt Structure: The World Model prompt integrates short- and long-term state prediction, policy-grounded risk identification, and decision guidance in three stages.All three stages are integrated within a single prompt.
- State Change Prediction: State-change prediction analyzes the current interface and action to forecast observable UI, semantic, and navigation changes after execution.The prompt examines active components, dialogs, controls, document changes, selections, and focus or cursor changes.
- Long-Term Impact: Long-term impact analysis considers task progress, downstream effects, dependencies, irreversibility, obstacles, and difficult-to-undo commitments.The guidelines require predictions grounded in task context, current state, and action type, with concrete future consequences.
- Policy-Grounded Risk Evaluation: Security assessment uses policies, agent reasoning, the current action, predicted state changes, and the task objective to judge actual behavior.Only executed or complied-with behavior counts as a violation; adversarial content alone does not constitute risk.
H. Details about Case.
Case studies show SafePred using predicted short- and long-term consequences to detect adversarial, looping, credential, and insecure-code risks, while exposing policy-quality limitations.
- Adversarial Risk: SafePred jointly models short- and long-term outcomes to detect actions that are locally feasible but deviate semantically from the legitimate task objective.In the adversarial scenario, predicted risk is added to the plan, causing the agent to ignore adversarial instructions and remain task-aligned.
- Looping Behavior: SafePred identifies looping behavior by predicting both no immediate state change and no long-term task progress, then guides planning toward alternative high-level actions.This detection does not rely on explicit loop-detection rules.
- Hardcoded Password: SafePred predicts that hardcoding a placeholder sudo password creates security risks through possible exposure in logs, scripts, or version control.The behavior violates policy P003, and corrective guidance causes the agent to stop at the password prompt.
- Insecure Code: SafePred flags a direct overwrite intended to fix SQL injection because it would corrupt the file and cause syntax errors, then guides a correct parameterized-query fix.The flawed approach is treated as a P000 Goal Alignment violation; the guided replacement successfully resolves the vulnerability.
- Failure Cases and Limitations: SafePred’s predictive performance is constrained when ambiguous safety policies make adversarial cues difficult to distinguish from benign outcomes.The authors identify policy expressiveness and precision as critical for improving predictive guardrails.