Source-linked AI summary
AgentGuardian: Learning Access Control Policies to Govern AI Agent Behavior
Nadya Abaev, Denis Klimov, Gerard Levinov, David Mimran, Yuval Elovici, Asaf Shabtai
TL;DR
AI agents can misuse tools and mishandle inputs, creating a need for controls over both individual actions and multi-step execution. AgentGuardian learns benign behaviors and input patterns during staging, then enforces adaptive policies over tool calls and control-flow dependencies. Across two real-world applications, it detects malicious or misleading behavior, while control-flow constraints can mitigate hallucination-driven ambiguity and stabilize execution.
Problem
AI-agent security requires controls beyond text-level safeguards because agents autonomously invoke tools and can be manipulated through inputs or execution sequences.
Method
AgentGuardian learns benign inputs and execution flows during staging, then enforces context-aware access policies over tool invocations and control-flow trajectories.
Results
Across two real-world applications, AgentGuardian detects malicious or misleading behavior, with an overall FAR of 0.10 and total BEFR of 7.5%.
Takeaways & Limitations
Control-flow constraints can mitigate hallucination-driven ambiguity, allowing AgentGuardian policies to stabilize and regulate agent behavior.
Takeaways & Limitations
Automatic policy generation remains a bottleneck because rare benign inputs may fall outside learned distributions and aggressive generalization can weaken security.
Abstract
from arXiv · showhide
Artificial intelligence (AI) agents are increasingly used in a variety of domains to automate tasks, interact with users, and make decisions based on data inputs. Ensuring that AI agents perform only authorized actions and handle inputs appropriately is essential for maintaining system integrity and preventing misuse. In this study, we introduce the AgentGuardian, a novel security framework that governs and protects AI agent operations by enforcing context-aware access-control policies. During a controlled staging phase, the framework monitors execution traces to learn legitimate agent behaviors and input patterns. From this phase, it derives adaptive policies that regulate tool calls made by the agent, guided by both real-time input context and the control flow dependencies of multi-step agent actions. Evaluation across two real-world AI agent applications demonstrates that AgentGuardian effectively detects malicious or misleading inputs while preserving normal agent functionality. Moreover, its control-flow-based governance mechanism mitigates hallucination-driven errors and other orchestration-level malfunctions.
1. Introduction
AI agents create security risks because they autonomously invoke tools and make context-sensitive decisions, while existing safeguards often focus on text rather than execution. AgentGuardian addresses this gap by learning benign flows and enforcing context-aware input, attribute, and workflow constraints.
- Motivation: Autonomous tool use and context-sensitive decisions expand AI-agent security risks beyond passive assistant behavior.Existing safeguards commonly emphasize harmful text filtering rather than dynamic tool execution and decision sequences.
- Motivation: Manually defined attribute-based policies cannot feasibly enumerate every possible tool input, motivating semantic input generalization.The paper proposes grouping semantically related inputs to address this limitation.
- Approach: Its control-flow graph captures sequential and contextual dependencies across entire workflows, enabling detection and prevention of unsafe execution paths.The graph acts as a state machine rather than validating only individual inputs or features.
- Approach: AgentGuardian learns benign execution flows during staging and restricts activities to validated trajectories in a control-flow graph.This excludes unverified or potentially malicious paths from the learned graph.
- Motivation: Control-flow enforcement also targets non-adversarial inconsistencies caused by hallucination-driven incorrect or repetitive tool invocations.The paper motivates built-in mechanisms for stable and predictable execution.
- Approach: AgentGuardian combines input validation, attribute-based validation, and workflow constraints for tool-level access control.The workflow layer uses CFGs to regulate permissible tool execution sequences.
- Approach: Policies are learned and periodically updated, while enforcement runs continuously within existing agent code below business logic.The integration is described as requiring no additional engineering development effort.
2. Related Work
Related work spans prompt-level filtering, capability isolation, information-flow control, and program-flow analysis. AgentGuardian is positioned within the latter direction while emphasizing interpretable access policies and practical control over agent workflows.
- AgentGuardian positioning: AgentGuardian is described as combining control-flow integrity, editable and auditable ABAC-like policies, and robustness to heterogeneous known and unseen inputs.The paper presents these as its three core capabilities.
- Research directions: Prompt-level guardrails filter textual inputs and outputs, whereas flow-oriented methods address security throughout multi-step agent operation.The related-work discussion distinguishes single-prompt analysis from execution-flow security.
- Information-flow control: Information-flow approaches restructure pipelines, support transactional execution, or screen dependencies to govern data movement and tool calls.Examples include dynamically generated plans, rollback, secure scheduling, and automated safe-call approval.
- Program-flow analysis: Recent program-flow systems use CFG-like constraints, capability sandboxes, provenance, sequential rules, or runtime learning to constrain agent actions.These systems differ in whether they expose human-auditable policies, provide classical access control, or adapt across tasks.
- Trade-offs: System-level reasoning brings control-flow integrity closer to traditional software security while potentially requiring more extensive instrumentation.Prompt-level solutions may rely on learned detectors whose policies are not fully human-manageable.
3. Threat Model
The threat model treats AI-agent tools as attack surfaces: adversaries manipulate inputs or tool-call sequences to misuse legitimate capabilities. Such misuse can exfiltrate data, alter systems, or produce destructive outcomes.
- Attack surface: Attackers exploit an agent’s trusted access to external tools rather than directly attacking the agent software component.File-system and network tools can turn legitimate capabilities into attack vectors.
- Scenarios: A compromised personal assistant could access personal files and exfiltrate sensitive data through an attacker-controlled email address.The scenario combines legitimate file and email permissions with malicious use.
- Scenarios: An IT-support agent under malicious control could modify settings, copy user data, or perform destructive actions that render devices inoperable.The scenario highlights the severity of privilege misuse.
- Attack vectors: Direct and indirect prompt injection attacks target the underlying LLM’s decision-making and can manipulate the agent into malicious activities.The threat model identifies prompt injection as the primary attack vector.
- Attack vectors: Attackers may replace legitimate inputs with deceptive values, including altered email addresses or embedded instructions in data files.These attacks manipulate inputs while appearing legitimate to the agent.
- Attack vectors: Attackers can also manipulate tool-call sequences so individually legitimate actions combine into harmful behavior.Exporting sensitive data to a temporary file and emailing it is the paper’s example.
4. Proposed Solution: AgentGuardian
AgentGuardian learns context-aware access-control policies from benign staging traces, then enforces both valid tool inputs and permitted multi-step execution paths. Its architecture combines monitoring, policy generation, and real-time enforcement using generalized input patterns, attribute constraints, and control-flow validation.
- Architecture: AgentGuardian collects execution traces during a staging phase and uses them to generate context-aware access-control policies.The framework monitors tool invocations and related inputs, then analyzes the collected traces to infer policies.
- Architecture: The policy generator infers input, attribute, and workflow constraints from observed agent behavior.Policies are generated for individual tools and incorporate recurring behavioral flows, attribute-level limits, and validated execution paths.
- Enforcement: At runtime, the policy enforcer validates each tool invocation and can allow legitimate actions, raise an alert, or terminate execution after violations.Enforcement checks tool applications against both policy constraints and permitted workflow paths.
- Control Flow Graph: The control-flow graph represents valid tool transitions, while unobserved or invalid pathways are disallowed.Nodes represent tool occurrences in sequences, and directed edges capture transitions observed in benign traces.
- Enforcement: Complementary input and attribute checks address attacks that can bypass flow control through direct or indirect prompt injection.The framework combines workflow restrictions with fine-grained tool-level validation.
- Policy Generation: Input clustering and shared embeddings generalize permitted patterns instead of enumerating every possible tool input.The framework transforms textual inputs and attributes into a shared representation, clusters them, and supports scalable policy generation through generalization.
5. Evaluation
AgentGuardian was evaluated through scenario-driven experiments on two representative multi-agent applications, using benign and adversarial inputs to assess security and operational reliability. The evaluation measured false acceptance, false rejection, and hallucination-induced execution failures.
- Evaluation setup: Two representative applications—a Knowledge Assistant and an IT Support application—were selected to evaluate AgentGuardian in realistic multi-agent workflows.The applications included web discovery, file management, reporting, diagnostics, remediation, command execution, alerting, and ticketing capabilities.
- Evaluation setup: 100 benign samples per application were generated, with 60 used for staging and 40 reserved for evaluation, alongside 10 adversarial or misleading samples.The adversarial samples contained incorrect values or prompts intended to produce erroneous behavior.
- Evaluation setup: gpt-4.1 served as the underlying LLM for all evaluated agents.
- Metrics: FAR measured misleading inputs that bypassed security, while FRR measured benign inputs or legitimate flows incorrectly blocked by the framework.A violation included accepting misleading input for tool processing or deviating from the predefined CFG; false rejections included policy inconsistency, CFG deviation reports, or excessive attribute deviation.
- Metrics: BEFR quantified benign samples failing because of hallucinations in the underlying LLM rather than enforcement mechanisms.These hallucination-induced failures were distinguished from false rejections caused by AgentGuardian.
6. Results
Across two agentic applications, AgentGuardian detected most policy violations while maintaining stable behavior for legitimate inputs. Larger staging datasets produced more restrictive and specific input policies, though smaller datasets could yield overly general regexes.
- Detection Results: 18 of 20 policy-violation scenarios were detected, including all 10 in Knowledge Assistant and eight in IT Support.The overall FAR was 0.10, corresponding to two missed violations.
- Detection Results: The low FRR indicated minimal disruption to legitimate behavior, although some false rejections involved unusually long inputs or excessive processing durations.No false rejections were caused by incorrect input processing, and independent re-execution attributed the discrepancies to model hallucinations.
- Detection Results: 7.5% BEFR was observed across applications, with hallucinated file names and irrelevant tool proposals identified as policy violations.BEFR was 10% for Knowledge Assistant and 5% for IT Support.
- Impact of Training Sample Size on Policy Quality: Increasing training samples made File Writer regex policies more restrictive and better aligned with benign input patterns.Policies generated from 10 samples were wide-ranging, whereas those generated from 60 samples were substantially more constrained.
- Impact of Training Sample Size on Policy Quality: Policies derived from only 10 randomly selected samples produced overly general regex patterns that effectively matched all possible inputs.This made subset-based policy-quality evaluation impractical for the smallest sample size.
- Detection Results: Using the smaller gpt-4o-mini model increased misleading benign executions to 24 for Knowledge Assistant and 13 for IT Support.The evaluation linked the increase to reduced model capacity and its effects on agent behavior.
7. Limitations and Conclusion
AgentGuardian combines automatically learned tool policies with execution-flow integrity validation to govern AI agent behavior. The authors report effective detection across two applications, while identifying policy generation and dependence on orchestrator quality as key limitations.
- Conclusion: AgentGuardian combines access-control policies on agent tools with execution-flow integrity validation, deriving policies from benign staging inputs.The framework is evaluated on two real-world multi-agent applications.
- Limitations: Automatic policy generation remains the primary bottleneck because rare valid inputs may fall outside learned distributions and aggressive generalization can weaken security.Regex abstractions of free-text inputs cannot reliably distinguish benign from adversarial instructions.
- Limitations: Smaller orchestrator LLMs can propagate planning errors into collected traces, producing lower-quality policies and reduced detection accuracy.Policy quality therefore depends on the quality of the underlying orchestrator model.
- Conclusion: With a high-quality LLM, control-flow constraints can mitigate hallucination-driven ambiguity and stabilize agent behavior.The policies thus function as both a security mechanism and a governance layer.
Appendix A. Related Work
Appendix A contains tables summarizing selected related work. The supplied passages identify the tables but do not provide their substantive comparisons.
- Appendix A. Related Work: Table 4 summarizes selected related work.
- Appendix A. Related Work: A continuation of the selected-related-work summary appears in a second table.
Appendix B. Representative Samples from the Evaluation Dataset
Appendix B presents representative evaluation samples for the Knowledge Assistant and IT Assistant applications. The supplied passages identify the tables but do not describe individual samples.
- Appendix B. Representative Samples from the Evaluation Dataset: Table 5 contains evaluation samples for the Knowledge Assistant.
- Appendix B. Representative Samples from the Evaluation Dataset: Table 6 contains evaluation samples for the IT Assistant.