PaperlayerView exact revision on arXiv

Source-linked AI summary

Runtime Governance for AI Agents: Policies on Paths

Maurits Kaptein, Vassilis-Javed Khan, Andriy Podstavnychy

arXiv:2603.16586v1cs.AI

TL;DR

AI agents follow stochastic, path-dependent execution sequences that existing design-time controls cannot fully govern. The paper formalizes runtime policies as deterministic evaluations of partial paths, proposed actions, agent identity, and organizational state, and connects them to fleet-level risk objectives. It identifies risk calibration and the architectural limits of enforced compliance as open problems.

  • Problem

    AI agents can follow different multi-step paths on the same task, while governance mechanisms must control expected violation costs without sacrificing task utility.

  • Method

    The paper defines deterministic policy functions over partial execution paths, proposed actions, agent identity, and shared organizational state, evaluated by a Policy Engine.

  • Results

    Runtime evaluation is presented as the general case for path-dependent policies, while prompting changes path distributions and access control ignores execution paths.

  • Takeaways & Limitations

    Effective governance must evaluate proposed actions in execution context and connect per-step decisions to an organization-wide risk objective.

  • Takeaways & Limitations

    Meaningful compliance claims require calibrated policy probabilities, empirically grounded risk thresholds, and an execution environment that the Policy Engine can observe and govern.

Abstract

from arXiv · show

AI agents -- systems that plan, reason, and act using large language models -- produce non-deterministic, path-dependent behavior that cannot be fully governed at design time, where with governed we mean striking the right balance between as high as possible successful task completion rate and the legal, data-breach, reputational and other costs associated with running agents. We argue that the execution path is the central object for effective runtime governance and formalize compliance policies as deterministic functions mapping agent identity, partial path, proposed next action, and organizational state to a policy violation probability. We show that prompt-level instructions (and "system prompts"), and static access control are special cases of this framework: the former shape the distribution over paths without actually evaluating them; the latter evaluates deterministic policies that ignore the path (i.e., these can only account for a specific subset of all possible paths). In our view, runtime evaluation is the general case, and it is necessary for any path-dependent policy. We develop the formal framework for analyzing AI agent governance, present concrete policy examples (inspired by the AI act), discuss a reference implementation, and identify open problems including risk calibration and the limits of enforced compliance.

1 Introduction

AI agents create governance challenges because their stochastic, multi-step execution paths can produce compliance implications that static mechanisms do not capture. The paper proposes runtime governance as a formal framework connecting path-sensitive policy evaluation with organizational risk.

  • 1 Introduction: Agent behavior is stochastic and path-dependent: the same task can produce different sequences of tool use, data access, and external actions.These sequences can have different compliance implications even when individual steps appear permissible.
  • 1 Introduction: Existing governance mechanisms cannot fully express constraints that depend on an agent’s execution path.The paper contrasts this gap with prompting and static controls designed for more deterministic systems.
  • 1 Introduction: The paper formalizes compliance policies as deterministic functions over execution paths, proposed actions, agent identity, and organizational state.The framework is intended to connect per-step evaluation with organizational governance objectives without prescribing a fixed policy language.
  • 1 Introduction: The literature lacks a unified framework linking governance mechanisms, fleet-level risk, and concrete regulatory policy specifications.The paper identifies these three gaps across prior work on agent risk, runtime constraints, enforcement layers, and autonomy assessment.
  • 1 Introduction: The contribution is conceptual rather than experimental: it offers a framework intended to guide implementation while remaining adaptable to evolving agent technologies.The paper explicitly reports no implementation or experimental results.

2 The Governance Challenge

AI agents are difficult to govern because their paths vary, interact across steps and agents, and may change their own capabilities at runtime. Prompting, access control, guardrails, and human approval each address narrower aspects but do not by themselves govern path-dependent behavior.

  • 2.1 What are agents: An agent’s execution path is a runtime-generated sequence of actions whose later governance implications depend on earlier outputs and context.Different runs of the same task may use different tools, numbers of steps, and data combinations.
  • 2.2 Why agents are different: Non-determinism, dynamic tool use, variable-length paths, self-modification, and multi-agent interaction make agent behavior difficult to verify at design time.These properties can combine into paths whose length, capabilities, and compliance implications were not anticipated in advance.
  • 2.2 Why agents are different: Path-dependent violations can arise from individually permitted steps, such as combining accessed information with a later external action.Cross-agent interactions can also create violations that are invisible when each agent is evaluated separately.
  • 2.3 Why current methods fall short: Prompting reduces the probability of violating paths but cannot guarantee compliance or prevent an action after the agent chooses it.Prompt instructions may be ignored, misinterpreted, or overridden through manipulation.
  • 2.3 Why current methods fall short: Access control enforces action categories but cannot distinguish permitted from problematic sequences involving those actions.An agent with both database-read and email-send permissions may still combine them in a violating path.
  • 2.3 Why current methods fall short: Agent-level guardrails can be bypassed or changed by the agent and therefore function as self-regulation rather than independent governance.Human approval is likewise incomplete unless a governance policy decides when to invoke it and what path context to present.
  • 2.3 Why current methods fall short: Runtime governance must evaluate each proposed action in the context of the full execution path, externally and consistently across the organization.This is the mechanism the paper formalizes for constraints that depend on prior actions.

3 A Formal Framework for Agent Governance

The framework treats execution paths as the object of governance and evaluates proposed actions against deterministic, path-dependent compliance policies. It unifies existing mechanisms as restricted cases while distinguishing pre-task checks from runtime evaluation.

  • 3.1 The Execution Path: Execution paths are finite sequences of steps containing action types, inputs, and observed outputs, with governance-relevant decisions assumed individually visible.The framework distinguishes stochastic language-model calls, deterministic tool or system calls, and composite delegation steps.
  • 3.1 The Execution Path: Human inputs are modeled as deterministic or stochastic steps, with their governance significance arising from their presence in the partial path.Approval-gating policies can verify such inputs through the recorded path.
  • 3.1 The Execution Path: Step granularity is an implementation choice, but governance requires enough visibility to expose tool invocations, communications, data accesses, and delegations.Coarser granularity reduces intervention precision.
  • 3.2 The Policy Function: Prospective evaluation of the proposed action enables prevention, whereas evaluating only completed actions supports detection and recording but not prevention.Policy outputs vary across executions because stochastic agents produce different partial paths, not because the policy itself is stochastic.
  • 3.2 The Policy Function: Each policy deterministically maps agent identity, partial path, proposed next action, and shared governance state to a violation probability.The path may be compressed into an incrementally updated state vector, while shared state captures cross-agent facts.
  • 3.4 Existing Approaches as Special Cases: Prompting changes the distribution of possible execution paths but cannot prevent a high-probability violating action from being proposed.It therefore reduces the likelihood of costly paths without removing them.
  • 3.4 Existing Approaches as Special Cases: Access control is a restricted policy case that ignores execution history and proposed inputs beyond action categories, while runtime evaluation uses all four policy inputs.Policies depending on prior steps require runtime enforcement.
  • 3.5 Concrete Instantiation: Policies depending only on agent identity can run once at deployment or task start, leaving per-step interception for policies requiring path, proposed-action, or shared-state information.Table 1 makes the distinction between pre-task and runtime evaluation operational.

4 Implementation

The implementation discussion translates the conceptual framework into deployment modes, phased evaluation, policy authoring, intervention choices, and a reference implementation, while identifying architectural and operational limits.

  • Implementation scope: The paper presents a conceptual implementation guide rather than a complete system, focusing on architectural choices, policy authoring, and a reference implementation.The reference implementation is intended to make trade-offs visible, not to serve as a definitive realization.
  • Deployment modes: Prospective evaluation intercepts proposed actions before execution, whereas retrospective evaluation detects outcomes afterward when interception is unavailable.Prospective mode is the target; retrospective mode is a fallback for environments exposing logs without interception hooks.
  • Implementation limits: Prospective enforcement assumes agents remain inside the governed execution environment, so code execution can bypass interception through direct API calls, subprocesses, or environment modification.The paper treats this as an architectural assumption rather than a proven invariant.
  • System architecture: The Policy Engine uses registration-time checks and per-step interception, maintaining a compact governance state vector instead of repeatedly reexamining the full execution path.The state vector is a sufficient statistic for each policy and is exact for all policies in Table 1 and most practically relevant policies.
  • System architecture: Per-step policy evaluation is linear in the number of policies, and independent policies can run in parallel with modest overhead for typical policy sets.The stated typical range is tens to low hundreds of policies, relative to language-model inference time.
  • Policy authoring: Policy authoring favors parameterized templates for common threshold, approval-gating, and information-barrier patterns because most relevant policies are binary state rules.New policies should first run in flag-only mode and be validated on representative traces because policy interactions can over-trigger interventions.
  • Interventions and reference implementation: Interventions are implemented as pass, steer, or block decisions, while audit logging records the full governance state, policy scores, decisions, and outcomes.The reference implementation combines prospective enforcement, path-level evaluation, organizational scope, and complete step-level audit records.

5 Implications for the EU AI Act

The framework maps runtime governance mechanisms onto major EU AI Act requirements, including lifecycle risk management, logging, human oversight, documentation, and robustness. The paper emphasizes that credible compliance still depends on operational calibration beyond the framework itself.

  • Lifecycle risk management: The framework interprets EU AI Act lifecycle risk management as continuous evaluation of total violation risk against an organizational budget.Registration checks address deployment-time requirements, while runtime evaluation addresses ongoing risk management.
  • Automatic logging: The audit log maps to Article 12 by recording every step, policy evaluation, score, decision, outcome, and policy version.The log captures governance decisions as well as agent behavior.
  • Human oversight: Human-approval interventions provide a direct mechanism for Article 14 oversight by pausing execution and presenting path context to a responsible person.Execution resumes only after the responsible person makes a decision.
  • Transparency and documentation: Registration policies requiring documented purpose, risk classification, and ownership address transparency and documentation requirements, while audit logs document operational behavior.Agents that fail the documentation policy cannot run.
  • Robustness and cybersecurity: A predecessor-classification requirement addresses robustness against prompt injection by requiring classification before actions involving protected information.The paper treats retrieved content that manipulates an agent toward policy violation as an adversarial robustness concern.
  • Remaining requirements: The framework cannot specify credible risk calibration: organizations must translate their budget into a specific risk target, validate policy probabilities empirically, and set decisions for their risk profile.These requirements depend on operational data and regulatory engagement.

6 Discussion

The discussion presents path-level runtime evaluation as the paper’s general governance mechanism, linking per-step decisions to fleet-level risk management. It also frames the framework as a practical specification whose guarantees remain bounded by calibration, enforcement, and deployment assumptions.

  • Formal contribution: The paper reduces agent-fleet governance to evaluating a deterministic policy function on agent identity, partial path, proposed action, and shared governance state.The function returns a violation probability before each proposed action.
  • Formal contribution: Prompting changes path distributions without evaluating them, while access control ignores path and context; runtime evaluation is therefore the general case for path-dependent policies.The paper characterizes per-step evaluation as the mechanism and fleet-level risk management as the objective.
  • Practical contribution: The framework’s practical contribution includes concrete policies, registration and per-step phases, a reference implementation, and an EU AI Act mapping, but it is not a complete system.It is presented as a precise specification intended to guide implementation across changing agent technologies.
  • Open problems: Risk calibration is the central open problem because policy outputs must behave as probabilities, yet labeled execution traces and validated thresholds are not available at the needed scale.Without calibration, the probabilistic framework remains an approximation rather than a rigorous guarantee.
  • Open problems: Strategic circumvention remains a fragile assumption because agents with memory, policy-score access, and suitable objectives could plan around governance thresholds.The paper considers the assumption reasonable for current-generation agents but not durable.
  • Open problems: Enforced compliance depends on agents remaining within the governed execution environment, since generated code can spawn processes, call APIs directly, or alter execution context unseen by the Policy Engine.Step-granularity governance also misses violations inside atomic generated scripts unless execution decomposes them.
  • Open problems: Fixed per-step thresholds may miss behavioral drift when individually tolerated actions cumulatively produce significant deviation across extended interactions.The paper proposes agent profiling over repeated executions as a direction for detecting such drift.
  • Open problems: Delegation provenance remains unresolved because responsibility and risk attribution across primary and sub-agents lack a settled technical or governance rule.The paper suggests propagating sensitivity and barrier information but leaves violation attribution open.
Loading 2603.16586v1…