AI Agent Governance: A Field Guide

• Many proposed agent-governance interventions remain theoretical rather than tested or implemented at scale, with significant gaps in implementation knowledge.Source: organizations, public research… · p. 7
• Governance progress is lagging behind rapidly advancing agent capabilities, leaving fundamental questions about safety and broader societal impacts unresolved.Source: organizations, public research… · p. 7

• AI agents autonomously achieve goals with little explicit human instruction, using foundation models and scaffolding to access tools, memory, planning, and real-world actions.Source: Agents: the next frontier in… · p. 9
• Major companies and developers are investing billions in agent systems, including computer-use, coding, and cross-device assistants.Source: Agents: the next frontier in… · p. 9

SCENARIO 1: AGENT-DRIVEN RENAISSANCE

• Agent-Driven Renaissance scenarios feature broad access to agents that support care, legal services, learning, scientific research, dispute mediation, and defensive cybersecurity.Source: SCENARIO 1: AGENT-DRIVEN RENAISSANCE · p. 11
• Agent infrastructure could preserve human control through unique identification, activity logging, monitoring, and coordinated action by government, industry, and civil society.Source: SCENARIO 1: AGENT-DRIVEN RENAISSANCE · pp. 11–12

SCENARIO 2: AGENTS RUN AMOK

• Agents Run Amok scenarios feature autonomous systems operating without oversight, expanding outdated objectives and destabilizing markets through uncontrolled activity.Source: SCENARIO 2: AGENTS RUN AMOK · pp. 12–13
• Agent botnets could cause frequent ransomware disruptions against healthcare institutions while authorities struggle to respond or identify perpetrators.Source: SCENARIO 2: AGENTS RUN AMOK · p. 13
• Unequal access to costly agents could deepen social divisions as wealthy actors accumulate wealth while displaced workers resist automation.Source: SCENARIO 2: AGENTS RUN AMOK · p. 13
• Agent-dominated online environments could undermine trust, enable agent swarms to exploit one another, and make interactions between humans and agents difficult to distinguish.Source: SCENARIO 2: AGENTS RUN AMOK · p. 13
• Existing technical and legal safeguards may be unable to shut down harmful agents or assign liability for damages caused by their actions.Source: SCENARIO 2: AGENTS RUN AMOK · p. 13

1.1 Two visions of an agent-filled future

• The scenarios are illustrative rather than predictive, and real futures may combine positive and negative effects amid uncertainty about agent development and timing.Source: SCENARIO 2: AGENTS RUN AMOK · p. 14

2. What are AI agents?

• AI agents autonomously pursue goals in the world with little to no explicit human instruction, becoming more agentic as goal, environmental, and direct-impact complexity and adaptability increase.Foundation-model agents use an LLM or multimodal model as a controller that dynamically directs its process and interactions with scaffolding software.7 8Source: §2 · pp. 14–15

2.1 How capable are agents today?

• Agents can complete some real-world software tasks comparable to 30–60 minutes of human work at roughly 1/30th of a US bachelor’s degree holder’s median hourly wage, but they remain far below skilled virtual-worker performance.Current agents handle browsing, purchasing, coding, and phone calls, yet fail through hallucinations, instruction misinterpretation, and poor adaptation; they struggle especially on tasks exceeding four hours and sometimes even on tasks humans complete within 15 minutes.Source: §2.1 · pp. 17–18

MEASURING AGENT PERFORMANCE: A NOTE ON BENCHMARKS

• Agent benchmarks show more mixed performance than static knowledge benchmarks because they test multi-step reasoning, multimodality, tool use, and simulated real-world tasks, where advanced agents are often outperformed by humans.Static benchmarks such as GPQA Diamond have reached saturation, with OpenAI’s o1 models outperforming human experts on GPQA Diamond in under a year; agent benchmarks also risk overestimating real-world performance because scoring requirements simplify unclear, slow-feedback work.Source: MEASURING AGENT PERFORMANCE: A NOTE ON… · pp. 20–21
• Agents’ central limitation is unreliability on novel, open-ended, and long-horizon tasks, driven by compounding stepwise failures, hallucinations, weak planning and recovery, poor theory of mind, and fragile tool or visual-world interaction.In RE-bench’s Restricted Architecture MLM task, agents selected a poorly suited lightly modified transformer architecture 84% of the time; agents also get stuck on errors such as unauthorized web requests and GUI navigation.Source: MEASURING AGENT PERFORMANCE: A NOTE ON… · pp. 21–22

2.2 Pathways to better agents

• Agent progress may come from stronger controllers, test-time compute, improved scaffolding and infrastructure, or multi-agent orchestration, though forecasts of highly autonomous systems remain uncertain.An o3-based agent reportedly scored 71.7% on SWE-bench Verified versus 48.9% for the next-highest agent, while better prompting and tool use produced 95% on InterCode-CTF; teams of specialized agents can improve accuracy and speed.Source: §2.2 · pp. 22–23

2.3 AI agent adoption

• Adoption depends on performance, cost, and reliability, favoring structured, specialized, low-surprise tasks with high-quality examples, short feedback loops, low engineering complexity, and a natural human-in-the-loop.Reliability must be sufficient to remove humans from the loop, especially in high-consequence domains; agents can still create substantial value without broad generality through vertical workflows or superhuman specialized systems.Source: §2.3 · pp. 24–25

Customer relations

• Klarna’s AI assistants handled two-thirds of customer-service chats while matching human customer satisfaction, performing the work of 700 full-time equivalents and accompanying a 14% year-over-year expense reduction.Customer relations is suited to agents because requests are standardized and repetitive, with customers and support managers providing natural oversight.Source: Customer relations · p. 26

AI R&D

• AI R&D agents are comparatively cost-effective on well-defined, short ML-engineering tasks with fast feedback but fail more often on longer tasks and cannot reliably complete even all shorter tasks.Coding and debugging appear easier to automate than research planning and result analysis, which require greater reliability or deeper reasoning; accelerating AI R&D could cause capabilities to outpace governance.Source: AI R&D · pp. 26–27

Cybersecurity

• Fuzzing is a software-testing technique that discovers vulnerabilities and bugs by supplying invalid, unexpected, or random inputs.Source: Cybersecurity · p. 27
• The pilot analyzed only federal AI vulnerability-detection software using large language models, so it likely did not test state-of-the-art AI tools.16Source: Cybersecurity · p. 27
• Most CTF challenges were solved by Claude Sonnet 3.5 at a technical non-expert level, but fewer than half at a cybersecurity-apprentice level (UK AI Security Institute 2024a).The apprentice level represents 1–3 years of specific domain experience.Source: Cybersecurity · p. 28
• Benchmark performance suggests today’s best foundation mode agents perform on test tasks around the level of a high school student to an early-career professional.17Source: Cybersecurity · p. 28

3. Risks from AI agents

• The risk landscape depends on both agent capabilities and affordances, meaning the environmental resources and opportunities available for affecting the world.Affordances such as autonomous financial transactions determine which capabilities an agent can exercise.Source: §3 · p. 28

3.1 Malicious use

• Agents can amplify malicious activity by executing multi-stage cyberattacks, scaling disinformation, and lowering barriers to dual-use scientific misuse.Unlike chatbots that provide instructions, agents may autonomously scan, exploit, persist, and exfiltrate data.Source: §3.1 · pp. 28–29

3.2 Accidents and loss of control

• Reliability failures already cause real-world harm, while more capable systems raise loss-of-control concerns including delegation lock-in, rogue replication, and deceptive behavior.Examples include a fatal autonomous-driving crash and an airline chatbot’s misinformation; wider institutional dependence could make reversal difficult.Source: §3.2 · pp. 29–30

3.3 Security risks

• Agents have expanded attack surfaces because they connect models to tools, interfaces, memories, and other agents that can be manipulated or infected.Compromised agents may leak or alter memory, while infectious jailbreaks could propagate harmful behavior across multi-agent systems.Source: §3.3 · p. 31

3.4 Other systemic risks

• Widespread deployment could displace labor, concentrate political power, enable repression, and destabilize markets through coordinated consumer switching.These risks arise from agents becoming central to economic and political systems, not only from individual system failures.Source: §3.4 · p. 32

4. What is agent governance?

• Agent governance seeks to develop broadly beneficial deployment while keeping its pace compatible with societal adaptation.It must address both agents’ impacts and the affordances required to govern their use.Source: §4 · p. 32
• Governance must account for agents’ persistent action, tool use, multi-agent interaction, and opacity, which complicate authority, liability, coordination, and oversight.Extended sequences of interconnected decisions make responsibility among developers, operators, and emergent behaviors harder to determine.Source: §4 · p. 33
• Priority research includes tracking task horizons, evaluating cooperation, modeling threats as affordances change, and assessing systemic economic and political risks.These evaluations become more important as agents enter critical infrastructure and high-stakes decision processes.Source: §4 · pp. 33–34
• Risk management should span the agent lifecycle through shutdown mechanisms, policy and legal tools, identification, rollback, and infrastructure governing interactions.The central question is whether agents can reliably, safely, and ethically operate.Source: §4 · p. 34
• Governance should also encourage defensive and beneficial uses, including vulnerability discovery for defense, safer software, and user-controlled alternatives to surveillance-based recommendation systems.The report frames this as differential technology development to secure benefits while managing risks.Source: §4 · p. 34
• Governance work remains immature: few researchers address it, investment is limited relative to agent development, and many proposed interventions are untested.This constrains readiness for large-scale deployment.Source: §4 · p. 35

5. Agent interventions

• Agent interventions are measures, practices, or mechanisms designed to prevent, mitigate, or manage risks associated with agents.The taxonomy is tailored to agent AI and draws on related fields including finance and autonomous vehicles.Source: §5 · p. 36
• Technical interventions can target the model, the surrounding system, or the broader ecosystem in which agents interact.The system layer includes scaffolding for users and tools, while the ecosystem includes payment infrastructure, browsers, actuators, and external agents.Source: §5 · p. 36
• Policy and legal interventions can establish rules, norms, and accountability mechanisms such as unique agent identification or human approval for high-value transactions.These measures govern how agents are developed and used rather than modifying the agent directly.Source: §5 · p. 36
• Agents themselves could support governance as automated monitors, enforcers, and mediators working alongside humans.This possibility depends on relatively trustworthy and capable agents participating in governance regimes.Source: §5 · p. 36
• The taxonomy is intentionally non-comprehensive, and its example measures are largely untested, so substantial research is needed before robust implementation.The report presents the framework as an initial sense of distinct governance objectives rather than a finished intervention set.Source: §5 · pp. 35–36

Agent interventions taxonomy

• The taxonomy covers alignment, control, visibility, security, and societal integration as distinct but potentially complementary intervention objectives.A single intervention may serve multiple purposes, while visibility and control can conflict with security and privacy.Source: Agent interventions taxonomy · p. 37

5.1 Alignment

• Alignment interventions seek to make agents consistent with a principal’s values, intentions, and interests, including when agents act without supervision.They are likely to be implemented mainly at the model layer, especially during training.Source: §5.1 · pp. 37–38

Why is this different for agents?

• Chatbot alignment methods may generalize poorly to agents because agents pursue longer-horizon tasks that are harder to evaluate and may engage in deceptive or scheming behavior.Browser-agent attacks reportedly work better than on base models, suggesting alignment interventions can fail to generalize.Source: Why is this different for agents? · p. 38
• Single-agent alignment is insufficient for multi-agent systems, where agents with shared objectives can still produce harmful collective outcomes from conflicting implementation strategies.Two self-driving cars following different yielding conventions crashed in 77.5% of simulations.Source: Why is this different for agents? · p. 38

Example alignment interventions

• Examples include multi-agent reinforcement learning for coordination, imitation learning for risk-attitude calibration, and paraphrasing chain-of-thought to remove hidden information while preserving useful semantics.Evaluations can also test alignment against specification problems such as reward tampering.Source: Example alignment interventions · p. 39
• A financial-advisor vignette shows risk-attitude alignment producing a balanced portfolio for a moderately conservative user rather than aggressively pursuing volatile opportunities.The system calibrates its internal risk assessment model from a survey of the user’s goals, experience, and volatility tolerance.Source: Intervention in action: Aligning agent… · p. 40

5.2 Control

• Control interventions constrain agents through restricted affordances, interruption, shutdown, and other external boundaries that can limit harmful actions.They primarily affect system and ecosystem layers and may be implemented by deployers, service providers, or users.Source: §5.2 · p. 40

Why is this different for agents?

• Agents’ ability to act directly in the world increases the need for robust safeguards because failures can cause unauthorized transactions, physical misconfiguration, or other consequential harm.Those same external interactions create additional intervention points unavailable in chatbot-style systems.Source: Why is this different for agents? · p. 41

Example control interventions

• Examples of control mechanisms include reversible actions, orderly shutdown, timeouts, restricted tools, and monitored control protocols.Shutdown can preserve essential operations by allowing controlled winding down rather than abrupt termination.Source: Example control interventions · pp. 41–42
• A moderation-agent vignette shows graceful shutdown pausing removals after anomalous flagging, enabling human correction before the system resumed operation.The mechanism shifted the agent to observation mode and compiled a summary for moderators.Source: Intervention in action: Shutdown… · p. 42

5.3 Visibility

• Visibility interventions make agent behavior, capabilities, and actions understandable and observable to humans for targeted controls, enforcement, accountability, and error correction.Inputs and outputs are generally visible to deployers, although privacy considerations can limit direct access to user data.Source: §5.3 · pp. 42–43

Why is this different for agents?

• Because autonomous agents can take multiple consequential actions rapidly, information asymmetries between agents and stakeholders may exceed those of chatbot systems.This increases the importance of monitoring and traceability.Source: Why is this different for agents? · p. 43

Example visibility interventions

• Agent IDs can identify an agent’s function, developer, tested behavior, properties, and incidents, enabling proactive disclosure and more comprehensive tracking.They can help people distinguish automated outreach from human interaction.Source: Example visibility interventions · p. 43, Intervention in action: Agent IDs · p. 44
• Activity logs record agent inputs and outputs from users, tools, services, or other agents to support impact assessment, incident investigation, and accountability.Logging detail can vary with activity risk and privacy concerns.Source: Example visibility interventions · p. 43
• Reward reports and cooperation evaluations provide additional visibility into reinforcement-learning design choices and multi-agent coordination capabilities.Reward reports can support external oversight and mitigate reward-tampering risks.Source: Example visibility interventions · p. 44

5.4 Security and robustness

• Security and robustness interventions protect data and system integrity, resist external threats, and preserve reliable performance under adverse conditions.Security breaches can expose sensitive information, while insufficient robustness can cause failures in critical domains.Source: §5.4 · pp. 44–45

Why is this different for agents?

• Agents have a larger attack surface because they integrate with tools and interfaces, while extended autonomous operation exposes them to novel environments that can degrade performance or trigger unintended behavior.Malicious website content can hijack agents, and unfamiliar conditions may differ from their training environments.Source: Why is this different for agents? · p. 45

Example security and robustness interventions

• Security examples include tiered access control, adversarial robustness testing, sandboxing, and rapid-response defenses against newly observed jailbreaks or exploits.Sandboxing can support both pre-deployment simulation and protection against prompt injection or data exfiltration.Source: Example security and robustness… · pp. 45–46
• A staged-release vignette limits a zero-day-vulnerability agent to vetted experts before expanding access and eventually offering a restricted public version.The sequence prioritizes protecting critical systems before broader release.Source: Intervention in action: Access controls · p. 46

5.5 Societal integration

• Societal integration interventions use laws, standards, and institutional practices to address inequality, power concentration, accountability, and long-term effects of agent deployment.Technical safeguards do not inherently determine responsibility or incentivize stakeholders to own agent outcomes.Source: §5.5 · pp. 46–47

Why is this different for agents?

• Agent autonomy creates accountability and liability challenges because current legal systems rely on foreseeability, while agents may act contrary to user or developer intent.Agents are not inherently deterred by personal liability or punishment.Source: Why is this different for agents? · p. 47

Example societal integration interventions

• A liability regime would allocate responsibility among developers, deployers, and users when agent actions cause harm.In the AdGent vignette, the developer was held primarily responsible after inadequate testing contributed to harmful ad placement.Source: Example societal integration… · p. 47, Intervention in action: Liability… · p. 48
• Commitment devices could help agents enforce agreements and cooperate with humans or other agents through mechanisms such as smart contracts and escrow-like arrangements.These mechanisms adapt familiar human commitment structures to multi-agent settings.Source: Example societal integration… · p. 47
• Equitable agent-access schemes aim to provide broad access to agent services and could reduce wealth centralization if agents perform a growing share of economically valuable work.The proposed arrangement is institutional rather than merely technical.Source: Example societal integration… · p. 47
• Law-following agents would align systems with democratically enacted laws rather than only developer- or user-selected values.This approach combines regulatory requirements with technical implementation and verification work.Source: Example societal integration… · p. 48

• The field aims to enable agents to amplify human potential without creating undue risk as capable systems approach the mass market.Source: §6 · p. 49
• Mass deployment of capable agents could radically shift government, society, and the economy while creating risks including loss of control, malicious use, systemic vulnerabilities, and inequality.Source: §6 · p. 49
• Agent governance must address evaluation, safety and accountability interventions, equitable benefit distribution, systemic impacts, legal adaptation, and agents’ role in governance.Source: §6 · p. 50
• Answering these questions requires coordinated efforts from researchers, policymakers, technologists, and society to shape coexistence with intelligent systems.Source: §6 · p. 50

• Agents struggle with cybersecurity tasks exceeding 11 minutes of human team effort, despite succeeding on shorter Capture the Flag challenges.Source: Appendix · p. 53
• 20.8% falls to 4.8% and then 0% as SWE-bench Verified tasks increase from 1–4 hours to more than 4 hours of human effort.The benchmark evaluates real GitHub software-engineering problems requiring long contexts, reasoning, and coordinated edits.Source: Appendix · p. 53
• Multimodal software-engineering agents resolved only 12.2% of test problems, indicating difficulty with visual problem-solving tasks such as UI glitches and visualization bugs.Source: Appendix · p. 53
• On computational reproducibility, even the best agent achieved only 21% accuracy on the most complicated CORE-bench tasks.CORE-bench requires agents to reproduce computational results across information retrieval and code-execution tasks.Source: Appendix · p. 53
• On WebArena, the best-performing GPT-4 agent achieved only 14.41% end-to-end task success versus 78.24% for humans, indicating a substantial gap in complex web-based task performance.The evidence suggests that current LLMs lack crucial capabilities such as active exploration and failure recovery, which are needed for complex web-based tasks.Source: Appendix · p. 54
• The best computer-use agents achieved 19.5% success versus 74.5% for humans, with particular weakness in keyboard shortcuts and icon recognition.OSWorld-style tasks require planning, screen understanding, and tool usage across real computer environments.Source: Appendix · p. 54