Source-linked AI summary
Visibility into AI Agents
Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, Lennart Heim, Markus Anderljung
TL;DR
As AI agents become more capable and widespread, delegating human activities to them may exacerbate existing risks and introduce new ones, creating a need for visibility into their use. The paper assesses identifiers, real-time monitoring, and activity logs across deployment contexts, while examining privacy, power concentration, and the limits of visibility for governance.
Problem
Increasing delegation of commercial, scientific, governmental, and personal activities to AI agents may exacerbate existing risks and introduce new ones, requiring information about where, why, how, and by whom agents are used.
Method
The paper assesses agent identifiers, real-time monitoring, and activity logs, considering implementations with different intrusiveness and informativeness across centralized and decentralized deployments.
Results
The paper identifies visibility measures for accountability, incident investigation, real-time intervention, and post-hoc analysis, including use of agent cards and infrastructure providers in decentralized deployments.
Takeaways & Limitations
Visibility can inform governance and risk management, but must be complemented by public influence, policy levers, and infrastructure and practices that prevent or defend against harms.
Takeaways & Limitations
Visibility alone is insufficient because people harmed by AI agents may lack the power to intervene and reduce risks.
Abstract
from arXiv · showhide
Increased delegation of commercial, scientific, governmental, and personal activities to AI agents -- systems capable of pursuing complex goals with limited supervision -- may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents.
1 INTRODUCTION
As AI agents become more capable and widely deployed, they may create or amplify risks that are difficult to detect, attribute, and govern. The paper therefore develops visibility measures spanning identifiers, monitoring, and logging, while examining their scope and trade-offs.
- 1.1 Risks from AI Agents: AI agents may automate consequential activities at scale, increasing risks from malicious use, overreliance, delayed effects, systemic interactions, and agent-to-agent delegation.These risks can arise when agents operate with limited human involvement across complex, long-horizon tasks.
- 1.2 The Case for Visibility into AI Agents: Visibility—information about where, why, how, and by whom agents are used—supports governance, adaptation of rules, and accountability for harms.Regulators may need such information to understand novel strategies and identify responsible users.
- 1.3 Contributions: The paper evaluates agent identifiers, real-time monitoring, and activity logs as complementary ways to reveal agent involvement, ongoing behavior, and post-incident evidence.Identifiers can distinguish agents, monitoring can enable intervention during activity, and logs can support forensic analysis afterward.
- 1.3 Contributions: The analysis covers centralized through decentralized deployments and considers developers, deployers, compute providers, and tool or service providers.The authors examine how visibility can extend across the supply chain, including distributed enforcement through service providers.
- 1.3 Contributions: The paper treats privacy and concentration of power as important trade-offs and calls for further study rather than immediate implementation of the measures.Visibility alone may not give affected parties sufficient power to intervene or reduce harms.
- 1.3 Contributions: The measures extend deployment-visibility practices to agent actions, external-service interactions, multi-agent interactions, and delayed or diffuse effects.Agent identifiers generalize watermarks beyond generated media to all agent outputs, including external actions.
2 DEFINITIONS
The paper defines agency and related roles and infrastructure in an AI-agent deployment. These definitions distinguish agents from less autonomous systems and clarify how agents interact with users, providers, tools, and services.
- Agency: Agency is the degree to which an AI system acts directly in the world toward long-horizon goals with little human intervention.Agents have relatively high agency, whereas systems that mainly predict without acting in the world have relatively low agency.
- Scaffolding: Scaffolding structures calls to an AI system through prompts, memory, tools, or planning mechanisms to facilitate goal pursuit.Such scaffolding can make a foundation model more agentic.
- Actors: Developers construct AI systems, users provide instructions, and deployers operate systems and serve them to users; these roles may belong to different entities.A deployer may serve a foundation model that users combine with scaffolding and tool access to create a more agentic system.
- Compute Providers: Compute providers supply and maintain the hardware infrastructure on which AI systems operate and may also be developers or deployers.They may become important partners for overseeing large-scale deployments that are not run directly by deployers.
- Tools, Services, Inputs, and Outputs: Tools and services are external systems or platforms that agents use to perform tasks, often through dedicated APIs.Providers maintain these systems, while deployers generally have broader access to agent inputs and outputs than service providers do.
3 MEASURES TO IMPROVE VISIBILITY
The paper proposes three complementary measures for visibility into AI agents: identifiers, real-time monitoring, and activity logs. These measures vary in informativeness and intrusiveness and can support accountability across deployment contexts.
- 3 MEASURES TO IMPROVE VISIBILITY: Three complementary measures improve visibility: agent identifiers, real-time monitoring, and activity logs.Identifiers indicate agent involvement, monitoring supports intervention during activity, and logs support post-incident attribution and forensics.
- 3 MEASURES TO IMPROVE VISIBILITY: Deployers can implement these measures because they operate agents and generally have access to their inputs and outputs.The paper focuses on agents run by deployers and served to users, while also considering decentralized deployments and other supply-chain actors.
- 3.1 Agent Identifiers: Agent identifiers can reveal whether and which agents participate in interactions, with formats including watermarks for images and headers for API requests.Identifiers may be visible to different actors and can distinguish agent involvement across output formats.
- 3.1 Agent Identifiers: Unique identifiers can link an action to an agent, its user, developers, and deployer, thereby facilitating accountability and investigation.Cryptographic methods such as software attestation may be needed to assure provenance for each deployed instance.
- 3.1.2 Attaching Additional Information to Agent Identifiers: Additional information, organized as an agent card, can describe the underlying system, deployed instance, and actors involved in development and deployment.More comprehensive cards may be more useful but also more intrusive and resource-intensive, so comprehensiveness should scale with activity risk.
3.2 Real-Time Monitoring
Real-time monitoring analyzes agent activity as it occurs to flag and potentially filter problematic behavior. Its immediacy makes it suited to recognizable rules, anomalies, and collective or high-impact activity.
- 3.2 Real-Time Monitoring: Real-time monitoring analyzes agent activity automatically to flag and potentially filter problematic behavior without requiring activity-log storage.Automation is needed to handle the speed and scale of agent activities.
- 3.2 Real-Time Monitoring: Monitoring is best suited to violations identifiable immediately, including prohibited activities, excessive permissions, sensitive-information leakage, usage caps, and anomalies.Deployers can intervene after flagging, while users may be given an opportunity to request human review.
- 3.2 Real-Time Monitoring: Monitoring may combine information across agents to detect collective problems such as collusion or system overload.Agent identifiers can track participating agents, and correlation in price movements is one possible collusion indicator.
- 3.2 Real-Time Monitoring: Monitoring can flag agents whose substantial transactions, compute use, sub-agent counts, or physical-system interfaces magnify the impact of failure.Significant influence may not itself be harmful, but it can increase the consequences of malfunction.
3.3 Activity Logs
Activity logs record selected agent inputs, outputs, and state changes to support attribution, forensics, audits, and research. Their detail, retention, and scope must balance usefulness against cost, expertise, storage, and privacy concerns.
- 3.3 Activity Logs: Activity logs record selected inputs, outputs, and agent-state changes, with detail determined by the behavior being tracked, privacy, and storage needs.Logs may cover user and tool interactions, weight updates, and only actions above a chosen significance threshold.
- 3.3 Activity Logs: Detailed logs help investigators trace harms to specific user choices and understand agent impacts during audits and post-incident forensics.Researchers can also use logs to study agents and develop improved deployment controls.
- 3.3.1 Agent-Specific Information: Separate logs for tool use, internal reasoning, self-critique, and memory management can serve different investigative and operational purposes.Tool-use logs can expose external-tool access and outputs, helping deployers and tool providers understand interactions and prevent abuse.
- 3.3.1 Agent-Specific Information: Longer retention and cross-log analysis can reveal delayed impacts, persistence, sub-agent dynamics, malfunction propagation, and undesirable inter-agent communication.Relevant records may include runtime, external-memory access, and compute consumed.
- 3.3.1 Agent-Specific Information: More detailed logging is more useful but increases deployer costs, analytical requirements, and privacy concerns.In high-risk environments, regulators may require detailed records of all agent behavior.
3.4 Risks
Visibility measures create a tension between obtaining detailed information about agent activity and protecting privacy. The paper therefore emphasizes risk-proportionate collection and controlled access to information.
- 3.4 Risks: Detailed visibility can conflict with privacy assurances, data-protection requirements, and user control over collection, retention, and deletion.Examples include no-logging APIs, limits on training use, customer-cloud storage, and log deletion options.
- 3.4 Risks: Because agent information may reveal information about users, deployers and governments could abuse centralized visibility for excessive or unjustified surveillance.The paper justifies limiting collection according to the agent’s activity risk or deployment domain.
- 3.4 Risks: Access should be limited to the minimum granularity and quantity necessary for legitimate objectives.Identifiable information may be disclosed for specific investigations upon compelling need or third-party adjudicator approval, while regulators may need it for some high-risk activities.
4 DECENTRALIZED DEPLOYMENTS
Decentralized deployments complicate visibility because users can combine or run agents outside deployers’ control, shifting oversight toward compute, tool, and service providers. These leverage points can improve accountability but raise privacy, concentration-of-power, and circumvention concerns.
- 4.1 Compute Provider Oversight: Compute providers may oversee large-scale agent deployments by identifying agent activity and requiring evidence that visibility measures are implemented.Large deployments are notable because they consume substantial resources, while cloud compute is often cheaper than user-owned hardware.
- 4.2 Tool and Service Provider Oversight: Tool and service providers can condition access on agent identifiers, creating incentives for deployers to adopt visibility measures.Identifiers could confirm permissions to access sensitive services, including financial services.
- 4.2 Tool and Service Provider Oversight: APIs may make decentralized deployments easier to monitor because users often prefer standardized, convenient interfaces over direct tool interaction.This preference can reduce the difficulty of obtaining visibility even though agents may circumvent APIs by mimicking human behavior.
- 4.3 Implications and Mitigations: Provider-based oversight risks exposing sensitive user information and concentrating power among dominant compute and certified service providers.Users may face pressure to use certified deployers, whose practices could become invasive or unresponsive to user interests.
- 4.3 Implications and Mitigations: Potential mitigations include voluntary standards, open-source identifier frameworks, targeted requirements for high-risk uses, liability incentives, and decentralized custody of provider logs.Voluntary standards support experimentation but may not reveal malicious use or achieve universal adherence.
5 CONCLUSION
The paper frames visibility as a foundation for governing increasingly agentic systems through identifiers, monitoring, and activity logs across centralized and decentralized deployments. It also emphasizes that visibility alone cannot reduce harms, so further work must address power, policy, and protective infrastructure.
- 5 CONCLUSION: The paper assesses agent identifiers, real-time monitoring, and activity logs as complementary mechanisms for visibility and accountability.Identifiers indicate agent involvement, monitoring flags problematic behavior during operation, and logs support post-hoc investigation.
- 5 CONCLUSION: These measures are extended to decentralized deployments through compute providers and tool and service providers, alongside analysis of privacy and concentration-of-power implications.The paper does not advocate immediate implementation; it calls for further understanding and mitigation of negative impacts.
- 5 CONCLUSION: Visibility can inform risk-management actions but is insufficient when affected people lack power to intervene.Future work should examine public influence, policy levers, and infrastructure or practices that prevent or defend against harms.
A.1 The Underlying System
The underlying system is characterized by broad capability and risk-relevant properties that help determine what visibility information should capture.
- A.1 The Underlying System: The underlying system’s degree of agency is evaluated as part of the information relevant to visibility.
- A.1 The Underlying System: Its generality is evaluated by whether it can accomplish a broad array of tasks above a specified performance threshold.
- A.1 The Underlying System: Red flags and dependencies, including prior dangerous-capability results and relationships to other models, are also recorded.
A.2 The Specific Instance of the Agent
Visibility into a specific agent instance includes how it was created, what it is trying to achieve, and the capabilities and deployment conditions shaping its potential impact.
- A.2 The Specific Instance of the Agent: The specific instance’s creation pathway and goal are relevant, including whether another agent created it and what it appears to be pursuing.
- A.2 The Specific Instance of the Agent: Its accessible tools, permissions, persistence, and deployment sector describe the agent’s operational scope.
- A.2 The Specific Instance of the Agent: Visibility should also capture the number and severity of people the agent can affect and how easily humans can oversee it.
A.3 The Actors Involved in the Creation and Operation of the Agent
AI agents involve multiple actors across their creation and operation, including users, compute providers, developers, people they interact with, and tool or service providers.
- AI agents involve users, compute providers, system and scaffolding developers, human interaction partners, and tool or service providers.