Source-linked AI summary
Levels of Autonomy for AI Agents
K. J. Kevin Feng, David W. McDonald, Amy X. Zhang
TL;DR
AI agents’ autonomy offers benefits while creating risks, raising the question of how developers should calibrate it. The paper treats autonomy as a design decision distinct from capability and environment, proposes five user-centered levels, autonomy certificates, and assisted evaluations. It concludes that these tools can support refined human-agent collaboration, governance, and separate evaluation of autonomy and capability.
Problem
Developers need ways to calibrate agent autonomy because autonomy can amplify both AI’s benefits and risks, while capability benchmarks do not measure autonomy separately.
Method
The paper defines five user-centered autonomy levels, proposes autonomy certificates, and introduces assisted evaluations that record user involvement during tasks.
Results
The paper presents autonomy as a design decision independent of capabilities and operational environment, with applications to user experience, governance, and multi-agent systems.
Takeaways & Limitations
Separating autonomy from capability can help developers design human-AI collaboration, multi-agent communication protocols, and approaches to failure modes and resolution.
Takeaways & Limitations
The paper broadly uses “intention” as an impetus for action because whether artificial agents formulate true intention is contested.
Abstract
from arXiv · showhide
Autonomy is a double-edged sword for AI agents, simultaneously unlocking transformative possibilities and serious risks. How can agent developers calibrate the appropriate levels of autonomy at which their agents should operate? We argue that an agent's level of autonomy can be treated as a deliberate design decision, separate from its capability and operational environment. In this work, we define five levels of escalating agent autonomy, characterized by the roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer. Within each level, we describe the ways by which a user can exert control over the agent and open questions for how to design the nature of user-agent interaction. We then highlight a potential application of our framework towards AI autonomy certificates to govern agent behavior in single- and multi-agent systems. We conclude by proposing early ideas for evaluating agents' autonomy. Our work aims to contribute meaningful, practical steps towards responsibly deployed and useful AI agents in the real world.
1 Introduction
AI agents are becoming more capable of acting autonomously, creating both promising applications and heightened risks. The paper frames autonomy as a deliberate design choice and introduces a five-level framework, autonomy certificates, and user-assisted evaluation ideas.
- Frontier generative AI models enable agents to resolve GitHub issues, shop online, and use computers in human-like ways.
- Autonomous agents can amplify AI’s benefits while increasing risks including scams, privacy leaks, deskilling, and reduced accountability.
- Developers can choose an agent’s autonomy level independently of fixed capabilities and operational environments.
- The framework defines five user roles—operator, collaborator, consultant, approver, and observer—to characterize escalating autonomy.
- Autonomy certificates are proposed as a governance mechanism for communicating agent behavior and supporting risk assessment, safety design, and multi-agent engineering.
- The paper proposes evaluating autonomy separately from capability by measuring user involvement during task completion.
2 Terminology
The paper distinguishes agents, users, agency, and autonomy as related but separate concepts. It defines autonomy through the extent of designed operation without user involvement while treating intention broadly for practical use with AI agents.
- AI agents: An AI agent perceives its environment through sensors and acts on it through effectors, including website retrieval or interface interaction.
- Users: A user is a human or AI entity that issues an agent’s initial service request and can itself be another agent in a multi-agent system.
- Agency and autonomy: The paper treats agency and autonomy as distinct concepts rather than interchangeable terms.
- Agency and autonomy: Agency concerns the capacity to act intentionally, while autonomy concerns the extent to which an agent is designed to operate without user involvement.
- Agency and autonomy: Autonomy includes forms of user involvement ranging from direct control to light supervision and is shaped by model and interface constraints.
- Agency and autonomy: The paper uses “intention” broadly as an impetus for action so the concept can apply practically to AI agents.
3 Five Levels of Autonomy for AI Agents
The framework treats autonomy as a design choice independent of fixed capability and environment, using five user roles to structure agent behavior. Across levels, increasing autonomy shifts planning and control from users to agents while creating tradeoffs among utility, efficiency, accountability, cost, and risk.
- Framework design: The framework varies autonomy while holding the agent’s model, tools, environment, and user request constant, separating autonomy from capability.The running example uses a computer-using model with UI navigation, web browsing, code execution, and document-writing tools.
- Level 1: Operator: L1 users drive long-term planning and retain ownership of the workflow, while agents provide contextual assistance on demand or through proactive suggestions.Developers must determine the boundary between long- and short-term planning and detect when preference-based decisions require user involvement.
- Level 2: Collaborator: L2 agents and users jointly plan, delegate, and execute tasks through frequent communication, enabling more complex workflows but reducing continuous on-demand availability.Users can review and directly edit an initial agent plan, with visibility into the agent’s work and progress.
- Level 3: Consultant: L3 agents take greater responsibility for extended planning and execution, while users provide feedback, preferences, and higher-level direction rather than direct control.Unlike L2, users may not directly take over or freely edit outputs, instead requesting changes indirectly through messages.
- Levels 4–5: Approver and observer: L4 users act as approvers who intervene only when agents encounter blockers, need credentials, or require sign-off on consequential actions; L5 users can only monitor logs or use an emergency off-switch.L4 suits lower-stakes decisions but increases security concerns and potential attack surfaces, while L5 agents plan and execute over long horizons without user input.
- Design tradeoffs: The framework presents autonomy as involving tradeoffs rather than a simple quality ranking, with powerful capabilities potentially supporting effective agents at every level.Relevant considerations include utility, efficiency, accountability, cost, and the risks associated with increasingly autonomous behavior.
4 Autonomy Certificates for AI Agents
The paper proposes autonomy certificates as third-party governance documents that specify an agent’s maximum permitted autonomy for defined capabilities and environments. Certificates could support targeted risk assessment, safety-framework design, and multi-agent coordination.
- Certificate definition: An autonomy certificate prescribes the maximum autonomy level an agent may operate at for specified technical capabilities and an operational environment.The certificate is stored with agent metadata so other agents and developers can retrieve it during deployment.
- Why autonomy certificates?: Certificates communicate developers’ autonomy decisions to stakeholders, including other developers and third-party auditors.The paper identifies targeted risk assessment, safety-framework design, and multi-agent systems engineering as applications.
- Risk assessment: Risk assessment can target long-term planning in L4–L5 agents and user-feedback mechanisms in L2–L3 agents.The proposed focus follows from the behavioral constraints associated with each autonomy level.
- Safety-framework design: Autonomy certificates encourage safety evaluations to distinguish degrees of autonomous task completion rather than relying only on binary success or failure.The paper notes that an L4 agent requiring simple approval to generate revenue may pose risks similar to an L5 agent that generates revenue independently.
- Multi-agent systems: Mixed-autonomy multi-agent systems may balance responsiveness with steerability, debugging, and auditability better than systems composed entirely of L1 or L5 agents.All-L1 systems may wait for operator assignments, whereas all-L5 systems may communicate sparsely.
- Certificate issuance: A proposed issuance process has developers submit an operational agent and autonomy case for private third-party evaluation before certification.The governing body evaluates whether the agent satisfies the autonomy case’s objectives and issues or withholds the corresponding certificate.
- Certificate renewal: Changes to an agent’s technical specifications or operational environment can alter its interactive behavior and invalidate its certificate.The paper therefore treats certificates as tied to a particular agent configuration and deployment setting.
5 Evaluating Autonomy: A User-Assisted Approach
The paper argues that autonomy should be evaluated separately from capability by measuring how much user involvement an agent needs to meet a task-performance threshold. It proposes assisted evaluations, in which a standby user’s involvement is progressively adjusted and logged to classify the agent’s autonomy level.
- Evaluation problem: The central evaluation question is how to verify the autonomy level an agent claims or appears to operate at.The paper presents this as an unresolved issue exposed by its autonomy framework and certificates.
- Why capability benchmarks are insufficient: Capability benchmarks measure task accuracy or completion, but they do not by themselves capture user-agent interaction or the user’s role.The paper argues that autonomy evaluations should measure the user involvement requested for successful task completion.
- Assisted evaluation: Assisted evaluations give an agent access to a standby human or AI user who records the assistance provided during task completion.The accumulated logs are used to determine the minimum user involvement needed to exceed a specified accuracy or pass-rate threshold.
- Evaluation procedure: The proposed procedure begins without user involvement, increases assistance until the agent exceeds threshold T, and classifies autonomy from the recorded involvement.The rounds move from fully autonomous operation through approvals, rejections, and greater user involvement as needed.
- Evaluation cost: Starting with no user involvement and adding assistance iteratively may be costly because it can require multiple rounds of task completion.The paper gives five rounds as an example of the potential evaluation burden.
6 Conclusion
The conclusion frames autonomy as an independent design choice that can help developers shape human-agent interaction, multi-agent communication, and failure analysis. It presents five autonomy levels, autonomy certificates, and assisted evaluations as practical steps toward useful, reliable, and safe agents.
- Framework: Autonomy can be designed independently of an agent’s capabilities and operational environment, even though both can influence it.The framework centers this design choice on five user roles: operator, collaborator, consultant, approver, and observer.
- Design implications: The framework is intended to support refined human-AI collaboration, communication protocols among agents, and systematic reasoning about failure modes and resolutions.These benefits are described for agents with specified capabilities and target deployment environments.
- Governance: Autonomy certificates specify the maximum permitted autonomy level and can support autonomy-risk assessment and multi-agent system design.The conclusion presents certificates as a potential governance mechanism rather than a completed regulatory system.
- Evaluation: Assisted evaluations provide a setup for evaluating autonomy alongside, but distinctly from, agent capabilities.The conclusion identifies this as the paper’s response to the open question of autonomy evaluation.
- Practical scope: Careful design of agents’ interactive behaviors is needed as agents move from research settings into the real world.The stated goal is to realize AI’s potential while bolstering rather than eroding human agency.