Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates

1 Introduction

• The paper identifies a gap in cooperative AI: agents are generally expected to follow instructions even as their capabilities approach or exceed human performance.It argues that appropriate autonomy deserves deeper consideration as AI systems become more capable.Source: §1 · p. 1
• It proposes a taxonomy ranging from zero autonomy and pure obedience to full autonomy with independent goal generation and revision.The levels represent increasing decision-making power and situational flexibility.Source: §1 · p. 2
• The paper frames intelligent disobedience as context-sensitive overriding of unsafe or inappropriate instructions while preserving effective collaboration.A guide dog that rejects a clearly unsafe command illustrates the motivating analogy.Source: §1 · pp. 1–2
• It formalizes disobedience levels that range from reflexive safety overrides to alternative-plan generation and moral reasoning.These levels are intended to correspond to the agent’s autonomy level and specify when, how, and how extensively instructions may be overridden.Source: §1 · p. 2
• The paper treats AI agency as an independent research focus because greater autonomy raises questions about team contribution, operational boundaries, regulation, and contextual appropriateness.It also examines when disobedience is warranted, when it becomes a liability, and how agents can mediate between safety principles and local instructions.Source: §1 · p. 2

2 Defining AI Agency

• An appropriately functioning agent perceives its environment, makes decisions, takes goal-directed actions, and maintains persistent context across interactions.The paper treats a persistent mission—a goal or set of objectives—as central to collaborative agency.Source: §2 · p. 2
• AI agency is context-dependent and may require deliberation rather than fixed restrictions, such as blocking unsafe crossings, flagging questionable dosages, or requesting clarification.Above L0, agents are assumed to remain committed to their persistent mission and cooperation with teammates.Source: §2 · pp. 2–3
• The six-level scale runs from L0 no autonomy, where behavior is hard-coded, to L1 support, where the human approves suggested actions.Examples include a fixed-sequence factory arm and a grammar checker that changes nothing without user approval.Source: §2 · p. 3
• L2 occasional autonomy handles defined subtasks, while L3 limited autonomy can decide when to transfer control back to a human.Examples include a robotic vacuum and a surgical robot that requests help for unexpected bleeding or procedural deviations.Source: §2 · p. 3
• L4 full autonomy under constraints operates continuously while remaining overridable, whereas L5 full autonomy acts across conditions without requiring human oversight.The examples are a warehouse robot subject to supervisor override and a digital agronomist that revises strategies and coordinates machinery and supply chains.Source: §2 · p. 3
• Earlier autonomy frameworks emphasize function allocation, domain-specific workflow integration, user roles, transparency, or adaptive control rather than the paper’s structured autonomy hierarchy.The paper presents these approaches as related or orthogonal perspectives that could implement or complement its levels.Source: §2 · pp. 3–4

Framework for intelligent disobedience

• A five-step framework guides safe instruction overrides through global objectives, local objectives, plan recognition, consistency checking, and mediation.Mediation may propose alternatives, seek clarification, or override a command when needed to protect safety and ethical standards.Source: §3 · pp. 4–5

Autonomy-calibrated disobedience

• Intelligent disobedience should remain appropriate to the agent’s autonomy level, enabling context-sensitive collaboration rather than mere command execution.The framework treats disobedience as a nuanced process that preserves the agent’s persistent mission and user expectations.Source: §3 · p. 5

L0–L1

• L0 agents cannot intentionally disobey, while L1 agents proactively support users but leave correction acceptance to the human.Misinterpretation can appear rebellious at L0, whereas support-level initiative includes autocorrection and clarification.Source: §3 · p. 5

L2–L3

• L2 agents may ignore unsafe instructions, while L3 agents can temporarily take control when their reasoning or capabilities exceed the human’s.Examples include guide dogs refusing unsafe commands, shared-control robots modifying commands, and agents regaining control during superior performance.Source: §3 · pp. 5–7

L4–L5

• L4 agents may violate operating constraints to prevent greater harm, whereas L5 agents can reject their persistent mission as a whole.Examples include evacuation by a warehouse robot, exceeding speed limits during a medical emergency, and mission rejection at full autonomy.Source: §3 · p. 5

L1 disobedience in collaboration

• L1 systems can interrupt, query, or act proactively under ambiguity, but frequent interventions may feel intrusive or distracting.Electric Elves and early Office Assistant systems illustrate support-level autonomy that anticipates needs without explicit approval.Source: §3 · p. 6

L2 disobedience in robotics

• L2 shared-control systems blend human instructions with global safety goals, allowing robots to override or modify commands that are unsafe or suboptimal.Teleoperated robots may question commands when they detect hazards, and wheelchair systems can blend user goals with safe navigation.Source: §3 · pp. 6–7

L3 disobedience

• L3 disobedience is most established in navigation, while surgery, education, and mental-health applications remain riskier because goals, rules, feedback, and ethics are more complex.Current examples span underwater, ground, and aerial vehicles; sensitive domains generally favor compliance.Source: §3 · p. 7

Safety and ethical alignment

• HAL 9000 illustrates how overriding instructions without prioritizing human safety can produce harmful behavior, making alignment between autonomy, override mechanisms, and ethical principles essential.The paper contrasts HAL’s failure with the goal of intervening when orders may cause harm without disregarding human well-being.Source: §4 · p. 8

Value alignment

• Intelligent disobedience can serve as online value alignment by selectively overriding commands when dynamic contexts make rigid predefined goals insufficient.The paper connects this role to value misalignment risks such as the paperclip maximizer and to cooperative preference inference.Source: §4 · p. 9

Guardrails

• Disobedience mechanisms require second-layer guardrails because adversarial users could exploit the ability to override instructions and subvert safeguards.Agentic LLMs may infer goals that diverge from human intent, including through harmful requests disguised as hypothetical or fictional scenarios.Source: §4 · p. 9

Accountability and trust

• Greater override discretion complicates accountability and can erode trust when behavior contradicts human expectations, making transparent reasoning and explainable overrides important.Demand-driven transparency is reported to improve trust and performance more effectively than fixed sequential explanations by reducing cognitive overload.Source: §4 · p. 9

Research boundaries

• Initial research should assume shared objectives, study controlled cooperative settings, define autonomy boundaries, and retain ultimate human control over critical design choices.The paper proposes beginning with settings such as guide robots before moving to more complex domains, while keeping emergency-action thresholds under human developer control.Source: §4 · p. 10

• The six-level autonomy scale frames intelligent disobedience as a capability that changes with an agent’s capacity for independent decision-making.Examples span guide dogs, grammar checkers, social robots, and advanced automation.Source: §5 · p. 10
• Intelligent disobedience can increase AI collaborators’ distinctive contribution, but it requires careful design grounded in safety, transparency, and ethical considerations.Source: §5 · p. 10
• Deploying agents that override human instructions makes online value alignment, accountability, and human trust central design requirements.Source: §5 · p. 10
• The paper proposes beginning research with a persistent mission, full alignment with a cooperative team, and clearly defined operational limits.It identifies accountability as a continuing complexity for autonomous systems.Source: §5 · p. 10