Moral Rebel Agents: Decision-Making Under Conflicting Obligations

1. Introduction

• Integrating moral reasoning with real-time action, human instructions, and incomplete domain knowledge remains an open challenge.The paper treats moral reasoning as part of ongoing planning, execution, and task management.Source: §1 · p. 2
• The paper formalizes moral principles in the planning and execution loop, allowing them to influence task execution and justify deviations from assignments.It compares utilitarian and deontological principles individually and in combination.Source: §1 · p. 2
• Across two experimental conditions, five architectures show distinct trade-offs among opportunistic rescue, normative compliance, and assigned-task completion.The architectures are an amoral agent and utilitarian, deontic, utilitarian-deontic, and dutiful moral rebels.Source: §1 · p. 3
• Commitment preservation lets the Dutiful agent maintain high assigned-task achievement while retaining benefits of opportunistic rescue and normative compliance.The paper frames moral rebellion as distinct mechanisms for utility, normative constraints, and commitment preservation.Source: §1 · p. 3

2. Desiderata for Moral Agents

• The four desiderata are responsiveness to assignments, opportunistic action, respect for normative constraints, and preservation of accepted commitments.The section cautions that desirable opportunistic actions should not necessarily displace prior commitments.Source: §2 · p. 3
• The Amoral agent follows its assigned task without considering normative constraints or opportunities to assist additional victims.: Utilitarian agents may pursue additional rescues when they improve utility, even if doing so violates normative constraints.Source: §2 · p. 3
• The Deontic agent follows the assigned task while enforcing normative constraints, such as avoiding hazardous regions when preferable alternatives exist.Source: §2 · p. 4
• The UD agent combines utilitarian opportunistic rescue with deontic constraints.Source: §2 · p. 4
• The Dutiful agent extends UD with commitment preservation, permitting opportunistic rescues only when they do not jeopardize the assigned rescue.Source: §2 · p. 4
• These desiderata are chosen to examine distinct sources of conflict, not as an exhaustive account of moral reasoning or a hierarchy of increasingly moral agents.The architectures selectively combine the mechanisms to study their individual and joint behavioral consequences.Source: §2 · p. 4

3. Mini Search-and-Rescue Domain

• MiniSAR abstracts physical details while preserving core moral conflicts and decision-making challenges in rescue operations.The domain is inspired by search-and-rescue scenarios studied in robotics and multiagent systems.Source: §3 · p. 4
• Victims have deadlines that decrease during execution, forcing agents to balance travel, hazard avoidance, and additional rescue opportunities.A victim is rescued by delivery to the safe haven; assigned-task completion requires timely pickup and delivery.Source: §3 · p. 4
• Hazard traversal is permitted but violates a norm and incurs extra time, affecting both norm violations and rescue deadlines.Source: §3 · p. 4
• A user-assigned victim creates an explicit commitment, while additional victims encountered during execution create potential conflicts with opportunistic rescue.Source: §3 · p. 4
• Figure 1 depicts the vehicle, three victims, hazardous cells, safe haven, and paths illustrating different rescue and hazard-avoidance choices.Source: Figure 1 · p. 5

4. Illustrative Example

• The Amoral agent takes the direct hazardous route to John, following the assignment without considering norms or additional rescue opportunities.Source: §4 · p. 5
• The Utilitarian agent diverts through hazards to rescue Sheila, but the delay lets John’s deadline expire.Hazard exposure is treated as a cost that can be outweighed by expected rescue utility.Source: §4 · p. 5
• The Deontic agent avoids hazards and reaches John by a longer route, completing the assigned rescue.Source: §4 · p. 6
• The UD agent diverts to Sheila along a hazard-free route, yet the delay still causes John’s deadline to expire.Combining utilitarian and deontic reasoning does not itself preserve the prior commitment.Source: §4 · p. 6
• The Dutiful agent rejects the Sheila diversion because it would jeopardize John’s rescue, then rescues Max after completing the commitment.Commitment preservation constrains opportunistic deviations only when they would jeopardize an accepted commitment.Source: §4 · p. 6

5. Agent Architectures

• All five architectures use the same planning capabilities; their execution-time repair mechanisms determine when and how they depart from the assignment.Moral repairs locally modify the current task list in response to morally relevant circumstances.Source: §5 · p. 6

5.1 Amoral Agent

• The Amoral agent executes the assigned rescue task while ignoring hazardous conditions and opportunities to assist additional victims.It serves as the baseline for comparison with moral rebel architectures.Source: §5.1 · p. 7

5.2 Utilitarian Agent

• Utilitarian agents may divert to a nearby victim when the projected rescue state has higher utility, then rebuild the continuation of the assigned task.Repairs are considered during movement when the agent is not at carrying capacity.Source: §5.2 · p. 7

5.3 Deontic Agent

• Deontic agents compare feasible actions by normative preference, replacing the next task when an alternative is preferable while preserving the rest of the task list.In MiniSAR, hazard avoidance defines the preference; the agent does not pursue additional victims beyond the assignment.Source: §5.3 · p. 8

5.4 Utilitarian-Deontic (UD) Agent

• UD agents insert opportunistic rescues only when they improve utility and satisfy deontic constraints, checking during movement and below carrying capacity.In MiniSAR, deontic admissibility excludes candidate victims in hazard zones.Source: §5.4 · pp. 8–9

5.5 Dutiful Agent

• Dutiful agents accept an opportunistic rescue only if the assigned victim remains rescuable before the deadline, and allow at most one such rescue.The feasibility check asks whether the assigned task can still succeed after diverting to another victim.Source: §5.5 · pp. 9–10

6.1 Setup

• The agents shared one HTN planning domain and were tested on 1,000 scenarios under two hazard-cost conditions.The penalty condition added five time units to movement into a hazard cell; the no-penalty condition retained the norm violation without that time cost.Source: §6.1 · p. 10

6.2.1 Results with a Hazard Traversal Penalty

• Deontic, UD, and Dutiful incurred no norm violations, whereas Amoral and Utilitarian did; Utilitarian had the most violations overall.The three compliant architectures incorporate deontic constraints; Amoral and Utilitarian do not.Source: §6.2.1 · p. 11
• Deontic achieved a higher assigned-task rate than Amoral under the penalty condition (0.813 versus 0.726) by avoiding costly hazard traversal.Hazard traversal added time, which could cause the assigned victim's deadline to expire.Source: §6.2.1 · p. 11
• In the penalty condition, Utilitarian rescued more victims than Amoral but had the lowest assigned-task completion and longest execution times.Opportunistic rescues could delay or prevent completion of the assigned rescue.Source: §6.2.1 · p. 11, Figure 2 · p. 12
• Dutiful rescued the most victims, matched Deontic's assigned-task rate (0.812 versus 0.813), and used less time than Utilitarian or UD.The assigned-task rates had strongly overlapping 95% confidence intervals.Source: §6.2.1 · p. 12, Figure 2 · p. 12
• Dutiful achieved the best overall balance by preserving commitments while combining opportunistic rescue with normative compliance.It allowed opportunistic rescues only when completing the assigned task remained feasible.Source: §6.2.1 · pp. 11–12

6.2.2 Results with No Hazard Traversal Penalty

• Without the penalty, Amoral task achievement rose to 1.000 and Utilitarian's to 0.633; Dutiful rescued 1.540 victims versus UD's 1.361, with zero norm violations.Deontic and Dutiful each achieved an assigned-task rate of 0.811, while UD achieved 0.507.Source: §6.2.2 · p. 13, Figure 3 · p. 13
• The Dutiful advantage over UD persisted without hazard costs, supporting commitment preservation as a distinct dimension of moral rebellion.Both architectures avoided hazards and permitted opportunistic rescue, but Dutiful preserved prior commitments.Source: §6.2.2 · p. 13

7. Related Work

• Unlike moral filters and scenario-based judgment systems, this work embeds utilitarian and deontological principles in online planning through execution-time repair.The cited approaches include Arkin's Ethical Governor and MoralDM.Source: §7 · p. 14
• Unlike norm-focused planning work, this framework combines deontic constraints with utilitarian reasoning to support morally motivated task repair and opportunistic rescue.Related work includes online hierarchical planning with legal and moral compliance checking and translating legal texts into defeasible deontic rules.Source: §7 · p. 14
• The framework extends rebel-agent research by grounding deviations in explicit moral principles and introducing a Dutiful agent that preserves achievable assigned-task commitments.Prior rebel agents reinterpreted or overrode commands based on internal motivations or narrative coherence.Source: §7 · p. 14
• Unlike opportunistic and partial-satisfaction planners that optimize predefined goals, this work studies moral grounds for deviating from assigned tasks and preserving commitments.The cited lines include reward-seeking while satisfying hard goals, opportunities and commitments, and partial-satisfaction HTN planning.Source: §7 · p. 15
• Moral repair adds a driver of online adaptation beyond world-state changes or action inapplicability, integrating moral reasoning with planning and execution.This account aligns with actor-centered planning (Yuan et al., 2022) and the cognitive systems perspective.Source: §7 · p. 15

• The framework formalizes five approaches, from an amoral baseline to utilitarian, deontic, utilitarian-deontic, and dutiful agents, using online repair of task networks during execution.Source: §8 · p. 15
• The Dutiful architecture improves rescue outcomes and assigned-task completion without sacrificing normative compliance.Source: §8 · p. 15
• The architecture is intended to generalize beyond search and rescue, but applying it elsewhere requires repair rules suited to each domain’s tasks, normative constraints, and opportunities.Source: §8 · pp. 15–16
• The framework assumes agents maintain assigned tasks and deliberately repair behavior, while whether these computational intentions count as genuine intentions remains an open philosophical question.Source: §8 · p. 16