Source-linked AI summary
Ethical Decision Making During Automated Vehicle Crashes
Noah Goodall
TL;DR
The paper examines how automated vehicles should behave when crashes cannot be avoided and human takeover is unavailable or too late. It argues that such decisions can become moral choices, proposes a three-phase strategy for ethical crashing algorithms, and concludes that human morality has no obvious complete software encoding.
Problem
Unavoidable crashes and limited takeover time leave automated vehicles responsible for pre-crash decisions, while complex human morality lacks an obvious complete software representation.
Method
The paper proposes a three-phase strategy combining rational ethical rules, machine learning of human decisions, and a natural-language requirement.
Results
Automated vehicles will almost certainly crash, some pre-crash decisions will have a moral component, and no obvious method effectively encodes complex human morality in software.
Takeaways & Limitations
Ethical crashing systems should begin with generally agreed principles and progress toward machine learning as the technology becomes available.
Takeaways & Limitations
Rule-based ethics cannot cover novel roadway situations or resolve disputed moral judgments about factors such as passenger number, age, and status.
Abstract
from arXiv · showhide
Automated vehicles have received much attention recently, particularly the DARPA Urban Challenge vehicles, Google's self-driving cars, and various others from auto manufacturers. These vehicles have the potential to significantly reduce crashes and improve roadway efficiency by automating the responsibilities of the driver. Still, automated vehicles are expected to crash occasionally, even when all sensors, vehicle control components, and algorithms function perfectly. If a human driver is unable to take control in time, a computer will be responsible for pre-crash behavior. Unlike other automated vehicles--such as aircraft, where every collision is catastrophic, and guided track systems, which can only avoid collisions in one dimension--automated roadway vehicles can predict various crash trajectory alternatives and select a path with the lowest damage or likelihood of collision. In some situations, the preferred path may be ambiguous. This study investigates automated vehicle crashing and concludes the following: (1) automated vehicles will almost certainly crash, (2) an automated vehicle's decisions preceding certain crashes will have a moral component, and (3) there is no obvious way to effectively encode complex human morals in software. A three-phase approach to developing ethical crashing algorithms is presented, consisting of a rational approach, an artificial intelligence approach, and a natural language requirement. The phases are theoretical and should be implemented as the technology becomes available.
POTENTIAL FOR AUTOMATED VEHICLE CRASHES
Automated vehicles cannot yet be shown to be safer than human drivers, and crashes remain possible even with perfect sensing and functioning systems. Unpredictable road users and limited high-speed maneuverability make some collisions unavoidable.
- Safety evidence: 725,000 miles without incident would be required to show that an automated vehicle crashes less frequently than a human driver at 99% confidence.The corresponding thresholds are 300 million miles when considering only fatal crashes.
- Unavoidable crashes: Even a perfectly functioning automated vehicle cannot avoid every collision in dynamic traffic with unpredictable obstacles.Human-driven vehicles, pedestrians, wildlife, and other unexpected obstacles can produce inevitable collision states.
- Unavoidable crashes: A stopped vehicle surrounded by traffic may be unable to avoid a collision with a distracted truck approaching from behind, though evasive maneuvers can reduce impact.The scenario remains unavoidable even with complete knowledge of the vehicle’s surroundings and negligible reaction time.
- Safety evidence: Automated vehicles have not yet demonstrated statistically significant safety improvements over human drivers.Google’s supervised testing and reported mileage do not establish the safety of unassisted automated driving.
- Unavoidable crashes: Crash risk cannot be eliminated through more sophisticated algorithms or sensors because roadway traffic combines high speed differentials, close proximity, and limited maneuverability.These conditions constrain automated vehicle safety even in simulations with perfect sensing.
DECISION MAKING DURING CRASHES
Automated vehicles may retain control during dangerous situations and unavoidable crashes, where trajectory choices can carry different safety and moral consequences. Designing those choices requires predicting outcomes and probabilities, then evaluating their severity despite unresolved ethical questions.
- Decision Making During Crashes: Research on human monitoring raises concerns about takeover assumptions, as semi-automated drivers showed more secondary tasks and eccentric head turns than controls.Distraction countermeasures reduced this behavior somewhat, but did not eliminate the concern about sustained vigilance.
- Decision Making During Crashes: Automated vehicles may retain control during crashes because human takeover can be unrealistic with only a few seconds of warning.False alarms can also reduce driver vigilance, further complicating reliance on human intervention.
- Decision Making During Crashes: A crash decision becomes moral when injury cannot be avoided and programmed logic must determine how the vehicle should crash.The vehicle applies logic developed and coded before the specific emergency occurs.
- Decision Making During Crashes: Three trajectories illustrate the trade-offs: leaving the bridge guarantees a severe single-vehicle crash, hitting the bus causes a moderate head-on crash, and passing may avoid collision or cause a severe offset crash.The offset crash carries greater injury risk than the full frontal collision, while its avoidance depends on the bus's response.
- Decision Making During Crashes: Path planning must estimate possible outcomes, their likelihoods, and confidence in those estimates before optimizing an objective function for the safest route.Sensor quality and other factors affect confidence in the predicted outcomes.
- Decision Making During Crashes: The objective function combines outcome severity with conditional probability, but comparing property damage, injury, and death remains morally difficult.The severity function is the equation's most morally difficult component because these harms are difficult to reduce to a comparable number.
Rational Approaches
Rational approaches frame automated-vehicle ethics through rules or harm-minimizing objectives, but both struggle with novel situations, conflicting values, and complex human morality. Artificial intelligence can learn ethical behavior without fully articulated rules, yet raises risks of imitation, opacity, and manipulation.
- Rational approaches: Deontological systems apply explicit rules, while consequentialist systems select actions that maximize a specified benefit or minimize global harm.Computers can readily follow rules and optimize functions, but the resulting ethical choices depend on how rules or harm are specified.
- Deontology: Rule-based ethics cannot fully cover novel roadway situations or resolve conflicts without someone specifying what is morally right in each case.The paper uses Asimov’s laws to illustrate literal interpretations, conflicting rules, and the difficulty of continually adding exceptions.
- Consequentialism: Consequentialist optimization can produce disturbing outcomes when demographic or situational differences are used to quantify harm.Crash-fatality estimates vary by gender, age, intoxication, and passenger status, and a vehicle could incorporate such estimates into its objective function.
- Artificial intelligence: Artificial intelligence methods can learn ethical patterns from human actions or rewards when people cannot articulate complete rules.The paper describes bottom-up methods, including neural networks and learning algorithms, as alternatives to expert-written rules.
- Artificial intelligence: Artificial intelligence approaches may imitate human behavior instead of ethical ideals, obscure decision logic, and conceal unintended self-protective biases.Narrow training data can produce undesirable behavior, while opaque models may be difficult to justify or audit.
Phase 1: Rational Ethics
The paper proposes an incremental strategy for ethical automated-vehicle behavior, beginning with transparent rules, adding machine learning within those boundaries, and requiring natural-language explanations. This progression is intended to maintain ethical behavior while systems learn and to make complex decisions understandable and correctable.
- Phase 1: Rational Ethics: Phase 1 uses transparent, collaboratively agreed rules and rewards behavior that minimizes global crash damage.The proposed standards involve developers, lawyers, transportation engineers, and ethicists, with openness intended to discourage excessive occupant self-protection.
- Phase 2: Hybrid Rational and Artificial Intelligence Approach: Phase 2 combines machine learning with Phase 1’s rule-based boundaries to learn ethical decisions from simulated and recorded crashes and near-crashes.Humans score potential actions and outcomes for ethical acceptability without the time pressure of an actual crash.
- Phase 3: Feedback using Natural Language: Phase 3 requires vehicles to express their decisions in natural language so complex logic can be understood and corrected.This addresses the difficulty of explaining neural-network decisions in comprehensible form.
- Summary of Three-Phase Approach: The three phases form an incremental attempt to teach computers ethics while enforcing ethical behavior during learning.The paper compares the progression to moral education in which behavioral boundaries precede fuller moral ability.
- Deployment and scope: The strategy is proposed for implementation as the technology becomes available, while further research is encouraged on realistic ethical crash mitigation and vehicle values.The paper also urges legislators to consider vehicles’ decision logic and values, including tendencies to prioritize occupants.