Source-linked AI summary
A Voting-Based System for Ethical Decision Making
Ritesh Noothigattu, Snehalkumar 'Neil' S. Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, Ariel D. Procaccia
TL;DR
The paper tackles automated ethical decision making without formally specified ground-truth principles by learning societal preferences from human judgments. It combines preference learning, model summarization, and efficient voting-based aggregation, then evaluates the approach in autonomous-vehicle dilemmas. The resulting Moral Machine system is presented as capable of credible decisions when other technical and legal options have failed.
Problem
Automating ethical decisions lacks formal ground-truth ethical principles, which remain contested among ethicists and moral philosophers.
Method
The approach learns individual preference models, summarizes them into a societal model, and uses swap-dominance-efficient voting rules to aggregate runtime alternatives.
Results
The implemented system produces credible autonomous-vehicle ethical decisions when all other options have failed.
Takeaways & Limitations
Societal preference aggregation provides a practical basis for computational ethical decision making and can later incorporate ethical or legal principles.
Takeaways & Limitations
The Moral Machine dataset’s roughly 14 comparisons per voter may be insufficient for learning truly accurate individual voter models.
Abstract
from arXiv · showhide
We present a general approach to automating ethical decisions, drawing on machine learning and computational social choice. In a nutshell, we propose to learn a model of societal preferences, and, when faced with a specific ethical dilemma at runtime, efficiently aggregate those preferences to identify a desirable choice. We provide a concrete algorithm that instantiates our approach; some of its crucial steps are informed by a new theory of swap-dominance efficient voting rules. Finally, we implement and evaluate a system for ethical decision making in the autonomous vehicle domain, using preference data collected from 1.3 million people through the Moral Machine website.
1 Introduction
The paper proposes automating ethical decisions by learning individual and societal preferences, then aggregating them at runtime for specific dilemmas. It develops supporting theory, instantiates the framework algorithmically, and evaluates it with Moral Machine data.
- The paper addresses the lack of formal ground-truth ethical principles by proposing societal preference aggregation as an alternative basis for automated ethical decisions.
- The four-step approach collects pairwise comparisons, learns each voter’s preferences, summarizes them into a collective model, and aggregates preferences over runtime alternatives.
- The system selects the least catastrophic alternative only after technical avoidance options and applicable legal constraints have failed.
- 18,254,285 pairwise comparisons from 1,303,778 voters provide the autonomous-vehicle preference data used for the approach.
- Permutation processes model preferences over every finite subset of a potentially infinite alternative space, accommodating scenarios with continuously modeled features.
- The aggregation theory uses swap dominance and swap-dominance-efficient voting rules to identify desirable alternatives efficiently under stated assumptions.
- The concrete algorithm uses the Thurstone-Mosteller process with linear feature-based utility parametrization, while the framework permits more complex processes.
- Simulation results validate learning and summarization accuracy, while the Moral Machine implementation yields a computationally and statistically attractive decision-making system.
2 Related Work
Related work connects ethical decision making with computational social choice, machine learning, and domain-specific preference systems. The paper distinguishes its explicit voting-based aggregation framework from prior proposals and applications.
- Earlier work proposed connecting computational social choice with ethical decision making and modeling ethical principles as preferences of a dummy agent.
- Conitzer et al. discussed combining machine learning and social-choice techniques to aggregate moral views, potentially reducing individual idiosyncratic mistakes.
- Prior work also considered representing ethical dilemmas through key moral features for machine-learning algorithms.
- Freedman et al. learned societal weights for eight patient types from 289 workers to prioritize kidney-exchange patients, without explicit preference aggregation or voting.
- The paper repurposes the Moral Machine dataset from measuring human decisions to automating decisions in autonomous-vehicle dilemmas.
3 Preliminaries
The preliminaries formalize consistent permutation processes as distributions over rankings for finite subsets of alternatives. They relate these processes to utility models and introduce Thurstone-Mosteller and Plackett-Luce examples.
- A permutation process assigns a ranking distribution to every finite subset of a potentially infinite alternative set.
- Consistency requires that restricting the distribution for a larger alternative set to a subset yields the distribution defined directly for that subset.
- Consistent permutation processes can be interpreted through stochastic utilities whose sorted values generate the corresponding rankings.
- The framework differs from standard random utility models by defining distributions across all finite subsets rather than one fixed finite alternative set.
- The Thurstone-Mosteller process uses independent Gaussian utilities with identical variances and alternative-specific mode utilities.
- The Plackett-Luce process uses independent Gumbel utilities with identical scale and alternative-specific mode utilities.
4 Aggregation of Permutation Processes
The section formulates aggregation over anonymous preference profiles induced by permutation processes and develops swap-dominance theory to obtain efficient, stable winners. For TM and PL processes, common monotonic and neutral voting rules accept an alternative with maximum mode utility.
- Preference aggregation: Anonymous preference profiles represent the population distribution over rankings, allowing voting rules to aggregate permutation-process preferences without voter identities.For a finite alternative set, each ranking’s probability equals the fraction of voters represented by that ranking.
- Preference aggregation: The aggregation problem requires choosing an anonymous social choice correspondence and computing its outcome without explicitly constructing the full preference profile.Different voting rules can produce different outcomes on the same profile, while direct profile construction may be computationally expensive.
- Efficient aggregation: For TM or PL processes, any maximum-mode-utility alternative belongs to every anonymous, monotonic, and neutral correspondence’s selected set.Plurality, Borda count, positional scoring rules, Copeland, maximin, and Bucklin satisfy these correspondence properties.
- Swap-dominance theory: Swap-dominance requires that rankings placing a above b have at least as much weight as the corresponding rankings with a and b swapped.The relation is stronger than existing notions such as position dominance.
- Swap-dominance theory: An SwD-efficient correspondence applied to an SwD-compatible process selects an alternative that swap-dominates every other alternative in any finite set.This result supplies a general winner-selection principle for permutation-process aggregation.
- Stability: Borda count and Copeland are strongly SwD-efficient, yielding stability with TM or PL processes and, more generally, with consistent SwD-compatible processes.Stability means reducing a profile to a subset and then applying the rule agrees with applying the rule after directly reducing to that subset.
5 Instantiation of Our Approach
The paper instantiates its framework with a feature-based Thurstone–Mosteller model: it learns each voter’s preferences, summarizes them into one model, and aggregates alternatives by mode utility. Concavity makes per-voter maximum-likelihood learning efficient, while the summary model enables fast runtime decisions.
- Model: The instantiation uses a Thurstone–Mosteller process whose utility means are linear functions of alternative features.Each alternative is represented by a vector in R^d, with µ_x = β^⊺x.
- Step II: Learning: For each voter, maximum likelihood estimates β from ordered pairwise comparisons indicating which alternative the voter chose.The learned parameter completely describes that voter’s Thurstone–Mosteller process.
- Step II: Learning: The standard normal CDF makes the log-likelihood concave in β, so the voter-specific parameter can be maximized efficiently.The likelihood uses the probability of the observed comparison under utilities distributed as N(β^⊺x, 1).
- Step III: Summarization: The N voter models are summarized into one Thurstone–Mosteller process because runtime decisions may need to be made within a split second.The summary parameter is chosen so its utilities are close to the mean utilities induced by the individual voter models, using KL divergence.
- Step IV: Aggregation: At runtime, aggregation selects an alternative with maximum summary mode utility, which is also acceptable under any anonymous monotonic and neutral correspondence.For the summarized model, this procedure is equivalent to applying Borda count or Copeland and has the stated stability property.
6 Implementation and Evaluation
The evaluation tests learning and preference summarization on synthetic data, then applies the resulting system to Moral Machine data for autonomous-vehicle decisions. The system achieves high winner agreement while remaining computationally efficient for large voter populations.
- Synthetic Data: 84.3% accuracy is achieved with 30 pairwise comparisons per voter, rising to 92.4% with 100 comparisons in Step II learning.Accuracy is measured by matching winners computed from learned versus true voter parameters.
- Synthetic Data: 93.9% accuracy is reached as the number of voters increases in Step III summarization, indicating a small expected accuracy loss.The summarized model averages voter parameters and is compared with the desired winner based on the mean voter profiles.
- Moral Machine Data: The implemented system learns each voter’s parameters, averages their parameter vectors, and supports real-time choices among any finite subset of alternatives.This summarizes the individual models into a single model for efficient runtime decision making.
- Moral Machine Data: 95.1% of winners match at 10 alternatives per instance, compared with 98.2% at two alternatives on Moral Machine test instances.Accuracy is computed over 3,000 test instances and declines gradually as the number of alternatives increases.
7 Discussion
The paper presents a proof-of-concept system that can make credible autonomous-vehicle ethical decisions, while identifying limitations in its data, features, principles, and model choices.
- The Moral Machine implementation is presented as a proof-of-concept system that can make credible decisions on autonomous-vehicle ethical dilemmas when other options have failed.
- Moral Machine users may be poorly informed or insufficiently reflective, potentially producing inconsistent answers and poor voter models.The authors believe much of this noise cancels out during summarization and aggregation.
- The dataset averages roughly 14 pairwise comparisons per voter, which may be insufficient for learning truly accurate voter models.Later work cited by the authors reports improved accuracy from common-distribution assumptions and roughly 90% prediction accuracy with up to 100 queries per voter.
- The Moral Machine feature set is contentious because it includes attributes such as gender and profession while omitting probabilities and degrees of harm.The authors identify feature selection as a major issue for machine-learning-based ethical decision making.
- Extending the framework to incorporate ethical or legal principles remains a primarily conceptual challenge, especially for settings where such principles are easier to specify.The design can fall back on societal choice when principles do not determine a decision.
- The implementation assumes linear utility structure; richer utility representations would require new learning and summarization methods.
- Summarizing individual Thurstone-Mosteller models as a single model may limit accuracy, while mixtures of a relatively small number K of models pose unresolved aggregation and efficiency challenges.
A Robustness of the Empirical Results
Appendix experiments test robustness by varying the number of voters, alternatives, and features while holding other synthetic-data parameters fixed.
- With N = 40 voters, the appendix tests robustness of Step II relative to the baseline N = 20 setting.
- With 3 alternatives per instance, the appendix tests robustness relative to the baseline of 5 alternatives.
- With d = 20 features, the appendix tests robustness relative to the baseline d = 10 setting.
A.1 Number of Voters in Step II
With 40 voters, Step II accuracy rises quickly as pairwise comparisons increase, reaching 89.3% with 30 comparisons and 94.9% with 100.
- A.1 Number of Voters in Step II: 40 voters are used to test robustness with respect to the number of voters N.These experiments replace the baseline N = 20 setting.
- A.1 Number of Voters in Step II: 89.3% accuracy is achieved with just 30 pairwise comparisons.Accuracy increases quickly as the number of comparisons grows.
- A.1 Number of Voters in Step II: 94.9% accuracy is achieved with 100 pairwise comparisons.
A.2 Number of Alternatives
With three alternatives, Step II accuracy reaches 88.8% with 30 comparisons and 93.5% with 100, while Step III reaches 96.2% as voters increase.
- A.2 Number of Alternatives: Three alternatives are used to test robustness with respect to the number of alternatives.The experiments use |A| = 3 instead of |A| = 5.
- A.2 Number of Alternatives: 88.8% accuracy is achieved in Step II with just 30 pairwise comparisons.Accuracy increases quickly as the number of comparisons grows.
- A.2 Number of Alternatives: 93.5% accuracy is achieved in Step II with 100 pairwise comparisons.
- A.2 Number of Alternatives: 96.2% accuracy is reached in Step III as the number of voters increases.
A.3 Number of Features
With 20 features, Step II accuracy increases more slowly as comparisons grow, reaching 74.6% with 30 and 88.2% with 100; Step III reaches 94.7% as voters increase.
- A.3 Number of Features: 20 features are used to test robustness with respect to the number of features d.The experiments use d = 20 instead of d = 10.
- A.3 Number of Features: Step II accuracy increases more slowly with higher dimension as pairwise comparisons increase.The reported slowdown is attributed to the higher dimension.
- A.3 Number of Features: 74.6% accuracy is achieved with 30 pairwise comparisons, rising to 88.2% with 100 comparisons.
- A.3 Number of Features: 94.7% accuracy is reached in Step III as the number of voters increases.