PaperlayerView exact revision on arXiv

Source-linked AI summary

'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions

Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, Nigel Shadbolt

arXiv:1801.10408v1cs.HCcs.CY

TL;DR

Algorithmic decisions can affect people without providing the process information they need to judge whether outcomes are just. Across studies comparing explanation contexts and styles, the authors find that explanation effects depend on exposure: differences emerge with multiple styles but largely disappear when one style is repeated.

  • Problem

    Algorithmic decisions lack the process information people use to judge justice, making explanation styles’ effects on justice perceptions important to understand.

  • Method

    The authors use qualitative and experimental studies comparing justice perceptions across scenarios, explanation styles, and single- versus multiple-style exposure.

  • Results

    Explanation styles produced significant justice-perception differences under multiple-style exposure, but these effects largely disappeared when one style was repeated across cases.

  • Takeaways & Limitations

    People evaluate algorithmic decisions through justice-related considerations, but explanations may help fairness evaluation depending on how and when they are deployed.

  • Takeaways & Limitations

    The samples were not representative, and the hypothetical scenarios lacked the direct consequences and possible responses of real-world decisions.

Abstract

from arXiv · show

Data-driven decision-making consequential to individuals raises important questions of accountability and justice. Indeed, European law provides individuals limited rights to 'meaningful information about the logic' behind significant, autonomous decisions such as loan approvals, insurance quotes, and CV filtering. We undertake three experimental studies examining people's perceptions of justice in algorithmic decision-making under different scenarios and explanation styles. Dimensions of justice previously observed in response to human decision-making appear similarly engaged in response to algorithmic decisions. Qualitative analysis identified several concerns and heuristics involved in justice perceptions including arbitrariness, generalisation, and (in)dignity. Quantitative analysis indicates that explanation styles primarily matter to justice perceptions only when subjects are exposed to multiple different styles---under repeated exposure of one style, scenario effects obscure any explanation effects. Our results suggests there may be no 'best' approach to explaining algorithmic decisions, and that reflection on their automated nature both implicates and mitigates justice dimensions.

INTRODUCTION · BACKGROUND · Interpreting intelligent systems

Algorithmic decisions increasingly affect access to jobs, loans, and insurance, creating accountability and justice concerns that explanation systems may help address. Existing interpretability research offers varied explanation approaches, but their suitability for affected individuals and justice-related goals remains underexplored.

  • INTRODUCTION: Algorithmic decision-making increasingly uses predictions from models trained on prior customers or employees to allocate jobs, loans, and insurance.Such decisions may be made automatically and have significant consequences for affected individuals.
  • INTRODUCTION: Affected individuals care about justice in decisions, not only whether the outcomes benefit them, while human decision-makers can be required to rationalise their judgments.This creates an accountability challenge when consequential decisions are produced by automated systems.
  • INTRODUCTION: The GDPR requires meaningful information about the logic of certain significant automated decisions, creating HCI challenges about communicating outputs to affected individuals.Information useful for accountability may differ from explanations designed to support expert decision-making.
  • INTRODUCTION: The study preliminarily examines how novel explanation approaches affect perceptions of algorithmic decisions regarding fairness, accountability, and transparency.It asks whether different explanation approaches meet these regulatory aims in different ways.
  • BACKGROUND: Research on this topic connects justice perceptions in decision-making with interpretable machine learning, HCI, law, and the social sciences.These areas jointly frame the paper’s investigation of algorithmic explanations and justice.
  • Interpreting intelligent systems: Intelligent-systems research has long treated explanations as a way for systems to account for their operation and help users understand how tasks are accomplished.Earlier systems commonly explained outputs by presenting derivations from rules.
  • Interpreting intelligent systems: Explanation effects on trust and acceptance are inconsistent, and complex machine-learning models make it difficult to extract the knowledge underlying their outputs.Explanation quality lacks a formal definition and shared evaluation methodology, while evaluation depends on context and purpose.
  • Interpreting intelligent systems: Local, pedagogical explanations such as LIME and QII have not yet been tested for regulatory contexts involving loans, hiring, and insurance.Existing discrimination-correction tools primarily support data scientists rather than individuals affected by decisions, although justice-related uses have been proposed.

Perceptions of justice regarding decision-making · Key questions and contributions

The section frames justice perceptions in algorithmic decision-making through established dimensions from human decision-making, while highlighting unresolved questions about explanations and explanation styles. It asks whether explanation effects and justice relationships observed in human settings also apply algorithmic decisions.

  • Perceptions of justice regarding decision-making: The section establishes the need to understand psychological aspects of justice perceptions before evaluating how explanation facilities might serve justice in algorithmic decision-making.It therefore considers literature on justice perceptions concerning consequential human decisions, including courts, employment, and financial-product allocation.
  • Perceptions of justice regarding decision-making: Justice perceptions in consequential decisions can be analyzed through procedural, distributive, and interactional dimensions.Procedural justice concerns decision processes and logic; distributive justice concerns equitable or deserved outcomes given affected people’s circumstances, performance, or contributions.
  • Perceptions of justice regarding decision-making: Informational explanations may let people assess an algorithm’s procedural justice and thereby moderate judgments of distributive fairness.This rationale is presented as plausibly applying to regulatory requirements for explaining algorithmic logic.
  • Key questions and contributions: The paper notes that applicability of human-decision findings to algorithmic decisions and their proposed explanation systems remains largely unexplored.Kizilcec’s study of transparency and procedural justice is identified as a notable exception.
  • Key questions and contributions: As machine-learning explanations become practically achievable, research increasingly asks which explanation styles are desirable, not merely implementable.The section cautions that styles exploiting cognitive biases to inflate justice perceptions may face anti-paternalist critiques associated with ‘nudge’ philosophy.
  • Key questions and contributions: The second key question is how different explanation styles affect justice perceptions.This question narrows the inquiry from explanations generally to variation in explanatory presentation.

STUDY DESIGN AND METHODOLOGY · Scenarios and application contexts

The studies used fictional algorithmic decision scenarios to examine how information provision and explanation styles shape perceptions of fairness. Five consequential application contexts were selected using criteria concerning familiarity, likely algorithmic use, and practical or economic impact.

  • STUDY DESIGN AND METHODOLOGY: The experimental design focused on people’s responses to algorithmic decisions and explanation styles rather than their ability to predict model outputs or answer logic questions.The authors note that many prior explanation systems had not undergone user evaluation, while existing evaluations often emphasized predictive or logic-related tasks.
  • STUDY DESIGN AND METHODOLOGY: The study did not assess whether explanations faithfully represented model reasoning, despite acknowledging that faithfulness matters if explanations are to be more than comforting stories.This methodological focus isolates the effects of information provision and explanation styles on perceived fairness.
  • STUDY DESIGN AND METHODOLOGY: The researchers used fictional scenarios to test a wide range of novel explanation styles that had not been widely implemented with deployed machine-learning systems.They acknowledge that this approach may have lower ecological validity than field surveys but offers broader coverage of proposed explanation styles.
  • Scenarios and application contexts: The researchers selected 5 application contexts involving common interactions, likely present or future machine-learning use, and significant economic or practical effects for decision-subjects.Context selection also considered whether situations might fall under the previously mentioned legal framework.
  • Scenarios and application contexts: The contexts covered personal financial loans, workplace promotions, dynamically priced car insurance, airline re-routing, and bank-account freezing for suspected money laundering.The scenarios span financial, employment, insurance, travel, and banking decisions.
  • Scenarios and application contexts: For each context, the researchers created fictional cases in which an individual received an automated decision, drawing on established algorithmic ‘war stories’.Cases were adapted where necessary to fit the United Kingdom city from which in-person participants were recruited.

Explanation styles · Justice constructs

The paper distilled model-agnostic algorithmic explanations into four information-based styles and tested justice constructs adapted from human decision-making research. Justice ratings used five-point agreement statements, supplemented by a separate agreement item to distinguish justice perceptions from simple acceptance of the decision.

  • Explanation styles: The review produced four model-agnostic explanation styles intended to provide meaningful information about automated decisions.The styles were derived from technical interpretability research and legal transparency requirements.
  • Explanation styles: Input Influence lists variables with quantitative measures of their positive or negative influence on the decision.
  • Explanation styles: Sensitivity shows how much each input variable would need to differ to change the output class.This usage differs from sensitivity as an evaluation metric in machine learning.
  • Explanation styles: Case-based explanations present the most similar case from the model’s training data.
  • Explanation styles: Demographic explanations present aggregate outcome statistics for people sharing demographic categories such as age, gender, income, or occupation.
  • Explanation styles: The explanation styles were informally tested and presented as purely textual explanations to control for differences in representation.The literature also uses graphs and bullet points, but this study varied wording and information dimensions within text.
  • Justice constructs: Justice constructs across all three studies were adapted from psychology-of-justice research and prior studies of human decision-making for automated algorithmic scenarios.Participants rated agreement with five statements on a 5-point Likert scale.
  • Justice constructs: An additional agreement statement was placed before the justice items because participants tended to treat the constructs as a proxy for whether they agreed with the decision.This was intended to separate justice perceptions from simple decision agreement.

Phase 1: Lab study … Within-subjects study

The paper combined an in-person qualitative lab study with quantitative online studies to examine how people interpret and evaluate algorithmic decisions. The online work compared explanation conditions between subjects and tested whether directly comparing multiple explanations changes perceived justice.

  • Phase 1: Lab study: The lab study used semi-structured interviews and fictionalised cases to examine how participants interpreted, evaluated, and reasoned about algorithmic decisions across contexts and explanation styles.The study used concurrent think-aloud reflection and thematic analysis of transcribed interviews.
  • Phase 1: Lab study: Nineteen UK participants considered 15 cases spanning 5 contexts, rating five justice measures on a 5-point Likert scale while verbalising their reasoning.Cases included four explanation styles and a no-explanation control condition.
  • Phase 2: Online studies: Two online studies followed the lab study to generate quantitative data for testing explanation-style effects and relationships among justice constructs in algorithmic decision-making.The protocols were pared-down versions of the lab study and recruited UK adults through Prolific Academic.
  • 1. Do different explanation styles result in differences in perceived levels of justice?: The online research used a between-subjects design and added a within-subjects experiment because direct comparisons between explanation styles might alter justice perceptions.The follow-up tested whether exposure to multiple styles for one case produced different effects.
  • Between-subjects study: In the between-subjects study, 325 participants were randomly assigned to 1 of 5 conditions, receiving 12 cases with one explanation style or no explanation.Each group contained n = 65, and participants rated the same five measures used in phase 1.
  • Within-subjects study: The within-subjects study involved 65 participants and focused on loan and insurance cases because other scenarios could not support realistic presentation of four explanations for one case.Participants compared four negative decisions from different lenders or insurers, each explained in a different randomly ordered style.
  • Within-subjects study: The airline context was excluded because participants almost universally judged its cases very unfair and undeserved, preventing useful conclusions.The exclusion was made at the within-subjects stage.
  • Within-subjects study: Both online studies used Spearman’s rho correlations and ANOVA with Tukey’s post-hoc paired tests to examine justice-construct relationships and explanation-style effects.The analyses assessed how explanation styles related to different justice constructs.

RESULTS … The (lack of) human touch

The lab study’s think-aloud responses identified five major themes, including the lack of human touch, while participants often viewed algorithmic decisions as impersonal, dehumanising, and lacking negotiation or human interaction. These reactions suggest that interactional justice may be relevant when evaluating algorithmic decisions.

  • Participant information: The lab study included 19 participants, with an average age of 28.8 years and varied educational attainment.Participants included 11 male and 8 female individuals; education ranged from high-school/A-levels to PhD.
  • Participant information: The between-subject online study had 64% female participants, an average age of 37.6 years, and varied employment and education profiles.48% were in full-time employment, 26% unemployed, and 21% in part-time employment.
  • Participant information: The within-subject online study had 64% female participants, an average age of 39.77 years, and an average completion time of 8.1 minutes.Participants had varied employment and education profiles, including 42% with an undergraduate degree.
  • Qualitative Results: Themes and reflections: Think-aloud responses produced five major themes spanning explanation styles and scenario types, including human touch, system reasoning, statistical inference, actionability, omissions, and moral concepts.The passage lists five major themes while naming six thematic areas, with many occurring across different styles and scenarios.
  • The (lack of) human touch: Many participants described algorithmic decision-making as impersonal, dehumanising, undignified, or ‘weird’ compared with human decision-making.These reactions arose when an employee was automatically rejected for promotion.
  • The (lack of) human touch: Participants criticised automated systems for providing no sense of negotiation or opportunity for human interaction, while some anthropomorphised the system as ‘rude’.The responses contrasted the system’s apparent lack of humanity with attempts to humanise it.
  • The (lack of) human touch: Specific figures in explanations led one participant to infer arbitrary thresholds, making the decision seem ‘mean’ and difficult to understand.The participant questioned why the cut-off was so specific and said she did not understand its basis.
  • The (lack of) human touch: These reactions suggest that interactional justice may be a relevant dimension for evaluating algorithmic decisions when systems are judged by norms of social behaviour.The inference follows participants’ concerns about impersonal treatment, absent negotiation, and unexplained cut-offs.

Interpreting the system’s ‘reasoning’ · Acceptability of statistical inference

Participants judged algorithmic decisions partly by whether they could reconstruct the system’s reasoning, but found decisions unfair when understandable premises did not connect to the overall outcome. They also questioned statistical generalisation as a basis for judging individuals, while accepting some inferences from relevant past behaviour.

  • Interpreting the system’s ‘reasoning’: Participants assessed algorithmic reasoning by comparing it with their own reasoning or knowledge.Some subjects viewed the computer as using the same reasoning they would use, while others accepted decisions based on known statistical patterns.
  • Interpreting the system’s ‘reasoning’: Others interpreted the system’s reasoning as rules linking behaviours to consequences.For example, breaking the speed limit was understood as violating standards for the cheapest insurance.
  • Interpreting the system’s ‘reasoning’: Understandable individual premises did not always explain how the algorithm reached its overall decision.Participants could see the relevance of each point but could not determine how those points fit into the “big picture”.
  • Interpreting the system’s ‘reasoning’: When the reasoning process seemed inexplicable, some participants concluded that the system was “making it up” and therefore unfair.The perceived arbitrariness of the decision undermined its acceptability.
  • Acceptability of statistical inference: Participants questioned whether statistical inference was scientifically rigorous, including whether predictions used sufficiently large samples.They wanted to know how many previous customers formed the basis for the decision.
  • Acceptability of statistical inference: Many participants objected to statistical inference because it reduced individuals to probabilities rather than considering their actual abilities, successes, or circumstances.Concerns included treating gender as determinative and using comparisons with other people as “random stats” rather than reasons.
  • Acceptability of statistical inference: Participants noted that similarity to past individuals could conceal extenuating circumstances, particularly in case-based explanations.Poor performance by one person was not necessarily evidence that another person would perform poorly.
  • Acceptability of statistical inference: Participants were sometimes more comfortable judging future behaviour from an individual’s own past behaviour than from comparisons with other individuals.One participant accepted denying a loan to an applicant with existing debts because they had “proven not to be able to pay it back”.

Actionability

Participants judged algorithmic decisions more justifiable when explanations identified realistic actions that could change the outcome, but less deserved when people lacked control or actionable guidance. They also noted that algorithmic explanations might encourage gaming the system.

  • Actionability: Negative decisions were more acceptable when sensitivity-based explanations identified a reasonable alternative action that would have produced a different outcome.Participants endorsed rejecting applicants when explanations suggested concrete changes, such as increasing income or borrowing less.
  • Actionability: Participants saw decisions as particularly justified when individuals were responsible for risky or undesirable past behaviour.One participant considered denying cheap insurance deserved after repeated accidents, unless they were entirely due to bad luck.
  • Actionability: Decisions were viewed as undeserved when based on circumstances outside an individual’s control or on unrealistic suggested actions.Participants specifically identified birthplace, gender, and the absence of poor decisions as factors people could not reasonably change.
  • Actionability: Explanations were expected to guide future action, a function participants found particularly lacking in demographic explanations.A demographic statistic such as “1.5% of women have had their accounts frozen” was criticized as providing nothing actionable.
  • Actionability: Participants also worried that actionable knowledge about an algorithm could incentivize people to discover how to game it.This concern framed behaviour change in response to algorithmic decisions as potentially dangerous.

Unaccounted aspects

Participants criticized algorithmic decisions for omitting important personal information, insufficient complexity, and proportionality between inputs and outcomes. These omissions made decisions appear incomplete or out of proportion.

  • Missing information: Participants said decisions lacked important information, including credit scores and greater knowledge about the individual.Some also noted that circumstances such as sudden terminal illness could explain an inability to repay a loan.
  • Missing complexity: Some participants viewed the algorithm as insufficiently complex and suggested asking more questions to identify cases where applicants were actually fine.They proposed making the algorithm more complicated as one way to address this omission.
  • Missing proportionality: Participants also criticized explanations when relevant inputs were weighted disproportionately in the decision.One participant considered existing debts relevant but objected to assigning them a minus 7 influence on loan denial.

Meaning and relevance of moral concepts · Quantitative results · Do justice correlations from human decision-making settings

Participants questioned whether moral concepts such as fairness and desert meaningfully apply to computer-driven decisions, while quantitative findings showed that explanation effects depended strongly on exposure to multiple styles. Justice correlations broadly followed expected construct relationships, but comparison revealed additional significant links and lower evaluations for case-based explanations.

  • Meaning and relevance of moral concepts: Participants sometimes questioned whether fairness and desert could meaningfully apply to computer-driven decisions.One participant said computer decisions were difficult to judge morally because a computer simply executes instructions.
  • Meaning and relevance of moral concepts: Fairness was also viewed as potentially inapplicable when systems primarily pursue organisational efficiency.A participant described such decisions as understandable from the business perspective despite not being fair.
  • Meaning and relevance of moral concepts: A truly random system was described as potentially fair under some definitions of fairness.The participant linked this view to a conception of fairness as chaos or random allocation.
  • Quantitative results: The quantitative study involved 325 Prolific Academic participants rating scenarios and explanation styles on 5-point Likert scales.Participants responded to questions concerning justice perceptions across different decision scenarios and explanation conditions.
  • Do justice correlations from human decision-making settings: Justice correlations partially confirmed expected relationships, while significant positive correlations also appeared between two relationships that were not expected.Expected links included connections among fair process, deserved outcome, understanding, and appropriateness of factors.
  • Do justice correlations from human decision-making settings: In the between-subjects study, explanation styles generally did not significantly affect justice perceptions, except for fair process in loans only (loan F(3, 258) = 2.71, p = 0.046) and appropriate factors (F(3, 258) = 5.35, p = 0.001).The study compared single-style exposure with multiple-style exposure using ANOVA tests followed by Tukey’s post-hoc paired tests.
  • Do justice correlations from human decision-making settings: In the within-subjects study, explanation styles significantly affected fair process and appropriate-factor perceptions in both loan and insurance scenarios.Fair process effects were loan scenario F(3, 260) = 7.52, p < .001 and insurance scenario F(3, 260) = 4.5, p = .004; appropriate-factor effects were loan F(3, 260) = 3.312, p = .02 and insurance F(3, 260) = 6.44, p = .0003.
  • Do justice correlations from human decision-making settings: Case-based explanations produced lower perceptions of appropriateness, fair process, and, for loans, deservedness than sensitivity-based styles.These effects appeared primarily when participants compared multiple explanation styles for the same decision.

DISCUSSION

Algorithmic decisions elicited conflicting justice responses: some participants viewed data-driven generalisation as unfair, while others treated statistical accuracy as sufficient for fairness. Explanation effects depended on exposure, and justice constructs correlated in ways broadly consistent with prior literature.

  • Justice perceptions: Participants disagreed about algorithmic justice, viewing past-data decision-making as unfair or treating accurate statistical inferences as statistically fair.Some participants objected to generalising from others’ characteristics, whereas others considered fairness irrelevant once the system was accurate.
  • Justice perceptions: Algorithmic decisions implicated justice while sometimes mitigating it, as people alternated between moral and computational standards for evaluating outcomes.Participants could distinguish whether an outcome was deserved according to a model from whether it was deserved in a court of law.
  • Explanation styles: Explanation styles significantly differed in justice perceptions within subjects, particularly because case-based explanations negatively affected justice-related judgements.The importance of explanation style was complicated because participants engaged closely with each explanation’s details.
  • Explanation styles: Repeated exposure may habituate participants to an explanation style, whereas comparing multiple styles makes differences more salient and increases attention to explanation features.Under repeated exposure, attention may shift toward case features rather than the explanation itself.
  • Justice constructs: 0.69 was the correlation coefficient between procedural and distributive justice, compared with 0.48 for the same constructs in a 2001 meta-analysis.The observed correlations largely aligned with justice-psychology findings accumulated since the 1970s.

LIMITATIONS AND FUTURE WORK … CONCLUSION

The studies combine multiple methods but face important validity limitations, including nonrepresentative samples and restricted scenarios. The paper calls for further research on justice measures and interpretable systems, concluding that people assess algorithmic decisions through both familiar and novel justice considerations.

  • LIMITATIONS AND FUTURE WORK: The three-study design combines different methods, but the authors note important limitations associated with any single approach.The methodological combination only partly addresses these problems.
  • Threats to validity: The samples were not representative: the lab sample came from an affluent, academically dominated UK city, while online studies were not gender balanced.The online studies included almost twice as many females as males, and decision-context experiences may differ across countries.
  • Future work: Future work should apply additional psychology-of-justice measures, including interactional justice, to algorithmic decision-making.Participants’ descriptions of a computer as “rude” suggest interactional justice may matter even without direct human communication.
  • Future work: Future research should examine systems that make machine-learning outputs interpretable in different ways for multiple end-users and purposes.The passage identifies this as another opportunity arising from the discussion and limitations.
  • CONCLUSION: As algorithmic decisions become more consequential, understanding how people assess their fairness and what explanations they need for accountability remains increasingly significant.The conclusion notes that much remains to learn despite repeated calls for greater transparency.
  • CONCLUSION: People consider justice-related aspects of algorithmic decisions much as they do in manual processes, but explanations help evaluate fairness only depending on how and when they are deployed.Algorithmic systems also introduce novel considerations not captured by traditional justice-perception research.
Loading 1801.10408v1…