Source-linked AI summary
Human-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic Advice
Saar Alon-Barkat, Madalina Busuioc
TL;DR
The paper asks how public decision-makers process algorithmic advice when algorithms are used in high-stakes administrative settings. Using three survey experiments in the Netherlands, it finds no greater algorithmic deference than human-expert deference, but selective adherence when advice aligns with negative ethnic stereotypes. These patterns raise concerns for oversight and disadvantaged citizens, while the findings remain bounded by the study contexts and prior limitations it addresses.
Problem
The paper examines whether human processing of algorithmic advice produces automation bias or selective adherence, a question important for high-stakes public-sector decisions and vulnerable citizens.
Method
The authors conduct three survey experiments in the Netherlands, comparing algorithmic and human-expert advice and testing contradictory information, ethnic cues, and civil-servant responses.
Results
Participants were not more likely to follow algorithmic than human-expert advice, while selective adherence occurred when low scores were assigned to negatively stereotyped ethnic-minority teachers.
Takeaways & Limitations
Replacing human advice with algorithmic advice does not eliminate selective adherence, creating potential oversight blind spots when algorithmic recommendations disadvantage minorities.
Takeaways & Limitations
The study calls for further testing across additional policy sectors and national jurisdictions, as prior evidence was concentrated in one policy context.
Abstract
from arXiv · showhide
Artificial intelligence algorithms are increasingly adopted as decisional aides by public bodies, with the promise of overcoming biases of human decision-makers. At the same time, they may introduce new biases in the human-algorithm interaction. Drawing on psychology and public administration literatures, we investigate two key biases: overreliance on algorithmic advice even in the face of warning signals from other sources (automation bias), and selective adoption of algorithmic advice when this corresponds to stereotypes (selective adherence). We assess these via three experimental studies conducted in the NetherlandsWe discuss the implications of our findings for public sector decision making in the age of automation. Overall, our study speaks to potential negative effects of automation of the administrative state for already vulnerable and disadvantaged citizens.
Introduction
AI algorithms increasingly support high-stakes public-sector decisions, but bias may arise not only in algorithms themselves, but also in how humans process algorithmic advice. This study examines automation bias and selective adherence as competing patterns of human–AI decision-making.
- Introduction: AI tools are increasingly used as decisional aides in high-stakes public-sector domains, where human decision-makers remain involved in interpreting algorithmic outputs.Examples include policing, welfare, criminal justice, healthcare, immigration, and education.
- Introduction: Algorithmic decision-making may reproduce systemic bias through data and models, while human processing of algorithmic outputs can introduce additional bias.The paper focuses on bias arising at the human–AI interaction rather than solely within algorithmic systems.
- Introduction: Automation bias is the tendency to defer automatically to automated systems despite contradictory information or warning signals from other sources.This reflects reliance on automation as a shortcut that can replace vigilant information seeking and processing.
- Introduction: Selective adherence is the tendency to adopt algorithmic advice when it matches pre-existing stereotypes about decision subjects.The paper extends motivated-reasoning and bureaucratic-discrimination literatures to advice produced by AI algorithms.
- Introduction: Three survey experiments in the Netherlands test these biases among citizens and civil servants, including comparisons between algorithmic and human-expert advice.Study 1 tests contradictory algorithmic advice, study 2 examines ethnic cues, and study 3 replicates the findings with Dutch civil servants.
- Introduction: Prior studies provide tentative evidence that public decision-makers process algorithmic advice selectively but do not strongly establish automatic deference to it.The authors address limitations in prior work by comparing algorithmic advice with equivalent human-expert advice and adding contradictory information.
Research Design
Across three Dutch experimental studies, the paper tests whether people defer to algorithmic advice and whether stereotypes shape selective adherence. The studies find no automation bias, but selective adherence produces group disparities across both algorithmic and human advice.
- Research Design: The experiments presented qualitative evaluations alongside ILE numeric predictions, then measured whether participants followed the lowest score by not renewing that teacher’s contract.The ILE prediction ranged from 1 to 10, with 1 indicating the lowest prediction.
- Study 1: Study 1 found very small, statistically insignificant differences between algorithmic and human-advice conditions.Under both conditions, most participants overrode the ILE score and chose the teacher with the poorest qualitative evaluation.
- Study 2: Study 2 replicated the absence of automation bias: pooled results showed 11.1% versus 10.5% adherence, with OR = 1.07 and statistically insignificant differences.The result remained unchanged after adding covariates and restricting analyses to participants passing manipulation checks.
- Study 2: Participants were 50% more likely not to renew a Moroccan-Dutch teacher’s contract than a Dutch teacher’s contract at the same low ILE score.The reported odds ratio was OR = 1.50, p = .04; descriptive probabilities were 12.3% versus 8.6%.
- Study 2: Selective adherence appeared across both human and algorithmic advice, but the interaction did not show that algorithms amplified this bias.The pattern was consistent across incongruence conditions, while the authors acknowledge statistical-power limitations for the interaction analysis.
- Limitations: The authors caution that the study occurred shortly after a scandal that made participants unusually aware of algorithmic-bias risks.Sixty-three percent reported familiarity with algorithms used by public organizations, and 33% mentioned the case as an example.
- Study 3: Study 3, conducted with civil servants after the scandal, again found no automation bias and found lower adherence to algorithmic than human advice.Civil servants were also less likely to sanction the Moroccan-Dutch teacher regardless of advice source.
Discussion and Conclusion
Across three Dutch experiments, the authors found no general automation bias, but selective adherence appeared when algorithmic advice aligned with negative ethnic stereotypes. They argue that human–AI interaction can preserve bias and create oversight concerns for vulnerable citizens.
- Automation Bias: Across three studies with 2,854 participants, researchers found no general pattern of automatic adherence to algorithmic advice.Participants were not consistently more likely to follow algorithmic than human-expert advice.
- Automation Bias: In studies 1 and 2, algorithmic and human-expert advice produced small, statistically insignificant differences in adherence.In study 3, participants were less likely to follow algorithmic advice after the childcare benefits scandal.
- Limitations and Future Research: The authors identify limited prior comparisons between algorithmic advice and equivalent human advice as a methodological gap addressed by this study.They also note that skepticism or limited exposure to AI may help explain differences from automation-bias findings in aviation and healthcare.
- Implications: The findings suggest that algorithmic tools did not generally supplant human discretion, while selective adherence may create blind spots in meaningful oversight.The authors connect these concerns to possible negative effects of administrative-state automation for vulnerable and disadvantaged citizens.
- Selective Adherence: When a low prediction score was assigned to a negatively stereotyped ethnic-minority teacher, participants were more likely not to renew that teacher’s contract.Study 2 found this selective adherence pattern across both human and algorithmic advice conditions.
- Selective Adherence: Selective adherence was not stronger for algorithmic advice than human advice, but replacing human advice with algorithmic advice did not eliminate the bias.The interaction between advice source and ethnicity was insignificant, while the ethnicity effect was positive and significant.
Funding
The project received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation programme.
- Funding: The project received European Research Council funding through the European Union’s Horizon 2020 research and innovation programme.The grant agreement number is 716439.
Randomization Groups
The supplied material records questionnaire exclusions for attention-check failures, very short completion times, and non-Dutch descent in selective-adherence analyses.
- Exclusion Criteria: Participants failing the attention check or completing the questionnaire in less than 3 minutes were excluded.Participants not of Dutch descent were excluded from analyses of selective-adherence hypotheses H2 and H3.
Sample Characteristics
The studies comprised 605 participants in Study 1, 904 in Study 2, and 1,345 in Study 3, with Dutch civil-service data also identified in the table.
- Study Samples: Study 1 included 605 participants, Study 2 included 904, and Study 3 included 1,345.The table identifies Study 3 as involving the Dutch Civil Service.
- Study Samples: Dutch civil-service data from 2018 covered 412,999 national and local civil servants, including defense and police.The table note identifies the Dutch civil-service source.