Source-linked AI summary
Too Much of the Same: From Algorithmic to Human Bias in Learning to Defer
Dario Pesenti, Alessandro Bogani, Stefano Teso, Andrea Pugnana
TL;DR
Learning to Defer can create class-imbalanced rejection sets, yet human performance is not invariant to those deferral distributions. The paper evaluates LtD strategies and a user study, finding lower accuracy on whichever class dominates the deferred cases and exploratory evidence consistent with the Test-taker’s effect.
Problem
LtD research has largely neglected how algorithmic deferral biases may interact with human cognitive biases during downstream decision making.
Method
The paper evaluates three LtD strategies computationally and conducts a user study comparing balanced and imbalanced rejection sets.
Results
Imbalanced rejection sets made participants less accurate on whichever class constituted the majority of their assigned cases.
Takeaways & Limitations
LtD deployment should account for rejection-set distributions and their effects on human decision making, not only which cases are deferred.
Takeaways & Limitations
Stricter participant exclusions reduced the final sample below the power-analysis target, although the result pattern remained unchanged.
Abstract
from arXiv · showhide
Learning to Defer (LtD) extends supervised learning by allowing a Machine Learning (ML) model to defer harder or less confident decisions to a human expert. Despite being geared for human-AI collaboration, LtD strategies neglect the potential negative interference of human cognitive biases. Our contribution is twofold. First, we demonstrate that standard LtD strategies show class-dependent sampling bias in classification tasks in practice, and thus may disproportionately defer the minority classes when applied to imbalanced datasets. Second, we show that such asymmetries in task delegation may trigger human biases, ultimately leading to poorer downstream decision making. Specifically, we conduct a user study ($N=226$) where participants complete a classification task on a set of deferred items, with conditions presenting different levels of class imbalance. Our results show that participants exposed to a highly imbalanced rejection set achieved lower classification accuracy in the majority class compared to those exposed to a more balanced set, regardless of which class constituted the majority. Exploratory analyses suggest that this may be an instance of the Test-taker's effect, which stems from a mismatch between the actual distribution of classes and the participants' expectations about that distribution. Finally, we discuss the implications of these findings for the deployment of LtD algorithms.
1 Introduction
Learning to Defer supports hybrid human-AI decision making by routing difficult cases to human experts, but its deferral policies can create class-dependent sampling bias and affect downstream human decisions.
- Learning to Defer: LtD routes difficult or risky items to human experts when the ML model is insufficiently confident.Deferral may use confidence or novelty thresholds, or a separately trained rejector model.
- Learning to Defer: LtD can reduce cognitive load and improve human-AI team performance compared with either humans or models working alone.The approach has been applied in domains including biomedicine and vehicle design testing.
- Problem: Algorithmic deferral biases may trigger human cognitive biases, affecting downstream decision making.Imbalanced tasks can lead LtD strategies to defer challenging minority classes disproportionately, creating an abundance of same-class items for human decision makers.
- Empirical approach: Three LtD strategies were evaluated across one naturally imbalanced dataset and three datasets with manipulated training-class composition.The study examined Realizable Surrogate, Selective Prediction, and Compare Confidence, alongside a user study with N = 226 participants and 33,900 observations.
- Findings: Imbalanced rejection sets reduced participant accuracy on the majority class, while exploratory analyses favored the Test-taker’s effect over the Prevalence effect.The study compared heavily imbalanced deferred items with a more balanced control condition.
- Contributions: The contributions establish biased deferral sampling, empirical imbalanced rejection sets, and downstream performance effects in a user study.Together, these findings connect LtD’s algorithmic delegation patterns with possible cognitive effects.
2 Preliminaries & Related Work
The paper situates LtD within hybrid decision making and examines how its sampling patterns interact with established cognitive biases, especially when deferred classes are imbalanced.
- Learning to Defer: LtD extends supervised learning by allowing an ML model to defer predictions to another predictor, such as a human expert or a stronger ML model.The framework includes heuristic methods and more recent approaches with theoretical guarantees.
- Terminology: Coverage c denotes the percentage of cases handled autonomously, so 90% coverage implies approximately 10% deferred test instances.The deferred items collectively form the rejection set.
- Sampling bias: In naturally imbalanced ChestXRay tasks, LtD disproportionately defers minority-class instances across target coverage levels.Figure 1 encodes target coverage on the x-axis and class-specific deferred percentages on the y-axis, with minority instances shown in orange.
- Cognitive effects: Human-AI interaction can produce cognitive biases such as automation bias, trust misplacement, and ordering effects that may impair decision making.Prior LtD work also reported improved self-efficacy and task satisfaction on genuinely deferred items, alongside framing-related anchoring effects.
- Cognitive effects: The Test-taker’s effect describes choosing answers that fit expectations about overall answer distributions rather than relying on individual judgments.Repeated answer streaks can create uncertainty and prompt revisions toward an expected random sequence.
- Cognitive effects: The Prevalence effect is an increased visual-search error rate when targets make up no more than 10% of stimuli.It is associated with more omission errors and shorter reaction times when targets are absent.
- Research gap: Prior work had not connected algorithmic sampling biases in LtD with their direct effects on users interacting with these systems.This paper brings the algorithmic and psychological perspectives together.
3 Can LtD lead to Human Mistakes?
The paper hypothesizes that imbalanced training data can bias LtD rejection sets and that these distributions can systematically affect human decision making through cognitive mechanisms.
- Motivation: LtD frameworks often assume human experts are reliable, but deferred-case patterns may encourage inaccurate heuristics in human decision making.The paper highlights recognizable patterns concentrated in heavily deferred classes as one possible source of such heuristics.
- Hypotheses: H1 states that imbalanced training data can produce imbalanced rejection sets for LtD systems.The hypothesis concerns training-set imbalance with respect to a particular class.
- Hypotheses: Greater predictor confidence for one class can make that class more likely to be deferred relative to others.This mechanism motivates H2, which concerns systematic effects of imbalanced rejection sets on human decisions.
- Cognitive mechanisms: The paper proposes reduced attention and mismatched expectations about class distributions as two cognitive paths from imbalanced rejection sets to human performance changes.The first draws on visual-search findings about infrequent targets, while the second concerns expectations about answer distributions.
- Study design: The empirical design tests H1 through LtD experiments linking training imbalance to rejection-set bias and H2 through a larger user study linking selection bias to cognitive biases.The two analyses are presented in Sections 4 and 5, respectively.
4 Learning to Defer Induces Sampling Bias
The evaluation shows that representative Learning to Defer strategies can create class-dependent rejection-set sampling bias, especially when training data are imbalanced. Minority or underrepresented classes are preferentially deferred across natural and manipulated imbalance settings.
- Evaluation setup: Three LtD strategies—Selective Prediction, Compare Confidence, and Realizable Surrogate—were evaluated across naturally and artificially imbalanced classification datasets.The study used ChestXRay, GalaxyZoo, HateSpeech, and Cifar10H, with training class composition manipulated for the latter three datasets.
- Natural imbalance: Up to 100% of deferred ChestXRay instances belonged to the minority class at low coverage.ChestXRay’s minority class constituted 3.21% of training instances, and minority-class overrepresentation increased at higher coverage levels.
- Natural imbalance: At 90% coverage, the minority class remained overrepresented in ChestXRay deferrals: 38.1% for RS, 48.3% for SP, and 46% for CC.These values report the percentages of deferred instances belonging to the minority class for the three strategies.
- Artificial imbalance: Manipulating training imbalance changed rejection-set composition across GalaxyZoo, Cifar10H, and HateSpeech, with ablation increasing the represented class’s share among rejected items.For example, the majority-class proportion rose from 51.3% to 89.7% for GalaxyZoo RS and from 47.2% to 92.5% for HateSpeech RS.
- Conclusion: Overall, LtD strategies preferentially defer instances of the class underrepresented in the training set on naturally or artificially imbalanced data.This supports the paper’s first hypothesis concerning LtD-induced sampling bias.
5 Sampling Bias Leads to Human Mistakes
The user study tested whether different class proportions among deferred GalaxyZoo images affect human classification accuracy. Participants performed worse on whichever class was presented as the majority in imbalanced conditions, with exploratory evidence consistent with a Test-taker’s effect rather than a Prevalence effect.
- Study design: 226 participants classified 150 deferred GalaxyZoo images as smooth or non-smooth under balanced, smooth-majority, or non-smooth-majority conditions.The conditions used 50%-50%, 90%-10%, and 10%-90% class splits, respectively.
- Study design: The experiment used the same image pools across conditions, varying only the sampling proportions to minimize stimulus-related confounds.In the smooth-majority condition, all 135 smooth images and 15 randomly sampled non-smooth images were presented.
- Accuracy results: χ2(2) = 278.95, p < .001 for the condition-by-stimulus-ground-truth interaction, indicating that accuracy depended jointly on deferred condition and class.The main effects of condition and stimulus class were not significant.
- Accuracy results: Smooth-image accuracy was 0.62 ± 0.48 in the Smooth-majority condition, versus 0.72 ± 0.44 in Balanced and 0.79 ± 0.40 in Non-smooth-majority.Smooth-majority performance was significantly lower than both comparison conditions.
- Robustness checks: Accuracy did not change over time, and no training effect was found, making early familiarization and end-of-task fatigue unlikely explanations for the observed pattern.Trial block and its interaction with condition were not significant.
- Interpretation: After longer streaks of identical responses, switching accuracy declined in both imbalanced conditions but not in the balanced condition.The consecutive-response coefficient was −0.08 ± 0.02 for Smooth-majority and −0.05 ± 0.01 for Non-smooth-majority.
6 Discussion and Conclusion
The findings link LtD’s rejection-set composition to human decision quality, challenging the assumption that human performance is invariant to deferral policy. The authors suggest accounting for distribution-dependent human performance when designing or deploying LtD systems.
- Discussion and conclusion: Training-time class imbalance influenced rejection-set composition across all tested LtD strategies.The conclusion summarizes the ablation experiments as supporting the link between training imbalance and deferred-case distributions.
- Discussion and conclusion: Participants were less accurate on whichever class constituted the majority of their assigned deferred cases.This pattern was observed in the user study across the imbalanced conditions.
- Discussion and conclusion: Exploratory analyses suggest a Test-taker’s effect, but the evidence is not conclusive and differs from the opposite minority-class pattern expected for Prevalence effects.In imbalanced conditions, longer streaks of identical responses were followed more often by incorrect switches than in the balanced condition.
- Implications: LtD systems should consider not only which cases are deferred but also how rejection-set distributions affect human decision making.Suggested mitigations include informing users about non-uniform deferred cases and modeling distribution-dependent changes in human performance.
Ethical Statement
The study received approval from the university’s ethics board.
- Ethics approval was granted under document identifier code 2026-028ESA.
A LtD Experimental Details
The supplementary table reports machine-learning accuracy at target coverage 1.
- Table 1 reports machine-learning accuracy at target coverage 1.
B User Study Instructions
Participants consented to a short voluntary galaxy-classification study involving images selected for human review by an AI model. They classified centered galaxies as smooth or non-smooth using visible morphological features.
- B.1 Page 1: The study presents images of galaxies for classification as smooth or non-smooth.
- B.1 Page 1: The task is designed to take approximately 10 minutes.
- B.1 Page 1: Participation is voluntary, and participants may withdraw at any time without giving a reason.
- B.1 Page 1: Participants who withdraw do not receive compensation.
- B.1 Page 1: Questions about the study or data handling can be directed to the anonymized contact.
- B.1 Page 1: Informed consent is required before proceeding.
- B.2 Page 2: An AI model classified galaxy telescope images and deferred 150 images to human decision makers.
- B.2 Page 2: Participants classify only the centered galaxy, distinguishing smooth galaxies by gradual fading from non-smooth galaxies by visible structures such as spiral arms, bars, or bulges.
C User study results
The supplied materials define conventions for interpreting the user-study analyses and identify tables modeling trial-level accuracy and response-pattern effects. They also specify how experimental blocks and response streaks are represented.
- Predictors in the reported tables use deviation coding, with reference levels shown in brackets.
- SE denotes Standard Error, and OR denotes Odds Ratio.
- Table 2 reports the main logistic linear mixed-effects model for trial-level accuracy and its post-hoc contrasts.
- Table 3 repeats the logistic linear mixed-effects analysis after retaining participants who never left the experiment tab.
- Table 4 examines trial-level accuracy when no participants are excluded from the analyses.
- Table 5 assesses changes in trial-level accuracy across experimental blocks, with each order block representing 50 items.
- Table 6 models trial-level accuracy as a function of response streak length and experimental condition.