Source-linked AI summary
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R. Varshney, Amit Dhurandhar, Richard Tomsett
TL;DR
AI-assisted decisions can be distorted by human cognitive biases, but existing human–AI communication research often sidelines decision-makers’ cognition. This paper models those biases, studies anchoring with time-based de-anchoring, and develops confidence-based time allocation; experiments support de-anchoring, especially for incorrect, low-confidence AI predictions, under stated assumptions.
Problem
Human perception and understanding of AI predictions can be distorted by cognitive biases, creating a need to account for cognition in human–AI collaborative decision-making.
Method
The paper develops a biased Bayesian framework, studies time-based mitigation of anchoring, and formulates a confidence-based time-allocation strategy for resource-constrained collaboration.
Results
Experiments validate that more time reduces anchoring and show that confidence-based information helps humans de-anchor from incorrect, low-confidence AI predictions.
Takeaways & Limitations
Time can mitigate anchoring, while confidence-based allocation can support human–AI collaboration by directing adjustment effort under limited time.
Takeaways & Limitations
Participants were not experts in student assessment, and the study searched only four time settings because of limited resources.
Abstract
from arXiv · showhide
Several strands of research have aimed to bridge the gap between artificial intelligence (AI) and human decision-makers in AI-assisted decision-making, where humans are the consumers of AI model predictions and the ultimate decision-makers in high-stakes applications. However, people's perception and understanding are often distorted by their cognitive biases, such as confirmation bias, anchoring bias, availability bias, to name a few. In this work, we use knowledge from the field of cognitive science to account for cognitive biases in the human-AI collaborative decision-making setting, and mitigate their negative effects on collaborative performance. To this end, we mathematically model cognitive biases and provide a general framework through which researchers and practitioners can understand the interplay between cognitive biases and human-AI accuracy. We then focus specifically on anchoring bias, a bias commonly encountered in human-AI collaboration. We implement a time-based de-anchoring strategy and conduct our first user experiment that validates its effectiveness in human-AI collaborative decision-making. With this result, we design a time allocation strategy for a resource-constrained setting that achieves optimal human-AI collaboration under some assumptions. We, then, conduct a second user experiment which shows that our time allocation strategy with explanation can effectively de-anchor the human and improve collaborative performance when the AI model has low confidence and is incorrect.
1 Introduction
The paper frames cognitive biases as a central part of human–AI communication and develops a biased Bayesian framework to model them. Focusing on anchoring, it validates time-based de-anchoring and proposes confidence-based time allocation for resource-constrained collaboration.
- Human perception of AI communication is central to the human–AI collaboration gap, motivating explicit models of cognition in decision-making.
- Anchoring bias limits exploration of alternative hypotheses because decision-makers form skewed perceptions around AI decisions and rely on them blindly.
- In a second experiment, confidence-based information helped participants de-anchor from incorrect, low-confidence AI predictions, although overall policy performance was roughly similar.
- The biased Bayesian framework models cognitive biases and situates confirmation, availability, representativeness, selective-accessibility, anchoring, and weak-evidence effects.
- More decision time reduces anchoring in AI-assisted decisions, validating time as a resource that helps people adjust away from an AI prediction when needed.
- The paper formulates a time-allocation problem accounting for anchoring and AI-accuracy variance, then proposes a confidence-based policy with optimality conditions.
2 Related work
Research on cognitive biases in human-AI collaboration remains limited, especially for anchoring bias. This work distinguishes its time-based de-biasing approach from prior settings and related reliance research.
- Prior studies examine trust, interpretability, explainability, and reliance on algorithmic systems as aspects of human-AI communication.
- Cognitive-bias research in human-AI collaboration is still rare, with prior work often addressing interpretability, rule-based models, or business decision processes.
- Anchoring in AI-assisted decisions has been discussed as automation bias and linked to over-reliance, poor reliance calibration, and difficulty detecting algorithmic errors.
- The paper presents the first empirical and analytical study of a time-based de-biasing strategy for anchoring bias in AI-assisted decision-making.
- Unlike work that delays the AI prediction until after human deliberation, this study provides the prediction beforehand and treats time as a limited resource for allocation.
3 Problem setup and modeling
The paper models AI-assisted binary classification as Bayesian decision-making by a human who combines task features, AI output, and prior beliefs. Cognitive biases are represented as exponents that reweight these factors in the final decision ratio.
- The setup is a binary classification problem in which humans receive feature information and AI output before making the final prediction.
- A rational decision-maker combines the data likelihood, perceived AI-output likelihood, and prior probability under a conditional-independence assumption between features and AI output.
- The Bayesian model introduces exponential bias parameters α, β, and γ to represent distortions in different inference factors.
- The human chooses class 1 when the posterior ratio exceeds 1 and class 0 otherwise; the ratio is a product of reweighted prior, data, and AI-prediction terms.
- Anchoring bias corresponds to β > 1, confirmation bias to γ > 1, selective accessibility to distortion of the data likelihood, and weak evidence effects to β < −1.
4 Anchoring bias
The paper treats anchoring as insufficient adjustment toward an AI prediction caused by limited cognitive resources, and studies whether allocating more time enables de-anchoring. Experiment 1 supports this mechanism by showing greater disagreement with incorrect AI predictions at longer time limits.
- Anchoring bias: Anchoring-and-adjustment describes insufficient movement away from an AI anchor because adjustment is effortful, framing anchoring as a time-accuracy trade-off under bounded rationality.
- Anchoring bias: The paper defines de-anchoring as rational adjustment away from the AI anchor and conducts two user studies using time as the relevant resource.
- Experiment 1: Experiment 1 measures anchoring through human agreement with the AI label in an AI-assisted binary student-performance prediction task.
- Experiment 1: 58.3%: accuracy of the AI predictions shown in unmodified test trials, compared with the 85% accuracy stated during training.
- Experiment 1: 48% to 67%: average disagreement on probe trials increased from the 10-second condition to the 25-second condition.
- Experiment 1: 0.01: the linear-regression coefficient relating average probe-trial disagreement to allocated time, with bootstrap 95% confidence interval [0.001, 0.018].
5 Optimal resource allocation in human-AI collaboration
The paper formulates confidence-based time allocation as a resource-constrained optimization problem that gives more time to low-confidence tasks, where anchoring is more harmful. In Experiment 2, confidence-based time with explanation improved accuracy specifically when AI predictions were incorrect and low-confidence, although the experiment could not separate the effects of time allocation and explanation.
- Resource allocation problem: The framework models de-biasing as resource allocation, using time to trade off anchoring reduction against human-AI accuracy under a finite budget.The allocation problem aims to optimize human-AI team performance while accounting for anchoring bias and variation in AI accuracy.
- Confidence-based policy: AI confidence is estimated from predicted class probabilities and divided into low-confidence CL and high-confidence CH intervals for assigning time.The study used a threshold of bCi ≥0.75 for CH, with 20 questions in each confidence class.
- Confidence-based policy: The policy is intended to combine human expertise in low-confidence regions with AI decisions in high-confidence regions, producing the idealized “best of both worlds.”The setup assumes human accuracy exceeds AI accuracy in CL and is lower than AI accuracy in CH.
- Confidence-based policy: Under the stated assumptions, the optimal policy assigns tmax to low-confidence tasks and tmin to high-confidence tasks.This allocation gives more time where anchoring to the AI prediction is associated with a lower probability of accuracy.
- Experiment 2 results: Anchoring reduced team accuracy when AI was incorrect, while “Confidence-based time with explanation” significantly improved performance in incorrect, low-confidence trials.The comparison between “Human only” and “Confidence-based time” within low-confidence trials used the same 25-second allocation and remained significant for agreement and accuracy.
- Experiment 2 results: 43.8% accuracy for “Confidence-based time with explanation” exceeded “Random time” at 36.2%, “Constant time” at 36.4%, and “Confidence-based time” at 37.5% on incorrect, low-confidence AI trials.The same condition also showed a significant reduction in anchoring bias in these trials (p = 0.003, t(383) = 2.70).
- Experiment 2 results: The experiment does not separately quantify the effects of time-based allocation and confidence-based explanation.The authors identify this as a limitation of the selected experimental conditions.
6 Discussion
The discussion reports that explanations reduce adverse behavioral effects of confidence-based time allocation and emphasizes the importance of modeling evolving human perceptions and cognitive biases. It also identifies non-expert participants and limited time settings as study boundaries.
- Lessons learned: Confidence-based time allocation can produce lower agreement than human-only and constant-time strategies on some high-confidence trials.The reported pattern suggests participants may distrust the AI when longer times are assigned only to low-confidence trials.
- Lessons learned: Explanations reduce the impression that the AI is worse than it really is under confidence-based time allocation.The authors connect this effect to reduced distrust and emphasize accounting for human behavior in collaborative decisions.
- Future directions: Dynamic cognitive models could represent how decision-makers update their perception of the AI as interaction progresses.The discussion identifies sequential learning during human-AI interaction as a direction for more informed collaboration policies.
- Limitations: The experiments used non-expert participants in a non-critical student-assessment task, with training and domain statistics intended to mitigate limited expertise.The authors describe this as a human-grounded evaluation using lay participants as proxies for general behavioral patterns.
- Conclusions: The paper foregrounds cognitive biases, mathematically models their effects, and validates time as a strategy for mitigating anchoring bias.Future work includes studying additional biases such as confirmation bias and the weak evidence effect.
A Additional details of optimal resource allocation
The appendix explains why confidence-based time allocation can outperform constant and random allocation under Assumption 1. The argument relies on confidence-conditioned accuracy improving with extra time when AI confidence is low and worsening when confidence is high.
- Reasoning for Assumption 1: For low-confidence trials, additional time is expected to increase accuracy because disagreement with an incorrect AI anchor becomes more likely.The derivation states that the disagreement term increases or stays the same with time and can dominate when low confidence correlates with low AI accuracy.
- Reasoning for Assumption 1: For high-confidence trials, additional time may decrease accuracy because agreement with a correct AI prediction tends to dominate.The appendix describes the high-confidence conditional accuracy as decreasing with time under the stated reasoning.
- Optimality result: Under Assumption 1, the confidence-based time allocation policy has accuracy greater than or equal to constant and random time allocation.The corollary considers N trials and total time T.
- Proof: The policy’s expected accuracy is the probability-weighted sum of conditional accuracies for low- and high-confidence trials at their assigned times.The low- and high-confidence groups receive times tL and tH, respectively.
- Proof: The proof compares confidence-based allocation with constant and random allocation by expanding each policy’s expected per-round accuracy.It concludes Eπ[Ri] ≥ Erand[Ri] under the assumptions.
- Assumption 1: Assumption 1 requires conditional accuracy under assigned low- and high-confidence times to be at least as large as under constant time T.These inequalities support the comparison with constant allocation.