Source-linked AI summary
To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
Zana Buçinca, Maja Barbara Malaya, Krzysztof Z. Gajos
TL;DR
People often overrely on incorrect AI recommendations, and standard explanations do not reliably prevent this behavior. This paper tested three cognitive forcing interventions against explainable AI baselines and no AI in an experiment with 199 participants. Cognitive forcing reduced overreliance, but the most effective designs were less acceptable and benefited high-Need-for-Cognition participants more.
Problem
People frequently follow incorrect AI recommendations even when they would make better decisions unaided, while standard explanations have not substantially reduced overreliance.
Method
The authors compared three cognitive forcing interventions with two simple explainable AI approaches and a no-AI baseline in a 199-participant experiment.
Results
Cognitive forcing significantly reduced overreliance relative to simple explainable AI, but the most effective designs were more difficult, less preferred, and less trusted, with larger benefits for high-NFC participants.
Takeaways & Limitations
Effective AI assistance depends not only on explanation content but also on interaction design and users’ cognitive motivation.
Takeaways & Limitations
The study used a single non-critical decision-making task, so additional work is needed to determine whether the effects generalize across domains and settings.
Abstract
from arXiv · showhide
People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.
1 INTRODUCTION
Human–AI teams often underperform the AI because people overrely on incorrect recommendations. The paper proposes cognitive forcing interventions that promote analytical engagement with explanations, while examining acceptability and unequal benefits.
- Human–AI teams can underperform AI systems because people follow incorrect AI suggestions despite better unaided judgments.
- Explanations may fail because people use general heuristics about AI competence rather than analytically evaluating each recommendation.
- Cognitive forcing functions disrupt heuristic reasoning at decision time to encourage analytical engagement with AI recommendations and explanations.
- 199 participants were assigned to three cognitive forcing designs, two simple explainable AI approaches, or a no-AI baseline.
- Cognitive forcing reduced overreliance compared with simple explainable AI, but the strongest reductions occurred in less preferred, less trusted, and more difficult conditions.
- Cognitive forcing benefited participants with high Need for Cognition more than those with low Need for Cognition.
2 RELATED WORK
The related work frames overreliance as a consequence of fast heuristic decision-making and reviews interventions designed to elicit deliberation. It also highlights a recurring preference–performance tension for cognitively demanding interfaces.
- Dual-process theory distinguishes fast, heuristic System 1 thinking from slower, deliberative System 2 thinking, with System 1 vulnerable to predictable errors.
- Human–AI teams may systematically underperform AI alone when AI accuracy exceeds unaided human accuracy.
- Overtrust leads people to follow incorrect AI predictions even when they would have made better decisions independently.
- Cognitive forcing functions disrupt quick heuristic reasoning at decision time and include strategies such as checklists and diagnostic time-outs.
- People often prefer simpler interfaces even when more complex designs improve comprehension, recall, learning, or performance.
3 EXPERIMENT
The experiment tested whether cognitive forcing improves human–AI decision-making relative to explainable AI baselines and whether greater cognitive effort creates an acceptability trade-off. Participants completed a nutrition substitution task using meal images.
- Design and hypotheses: The experiment compared three cognitive forcing interventions, two simple explainable AI conditions, and a no-AI baseline.
- Design and hypotheses: H1a predicted better human–AI team performance with cognitive forcing when the AI’s top prediction was incorrect.
- Design and hypotheses: H1b predicted better overall human–AI team performance with cognitive forcing than with simple explainable AI.
- Design and hypotheses: H2 predicted a negative relationship between interface acceptability and team performance when the AI prediction was incorrect.
- Task: Participants replaced the highest-carbohydrate meal ingredient with a lower-carbohydrate ingredient chosen for similar flavor.
3.2 Conditions
The study used six conditions spanning no AI, simple explanations, uncertainty information, and three ways of forcing engagement. These conditions varied when participants saw, requested, or could revise AI recommendations.
- Overview: The six conditions varied whether and how a simulated AI assisted participants’ decisions.
- Baselines: The no-AI condition provided only the meal image and menus for selecting an ingredient to remove and replace.
- Baselines: The explanation condition immediately presented recognized ingredients, four substitutions, and feature-based estimates of carbohydrate reduction and flavor similarity.
- Baselines: The uncertainty condition added an AI-confidence prompt to the explanation interface when the AI was uncertain.
- Cognitive forcing functions: The on-demand condition hid the AI suggestion until participants clicked a button to view it and its explanation.
- Cognitive forcing functions: The update condition required an initial unaided decision before showing the AI suggestion and explanation for possible revision.
- Cognitive forcing functions: The wait condition delayed the AI suggestion and explanation for 30 seconds, allowing participants to form a hypothesis before evaluating the AI.
3.3 The Simulated AI
The study used a simulated AI with controlled visual-recognition errors to recommend lower-carbohydrate ingredient replacements. Its explanations ranked alternatives by both carbohydrate reduction and flavor similarity.
- The simulated AI recognized the meal ingredient with the highest carbohydrate impact with 75% accuracy.
- The AI suggested replacing the highest-carbohydrate ingredient and explained four ranked alternatives using carbohydrate reduction and flavor similarity estimates.
- Replacement candidates came from ingredient-specific lookup tables built from flavor-molecule similarities across 936 ingredients.
- Candidates were ranked by the harmonic mean of flavor similarity and carbohydrate reduction to optimize both properties.
3.4 Procedure
Participants completed an online MTurk task containing two blocks of questions, with experimental conditions assigned across blocks and practice questions introducing each block.
- Participants completed 26 questions divided into two blocks of 13 questions.
- Each block began with a practice question to familiarize participants with the task.
- Participants were randomly shown a different condition in each block from nine available conditions.
3.5 Participants
The study recruited U.S.-based adults through Amazon Mechanical Turk and retained 199 participants after excluding invalid or exploratory-condition records.
- 260 adults residing in the United States were recruited through Amazon Mechanical Turk in three batches.
- 49 participants were excluded after selecting off-plate ingredients in more than 55% of questions and showing zero task accuracy.
- An additional 12 participants assigned only to unreported exploratory conditions were excluded, leaving 199 participants for analysis.
3.6 Design and Analysis
The study used a mixed between- and within-subject design and analyzed objective and subjective outcomes across conditions with mixed-effects models and corrected pairwise tests.
- Design: Each participant interacted with two of nine conditions in a mixed between- and within-subject design.
- Measures: Performance measures included overall performance, carb-source detection, carbohydrate reduction, flavor similarity, and overreliance.
- Measures: Subjective measures assessed preference, trust, mental demand, and system complexity using five-point ratings.
- Comparisons: Incorrect-prediction performance was compared across cognitive-forcing and simple-XAI conditions against performance under no AI assistance.
- Analysis: Mixed-effects models treated condition as fixed and participant as random, with ANOVA, corrected t-tests, marginal means, and effect sizes used for analysis.
4 RESULTS
Cognitive forcing functions improved performance on incorrect AI predictions relative to simple explainable AI, while no-AI participants performed better in that setting. Subjective ratings and their associations with performance revealed a trade-off between cognitive effort, acceptability, and reliance.
- Objective measures: Both AI-assisted categories improved all four performance measures over no AI when correct and incorrect predictions were combined, with no significant difference between cognitive forcing and simple explainable AI.There were also no significant differences among conditions within either category.
- Objective measures: On incorrect AI predictions, cognitive forcing functions improved overall performance, carb source detection, carb reduction, and flavor similarity significantly more than simple explainable AI.However, no-AI participants performed significantly better than both AI-assisted categories on these incorrect-prediction instances.
- Objective measures: χ2(2, N=663) = 13.30, p=.0013 for overall decisions: cognitive forcing participants made more correct decisions than simple explainable AI participants.They also overrelied less, although that difference was not significant; human errors did not differ significantly.
- Objective measures: χ2(2, N=663) = 44.35, p≪.0001 for carb-source decisions: cognitive forcing participants overrelied significantly less and made significantly more correct decisions than simple explainable AI participants.Human errors did not differ significantly between categories.
- Subjective measures: Participants preferred AI-assisted conditions and found no AI more mentally demanding, while simple explainable AI was perceived as less complex than cognitive forcing or no AI.Trust was higher with simple explainable AI than cognitive forcing, but not significantly so; within-category conditions did not differ significantly.
- Subjective measures vs. objective measures: Trust and preference were negatively correlated with performance on incorrect predictions, while trust was positively correlated with carb-source overreliance for incorrect predictions.Mental demand and system complexity were negatively correlated with performance on correct predictions; mental demand was positively correlated with carb-source performance on incorrect predictions.
5 ETHICAL CONSIDERATIONS: INDIVIDUAL DIFFERENCES IN COGNITIVE MOTIVATION
The paper audits whether cognitive forcing functions create intervention-generated inequalities by comparing outcomes for participants with different Need for Cognition levels. High-NFC participants generally performed better and benefited more from cognitive forcing, while also rating simple explainable AI more favorably.
- Audit rationale: Intervention-generated inequalities arise when an intervention’s benefits disproportionately accrue to an already privileged group, increasing gaps between groups.The audit therefore examined whether cognitive forcing functions were equally effective across NFC levels.
- Individual differences: Need for Cognition is a stable personality trait reflecting how much a person enjoys engaging in cognitively demanding activities.The study treated intrinsic cognitive motivation as the relevant individual difference for this audit.
- Background: Prior evidence describes high-NFC participants as seeking more information and processing it more deeply, whereas low-NFC participants more often use cognitive shortcuts.High-NFC participants also tend to adopt complex productivity features and methods requiring greater cognitive exertion.
- Objective measures by NFC: High-NFC participants had higher overall performance (M=0.44) than low-NFC participants (M=0.23), and higher carb-source detection performance (M=0.67 versus M=0.44).Both differences were statistically significant: overall performance, F1,172.5=29.13, p≪.0001; carb-source detection, F1,172.2=32.11, p≪.0001.
- Objective measures by NFC: On incorrect AI predictions, cognitive forcing significantly improved high-NFC participants’ overall and carb-source performance compared with simple explainable AI.For low-NFC participants, the category difference was significant for carb-source detection but not overall performance.
- Subjective measures by NFC: Low-NFC participants reported greater mental demand (M=3.10) than high-NFC participants (M=2.62), and greater system complexity (M=3.04) than high-NFC participants.These comparisons combined cognitive forcing and simple explainable AI conditions.
- Subjective measures by NFC: High-NFC participants trusted and preferred simple explainable AI more than cognitive forcing and perceived it as significantly less complex.This pattern indicates that the intervention’s stronger objective effects coincided with less favorable subjective ratings for high-NFC participants.
6 DISCUSSION
Cognitive forcing reduced overreliance on AI relative to simple explainable AI, but did not improve overall team performance and carried usability and inequality trade-offs.
- Cognitive forcing led participants to disregard incorrect AI suggestions and make optimal choices significantly more often than with simple explainable AI.
- Human+AI teams continued to perform worse than the AI model alone, indicating that cognitive forcing reduced but did not eliminate overreliance.
- There was no significant performance difference between cognitive forcing and simple explainable AI approaches.
- Participants performed best with conditions they preferred and trusted least, revealing a trade-off between intervention effectiveness and subjective acceptability.
- Cognitive forcing improved error detection in an actual decision-making task, whereas proxy-task evaluations may produce more optimistic human-performance results.
- Cognitive forcing benefits held mostly for high-NFC participants, while high-NFC participants trusted and preferred these interventions less than simple explainable AI approaches.
- The study was conducted in a single non-critical decision-making task, so generalization across domains and settings requires additional work.
- Improved human+AI performance came at the cost of reduced perceived usability, potentially limiting adoption and motivating adaptive deployment strategies.
7 CONCLUSION
The study examines cognitive forcing functions as interventions against human overreliance on AI in collaborative decision-making. It finds reduced overreliance alongside lower trust and preference, with disproportionate benefits for participants high in Need for Cognition.
- The study investigates cognitive forcing functions as interventions for reducing human overreliance on AI in collaborative human+AI decision-making.
- Cognitive forcing functions significantly reduced overreliance compared with simply presenting explanations for AI recommendations.
- Participants preferred and trusted less mentally demanding systems, even though they performed poorly with them.
- Cognitive forcing functions disproportionately benefited participants with high Need for Cognition.
- Explainable AI research should account for cognitive motivation and develop interventions that elicit analytical engagement with explanations when necessary.