Source-linked AI summary
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld
TL;DR
The paper examines whether AI explanations can produce complementary human-AI decision performance when human and AI accuracy are comparable, addressing evidence drawn mainly from AI-superior settings. Through mixed-method studies across three datasets and multiple explanation conditions, it finds that teams can be complementary but explanations do not significantly improve accuracy and can increase reliance on incorrect recommendations.
Problem
Prior studies report explanation benefits mainly when autonomous AI outperforms both humans and teams, leaving unclear whether explanations enable complementary performance.
Method
The authors conduct large-scale mixed-method experiments with human- and AI-comparable accuracy across three tasks, testing multiple explanation sources and strategies.
Results
Human-AI teams showed complementary performance, but no explanation condition produced significantly higher accuracy than confidence-based assistance, and explanations increased reliance even when AI recommendations were incorrect.
Takeaways & Limitations
Effective explanations must encourage appropriate reliance rather than simply increasing acceptance of AI recommendations.
Takeaways & Limitations
The studies evaluate decision accuracy on crowdsourced text-classification and question-answering tasks, so results may not generalize to high-stakes expert domains or other performance metrics.
Abstract
from arXiv · showhide
Many researchers motivate explainable AI with studies showing that human-AI team performance on decision-making tasks improves when the AI explains its recommendations. However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. Can explanations help lead to complementary performance, where team accuracy is higher than either the human or the AI working solo? We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions). While we observed complementary improvements from AI augmentation, they were not increased by explanations. Rather, explanations increased the chance that humans will accept the AI's recommendation, regardless of its correctness. Our result poses new challenges for human-centered AI: Can we develop explanatory approaches that encourage appropriate trust in AI, and therefore help generate (or improve) complementary performance?
1 INTRODUCTION
The paper asks whether explanations improve complementary human-AI performance when human and AI accuracy are comparable, rather than merely bringing teams closer to a superior AI. Across studies, teams achieved complementarity, but explanations did not improve performance over confidence-only information and increased reliance even on incorrect recommendations.
- The authors control experiments so AI accuracy is comparable to human accuracy, creating more opportunity for teammates to correct each other’s mistakes.They argue that blindly following the AI is unlikely to be optimal in this setting.
- Adaptive Explanation reveals the paper’s attempt to reduce trust when AI confidence is low by explaining alternative classes in those cases.It did not significantly improve final team performance, though it showed suggestive movement in agreement between AI predictions and human decisions.
- Qualitative analysis identifies expertise differences and explanation displays as factors that may affect collaboration strategies and complementary performance.
- Prior explainable-AI studies typically examined settings where AI accuracy exceeded both human and team accuracy, limiting evidence about complementary performance.
- Across three tasks and 1626 users, every human-AI teaming condition showed complementary performance.
- Explanations produced no significant team-performance increase over simply showing the AI’s confidence.Explanations improved accuracy when the AI was correct but decreased it when the AI erred, yielding minimal net change.
2 BACKGROUND AND RELATED WORK
The background literature reports team-accuracy gains from explanations, but these gains generally occur when autonomous AI would outperform the team. This motivates testing whether explanations can support genuine complementarity when human and AI capabilities are closer.
- Explanations have been studied for communicating predictions, teaching, augmenting analysis or creativity, satisfying legal imperatives, and supporting trust or debugging.
- Prior decision-making studies evaluate end-to-end human-AI team accuracy, with complementary performance defined as the team outperforming both solo human and AI.
- Across reviewed examples, explanations improved team accuracy, but autonomous models still achieved better performance than the teams.
- The authors identify task design as one possible reason prior work lacked complementarity: highly superior AI leaves little room for useful human correction.
3 SETUP AND PILOT STUDIES
The study enlarges the opportunity for complementarity by matching AI and human accuracy, then evaluates local explanations across text classification and question answering. Pilot studies find no aggregate accuracy benefit from explanations, while showing increased reliance on incorrect recommendations and motivating adaptive and expert-generated variants.
- Study design: The experimental design matches AI accuracy to an average human’s and evaluates multiple explanation styles across several domains.
- Study design: The studies focus on local explanations that explain each individual model recommendation rather than the model globally.
- Tasks: The tasks include sentiment analysis of beer and book reviews and LSAT logical-reasoning questions, selected for crowdsourcing and differing difficulty.
- Explanation formats: Sentiment explanations use inline saliency highlights, whereas LSAT explanations use narrative justifications because logical constructs are not readily conveyed by highlighting alone.
- Pilot conditions: The pilot compared confidence-only assistance with explanations of the predicted class and top two classes.
- Pilot findings: Explanations increased reliance on recommendations when they were incorrect; predicted-class explanations helped on correct recommendations but hurt on incorrect ones.
- Follow-up strategies: Adaptive explanations switch between one- and two-class explanations based on confidence, while expert-generated explanations were added to provide an upper bound on explanation quality.
4 FINAL STUDY
The final study tested explanation strategies across sentiment-analysis and LSAT tasks using controlled human-AI team conditions. The design compared explanations with confidence-only assistance while adapting AI accuracy and explanation sources to each task.
- The study tested hypotheses about predicted-class explanations, confidence-only assistance, and Adaptive explanations across sentiment-analysis and LSAT tasks.
- The study included unassisted humans and a Team (Conf) baseline showing the AI recommendation and confidence without explanations.The confidence-only condition was intended as a stronger baseline than displaying the recommendation alone.
- The explanation interface used inline highlights aligned with sentiment labels, while Adaptive explanations varied whether the predicted class alone or both classes were explained.The LSAT interface showed explanations for the top-2 predictions, and confidence scores were calibrated for the sentiment models.
- Sentiment datasets used RoBERTa classifiers fine-tuned on prepared training data, with hyperparameters selected on validation data.Beer and Amzbook labels were binarized, class imbalance was addressed by oversampling, and the data were split into training, validation, and test sets.
- For Beer and Amzbook, 50 study examples were selected so AI accuracy was 84%, comparable to human accuracy measured at 87% and 85%, respectively.The examples were unambiguous and balanced for false positives and false negatives.
- The LSAT study used a RoBERTa model on ReClor and 20 questions, achieving 65% AI accuracy compared with 67% human accuracy on the selected examples.Expert explanations were used because automated explanations for LSAT were not feasible; questions were chosen to avoid easy online search and participant fatigue.
5 RESULTS
AI augmentation achieved complementary performance across tasks, but explanations did not significantly improve team accuracy over showing confidence. Explanations instead altered reliance: they helped when AI recommendations were correct but could hurt when AI was wrong, while adaptive explanations changed agreement without significantly improving final accuracy.
- Team (Conf) achieved complementary performance across tasks, including Beer accuracy of 0.89 ± 0.05 versus AI accuracy of 0.84 and unassisted human accuracy of 0.82 ± 0.09.
- Explain-Top-1 and Explain-Top-2 produced no significant performance difference on Beer, Amzbook, or LSAT.The reported comparisons were Beer z=0.85, p=.40; Amzbook z=0.81, p=.42; and LSAT z=0.42, p=.68.
- Explanations did not significantly improve performance over confidence scores across Beer, Amzbook, and LSAT.For Beer, accuracies were 0.89 ± 0.05 with confidence and 0.88 ± 0.06 with Explain-Top-1 AI.
- Adaptive explanations did not significantly improve final team performance, although they encouraged greater agreement when AI was confident and less agreement when it was uncertain.The agreement shift did not reliably translate into correct decisions, especially when users had to solve uncertain cases themselves.
- When AI recommendations were correct, Explain-Top-1 improved accuracy, but when AI was incorrect, it worsened accuracy; adaptive explanations showed suggestive compensating effects.In Beer, Adaptive improved over Explain-Top-1 when AI was incorrect and over Explain-Top-2 when AI was correct, with smaller effects elsewhere.
- Team benefits varied by domain because AI expertise complemented human expertise more strongly in Beer and LSAT than in Amzbook.Amzbook showed less complementary potential, and Explain-Top-1 improved performance by 5% over confidence when AI was incorrect, unlike Beer and LSAT.
- Higher perceived explanation usefulness did not translate into better team performance.Expert-generated adaptive explanations received higher usefulness ratings than AI-generated explanations, but the preference was not reflected in performance.
- Participants generally used AI recommendations as partial guidance or a double-check, but ignored AI more often when its expertise did not supplement theirs.More than 70% reported partial consideration across datasets; AI was mostly ignored by 11% of LSAT, 23% of Beer, and 30% of Amzbook participants.
6 DISCUSSION & FUTURE DIRECTIONS
The paper identifies scope limits and design tensions for explanations in human-AI teams. It argues that future work should target informative, appropriately timed coordination strategies that balance accuracy, speed, and human independence.
- Limitations: The study evaluated complementarity only through decision accuracy, leaving benefits such as speed, effort, trust, and engagement for future research.
- Limitations: The findings may not generalize beyond crowdsourced text classification and question-answering tasks to high-stakes domains with expert users.
- Limitations: The study tested only keyword highlighting and natural-language arguments, so alternative explanation approaches may benefit team performance more.
- Implications: Explanations sometimes increased blind trust, creating ethical concerns when humans must remain responsible for decisions.
- Implications: Adaptive explanations did not significantly improve final team accuracy because users still had to detect high-confidence mistakes and solve uncertain cases.
- Future Directions: Explanations should inform users rather than merely persuade them, potentially by communicating doubts or adapting to human-AI agreement patterns.
- Future Directions: Complementary partners require more than comparable accuracy: minimally overlapping human and AI errors create greater opportunities for mutual correction.
- Future Directions: Presenting AI recommendations before independent reasoning risks anchoring users, while delaying recommendations may improve reflection but reduce speed.
7 CONCLUSIONS
The paper tests whether explanations improve complementary human-AI performance when AI and human accuracy are comparable. Across large-scale studies, teams were complementary, but explanations did not outperform confidence displays because they helped on correct AI decisions while hurting performance on errors.
- Prior explanation studies improved team accuracy only when the AI alone outperformed both the human and the best human-AI team.
- The authors conducted large-scale experiments with more than 1,500 participants using questions selected to give AI systems accuracy comparable to humans.
- All human-AI teams showed complementary performance, but no explanation condition significantly exceeded the baseline that displayed AI confidence.
- Explanations improved team performance when the AI was correct but decreased accuracy when the AI was wrong, leaving minimal net improvement.
- Future work should develop coordination and explanation methods that increase understanding and complementary performance beyond communicating AI confidence.