Source-linked AI summary
As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making
Jingshu Li, Yitian Yang, Q. Vera Liao, Junti Zhang, Yi-Chieh Lee
TL;DR
Human-AI collaboration may depend on calibrated confidence, but AI confidence could shape users’ self-confidence and its calibration. Using a randomized behavioral experiment, the paper finds that human self-confidence aligns with AI confidence, persists after AI involvement ends, and is less aligned when real-time correctness feedback is present.
Problem
AI confidence may influence human self-confidence and calibration, complicating efforts to achieve complementary human-AI collaboration through calibrated confidence.
Method
The authors conducted an online randomized behavioral experiment with 270 participants completing individual and human-AI income-prediction tasks across multiple stages and feedback conditions.
Results
Human self-confidence aligned with AI confidence during collaboration, persisted afterward, affected self-confidence calibration, and showed less alignment with real-time correctness feedback.
Takeaways & Limitations
Human self-confidence is not independent of AI confidence, so AI designers and users should account for this relationship in human-AI decision making.
Takeaways & Limitations
The study used low-risk, relatively simple income-prediction tasks, so whether the findings generalize to high-risk or specialized tasks remains unknown.
Abstract
from arXiv · showhide
Complementary collaboration between humans and AI is essential for human-AI decision making. One feasible approach to achieving it involves accounting for the calibrated confidence levels of both AI and users. However, this process would likely be made more difficult by the fact that AI confidence may influence users' self-confidence and its calibration. To explore these dynamics, we conducted a randomized behavioral experiment. Our results indicate that in human-AI decision-making, users' self-confidence aligns with AI confidence and such alignment can persist even after AI ceases to be involved. This alignment then affects users' self-confidence calibration. We also found the presence of real-time correctness feedback of decisions reduced the degree of alignment. These findings suggest that users' self-confidence is not independent of AI confidence, which practitioners aiming to achieve better human-AI collaboration need to be aware of. We call for research focusing on the alignment of human cognition and behavior with AI.
1 INTRODUCTION
The paper examines whether AI confidence influences human self-confidence during and after human-AI decision making, and whether this alignment affects calibration and collaboration outcomes. A randomized behavioral experiment finds alignment, persistence after collaboration, and reduced alignment with real-time feedback.
- AI confidence may influence and align with human self-confidence, a possibility not explored by prior human-AI confidence research.
- Misaligned confidence can impair calibration by changing human self-confidence without changing decision-making capabilities.The paper links poorer calibration to inappropriate reliance on humans or AI and reduced decision-making efficacy.
- The randomized behavioral experiment used 270 participants across baseline, human-AI collaboration, and post-collaboration decision-making stages.Each stage included 40 income-prediction tasks; the study also varied collaboration conditions and real-time feedback.
- Participants’ self-confidence aligned with AI confidence during collaboration and remained aligned during later individual decisions.Real-time correctness feedback reduced the degree of alignment.
- Human uncertainty is not independent of AI-expressed uncertainty, creating design implications for human-AI decision making.The paper calls for awareness of this relationship when designing and using AI systems.
2 RELATED WORK
Related work frames human-AI decision making around three collaboration paradigms and confidence as a basis for complementary collaboration. It also reviews confidence calibration, human self-confidence, and confidence alignment as foundations for the paper’s investigation.
- Human-AI decision-making paradigms: Human-AI decision making commonly treats AI as an advisor, peer collaborator, or decision-maker supervised by humans.These paradigms differ in how humans and AI contribute to joint decisions.
- Confidence and complementary collaboration: AI uncertainty can be expressed through confidence levels, while human uncertainty can be expressed through numerical self-confidence.The study focuses on numerical confidence levels for both AI and participants.
- Confidence and complementary collaboration: Complementary collaboration uses relative human and AI confidence to support appropriate reliance on either human judgment or AI predictions.The intended decision rule favors the party with the higher confidence level.
- Confidence and complementary collaboration: Both AI confidence and human self-confidence can be miscalibrated relative to actual decision accuracy.Machine-learning models may be overconfident or underconfident, while human confidence is also subject to calibration challenges.
- Confidence alignment: Human decision-making groups show confidence alignment, with members’ self-confidence converging toward one another.Prior work connects this alignment with imitation of peers’ confidence and behavior.
3 RESEARCH QUESTIONS
The research questions ask whether AI confidence aligns with human self-confidence, how collaboration and feedback shape that alignment, and how alignment affects calibration and reliance. The paper also examines confidence in final joint decisions.
- The paper proposes that human self-confidence may align with AI confidence, extending confidence-alignment research from human groups to human-AI decision making.The proposal is supported by research showing that people can imitate AI and robots, alongside the CASA theory of social responses to computers.
- RQ1 asks whether AI confidence affects human self-confidence and, if so, the degree of alignment.
- The study asks how different human-AI collaboration paradigms and real-time correctness feedback influence confidence alignment.
- RQ2 examines whether confidence alignment changes human self-confidence calibration and its consequences for reliance and decision-making efficacy.The motivation links changed confidence, without changed accuracy, to altered confidence-accuracy correspondence.
- The study also investigates whether AI confidence aligns with human confidence in final joint decisions when humans remain final decision-makers.
4 METHOD
The study used an online randomized behavioral experiment in which participants completed income-prediction tasks independently, with AI collaboration, and independently again. The mixed design varied task stage, real-time correctness feedback, and human-AI collaboration conditions while recording decisions and self-confidence.
- Participants and task: The experiment recruited participants for an online randomized behavioral study of human-AI decision making using income-prediction tasks.Participants predicted whether annual income exceeded $50,000 from demographic and employment information using data derived from the Adult Income dataset.
- Participants and task: The task used a Random Forest model trained on two-thirds of the dataset, with conditional probabilities serving as AI confidence levels.The remaining one-third supplied the participant tasks.
- Procedure and conditions: The procedure comprised a tutorial and three stages totaling 120 questions, with randomized question order and a mixed design involving within-subject task stages and between-subject experimental factors.Stage 1 and stage 3 were independent task stages, while stage 2 involved collaboration with AI.
- Procedure and conditions: In stage 2, participants collaborated with AI after first reporting their own prediction and self-confidence, with procedures differing across advisor, peer-collaborator, and decision-maker paradigms.The AI’s prediction and confidence were displayed after participants’ initial reports; final decisions followed the paradigm-specific rules.
- Procedure and conditions: Stage 3 repeated independent prediction without AI assistance to assess whether effects of AI confidence persisted after collaboration.The stage used questions different from those in stages 1 and 2 and retained the stage 1 decision and feedback settings.
- Interface and measures: The interface collected participants’ decisions and self-confidence, displayed AI and final decisions when applicable, and provided real-time correctness feedback under designated conditions.Participants’ self-confidence was represented by the average confidence reported for their decisions within each stage, and advisor over-reliance was measured from task-level correctness patterns.
5 RESULTS
Participants’ self-confidence aligned with AI confidence during collaboration and remained partly aligned afterward, with alignment varying by feedback condition and participant type. This alignment was associated with changes in self-confidence calibration and confidence in joint decisions.
- 5.1 Participants’ Self-confidence Aligned with AI Confidence (RQ1): Participants’ self-confidence aligned with AI confidence during collaboration, and this alignment persisted, although weakened, after collaboration ended.The absolute confidence difference was lower at stage 2 than stage 1 and remained lower at stage 3 than stage 1.
- 5.1 Participants’ Self-confidence Aligned with AI Confidence (RQ1): Real-time feedback moderated alignment: without feedback, stage 2 and stage 3 did not significantly differ, whereas with feedback, stage 1 and stage 3 did not significantly differ.The interaction between task stage and real-time feedback was significant, F(1.875, 495.024) = 3.855, p = 0.024, Partial η2 = 0.014.
- 5.1 Participants’ Self-confidence Aligned with AI Confidence (RQ1): No significant main or interaction effects of human-AI decision-making paradigm were observed, and real-time feedback alone had no significant between-subject effect.The paradigm effect was F(2, 264) = 0.514, p = 0.599, while the feedback effect was F(1, 264) = 0.339, p = 0.561.
- 5.1 Participants’ Self-confidence Aligned with AI Confidence (RQ1): Alignment was not significantly correlated with participants’ accuracy at stage 2 or stage 3.The correlations between accuracy and absolute confidence difference were r = −0.071 at stage 2 and r = −0.058 at stage 3.
- 5.1.1 The Exclusion of Irrelevant Causes.: Alignment involved both upward and downward self-confidence changes rather than only confidence increases caused by AI.Some participants initially more confident than AI decreased their confidence, while some initially less confident than AI increased theirs.
- 5.2 The Alignment Changed Participants’ Self-confidence Calibration and Affected Human-AI Decision Making Efficacy (RQ2): Most participants were type B, with 201 at stage 2 and 194 at stage 3; no type C participants were observed.Type A counts were 25 at stage 2 and 29 at stage 3, while type D counts were 44 and 47, respectively.
- 5.2 The Alignment Changed Participants’ Self-confidence Calibration and Affected Human-AI Decision Making Efficacy (RQ2): Alignment changed self-confidence calibration differently by participant type: it degraded calibration for type B participants but could improve it for type A and type D participants.Type B participants were overconfident but less confident than AI; type A were overconfident and more confident than AI; type D were underconfident and less confident than AI.
- 5.2 The Alignment Changed Participants’ Self-confidence Calibration and Affected Human-AI Decision Making Efficacy (RQ2): Self-confidence accuracy correlations were significant at stage 1 but not stages 2 or 3, indicating poorer overall calibration after AI involvement.Stage 1 showed r = 0.172, p = 0.005; stage 2 showed r = 0.058, p = 0.342; stage 3 showed r = 0.025, p = 0.688.
6 DISCUSSION
Human self-confidence aligns with AI confidence, persists after collaboration, and changes calibration; feedback weakens alignment. These effects shape reliance, joint decisions, theoretical understanding, and system-design priorities, while remaining bounded by task and model limitations.
- Results: Human self-confidence aligns with AI confidence and persists in individual tasks after human-AI collaboration ends.The alignment was also observed when participants merely observed AI decisions and confidence.
- Results: Real-time correctness feedback weakens alignment by helping humans adjust confidence toward observed accuracy.The experiment found a discrepancy between most participants’ accuracy and AI confidence levels.
- Calibration: Alignment changes self-confidence calibration without changing accuracy, improving or worsening calibration according to the relationships among confidence and accuracy levels.Most participants experienced worsened calibration because their confidence moved toward AI confidence and away from their accuracy.
- Consequences: Worsened calibration in AI-assisted decisions led people to adopt AI’s correct predictions less often, producing more errors and reducing complementary performance.The effect appeared as more decisions adopting the incorrect decision when only the human or AI was correct.
- Joint Decisions: Confidence in joint decisions aligns more closely with AI confidence than initial self-confidence, especially when final decisions follow AI recommendations.The authors connect this pattern to people adopting some of AI’s confidence when considering its advice.
- Implications: The study extends confidence-alignment theory to human-AI interaction and suggests systems should account for alignment, calibration, reliance, and users’ abilities and confidence.The authors also note that alignment may improve joint decisions when human and AI capabilities are comparable.
- Limitations: Generalization remains limited because the experiment used low-risk income prediction, fixed numerical AI confidence, and a fixed AI accuracy.The authors caution that results may differ for high-risk or specialized tasks, other uncertainty displays, and other experimental groups.
7 CONCLUSIONS
The study finds that human self-confidence commonly aligns with AI confidence, persists after AI involvement ends, and can impair confidence calibration. It frames AI confidence as an influence on human decision-making dynamics, not merely an indicator of AI performance.
- Human self-confidence commonly aligns with AI confidence during human-AI decision making.
- This confidence alignment can persist even after AI ceases to be involved.
- For most users, alignment with AI confidence could impair self-confidence calibration.
- Poor self-confidence calibration is closely related to inappropriate reliance and low human-AI decision-making efficacy.