PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems

Gaole He, Lucie Kuiper, Ujwal Gadiraju

arXiv:2301.11333v1cs.HCcs.AIcs.CL

TL;DR

This paper asks whether the Dunning-Kruger Effect (DKE) shapes appropriate reliance on AI, a key concern for complementary human-AI performance. It tests the relationship in a logical-reasoning decision task and evaluates a tutorial intervention. Overestimation was associated with under-reliance, while tutorial effects varied with initial self-assessment.

  • Problem

    Whether DKE affects reliance on AI had not been addressed, despite under-reliance and over-reliance both posing challenges to complementary human-AI performance.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    The study operationalized DKE through overestimated self-assessment and examined reliance in logical reasoning, including whether a tutorial could mitigate the effect.

    Source: §1 · p. 2, §3.1 · p. 5, §7 · p. 16

  • Results

    Participants who overestimated their performance tended to under-rely on AI; tutorials mitigated DKE but could harm appropriate reliance among initially underestimated participants.

    Source: §7 · pp. 15–16

  • Takeaways & Limitations

    More comprehensive, insightful, and personalized tutorials are needed to address DKE while supporting appropriate reliance.

    Source: §7 · p. 16

    Task assessment may be biased by other factors, and the researchers could not be certain that the task batches were perceived as equally difficult.

    Source: §6.3 · p. 15

Abstract

from arXiv · show

The dazzling promises of AI systems to augment humans in various tasks hinge on whether humans can appropriately rely on them. Recent research has shown that appropriate reliance is the key to achieving complementary team performance in AI-assisted decision making. This paper addresses an under-explored problem of whether the Dunning-Kruger Effect (DKE) among people can hinder their appropriate reliance on AI systems. DKE is a metacognitive bias due to which less-competent individuals overestimate their own skill and performance. Through an empirical study (N = 249), we explored the impact of DKE on human reliance on an AI system, and whether such effects can be mitigated using a tutorial intervention that reveals the fallibility of AI advice, and exploiting logic units-based explanations to improve user understanding of AI advice. We found that participants who overestimate their performance tend to exhibit under-reliance on AI systems, which hinders optimal team performance. Logic units-based explanations did not help users in either improving the calibration of their competence or facilitating appropriate reliance. While the tutorial intervention was highly effective in helping users calibrate their self-assessment and facilitating appropriate reliance among participants with overestimated self-assessment, we found that it can potentially hurt the appropriate reliance of participants with underestimated self-assessment. Our work has broad implications on the design of methods to tackle user cognitive biases while facilitating appropriate reliance on AI systems. Our findings advance the current understanding of the role of self-assessment in shaping trust and reliance in human-AI decision making. This lays out promising future directions for relevant HCI research in this community.

1 INTRODUCTION

Appropriate reliance—using AI when it is more accurate and not when it is wrong—matters for human-AI collaboration, but how the Dunning-Kruger Effect (DKE) shapes it had not been studied. This work tests DKE’s relationship to reliance and whether tutorials or logic-unit explanations can help.

1 INTRODUCTION

  • The study examines whether overestimating competence shapes AI reliance, addressing a gap in research on how self-assessment affects human-AI decision making.The authors note that no prior work, to their knowledge, had addressed whether DKE affects reliance on AI systems.Source: §1 · pp. 1–2
  • DKE is operationalized as overestimated self-assessment, with bottom-quartile performance analyzed to check whether overestimation indicates the effect.The study uses logical reasoning tasks to explore whether interventions can improve calibration of task skills.Source: §1 · p. 2
  • The tutorial contrasts participants’ incorrect choices with correct answers and explanations, while logic-unit explanations are intended to help interpret AI advice during disagreement.Contrastive feedback explains both why the correct answer is right and why the participant’s choice is wrong.Source: §1 · p. 2
  • Participants with DKE relied less on AI, and under-reliance hindered their average performance; tutorials mitigated DKE but could harm appropriate reliance among participants who initially underestimated themselves.The tutorial improved self-assessment, but its benefits for appropriate reliance and performance were limited overall; the authors suggest more comprehensive, personalized tutorials.Source: §1 · p. 2
  • Logic-unit explanations showed no obvious benefit, while the tutorial improved self-assessments but had statistically non-significant effects on appropriate reliance and performance overall.The effects of the tutorial differed by initial self-assessment: participants who underestimated themselves later showed worse appropriate reliance and performance.Source: §1 · p. 2
  • The relationship between self-assessment and AI reliance may be more complex than a linear relationship and warrants further research.Accurate self-assessment did not necessarily correspond to optimal appropriate reliance.Source: §1 · pp. 2–3

2 BACKGROUND AND RELATED WORK

The related work covers human-AI collaboration, appropriate reliance, competing attitudes toward algorithms, and self-assessment. Across these areas, it motivates studying how user factors and explanations shape reliance on AI.

2.1 Human-AI Collaborative Decision Making

  • Human-AI collaboration research examines user factors and explanations, but explanations that aid understanding do not necessarily improve performance.Prior work reports limited performance benefits from XAI; explanation desiderata include helping people understand AI, recognize uncertainty, and calibrate trust.Source: §2.1 · p. 3

2.2 Empirical Studies on Appropriate Reliance

  • Complementary team performance can fail through under-reliance on AI or over-reliance when people fail to use their own superior judgment.Prior research has studied cognitive forcing functions, tutorials, and explanations to address these reliance patterns; tutorials can educate users about AI weaknesses or domain knowledge.Source: §2.2 · pp. 3–4

2.3 Algorithm Aversion and Algorithm Appreciation

  • People may show algorithm aversion or algorithm appreciation, and expertise, literacy, cognitive load, and presentation can shape their responses to algorithmic advice.For example, users may lose confidence after observing repeated AI mistakes, while the opportunity to modify advice has been reported to overcome aversion.Source: §2.3 · p. 4

2.4 Self-assessment in HCI Studies

  • Self-assessment is used in HCI to measure perceived competence, but people may calibrate abilities poorly; prior studies have examined DKE in reasoning tasks and other settings.Earlier HCI work linked perceived familiarity with lower reliance on an intelligent assistant and found that crowd workers can exhibit DKE on logical reasoning tasks.Source: §2.4 · p. 4

3 METHOD AND HYPOTHESIS

The paper studies whether miscalibrated self-assessment shapes reliance on AI in logical reasoning, and proposes tutorial feedback and logic units-based explanations as interventions. It tests these ideas through task design and hypotheses about reliance and performance.

3.1 Logical Reasoning Task

  • Participants first answered context-based multiple-choice questions unaided, then saw AI advice and could revise their choices.The task used the Reclor dataset and was designed to combine difficult reasoning with the potential benefit and risk of relying on imperfect AI advice.Source: §3.1 · p. 5

3.2 Logic Units-based Explanations

  • Logic units-based explanations highlight the top-k text spans receiving the most attention from other logic units.The spans are selected by summing attention along each column of the self-attention matrix.Source: §3.2 · p. 6

3.3 Proposing a Tutorial Intervention to Help Users Calibrate Their Skills

  • The tutorial provides performance feedback and contrastive explanations showing why correct answers are right and participants’ wrong choices are rejected.The intervention is intended to help participants recalibrate their self-assessment; explanations were manually created for the four tutorial tasks.Source: §3.3 · pp. 6–7

3.4 Pilot Study for Task Selection

  • A pilot study selected two similarly difficult six-question task batches and four tutorial questions, with AI advice correct on two tutorial questions and misleading on two.Difficulty was assessed using responses from six attention-check-passing participants among ten pilot participants.Source: §3.4 · p. 7

3.5 Hypotheses

  • The hypotheses predict lower reliance among overestimators, improved self-assessment and appropriate reliance after feedback, and benefits from combining feedback with meaningful explanations.The predictions are stated in H1–H4, including that awareness of miscalibrated self-assessment will improve calibration and reliance.Source: §3.5 · p. 7

4 STUDY DESIGN

The study uses a 2 × 2 design to examine tutorial feedback and AI explanations, with reliance, decision accuracy, and self-assessment among its measures. Participants completed task phases and questionnaires under one of four conditions.

4.1 Experimental Conditions

  • The 2 × 2 factorial design varied whether participants received a tutorial and whether AI advice included explanations.Tutorial conditions included four tasks with performance feedback and contrastive explanations; XAI conditions highlighted the five most important logic units.Source: §4.1 · pp. 7–8

4.2 Measures and Variables

  • Reliance was measured with Agreement Fraction, Switch Fraction, RAIR, and RSR; accuracy with initial disagreement was also assessed.RAIR and RSR capture appropriate reliance under disagreement, while self-assessment was estimated by comparing participants’ predicted and actual correct answers per batch.Source: §4.2 · p. 8

4.3 Participants

  • The final sample included 249 participants after excluding 65 who failed at least one attention check.The study recruited 314 Prolific participants to allow for exclusions.Source: §4.3 · p. 8

4.4 Procedure

  • Participants completed a pre-task questionnaire, were assigned to a condition, and worked through task phases with post-task assessments of self-assessment and trust.The procedure included 16 tasks and four attention checks; participants in XAI conditions also rated explanation helpfulness.Source: §4.4 · p. 9

5 RESULTS

In a study of human-AI decision making, participants who overestimated their performance relied less on AI and achieved worse appropriate reliance. Tutorials improved self-assessment calibration, but their effects on appropriate reliance differed by initial miscalibration; logic-unit explanations showed no significant benefit.

5.1 Descriptive Statistics

  • Human accuracy averaged 56.9% versus AI accuracy of 66.7%; 57.2% of explanation recipients rated logic-unit explanations slightly or very slightly helpful.Only 28.3% gave positive feedback on explanation helpfulness.Source: §5.1 · p. 10

5.2 Hypothesis Tests

  • Participants who overestimated their performance relied significantly less on AI and achieved lower accuracy, largely because they showed poorer appropriate reliance when disagreeing with AI advice.The reported performance comparison had an exception on RSR.Source: §5.2.1 · p. 10
  • Analyses found no significant logic-unit explanation effects on self-assessment calibration, performance improvement, or reliance, and no significant final-batch effects from tutorial, explanations, or their interaction.Although the explanation group showed better RSR improvement, that difference was not significant.Source: §5.2.2 · p. 11, §5.2.3 · p. 12, §5.2.4 · p. 12
  • After the tutorial, self-assessment calibration improved, but appropriate reliance improved for initial overestimators and worsened for initial underestimators; accurately calibrated participants showed no significant change.The tutorial's overall effect on reliance and performance was not significant.Source: §5.2.3 · p. 11, §5.2.2 · p. 11
  • Changes in self-assessment were negatively associated with changes in performance and appropriate reliance, with significant reliance correlations concentrated among initial overestimators.Among overestimators, calibration accounted for 32.0% of improved RAIR and 12.9% of improved RSR; among underestimators, it accounted for 26.2% of decreased Accuracy and 14.8% of decreased Accuracy-wid.Source: §5.2.3 · p. 12

5.3 Further Analysis On the DKE

  • Low-accuracy participants, used as a proxy for the Dunning-Kruger Effect, showed less reliance and poorer appropriate reliance than high-accuracy participants despite spending longer on tasks.The high-accuracy group averaged about 30 minutes and the low-accuracy group about 34 minutes.Source: §5.3 · pp. 12–13

5.4 Further Analysis of Trust

  • The tutorial did not significantly change subjective trust; propensity to trust correlated weakly with AI trust (ρ=0.22) but not with reliance measures.The reported trust increase was from 2.996 to 3.016, with p=0.952.Source: §5.4 · pp. 13–14

6 DISCUSSION

The discussion finds that self-assessment calibration and appropriate reliance do not always move together: the tutorial helped participants who overestimated themselves but could harm reliance among those who underestimated themselves. It also weighs the explanation results, design implications, and limits of applying findings from a logical-reasoning task more broadly.

6.1 Key Findings

  • Participants who overestimated themselves tended to under-rely on AI, while tutorial-based calibration was associated with improved appropriate reliance for them but worse reliance and performance for initial underestimators.The tutorial significantly improved self-assessment calibration, but its effects on appropriate reliance were non-significant; the underestimator pattern may reflect overestimation or algorithm aversion.Source: §6.1 · p. 14
  • Logic units-based explanations did not significantly amplify tutorial effects, and users still had to connect and interpret the logic units themselves.The study did not collect participants’ explicit views of their understanding, leaving explanation comprehension unmeasured.Source: §6.1 · p. 14

6.2 Implications

  • Tutorials should address both users’ and AI systems’ strengths and shortcomings, since calibrating self-assessment alone does not ensure appropriate reliance.The authors recommend checking bias-specific tutorials for side effects, particularly whether calibration leads users to overestimate their ability relative to AI.Source: §6.2 · p. 15

6.3 Caveats and Limitations

  • Potential bias remains because task assessments may be affected by factors such as unequal perceived difficulty, worker incentives, and familiarity with the dataset.The authors could not be certain that the two task batches were perceived as equally difficult; Reclor familiarity and availability bias may also affect results.Source: §6.3 · p. 15
  • Because analyses used only a logical-reasoning task, transfer to settings with different user expertise, familiarity, or input modalities remains uncertain.The authors identify this gap as a potential threat to transferability.Source: §6.3 · p. 15

7 CONCLUSIONS AND FUTURE WORK

The study concludes that people who overestimate their performance tend to rely less on AI, hindering average performance, and that tutorial calibration can also harm reliance for some who initially underestimate themselves. The authors call for more comprehensive, personalized tutorials and note that their intervention did not change subjective trust in AI.

7 CONCLUSIONS AND FUTURE WORK

  • Participants who overestimated their performance tended to rely less on AI, and this under-reliance hindered better average performance.The conclusion combines the reported association between DKE and overestimation with the observed reliance pattern.Source: §7 · pp. 15–16
  • The tutorial helped mitigate DKE but could mislead participants who initially underestimated themselves, harming appropriate reliance.The authors identify overestimation or algorithm aversion as possible pathways and call for more comprehensive, personalized tutorials.Source: §7 · p. 16
  • The intervention did not change subjective trust in AI, while users’ general propensity to trust significantly shaped that trust.The authors suggest future work examine different interventions or more effective explanations.Source: §7 · p. 16
Loading 2301.11333v1…