PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

Gaole He, Nilay Aishwarya, Ujwal Gadiraju

arXiv:2501.17546v1cs.HCcs.AI

TL;DR

Opaque AI systems motivate explanations that can support informed human-AI decisions, but the effects of conversational XAI on understanding, trust, and reliance remain under-explored. The paper compares conversational XAI with an XAI dashboard in a loan approval study and finds better understanding and higher trust alongside overreliance. It further highlights the design challenge of supporting critical consideration rather than merely persuasive, plausible explanations.

  • Problem

    Opaque AI systems make critical automated decisions unreliable, while the effects of conversational XAI interfaces on understanding, trust, and reliance remain insufficiently studied.

  • Method

    The paper conducts a controlled loan-approval study comparing XAI dashboard and conversational XAI interfaces, including LLM agents and measures of user understanding.

  • Results

    Conversational XAI produced slightly better understanding and slightly higher trust than an XAI dashboard, but both interfaces increased overreliance on AI advice.

  • Takeaways & Limitations

    Effective conversational XAI design must promote appropriate reliance and critical consideration while supporting human-AI collaboration.

  • Takeaways & Limitations

    The study is a first exploration requiring confirmatory studies in different contexts, and its conversational interfaces used predefined rule-based dialogue with limited flexibility.

Abstract

from arXiv · show

Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI

1 Introduction

Critical AI decisions remain difficult to trust because deep learning systems are opaque, motivating XAI and conversational interfaces that may improve understanding. This study compares conversational XAI with an XAI dashboard for understanding, trust, and reliance, while examining the risk of overreliance.

  • Deep learning systems’ intrinsic opaqueness makes automating critical decisions far from reliable.
  • XAI methods provide explanations of AI predictions, while conversational interfaces may simplify information exploration and support more coherent mental models.
  • Existing research has not systematically examined how conversational XAI affects user understanding, trust, and reliance in critical decision making.
  • The study empirically compares an XAI dashboard and conversational XAI interfaces in a loan approval decision task, including LLM-powered conversation.
  • Conversational XAI users tended to rely more on the AI system, but increased reliance did not always translate into appropriate reliance.
  • Both XAI interfaces may function as persuasive technology, creating an illusion of explanatory depth that can lead users to overestimate AI capability.
  • The work highlights implications for promoting appropriate reliance and designing conversational interfaces that support human-AI collaboration.

2 Related Work

Related work frames human-AI decision making around appropriate reliance, explainability, and user-centered interaction. Conversational XAI is presented as a promising but under-explored way to improve explanation experiences and understanding, with limited systematic evidence about trust and reliance.

  • Human-AI Decision Making: Complementary team performance requires users to follow AI advice when the system is more capable and reject it when the system is less capable.
  • Human-AI Decision Making: Miscalibrated trust can produce over-reliance or under-reliance, while expertise, cognitive bias, confidence, risk perception, feedback, and explanations influence reliance.
  • Human-AI Decision Making: Existing XAI research has examined effects on understanding, trust, and reliance, but the role of interaction interfaces remains unclear.
  • Explainable AI: XAI methods include feature attribution, counterfactual, and contrastive explanations intended to address deep learning systems’ explainability concerns.
  • Explainable AI: Effective explanations should support coherent and complete mental models while remaining interactive enough to address contradictions.
  • Conversational User Interfaces: Conversational XAI has been proposed for human-AI collaboration, but its effects on trust and appropriate reliance remain under-explored empirically.
  • Conversational User Interfaces: A conversational user interface emulates conversation with a real human and can offer natural interaction, filtered information, engagement, understandability, and accessibility.

3 Task, Method, and Hypotheses

The study uses a loan-approval task to compare dashboard and conversational XAI interfaces, including evaluative and LLM-agent variants, and tests their effects on understanding, trust, and reliance. Participants receive multiple XAI explanations mapped to distinct information needs during a two-stage decision process.

  • Task: Participants decide whether loan applicants are credit worthy using eleven profile features presented in table and text formats.The task uses a publicly available loan prediction dataset and simulates a consequential human-AI decision setting.
  • Task: In the two-stage procedure, participants first decide without AI assistance and then may revise their choice using AI advice and explanations.They also report their three most important decision features and confidence at each stage.
  • XAI methods: The interfaces provide five XAI methods addressing six information needs: how, why, why not, how to be that, how to still be this, and what if.The methods include PDP, SHAP, MACE, WhatIf, and decision-tree explanations.
  • Interface design: The XAI dashboard lets users request each method individually, whereas the conversational interface guides users through categorized queries and corresponding responses.The conversational setup uses rule-based intent categories for AI advice, modified profiles, and global feature effects.
  • Interface design: The evaluative conversational interface steers users toward explanations related to their initially selected top-three features and compares those criteria with AI explanations.For PDP and WhatIf, the interface adapts prompts to incorporate the user's initial decision criteria.
  • Hypotheses: The hypotheses predict that conversational XAI will improve understanding, trust, and appropriate reliance, while evaluative steering will further increase trust and appropriate reliance.These predictions are stated relative to the XAI dashboard.

4 Study Design

The study compares five XAI conditions in a between-subjects loan-approval experiment. It measures understanding, explanation utility, trust, performance, reliance, confidence, and related decision-making outcomes.

  • Experimental conditions: Participants were randomly assigned to five conditions: Control, Dashboard, CXAI, ECXAI, or LLM Agent.The conditions differ in the interface used to explain AI advice, while participants complete the same loan-approval tasks.
  • Experimental conditions: The study treats the XAI interface as the sole independent variable in a between-subjects design with AI advice provided during the second decision stage.The experiment was conducted under institutional human-research ethics approval.
  • Measures: The dependent variables cover user understanding, explanation utility, trust, performance, reliance, and appropriate reliance.User confidence and other variables were also recorded to examine broader effects of the interfaces.
  • Measures: User understanding is defined as a function of interactive exploration of XAI interfaces, while perceived explanation utility is assessed separately.Both constructs are based on dimensions synthesized from prior literature.
  • Measures: Trust is measured through reliability/competence, understanding/predictability, and trust-in-automation subscales using average scores on five-point Likert items.These measures capture perceived system capabilities, predictability, and overall automation trust.
  • Measures: Performance and reliance are assessed from distinct standpoints, including overall accuracy, accuracy when initial choices disagree with AI advice, and agreement-based measures.This design addresses limitations of inferring reliance solely from agreement with AI advice.
  • Procedure: The procedure includes consent, machine-learning experience, a technology-interaction questionnaire, onboarding, and a practice example before the experimental tasks.Participants were randomly assigned before completing the study procedure.

5 Experimental Results

The empirical study found mostly limited differences among XAI interfaces in explanation utility and user understanding, while trust and reliance patterns showed both benefits and risks. Conversational interfaces increased some reliance measures without consistently improving appropriate reliance or team performance.

  • Performance overview: 64.5% average participant accuracy remained below the AI's 70% accuracy, while agreement and switching fractions were 0.847 and 0.522.These measures indicate that participants did not always blindly follow AI advice when they disagreed.
  • Explanation utility: Participants reported similar Explanation Completeness and Explanation Clarity across conditions, with slightly higher Explanation Coherence and Explanation Usefulness for conversational interfaces.The reported differences in perceived explanation utility were not statistically significant overall.
  • User understanding: No significant interface differences appeared across most user-understanding dimensions, but objective feature understanding differed significantly across interfaces (H=16.19, p=.001).The LLM Agent condition achieved significantly worse objective feature understanding in the reported post-hoc comparison.
  • User trust: XAI interfaces significantly increased trust in Understandability/Predictability relative to Control, while conversational and dashboard conditions did not significantly differ.The interface condition did not significantly affect Reliability/Competence or Trust, and the LLM Agent condition showed only a consistent non-significant trend.
  • Reliance and appropriate reliance: Conversational XAI increased reliance, but only CXAI significantly improved RAIR versus Control; dashboard and conversational interfaces worsened RSR, with LLM Agent producing severe over-reliance.The hypothesis that interfaces would improve appropriate reliance was not supported.
  • Team performance: Team performance showed no significant differences, although estimation plots indicated higher mean accuracy for CXAI and slightly better Accuracy-wid for ECXAI than comparison conditions.No significant trust or appropriate-reliance difference was found between CXAI and ECXAI, so H4 was not supported.

6 Discussion

Interactive XAI interfaces improved perceived understanding and trust but also acted as persuasive technology, encouraging over-reliance. More flexible, plausible LLM-agent conversations could further worsen objective understanding and reliance calibration, underscoring a trade-off between usability and critical reflection.

  • 6.1 Key Findings: Interactive XAI interfaces produced relatively high perceived understanding, trust, and reliance, but users still over-relied on AI advice.The authors link this pattern to an illusion of explanatory depth.
  • 6.1 Key Findings: LLM-agent conversations further reinforced over-reliance and produced significantly worse objective feature understanding than other interactive-XAI conditions.The authors interpret this as enhanced conversation quality inducing a stronger illusion of explanatory depth.
  • 6.1 Key Findings: Conversational XAI yielded relatively better understanding, team performance, trust, and appropriate reliance than the XAI dashboard, but further conversational enhancement was harmful in the LLM-agent condition.The comparison is reported across the conversational interface and dashboard conditions, with the LLM-agent condition showing decreased outcomes.
  • 6.3 Caveats and Limitations: The findings require confirmatory studies in different contexts and highlight a trade-off between seamless conversational interfaces and critical consideration of AI advice.The authors frame this work as an initial exploration requiring further empirical corroboration.
  • 6.2 Implications of Our Work: Effective conversational XAI should prioritize accessible exploration and faithful calibration of AI-advice trustworthiness over persuasive plausibility.The authors also identify AI literacy support as a promising direction.
  • 6.3 Caveats and Limitations: The study’s five selected XAI methods may have left some participants’ information needs uncovered, while rule-based dialogues lacked communication flexibility.These design choices constrain how broadly the observed trust and reliance effects should be interpreted.
  • 6.3 Caveats and Limitations: Evaluative conversations increased trust and reliance without increasing appropriate reliance, raising concerns about unequal benefits for users with different AI literacy levels.The discussion connects this concern to possible disparities affecting users with lower AI literacy or less critical mindsets.

7 Conclusion

The study found slightly better understanding and higher trust with conversational XAI than with an XAI dashboard, but interactive XAI interfaces also increased over-reliance. LLM-based conversational agents could further reinforce over-reliance while harming understanding and trust.

  • 7 Conclusion: Conversational XAI participants showed slightly better system understanding and slightly higher trust than participants using the XAI dashboard.These findings address RQ1 and RQ2 in the loan-decision study.
  • 7 Conclusion: Interactive XAI interfaces were persuasive and potentially created an illusion of AI capability that increased users’ over-reliance.The conclusion cautions that explanations may affect reliance without ensuring appropriate judgment.
  • 7 Conclusion: LLM-based conversational agents further reinforced over-reliance and hurt user understanding and trust despite improving conversation quality and flexibility.The reported result challenges the assumption that richer conversation automatically improves decision support.
  • 7 Conclusion: Future conversational-XAI design should balance engagement with seamful features that promote critical reflection and mitigate the illusion of explanatory depth.The authors recommend prioritizing mitigation before further improving user experience.

A.1 Implementation Details

The study used selected loan-approval tasks and questionnaires to evaluate interactive XAI in a controlled between-subjects experiment. Participants were screened, compensated for correct decisions, and assigned across multiple interface conditions.

  • Task Selection: Participants completed ten loan-approval tasks selected from a 4:1 training/test split and balanced between Credit Worthy and Not Credit Worthy cases.Tasks were selected using prediction correctness and model confidence after training an XGBoost classifier.
  • Task Selection: Table 3 defines task-selection criteria using prediction correctness and model confidence for Credit Worthy and Not Credit Worthy cases.CW and NCW denote Credit Worthy and Not Credit Worthy, respectively.
  • Sample Size Estimation: The power analysis required 244 participants using f=0.25, α=0.0125, power 0.8, and four experimental groups.The threshold reflects Bonferroni correction for multiple hypotheses.
  • Sample Size Estimation: The study recruited 352 participants across Control, Dashboard, CXAI, and ECXAI conditions, excluding 107 participants; the LLM Agent condition was a follow-up study.Participants were recruited through Prolific and randomly assigned continuously until the required sample size was reached.
  • Compensation: Participants received £4 plus £0.05 per correct loan decision, encouraging effort toward correct decisions and appropriate system reliance.The estimated completion time was 30 minutes, corresponding to an £8 hourly wage before bonuses.
  • Filter Criteria: Eligibility required English proficiency, age over 18, more than 40 completed Prolific tasks, and an approval rate above 90%, with attention checks for reliability.Attention checks covered both decision-making tasks and questionnaires.
  • Questionnaire: Questionnaires measured perceived feature understanding, understanding of the system, explanation usefulness, and explanation clarity.Items used five-point agreement scales.

A.2 Additional Exploratory Analyses

Exploratory covariate analyses found that trust propensity and attitude toward AI influenced user trust and several human-AI decision-making outcomes.

  • Additional Exploratory Analyses: TiA-Propensity to Trust significantly positively impacted most dependent variables related to user understanding and trust.The analysis used Spearman rank-order tests between covariates and dependent variables.

A.2.1 Impact of Covariates.

The study examined how perception-based variables, including user trust and understanding, relate to team performance and reliance behaviors.

  • Spearman rank-order tests assessed relationships between user trust, user understanding, explanation utility, team performance, and reliance behaviors.The analysis followed prior evidence that user trust can substantially affect reliance behaviors.

A.2.2 The Impact of User Perceptions on Their Behavior.

Improved user understanding, explanation utility, and trust were associated with greater agreement with AI advice but lower relative switching, indicating increased over-reliance without corresponding performance gains.

  • Agreement Fraction and RSR significantly correlated with most dimensions of user understanding, explanation utility, and user trust.These perception-based dimensions were positively correlated with Agreement Fraction but negatively correlated with RSR.
  • The correlations suggest that improved understanding, explanation utility, and trust with XAI interfaces can partially explain increased over-reliance on AI.The pattern links stronger perceptions of the system with behavior indicating greater reliance.
  • The analyzed perception-based variables did not significantly contribute to team performance.
  • Participants generally reported higher confidence after receiving AI advice and explanations.Confidence patterns varied by condition across the task order.

A.2.3 Confidence Dynamics.

The section examines engagement, XAI-method usage, conversation histories, and broader interface effects, with enhanced LLM conversations associated with greater exploration but also amplified confidence and over-reliance.

  • LLM Agent users reported the highest engagement mean at M=3.44, but the between-condition engagement difference was non-significant.Engagement means were Control 3.15, Dashboard 3.33, CXAI 3.20, ECXAI 3.28, and LLM Agent 3.44.
  • No significant differences in XAI-method usage frequency were found between the CXAI and ECXAI conditions across all five methods.
  • The LLM Agent condition allowed flexible exploration through randomized hint questions and free-text input for personalized information needs.Users could ask questions beyond the provided hints through free-text interaction.
  • The Sankey diagram visualized the sequence and dynamics of users’ XAI-method interactions in the LLM Agent condition.
  • Among 61 LLM Agent users, 1,946 queries were recorded, with approximately 40% based on provided hint questions.The queries included XAI-response requests, greetings, and opinion-seeking questions.
  • Conversational interfaces showed no significant superiority over the dashboard, although they produced slightly better perceived understanding and explanation utility.Conversational conditions also did not yield higher engagement than the dashboard.
  • Enhanced LLM conversations enlarged the confidence gap between decision stages and coincided with the lowest objective feature understanding, subjective trust, and appropriate reliance.The authors infer that LLM agents may amplify the illusion of explanatory depth and high over-reliance.
  • Crowdsourcing, confirmation bias, and rule-based conversational agents may have influenced trust, XAI-method usage, understanding, and reliance patterns.The rule-based agents may have contributed to similar usage patterns in the CXAI and ECXAI conditions.
Loading 2501.17546v1…