PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

Shuai Ma, Qiaoyi Chen, Xinru Wang, Chengbo Zheng, Zhenhui Peng, Ming Yin, Xiaojuan Ma

arXiv:2403.16812v2cs.HCcs.AI

TL;DR

In AI-assisted decision-making, people may engage little analytical thinking and have limited ways to discuss partial disagreements with AI. The paper proposes Human-AI Deliberation and Deliberative AI, and an exploratory graduate admissions study reports improved decision accuracy and appropriate reliance compared with conventional XAI assistants.

  • Problem

    People often do not engage analytical thinking when reviewing AI suggestions, while existing systems offer limited support for discussing conflicting human-AI opinions.

    Source: §1 · pp. 1–2

  • Method

    Human-AI Deliberation elicits dimension-level opinions, supports discussion of disagreements, and enables updates, with LLMs bridging people and domain-specific models.

    Source: §1 · p. 2

  • Results

    The exploratory assessment found that Human-AI Deliberation improves decision accuracy and fosters appropriate reliance on AI compared with conventional explainable AI systems.

    Source: §7 · p. 16

  • Takeaways & Limitations

    The work offers an initial design direction for integrating deliberation into AI-assisted decision-making.

    Source: §2.2 · p. 4

    The admissions task simplified nuanced evaluations into numerical strength values, and participants were not actual admissions committee members; findings may not generalize to PhD admissions.

    Source: §7.7 · pp. 17–18

Abstract

from arXiv · show

In AI-assisted decision-making, humans often passively review AI's suggestion and decide whether to accept or reject it as a whole. In such a paradigm, humans are found to rarely trigger analytical thinking and face difficulties in communicating the nuances of conflicting opinions to the AI when disagreements occur. To tackle this challenge, we propose Human-AI Deliberation, a novel framework to promote human reflection and discussion on conflicting human-AI opinions in decision-making. Based on theories in human deliberation, this framework engages humans and AI in dimension-level opinion elicitation, deliberative discussion, and decision updates. To empower AI with deliberative capabilities, we designed Deliberative AI, which leverages large language models (LLMs) as a bridge between humans and domain-specific models to enable flexible conversational interactions and faithful information provision. An exploratory evaluation on a graduate admissions task shows that Deliberative AI outperforms conventional explainable AI (XAI) assistants in improving humans' appropriate reliance and task performance. Based on a mixed-methods analysis of participant behavior, perception, user experience, and open-ended feedback, we draw implications for future AI-assisted decision tool design.

1 INTRODUCTION

The paper proposes Human-AI Deliberation to address limited analytical engagement and poor support for resolving partial human-AI disagreements. It evaluates Deliberative AI in an exploratory graduate admissions study.

1 INTRODUCTION

  • People rarely engage analytical thinking when shown AI suggestions, and may over-rely on incorrect or under-rely on correct recommendations.Prior approaches such as explanations and effortful-thinking prompts have produced mixed results.Source: §1 · p. 1
  • Because human and AI rationales often partially align, accept-or-reject interfaces offer little support for discussing disagreements or adopting only parts of an AI suggestion.The authors argue this may limit the use of both human and AI knowledge.Source: §1 · p. 2, Figure 1 · p. 2
  • Deliberation can encourage people to evaluate perspectives and refine viewpoints, but its design for human-AI conversation and its effects on decision-making remain underexplored.The authors propose it as a possible response to both limited reflection and limited conflict resolution.Source: §1 · p. 2
  • Human-AI Deliberation elicits human thoughts, supports discussion of conflicting opinions and rationales, and allows both people and AI to update decisions.Deliberative AI combines domain-specific models for information generation with LLMs for interaction and conversation.Source: §1 · p. 2
  • The study examined effects on task performance and reliance, perceptions of the AI partner, user experience, and views on the approach and its future design.Participants with graduate admissions experience predicted applicants’ chances of receiving an offer.Source: §1 · pp. 2–3
  • Human-AI Deliberation showed potential to improve decision accuracy and appropriate reliance on AI compared with traditional XAI assistants.The study used a graduate admissions task.Source: §1 · p. 3

2 RELATED WORK

The related work frames complementary human-AI performance as a goal, identifies gaps in leveraging both parties’ knowledge and deliberation, and distinguishes this work from prior approaches.

2.1 AI-Assisted Decision-Making: Objectives and Challenges

  • AI-assisted decision-making aims at complementary team performance, but systems have difficulty harnessing both parties’ domain knowledge and supporting its exchange when views conflict.A further challenge is helping people decide when to rely on AI recommendations rather than over- or under-relying.Source: §2.1 · p. 3

2.2 The Role of Deliberation in Human Decision Making

  • Deliberation can help people evaluate perspectives and refine viewpoints, yet its integration into AI-assisted decision-making has received little research attention.The paper takes an initial step by proposing Human-AI Deliberation and studying its effects on decision processes and outcomes.Source: §2.2 · p. 4

2.3 Existing Studies on Deliberation in AI-Assisted Decision Making

  • Prior work explored analytical-thinking interventions and human-AI dialogue, but was not tailored to direct deliberation between humans and AI in decision-making.The authors characterize their work as an initial step toward designing and evaluating such deliberation.Source: §2.3 · p. 4

3 DELIBERATIVE AI FOR HUMAN-AI DELIBERATION

The paper places Human-AI Deliberation in the evidence-weighing phase and structures it around dimension-level opinions, discussion, and updates. Deliberative AI combines LLM interaction with domain-specific information.

3.1 Integrating Deliberation into Decision-Making

  • Human-AI Deliberation is proposed for the evidence-weighing phase, where disagreements between human perspectives and AI suggestions can shape later opinions and decisions.The framework breaks decisions into an overall choice, dimensions, opinions on dimensions, and supporting evidence; opinions and Weight of Evidence are used interchangeably.Source: §3.1 · p. 4

3.2 An Architecture of Human-AI Deliberation

  • The architecture links four activities: eliciting thoughts, aligning human and AI viewpoints, discussing evidence and opinion weights, and updating thoughts.Elicitation can prompt human reasoning but may impose workload; the interface should let people revise their Weight of Evidence.Source: §3.2 · pp. 4–5

3.3 Deliberative AI: Design Considerations and Overall Structure

  • The design considerations call for equal participation, rational justifications, constructive updates, interactivity, and respectful discourse.The architecture depicts core activities alongside potential design choices.Source: §3.3 · pp. 5–6, Figure 2 · p. 5
  • Deliberative AI uses LLMs to mediate conversation and domain-specific models to generate and refine Weight of Evidence with domain knowledge and data-derived insights.Its three layers are Communication, Control, and Knowledge; the Communication layer uses LLMs, while the Control layer connects interactions to model knowledge and updates.Source: §3.3 · p. 6

4 INSTANTIATING THE ARCHITECTURE: GRADUATE ADMISSION PREDICTION

The graduate-admissions instantiation combines dimension-level human and AI opinions with LLM-mediated discussion grounded in a domain-specific model and its data. Its components support opinion elicitation, evidence retrieval, opinion updates, and an interface linking decisions with dialogue.

4.1 Task, Dataset and AI Model

  • The illustrative task uses 100 synthesized applicant profiles and a linear decision model that achieved 80% accuracy on the remaining 30% test set.Strong/weak accept and reject labels were binarized for accuracy assessment; study cases came from the test set.Source: §4.1 · p. 6

4.2 Human-AI Thought Representation and Alignment

  • Human and AI express each applicant attribute’s influence as a dimension-level contribution, with SHAP representing AI contributions and meta-explanations supplying relevant training-data evidence.The interface makes human and AI judgments comparable; SHAP values are converted to probabilities to improve direct interpretation.Source: §4.2 · pp. 6–7

4.3.1 I. Communication Layer.

  • The Intention Analyzer identified themes in participant statements with 96% accuracy, and the Argument Evaluator’s LLM scores showed substantial agreement with human scores (κ = 0.78).The pilot involved 226 deliberative statements; two authors coded them to iteratively refine the Intention Analyzer prompts.Source: §4.3.1 · pp. 7–8

4.3.2 II. Control Layer.

  • The control layer sequences elicitation and discussion, retrieves task-specific evidence for the LLM, regulates responses against DS-model knowledge, and updates AI opinions using argument strength and uncertainty.The opinion update adjusts the AI’s expressed viewpoint for the current case, not the domain-specific model itself.Source: §4.3.2 · pp. 8–9

4.3.3 III. Knowledge Layer.

  • The knowledge layer supplies DS-model predictions and dimension opinions alongside training-data patterns for real-time queries, grounding LLM-generated discussion in retrieved task information.The LLM handles language understanding and generation, while logic code retrieves opinions and evidence from the model and training data.Source: §4.3.3 · p. 9

4.4 Interface Design

  • The interface links applicant profiles, side-by-side human and AI opinions, and discussion, with changes in dimension-level sliders updating aggregate predictions and dialogue views.Dimension status indicators distinguish discussed, active, and untouched topics; viewpoint changes mentioned in dialogue are reflected in the opinion region.Source: §4.4 · pp. 9–10

5 EXPLORATORY USER STUDY

The exploratory mixed-methods study used graduate admissions to examine Human-AI Deliberation’s potential effects, not its effectiveness specifically for admissions. Participants compared deliberation with XAI and unaided decisions across conflict-focused cases, with quantitative measures and qualitative feedback.

5 EXPLORATORY USER STUDY

  • The exploratory study used graduate admissions as an illustrative testbed to investigate potential effects on AI-assisted decision-making, rather than assess admissions-specific effectiveness.The study examined task performance, reliance, perceptions of AI, user experience, and open-ended feedback.Source: §5 · p. 10

5.1 Task Setup

  • The four main cases were selected from a pilot for human-AI disagreement, often involving borderline applicants; both human and AI performance on these cases ranged from 50% to 60%.The cases were chosen to focus on ambiguous decisions requiring conflict resolution, not to represent the model’s general performance.Source: §5.1 · p. 10

5.2 Conditions

  • The Deliberative AI condition elicited participants’ dimension-level views before AI recommendations and offered discussion, while XAI supplied feature-contribution explanations and Human Alone provided no AI assistance.The study compared Deliberative AI with both traditional XAI and unaided human decisions.Source: §5.2 · pp. 10, 12

5.3 Procedure

  • Participants completed a tutorial and qualification questions before four randomly presented admissions cases, then provided perceptions, experiences, and feedback in an exit survey.The tutorial included one practice case and attribute distribution and summary statistics.Source: §5.3 · p. 12

5.4 Participants

  • Of 174 recruited participants, 153 valid responses remained after attention checks: 48 in Deliberative AI, 51 in XAI, and 54 in Human Alone.Recruitment targeted U.S. residents previously admitted to a U.S. graduate program, among other eligibility requirements.Source: §5.4 · p. 12

5.5 Measurement

  • Measures covered decision accuracy, reliance appropriateness, AI helpfulness, trustworthiness and understanding, decision confidence, workload-related experience, satisfaction, and open-ended feedback.Reliance measures included agreement, switching, over-reliance, and under-reliance; several perception and experience measures used 7-point Likert scales.Source: §5.5 · p. 12

5.6 Analysis Methods

  • Because data failed normality tests, quantitative comparisons used Mann-Whitney U or Kruskal–Wallis tests with Bonferroni post-hoc correction; two authors inductively coded feedback and conversation logs.The final qualitative themes were agreed through discussion and harmonization over several iterations.Source: §5.6 · p. 12

6 RESULTS

Deliberative AI improved decision accuracy and reduced agreement, switching, and over-reliance relative to XAI. It also drew less trust and satisfaction, highlighting a trade-off between deliberative thinking and user experience.

6.1.1 Decision Accuracy.

  • Decision accuracy was higher with Deliberative AI (M=0.598) than XAI (M=0.524, p<0.05).The evaluation focused on cases where tasks were challenging for both humans and AI.Source: §6.1.1 · p. 13

6.1.2 Reliance.

  • Participants agreed less with AI suggestions (M=0.57 vs. 0.68, p<0.05) and switched less to AI predictions (M=0.23 vs. 0.51, p<0.001) with Deliberative AI than XAI.The authors suggest deliberation may prompt more independent thinking and recognition of problems in AI’s perspective.Source: §6.1.2 · p. 13

6.1.3 Appropriateness of Reliance.

  • Over-reliance was lower with Deliberative AI than XAI (M=0.47 vs. 0.65, p<0.001), while under-reliance did not differ significantly (M=0.32 vs. 0.29).The authors interpret this pattern as more appropriate reliance without the under-reliance increase reported in prior cognitive-forcing research [16].Source: §6.1.3 · p. 13

6.2.1 Perceptions of AI.

  • Participants reported less trust in Deliberative AI than XAI (M=4.47 vs. 5.52, p<0.01), with conversation-log analysis suggesting they identified more AI flaws.Perceived helpfulness and understanding did not differ significantly between conditions.Source: §6.2.1 · p. 13

6.2.2 User experience.

  • M=6 (SD=1.10) confidence in XAI predictions exceeded Human Alone (M=5.59, SD=1.12, p<0.05), although XAI accuracy was lower than Human Alone.The authors characterize this pattern as potentially producing illusionary confidence.Source: §6.2.2 · p. 13
  • Satisfaction was lower with Deliberative AI than Human Alone, suggesting a trade-off between encouraging deliberation and optimizing user experience.The authors propose balancing deliberation’s benefits with maintaining a positive user experience.Source: §6.2.2 · p. 13

6.3 RQ3: How will humans perceive the effectiveness of the proposed Human-AI Deliberation and what can be improved for future Human-AI Deliberation design?

Open-ended feedback and conversation logs suggest participants often found deliberation useful, while also raising concerns about AI updates, reasoning, and interaction design.

6.3.1 Perceptions of Deliberative AI and the Discussion Process.

  • 43 of 48 participants said deliberation helped them make more informed decisions.Participants cited identifying AI limitations, considering different perspectives, and reflecting on or correcting their own mistakes.Source: §6.3.1 · p. 13
  • Participants doubted AI in 32% of dialogues, acknowledged its arguments in 17%, and self-reflected or corrected in 15%.Feedback also described deliberation prompting participants to question their own reasoning.Source: §6.3.1 · pp. 13–14
  • Participants credited deliberation with introducing new information (18), helping recognize AI bias (9), and balancing objective data with human judgment (15).Source: §6.3.1 · pp. 14–15
  • Some participants reported reduced agency (5), mental effort and verbose responses (4), data-heavy reasoning that missed human reasoning (15 and 10), and shortcomings in memory, context, or transparency.Source: §6.3.1 · p. 15
  • 25 of 48 participants said AI updates improved the experience; 19 felt heard and respected, while others described discussion as more engaging and collaborative.Source: §6.3.1 · p. 15
  • Update perceptions varied: 11 participants thought the AI changed views too often, 13 were unsure why or how, and 3 doubted its initial opinions because updates lacked transparency.The authors recommend considering preferences for update frequency and magnitude and improving transparency.Source: §6.3.1 · p. 15
  • Participants requested deeper understanding of human reasoning, memory of prior interactions, broader analysis, and clearer explanations of changing opinions.These requests included more nuanced, context-aware communication and moving beyond statistical reasoning.Source: §6.3.1 · pp. 15–16
  • Discussion-design suggestions ranged from more human-like conversation and concise responses to gradual argument disclosure, user steering, faster dialogue, and richer visuals.Source: §6.3.1 · p. 16

7 DISCUSSION

The discussion presents Human-AI Deliberation as a conflict-focused alternative to whole-suggestion acceptance or rejection, reporting improved decision accuracy and appropriate reliance while examining trade-offs and limits. It highlights reduced over-reliance, the effort involved in conflict resolution, and constraints on generalization and implementation.

7.1 Deliberation as a New Paradigm Complementing Existing (X)AI Assistance

  • Conflict-focused deliberation complements XAI by encouraging independent opinions, critical reflection, and active discussion, though it can require more time and effort.Participants valued reflection even in subjective decision contexts; the discussion identifies investment and hiring as examples where decision quality may outweigh speed or convenience.Source: §7.1 · p. 16

7.2 Reducing Human Over-Reliance by Exposing AI Mistakes

  • Deliberative AI reduces over-reliance on inaccurate suggestions without increasing under-reliance; 31% of conversations included participants questioning AI logic.The discussion attributes the reduction to deliberative engagement with conflicting opinions and notes that participants adjusted judgments through dialogue.Source: §7.2 · p. 16

7.3 Human-AI Conflict Resolution: Key to Enhancing Collaboration

  • Conflict resolution can improve accuracy and introspection, but Deliberative AI has lower user satisfaction than XAI and Human Alone despite similar mental demand, effort, and perceived complexity.The discussion identifies opinion articulation, dialogue effort, and emotional difficulty as sources of burden and proposes interface changes to address them.Source: §7.3 · pp. 16–17

7.4 Obstacles to Discussion: Humans and AI Think Differently

  • Human and AI reasoning can diverge because people use personal understanding and heuristics while AI relies on data, making strong subjective perspectives challenging to engage.The authors propose using DS-Models to guide LLMs on data-related discussion while leaving non-data matters to the LLM; data insights may still not match users’ reasoning.Source: §7.4 · p. 17

7.5 Ethical Concerns in Deliberative AI Design

  • DS-Model guidance does not eliminate LLM inaccuracies, and adaptive AI opinions raise accountability and expectation-management concerns.The discussion calls for caution and transparency about possible errors and for transparent update mechanisms that explain how opinions change.Source: §7.5 · p. 17

7.6 On the Generalizability of Human-AI Deliberation

  • Generalization is bounded by task demands and representation: discussing many features may be impractical, Weight of Evidence may be unintuitive, and image adaptation may require advances in vision-language models.The architecture is designed for tabular data and may extend to text; the LLM is used mainly for communication rather than complex reasoning, whose capabilities remain challenging.Source: §7.6 · p. 17

7.7 Limitations and Future Work

  • The exploratory admissions study may not generalize broadly because its simplified dataset and non-committee participants do not capture real-world admissions expertise or nuance.The authors also note only four decision tasks were used, the task lacks definitive ground truth, and the cases were deliberately conflict-prone and ambiguous.Source: §7.7 · pp. 17–18

8 CONCLUSION

The conclusion presents Human-AI Deliberation as an exploratory new paradigm, illustrated in graduate admissions, with potential to improve decision accuracy and appropriate reliance on AI.

8 CONCLUSION

  • The graduate-admissions exploratory study indicates potential for Deliberative AI to improve decision accuracy and promote more appropriate reliance on AI.The authors frame the work as an exploratory step toward a new paradigm of AI assistance for decision-making.Source: §8 · p. 18

A BASELINE XAI INTERFACE

The appendix identifies the baseline XAI interface.

A BASELINE XAI INTERFACE

B AN EXAMPLE DATAFLOW OF DELIBERATIVE AI

The example dataflow follows a user’s GPA-related argument through intention analysis, data retrieval, and a constrained LLM response.

B AN EXAMPLE DATAFLOW OF DELIBERATIVE AI

C DETAILED METRICS AND QUESTIONS

The section outlines the study’s measurement categories and illustrates the baseline XAI interface and Deliberative AI’s input-output process.

C DETAILED METRICS AND QUESTIONS

  • The baseline XAI interface shows an applicant profile, AI feature-contribution explanations, and controls for initial predictions, AI suggestions, and final decisions.AI suggestions and explanations appear only after humans make their initial predictions.Source: Figure 10 · p. 22
  • Figure 11 illustrates how Deliberative AI processes human inputs and generates outputs, but its prompts are simplified for illustration.The complete prompts are in the supplementary materials.Source: Figure 11 · p. 23
  • Table 2 lists the study’s metrics and questions and groups collected evidence into objective decision data, subjective questionnaire data, and qualitative open-ended feedback.The supplied passages identify the categories but do not specify the individual metrics or questions.Source: C DETAILED METRICS AND QUESTIONS · p. 21, Table 2 · p. 23
Loading 2403.16812v2…