PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making

Han Liu, Vivian Lai, Chenhao Tan

arXiv:2101.05303v4cs.AIcs.CYcs.HCcs.LG

TL;DR

The paper asks why human-AI teams rarely outperform AI alone and humans alone in challenging prediction tasks. It compares in-distribution with out-of-distribution examples and evaluates interactive explanations through pilot and randomized experiments across three tasks. Out-of-distribution settings narrow the team–AI gap, while interactive explanations improve perceived usefulness but do not reliably improve performance and may reinforce human biases.

  • Problem

    Human-AI teams seldom outperform AI alone in challenging prediction tasks, leaving complementary performance difficult to achieve.

  • Method

    The authors compare in-distribution and out-of-distribution examples and test interactive explanation interfaces in virtual pilots and randomized experiments across three tasks.

  • Results

    Human-AI teams remain below AI alone in-distribution, but the performance gap becomes smaller out-of-distribution; interactive explanations improve perceived usefulness without improving team performance.

  • Takeaways & Limitations

    Evaluations should include distribution shift, while interactive explanations require caution because perceived usefulness may coexist with limited performance gains and reinforced biases.

  • Takeaways & Limitations

    Complementary performance is not achieved, and the interactive-explanation implementation may be insufficient to improve team performance.

Abstract

from arXiv · show

Although AI holds promise for improving human decision making in societally critical domains, it remains an open question how human-AI teams can reliably outperform AI alone and human alone in challenging prediction tasks (also known as complementary performance). We explore two directions to understand the gaps in achieving complementary performance. First, we argue that the typical experimental setup limits the potential of human-AI teams. To account for lower AI performance out-of-distribution than in-distribution because of distribution shift, we design experiments with different distribution types and investigate human performance for both in-distribution and out-of-distribution examples. Second, we develop novel interfaces to support interactive explanations so that humans can actively engage with AI assistance. Using virtual pilot studies and large-scale randomized experiments across three tasks, we demonstrate a clear difference between in-distribution and out-of-distribution, and observe mixed results for interactive explanations: while interactive explanations improve human perception of AI assistance's usefulness, they may reinforce human biases and lead to limited performance improvement. Overall, our work points out critical challenges and future directions towards enhancing human performance with AI assistance.

1 INTRODUCTION

The paper studies why human-AI teams rarely achieve complementary performance, focusing on distribution shift and interactive explanations as two potentially important experimental factors. Across three tasks, it finds that out-of-distribution settings narrow the team–AI performance gap, while interactive explanations improve perceived usefulness without reliably improving performance.

  • Motivation: Human-AI teams should outperform both AI alone and humans alone, yet prior studies find they seldom outperform AI alone in challenging prediction tasks.This motivates the paper’s focus on complementary performance.
  • Research directions: The authors argue that standard random train-test splits favor AI by evaluating teams on in-distribution examples, whereas distribution shift can reduce AI performance and create opportunities for human insights.Their proposed setup explicitly compares in-distribution and out-of-distribution examples.
  • Research directions: Interactive explanations extend static one-way explanations into two-way interaction, including changing inputs to observe counterfactual AI predictions.The study evaluates this approach alongside distribution type across recidivism prediction and profession detection tasks.
  • Findings: Human-AI teams underperform AI alone in-distribution across tasks, but can occasionally outperform AI in out-of-distribution recidivism examples, making the performance gap smaller there.The result supports greater potential for complementary performance out-of-distribution, although the reported difference is small.
  • Findings: Distribution type changes human agreement with AI: agreement is generally higher in-distribution for recidivism, but the pattern reverses in BIOS, where task-specific intuitions may matter more.In recidivism, humans also agree less with wrong AI predictions out-of-distribution, while BIOS shows lower in-distribution team performance relative to AI alone.
  • Findings: Interactive explanations increase perceived usefulness but do not improve team performance and may reinforce demographic-feature biases while increasing mistakes during disagreement with AI.Participants with interactive explanations focused more on age and race and less on computationally important features in the exploratory recidivism study.

2 RELATED WORK AND RESEARCH QUESTIONS

Prior work finds that human-AI teams rarely outperform AI alone in challenging prediction tasks, motivating study of distribution shift and interactive explanations. The paper asks how distribution types affect team performance and agreement, and whether interactive explanations improve decision making.

  • 2.1 Performance of Human-AI Teams in Prediction Tasks: Human-AI teams typically improve over humans alone but seldom outperform AI alone in challenging prediction tasks.This pattern has been reported across recidivism prediction, deceptive review detection, and hypoxemia prediction.
  • 2.1 Performance of Human-AI Teams in Prediction Tasks: Complementary performance is difficult to achieve, although prior work reports exceptions when human and AI performance are comparable.Bansal et al. found human-AI teams surpassing AI in sentiment classification and LSAT question answering.
  • 2.1 Performance of Human-AI Teams in Prediction Tasks: Random train-test splits create in-distribution evaluations that favor AI, whereas distribution shift can make learned patterns fail on out-of-distribution examples.The paper defines in-distribution examples as matching the training distribution and out-of-distribution examples as differing from it.
  • 2.2 Agreement with AI: The paper investigates whether distribution types change human agreement with AI, including appropriate trust, overtrust, and undertrust based on AI correctness.It hypothesizes greater agreement in-distribution and examines whether humans provide more complementary insights out-of-distribution.
  • 2.3 Interactive Explanations: Interactive explanations are proposed as a two-way alternative to static, one-way explanations that may insufficiently support understanding of AI predictions.The paper tests whether interactivity improves human-AI performance and examines its effects on decision making.

3 METHODS

The study evaluates human-AI decision making across three prediction tasks, contrasting in-distribution and out-of-distribution examples and static with interactive explanations. It combines virtual pilot studies with large-scale crowdworker experiments using tailored data splits and interfaces.

  • 3.1 Prediction Tasks: The experiments cover three prediction tasks spanning tabular recidivism data and text-based profession prediction, with varying human-AI performance gaps.The in-distribution gap is about 7% for recidivism prediction and about 20% for profession prediction.
  • 3.2 In-distribution vs. Out-of-distribution Setup: The study simulates distribution shift by splitting datasets into in-distribution and out-of-distribution subsets using selected attributes and bins.For ICPSR, age is used to create the split; attributes such as prior arrests and prior convictions are rejected because their distributions limit possible splits.
  • 3.2.2 AI Performance in-distribution and out-of-distribution.: AI accuracy drops by about 10% in recidivism prediction and about 7% in BIOS on out-of-distribution examples versus in-distribution examples.The balanced test set uses a 50% baseline in ICPSR and COMPAS, and AI remains above that baseline out-of-distribution.
  • 3.3 Explanation Type: Static assistance highlights important features, using colored signs for tabular features and the top 10 important words for BIOS.The experiments use linear SVMs with one-hot features for recidivism tasks and unigram bag-of-words for BIOS.
  • 3.2.2 AI Performance in-distribution and out-of-distribution.: The workflow trains participants with up to six representative examples before evaluating human-AI decisions on 10 in-distribution and 10 out-of-distribution examples.The training phase includes a feature quiz requiring one positive and one negative feature choice after each example.

3.3 Interactive Explanations and Explanation Type

The study combines representative training examples with static and interactive explanations, letting users manipulate tabular or text features and observe changed AI predictions. Pilot studies show task-specific strategies, disagreement with AI when explanations conflict with intuition, and mixed engagement with interactivity.

  • 3.3 Interactive Explanations and Explanation Type: The experiment trains users with 5–6 representative examples, then evaluates predictions on 10 in-distribution and 10 out-of-distribution examples.
  • 3.3.1 Static Assistance: Interactive explanations let users create counterfactual profiles or remove words and observe how the AI prediction changes.Tabular interfaces use sliders, radio buttons, and select lists; BIOS supports word removal and dynamic feature highlighting.
  • 3.4 Virtual Pilot Studies: Pilot participants often disagreed with AI when its explanations contradicted their intuitions or when learned feature patterns conflicted with later examples.Participants also struggled to attend globally to features rather than relying on a few learned patterns.
  • 3.4 Virtual Pilot Studies: Participants used different strategies by task: keyword search in BIOS and attention to a few predictive features in ICPSR, while generally avoiding discussion of Race.
  • 3.4 Virtual Pilot Studies: Interactive-console use varied sharply, with some participants avoiding it and others experimenting excessively; some found its added value limited relative to highlights.Dynamic highlights could help users who relied on visual cues, while others preferred numerical coefficients.

3.5 Large-scale Experiments with Crowdworkers

The large-scale study compares human-only, static, and interactive assistance across training and prediction phases. Crowdworkers are randomly assigned to explanation conditions and evaluated on random in-distribution and out-of-distribution examples.

  • 3.5 Large-scale Experiments with Crowdworkers: Participants complete training followed by prediction, with assistance types allowed to differ between phases to test whether training can support performance without real-time assistance.
  • 3.5 Large-scale Experiments with Crowdworkers: The design compares six combinations of None, Static, and Interactive assistance across training and prediction, including a human-only None/None condition.
  • 3.5 Large-scale Experiments with Crowdworkers: Each participant is randomly assigned to an explanation type and evaluated on 10 random in-distribution plus 10 random out-of-distribution examples; each task uses 1,296 participants.
  • 3.5 Large-scale Experiments with Crowdworkers: Interactive conditions took 12 minutes versus 7 minutes for non-interactive conditions, with participants allowed unlimited time to explore the interfaces.

4 RQ1: THE EFFECT OF IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION EXAMPLES ON HUMAN PERFORMANCE

Out-of-distribution examples produce a clear performance shift across tasks, narrowing the gap between human-AI teams and AI. Nevertheless, teams generally do not surpass AI, with results varying by task and explanation condition.

  • 4 RQ1: THE EFFECT OF IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION EXAMPLES ON HUMAN PERFORMANCE: The study compares human-alone and AI-alone performance across three tasks spanning different performance gaps, with AI performance estimated from the random samples used in the human-only condition.The sample-based AI estimate differs by no more than 1.2% from the test-set estimate.
  • 4 RQ1: THE EFFECT OF IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION EXAMPLES ON HUMAN PERFORMANCE: Human-AI teams generally do not outperform AI out-of-distribution, although accuracy gain is sometimes positive in ICPSR and COMPAS without statistical significance.BIOS shows significantly negative accuracy gain across all explanation types (p< 0.001).
  • 4 RQ1: THE EFFECT OF IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION EXAMPLES ON HUMAN PERFORMANCE: Out-of-distribution examples significantly reduce the performance gap between human-AI teams and AI across tasks, with the strongest accuracy gains generally occurring in recidivism prediction.The authors report a strong distribution-type effect across all tasks (p< 0.001).
  • 4 RQ1: THE EFFECT OF IN-DISTRIBUTION AND OUT-OF-DISTRIBUTION EXAMPLES ON HUMAN PERFORMANCE: Interactive/None produces the highest accuracy gain in ICPSR, whereas Interactive/Interactive produces a tiny negative gain, suggesting real-time interactive assistance may hurt performance there.

5 RQ2: AGREEMENT/TRUST OF HUMANS WITH AI

Agreement with AI depends on both distribution type and task. Out-of-distribution examples can reduce overtrust in recidivism prediction, while BIOS shows stronger appropriate agreement out-of-distribution.

  • 5 RQ2: AGREEMENT/TRUST OF HUMANS WITH AI: Humans agree with AI more in-distribution than out-of-distribution in ICPSR and COMPAS, but BIOS reverses this pattern.
  • 5 RQ2: AGREEMENT/TRUST OF HUMANS WITH AI: When AI predictions are wrong, humans are significantly less likely to agree out-of-distribution than in-distribution in ICPSR and COMPAS, but not BIOS.
  • 5 RQ2: AGREEMENT/TRUST OF HUMANS WITH AI: Out-of-distribution appropriate agreement generally exceeds overtrust, whereas in-distribution appropriate agreement exceeds overtrust only in BIOS.Appropriate agreement means agreeing with correct AI predictions; overtrust means agreeing with incorrect predictions.
  • 5 RQ2: AGREEMENT/TRUST OF HUMANS WITH AI: In BIOS, appropriate agreement is stronger out-of-distribution, while in ICPSR it is stronger in-distribution and reduced overtrust primarily narrows the team–AI gap out-of-distribution.COMPAS shows no significant in- versus out-of-distribution difference in appropriate agreement.

6 RQ3: THE EFFECT OF INTERACTIVE EXPLANATIONS

Interactive explanations increase perceived usefulness of AI assistance but do not reliably improve human-AI team performance and may reinforce bias.

  • Overall results: Interactive explanations do not improve human-AI team performance over AI alone, despite increasing perceived usefulness in COMPAS and ICPSR.Perception effects were significant in COMPAS and ICPSR but not BIOS.
  • Performance: Real-time assistance improves performance over no assistance in BIOS, but interactive explanations do not outperform AI alone.Explanation type significantly affects human performance in BIOS, while the main gain comes from showing predicted labels in real time.
  • Human agreement and overtrust: Interactive explanations produce the highest overtrust in ICPSR and do not reduce overtrust relative to static explanations.The same absence of reduced overtrust holds in COMPAS across both distribution types.
  • Feature perception: Participants using interaction select more demographic features and fewer computationally important features in ICPSR and COMPAS.Interaction increased selection of Age and Offense Type in ICPSR, and Race and Sex in COMPAS.
  • Human agreement and overtrust: Users disagree with AI predictions less often under Interactive/Interactive than Interactive/None, potentially explaining limited performance improvement.The authors hypothesize that interaction increases mistakes when participants disagree with AI predictions.

7 DISCUSSION

The discussion finds that distribution shift narrows the human-AI versus AI performance gap, while interactive explanations remain insufficient and task-dependent.

  • Out-of-distribution design: The human-AI team performance gap is smaller out-of-distribution than in-distribution, although complementary performance is not achieved.This pattern appears in recidivism prediction and BIOS despite differences in task difficulty and AI advantage.
  • Out-of-distribution design: The study’s out-of-distribution split uses separate age or length bins, whereas real distribution shifts may be continuous and subtle.The authors note that this makes the setup realistic but limits evaluation practice.
  • Interactive explanations: Interactive explanations improve perceived usefulness but fail to improve team performance and may reinforce human biases.The authors describe this observation as preliminary and call for caution before deployment.
  • Choice of tasks: Task differences complicate generalization, including opposite patterns of human agreement with AI predictions across recidivism prediction and BIOS.The authors identify task difficulty, performance gaps, and human expertise or confidence as possible relevant factors.
  • Limitations: The experiments focus on linear models, whose explanations can still be difficult for humans and may not represent challenges from complex deep models.The authors leave the role of model complexity for future work.
  • Limitations: The findings may depend on the human subject population, because university students are not necessarily representative of task decision makers.Both pilot and randomized experiments nevertheless showed difficulty identifying important features or using AI-identified patterns.

A HUMAN PERFORMANCE IN ABSOLUTE ACCURACY

The appendix presents absolute human performance across distribution types and explanation conditions for ICPSR, COMPAS, and BIOS.

  • Absolute accuracy: Figure 12 compares human-AI team performance across explanation types and in-distribution versus out-of-distribution examples.The caption identifies distribution type by bar color and uncertainty with 95% confidence intervals.

B COMPAS FIGURES

The COMPAS appendix reports accuracy gain, agreement with AI predictions, agreement by AI correctness, perceived usefulness, and perceived feature importance.

  • COMPAS performance and agreement: Figure 13 reports accuracy gain across COMPAS conditions, while Figure 14 reports agreement with AI predictions.Both figures distinguish distribution types by bar color and use 95% confidence intervals.
  • COMPAS performance and agreement: Figure 15 separates agreement with correct and incorrect AI predictions by distribution type and explanation condition.Striped bars indicate wrong AI predictions.
  • Perception and features: Figures 16 and 17 show subjective usefulness ratings and the percentage of participants who consider each COMPAS feature important.Feature order follows decreasing Spearman correlation with groundtruth labels in Figure 17.
  • Distribution setup: Figure 18 presents the ICPSR in-distribution versus out-of-distribution setup.

C IN-DISTRIBUTION VS. OUT-OF-DISTRIBUTION SETUP

The ICPSR setup splits examples into in-distribution and out-of-distribution groups by binning a selected attribute, then choosing which bins belong to each group. The final choice uses age ≥25 as in-distribution and age <25 as out-of-distribution because it produced the largest performance gap.

  • Candidate attributes are divided into 10 equal-size bins to explore different training and distribution-shift settings.
  • The final ICPSR split assigns age ≥25 to in-distribution examples and age <25 to out-of-distribution examples.
  • The study compares multiple bin ranges, including bins 1–5, 4–7, and 4–10 as in-distribution groups.
  • Option 3 was selected because it produced the largest performance gap between in-distribution and out-of-distribution examples.

D USER INTERFACE DESIGNS

The experiments combine static and interactive assistance interfaces with attention checks, feature quizzes, surveys, and timing measurements. Interactive assistance lets users edit defendant-profile features and observe changed AI predictions, while the study records task completion time and user responses.

  • Assistance interfaces: Static assistance for COMPAS provides feature highlights and AI predictions, while the interactive interface adds direct profile manipulation.
  • Interactive assistance: Interactive COMPAS assistance lets users edit Sex, Charge Degree, and Race in a defendant profile to observe changes in the AI prediction.
  • Attention checks: Attention checks require correct answers before users proceed to training, and one confusing recidivism question was revised after pilot feedback.
  • Feature quizzes: Training-phase feature quizzes ask users to identify positive and negative features from candidate features after each prediction.
  • Timing: Median task times were 9’55” for ICPSR, 9’16” for COMPAS, and 8’59” for BIOS.Medians were used because some users remained inactive for long periods.

E SURVEY QUESTIONS

The survey asks participants about their perceived accuracy, important features, influence of AI assistance, feedback, and demographic characteristics. Separate survey materials are identified for the ICPSR and COMPAS tasks.

  • Task perceptions: Participants report how many answers they and the AI answered correctly, along with the three features they considered most important.
  • AI influence: The survey asks whether AI assistance influenced each participant’s decision and requests further elaboration.
  • Feedback: Participants are invited to provide general feedback about the study.
  • Demographics: Demographic questions cover gender, age, and highest completed education level, with a prefer-not-to-answer option.
  • Survey scope: The survey-question figure is designated for the ICPSR and COMPAS tasks.
Loading 2101.05303v4…