PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Human-AI Collaboration With Descriptions of AI Behavior

Ángel Alexander Cabrera, Adam Perer, Jason I. Hong

arXiv:2301.06937v1cs.HCcs.AI

TL;DR

People may rely too much or too little on AI predictions, and mental models of AI behavior can help guide when to accept or override them. This paper tests behavior descriptions—subgroup performance information—in three domains and finds they can improve human-AI accuracy through failure correction and greater reliance when AI is more accurate.

  • Problem

    People can under- or over-rely on AI predictions, while developing accurate mental models is challenging for black-box, stochastic systems whose behavior may differ from human expectations.

    Source: §1 · p. 1

  • Method

    The authors tested subgroup performance descriptions that inform users about AI behavior in user studies spanning fake review detection, satellite image classification, and bird classification.

    Source: §1 · p. 2, §8 · p. 16

  • Results

    Behavior descriptions significantly improved human-AI team performance by helping users correct AI failures and rely on AI when it is more accurate.

    Source: §1 · p. 2, §8 · p. 16

  • Takeaways & Limitations

    The findings highlight the importance of people’s mental models and indicate that informing users about high-level AI behaviors can improve performance with AI aids.

    Source: §6 · p. 13, §8 · p. 16

    The study focused on relatively simple classification tasks, while behavior descriptions for more complex tasks may look significantly different.

    Source: §7 · p. 15

Abstract

from arXiv · show

People work with AI systems to improve their decision making, but often under- or over-rely on AI predictions and perform worse than they would have unassisted. To help people appropriately rely on AI aids, we propose showing them behavior descriptions, details of how AI systems perform on subgroups of instances. We tested the efficacy of behavior descriptions through user studies with 225 participants in three distinct domains: fake review detection, satellite image classification, and bird classification. We found that behavior descriptions can increase human-AI accuracy through two mechanisms: helping people identify AI failures and increasing people's reliance on the AI when it is more accurate. These findings highlight the importance of people's mental models in human-AI collaboration and show that informing people of high-level AI behaviors can significantly improve AI-assisted decision making.

1 INTRODUCTION

People rely on mental models of AI behavior to decide when to accept or override predictions, but AI’s black-box, stochastic nature makes those models difficult to develop. The paper proposes behavior descriptions of subgroup performance and reports that they can improve human-AI accuracy through failure correction and increased reliance when AI is more accurate.

1 INTRODUCTION

  • People use mental models of AI behavior to decide when to rely on or override predictions, and inaccurate models can lead to systematic failures or worse-than-unassisted performance.The paper notes that clinicians shown incorrect AI predictions have made more errors than average.Source: §1 · p. 1
  • Black-box, stochastic AI and mismatches between human expectations and actual model behavior make it difficult for people to develop appropriate mental models.Developers must test and interpret what a model has learned before encoding those insights for end-users.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Behavior descriptions report how an AI performs on instance subgroups, describing output patterns rather than why a particular output was produced.The authors distinguish this approach from explainable AI and say the two can be used together.Source: §1 · p. 2
  • The study tested behavior descriptions in fake review detection, satellite image classification, and bird classification to examine effects across distinct domains.The authors argue that understanding end-users’ mental models is essential for understanding team performance and which decision aids may succeed.Source: §1 · pp. 2–3
  • Behavior descriptions can significantly improve human-AI accuracy by helping users correct AI failures and rely more on AI when it is more accurate.The reported effects are not universal: they depend on how obvious failures are and whether users can correct them.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The related work connects behavior descriptions to research on mental models, explainable AI, and tools for analyzing model behavior. It positions them as high-level descriptions of what patterns a model produces, complementary to approaches that explain why it produces particular outputs.

2.1 Human Factors and AI Aids

  • Behavior descriptions complement work on mental models and reliance by adding information about when to rely on an AI; they can also be used alongside tutorials and trust-calibration techniques.The paper contrasts its semantic, high-level descriptions with exemplar-based teaching of model behavior.Source: §2.1 · p. 3

2.2 Explainable and Interpretable AI

  • Behavior descriptions describe what output patterns a model tends to produce for subsets of instances, whereas post-hoc explanations such as LIME, Anchors, and MUSE approximate why outputs occur.The paper notes that post-hoc explanations have sometimes had small or detrimental effects on AI-assisted decision making.Source: §2.2 · p. 4

2.3 Behavioral Analysis of AI Systems

  • Behavior descriptions can be created from behavioral-analysis tools that surface model performance on subsets, including tools for discovering biases, errors, and high-loss slices.The authors note that analyses can be rerun after model updates to update descriptions and inform end-users about changes.Source: §2.3 · p. 4

3 BEHAVIOR DESCRIPTIONS

Behavior descriptions communicate how a model performs on meaningful subgroups so users can decide when to rely on or override its output. The paper’s initial design principles emphasize actionable, simple, and significant descriptions, and the authors present them as an intermediate aid when model failures are difficult to fix.

3 BEHAVIOR DESCRIPTIONS

  • Behavior descriptions report model performance, patterns, or potential failures for a subgroup of instances, with forms that can vary across tasks and datasets.Their goal is to help users rely on AI when it is more accurate and override it when it is more likely to fail.Source: §3 · pp. 4–5

3.1 Principles for Effective Behavior Descriptions

  • The study’s initial design principles are to make descriptions actionable, simple, and significant, focusing on concrete reliance or override decisions and impactful behaviors.The authors cite concerns about information overload, alert fatigue, and cognitive load as reasons to keep descriptions short and limited.Source: §3.1 · p. 5

3.2 Why Not Just Fix AI Failures?

  • Behavior descriptions can serve as an intermediate aid because stochastic, black-box model failures may be difficult to fix and updates can introduce new failures or disrupt users’ expectations.Fixing failures can require costly training data, while updated descriptions can accompany a new model to update users’ mental models.Source: §3.2 · p. 6

4 EXPERIMENTAL DESIGN

The study uses human-subject experiments across three classification tasks to test how behavior descriptions affect collaboration. Participants encounter AI-only, no-AI, or AI-plus-description conditions, with controlled low-accuracy subgroups; the hypotheses concern team accuracy, correction of subgroup errors, and perceptions of AI.

4 EXPERIMENTAL DESIGN

  • The experiments span tasks with different data types, task types, and human accuracies to reduce the chance that findings are tied to one domain.The three tasks are fake review detection, satellite image classification, and bird classification.Source: §4 · p. 6, §4.1.2 · p. 8

4.1 Experimental Setup

  • Each task compares no AI, AI alone, and AI plus behavior descriptions, with descriptions shown for 10 of 30 instances in designated groups.Participants labeled 20 overall-dataset instances and five instances from each of two lower-performing subgroups.Source: §4.1 · p. 6

4.1.1 Behavior descriptions and wizard-of-oz AI.

  • A wizard-of-oz system fixed observed AI accuracy by subgroup while randomizing the specific instances shown to participants.The 20-instance main group had 95% accuracy, while the two five-instance behavior-description groups had 40% and 20% accuracy; participants were told overall accuracy was 90%, though actual overall accuracy was 73.33%.Source: §4.1.1 · pp. 7–8

4.1.2 Classification tasks.

  • The tasks use text reviews and two image-classification datasets, with behavior-description groups chosen to represent potentially difficult cases in each task.Examples include short or exclamation-heavy reviews, cloud or glacier and meadow or golf-course images, and visually similar bird species.Source: §4.1.2 · pp. 8–9

4.2 Hypotheses

  • The hypotheses predict higher team accuracy with behavior descriptions, improved correction of errors in described groups, and higher trust, helpfulness, and future-use intentions.The proposed accuracy mechanism is that descriptions help participants detect and correct systematic AI failures.Source: §4.2 · p. 9

5 RESULTS

Behavior descriptions significantly increased human-AI accuracy in two of three tasks, but effects varied across tasks and instance groups. They changed how participants used AI predictions without significantly changing reported perceptions of the AI.

5 RESULTS

  • Accuracy conditions were assessed with ANOVA and Tukey tests, Likert responses with Mann–Whitney U tests, and learning with linear models and Bonferroni corrections.The significance cutoff was p = 0.05.Source: §5 · p. 10

5.1 Overall Accuracy

  • Two of three tasks showed significantly higher team accuracy with behavior descriptions: reviews and birds, but not satellite classification.Reviews improved relative to No AI, while birds achieved complementary human-AI accuracy; satellite accuracy did not significantly differ across conditions.Source: §5.1 · p. 10

5.2 Accuracy by Behavior Description Group

  • Figure 5’s subgroup breakdown was mixed: reviews and satellite gains centered on main-group AI reliance, while behavior descriptions improved bird group 2.Reviews and satellite showed no condition differences within behavior-description groups; bird group 2 accuracy was higher with AI + BD than with AI alone.Source: Figure 5 · p. 11, §5.2 · pp. 11–12

5.3 Qualitative Results

  • Participants’ reported trust, helpfulness, and willingness to use the AI again did not significantly differ between AI and AI + BD conditions.This held across tasks despite differences in how participants used AI predictions.Source: §5.3 · p. 12, Figure 6 · p. 12

5.4 Additional Findings

  • Instance-time measurements were inconsistent, with high variance and many outliers, likely because AMT workers took breaks or searched for new tasks.The authors suggest incentivizing quick responses in future studies to improve time data.Source: §5.4 · p. 12
  • AI + BD reviews and birds conditions showed significant learning effects, matching the tasks where behavior descriptions improved accuracy.The reported learning slopes were 0.0062 for reviews and 0.0031 for birds.Source: §5.1 · p. 10, Figure 7 · p. 13, §5.4 · p. 13
  • In satellite classification, the AI + BD learning curve started above No AI but ended at a similar point, suggesting descriptions helped bootstrap learning.The authors suggest participants learned correct labels or when the AI tended to fail over time.Source: §5.4 · p. 13
  • Exploratory comments were inconsistent overall, but bird-task participants using AI alone described a correct general failure pattern: main class right, subclass wrong.More AI + BD participants described failures, but comments showed few significant differences between conditions.Source: §5.4 · pp. 12–13

6 DISCUSSION

Behavior descriptions improved human-AI accuracy through increased AI reliance on instances where it was more accurate, while helping users correct AI failures inconsistently. Their effects also extended to learning and raised questions about how to create descriptions and understand users’ mental models.

6.1 Effectiveness of Behavior Descriptions

  • Behavior descriptions improved team accuracy through two complementary effects: users corrected some AI failures and relied more on AI for instances where it was more accurate.The more consistent effect was increased reliance on instances outside behavior-description groups in the reviews and bird tasks.Source: §6.1 · pp. 13–14
  • Failure correction varied across tasks and groups, and was often not the main driver of improved overall accuracy.Reviews showed no significant accuracy difference within either described group, whereas users corrected failures in one bird-classification group.Source: §6.1 · pp. 13–14
  • Satellite classification showed no significant accuracy increase, which the authors attribute to high human baseline accuracy and limited room for improvement.Participants also detected and corrected AI errors in described groups without prompting.Source: §6.1 · p. 14

6.2 Learning and Behavior Descriptions

  • In reviews and bird classification, participants in the AI-plus-descriptions condition learned more quickly, including when to override the AI.Descriptions also raised initial accuracy in satellite and bird tasks by making failures apparent from the start.Source: §6.2 · p. 14

6.3 Authoring Behavior Descriptions

  • Authoring tools could support descriptions beyond failures, including consistent output patterns or areas where models outperform people.The authors suggest that quality-assurance and domain teams, and potentially aggregated end-user reports, could help generate descriptions.Source: §6.3 · pp. 14–15

6.4 Understanding and Improving Mental Models of AI

  • More detailed mental models were associated with greater reliance on AI, including when users developed those models through experience.The authors propose cognitive models of how people learn AI behavior as one route to designing decision aids.Source: §6.4 · p. 15
  • Sparse decision trees and methods that identify clusters of high error could help models provide behavior descriptions directly.The authors present these as possible directions for designing more collaborative systems.Source: §6.4 · p. 15

7 LIMITATIONS AND FUTURE WORK

The study’s findings are bounded by its participants, simulated setting, use of subgroup accuracy as the sole behavior-description type, and focus on classification tasks.

7 LIMITATIONS AND FUTURE WORK

  • Participants were Mechanical Turk workers with limited expertise, so findings may differ for professionals such as doctors or lawyers.Experts may notice AI failures more often or be less influenced by behavior descriptions.Source: §7 · p. 15
  • The experiments used a controlled, simulated setting, so effects may differ in real-world situations with consequential outcomes.The authors note that higher error costs could change how people update mental models or act when shown descriptions.Source: §7 · p. 15
  • The study tested only subgroup accuracy descriptions, leaving other forms and their effective features for future comparison.Possible alternatives include highlighting high-accuracy groups, describing features correlated with failure, or suggesting alternative labels.Source: §7 · p. 15
  • The study focused on classification, while behavior descriptions for more complex tasks may look substantially different.For image captioning, the authors suggest descriptions could focus on grammatical issues and object references rather than statistical metrics.Source: §7 · pp. 15–16

8 CONCLUSION

The paper introduces behavior descriptions and reports that they can improve human-AI team performance across three studied domains. The findings point to mental models as an important part of collaboration and motivate behavior-based aids.

8 CONCLUSION

  • Across user studies with 225 participants in three domains, behavior descriptions improved team performance by helping people correct AI failures and rely on AI when it was more accurate.The authors conclude that methods directly improving people’s mental models can improve performance when using AI aids.Source: §8 · p. 16

A ADDITIONAL STUDY DETAILS

The appendix provides additional study materials and participant counts to support reproducibility.

A ADDITIONAL STUDY DETAILS

Loading 2301.06937v1…