PaperlayerView exact revision on arXiv

Source-linked AI summary

What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play

Shi Feng, Jordan Boyd-Graber

arXiv:1810.09648v3cs.AI

TL;DR

Interpretability remains difficult to evaluate, particularly in ways that reflect both model faithfulness and usefulness to people. The paper tests interpretations in cooperative Quizbowl play with experts and novices, finding that user skill matters because the groups trust and use assistance differently.

  • Problem

    Interpretability lacks an agreed evaluation standard balancing model faithfulness and usefulness to people, and predictions without justification can be hard for humans to interpret and act on.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    The authors measure how interpretations affect human performance in cooperative Quizbowl play with expert and novice participants.

    Source: §1 · p. 2

  • Results

    Experts and novices trust and use interpretations differently; experts can overrule wrong answers, while novices are more trusting and less able to discern useful help.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The results highlight the importance of considering user skill, and suggest that more intelligent combinations of interpretations and user adaptation could further improve cooperation.

    Source: §8 · p. 10

    Fixed visualization placement can produce uneven exposure, while changing visualizations from question to question may confuse users.

    Source: §7.4 · p. 9

Abstract

from arXiv · show

Machine learning is an important tool for decision making, but its ethical and responsible application requires rigorous vetting of its interpretability and utility: an understudied problem, particularly for natural language processing models. We propose an evaluation of interpretation on a real task with real human users, where the effectiveness of interpretation is measured by how much it improves human performance. We design a grounded, realistic human-computer cooperative setting using a question answering task, Quizbowl. We recruit both trivia experts and novices to play this game with computer as their teammate, who communicates its prediction via three different interpretations. We also provide design guidance for natural language processing human-in-the-loop settings.

1 INTRODUCTION

The paper argues that machine-learning interpretations should be evaluated with real users on real tasks, because their effectiveness for human decision-making remains difficult to assess. It evaluates interpretations in cooperative Quizbowl play and finds that experts and novices use computer assistance differently.

1 INTRODUCTION

  • In cooperative decision-making, predictions without justification and potentially overconfident neural-model scores can be difficult for people to interpret and act on.The paper frames interpretations as a way to augment human intelligence in pursuit of a shared goal.Source: §1 · pp. 1–2
  • Interpretability lacks standard evaluation data and ground truth, and its meaning is unsettled between model faithfulness and usefulness to people.Interpretation is also difficult to evaluate without the humans who will consume it.Source: §1 · p. 1
  • The authors evaluate interpretation by measuring how it improves human performance, using uncertainty, important-feature highlights, and relevant training examples.They focus on whether interpretations communicate model predictions effectively, leaving their faithfulness to future work.Source: §1 · p. 2
  • Experts can confidently overrule wrong computer answers, while novices trust assistance more and discern useful help from misleading help less well.Novices play more aggressively with computer assistance.Source: §1 · p. 2

2 RELATED WORK

Related work situates interpretation in human-AI cooperation, describes common ways to explain model predictions, and motivates evaluating interpretations through real tasks with real users.

2.1 Human-AI Cooperation

  • Human-AI cooperation depends on explanations that help people judge model predictions, while real-world use also raises fairness, accountability, and transparency concerns.The discussion connects interpretability to human-aware AI and human-robot teaming.Source: §2.1 · p. 2

2.2 Interpretation of Machine Learning Models

  • Interpretations commonly convey uncertainty, highlight important input features, or retrieve relevant training examples, with methods differing between linear and nonlinear models.For neural models, uncertainty estimates can be overconfident, and feature highlighting can use gradients or local linear approximations.Source: §2.2 · pp. 2–3

2.3 Evaluation of Interpretation

  • Interpretation evaluation remains unsettled because there is no agreed best balance between model faithfulness and usefulness to people.Application-grounded evaluation uses real tasks and users to assess whether interpretations improve human performance, but it requires a challenging task and motivated participants.Source: §2.3 · p. 3

3 INTERPRETATION TESTBED: QUIZBOWL

The paper uses Quizbowl as a cooperative testbed because it challenges both humans and computers and lets researchers observe how people use changing model interpretations while answering. Its linear model supports uncertainty scores, retrieved evidence, and question highlights.

3.1 Quizbowl and Computer Models

  • Quizbowl rewards correct early answers and penalizes incorrect buzzes, requiring players to balance aggressiveness with accuracy.Questions reveal difficult clues before easier ones, giving players repeated opportunities to decide when they have enough information.Source: §3.1 · p. 3
  • Humans and computers have complementary Quizbowl strengths, making cooperative play a test of whether interpretations help people judge the model’s guesses.The human, rather than the model, decides when the system has a good answer; each new word can change that judgment.Source: §3.1 · p. 3
  • The authors use an ElasticSearch-based linear Quizbowl guesser, whose relevant training examples provide a basis for natural interpretations.The system uses tf-idf features to retrieve training documents and takes their label as the answer.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Interpretation of a Question Answering Model

  • Unnormalized scores for the top ten guesses convey uncertainty, while retrieved training examples serve as evidence and supply words for question highlights.Evidence snippets let players judge relevance to the question; question highlights indicate whether the model attends to matching keywords.Source: §3.2 · p. 4

4 INTERFACE DESIGN

The interface centers the word-by-word question and places model interpretations around it. Guesses, evidence, and highlights are arranged to let players compare the question with the model’s cues.

4 INTERFACE DESIGN

  • The guesses panel lists the top ten answers in score order, helping players notice when the model appears uncertain.The scores are not probabilities; a low score across guesses can also signal uncertainty.Source: §4 · p. 4
  • Evidence presents relevant snippets from salient training examples for the top guess, positioned below the question for direct comparison.Each line displays a snippet from one selected document.Source: §4 · p. 4
  • Highlights mark salient words in both the question and evidence, and showing both together gives players cues in each text.The design recognizes that visualizations may not receive equal exposure because some are less intuitive to display.Source: §4 · p. 4
  • The word-by-word question and its highlights occupy the center panel, with guesses on the left and evidence below.The layout aims to minimize distraction from the question while keeping interpretations around it.Source: §4 · p. 4, Figure 1 · p. 5

5 SETUP

The setup pairs human Quizbowl players with a computer teammate and controls which interpretations they see. It compares novices and experts using new questions and randomized interpretation combinations.

5.1 Data and Participants

  • The evaluation uses 160 new questions and recruits 40 Quizbowl experts and 40 novices under different participation requirements.Source: §5.1 · p. 5

5.2 Human-AI Cooperation on Quizbowl

  • Players control a computer teammate that updates guesses and interpretations as a question appears word by word, and can buzz before the question is fully read.Source: §5.2 · p. 5

5.3 Controlling Which Interpretations to Show

  • Highlight and Evidence improve accuracy for novices and experts, while combining interpretations has a more positive effect for experts; guesses+evidence has a negative combo effect for novices.Source: Figure 2 · p. 6
  • Interpretation combinations are sampled to balance their expected distribution across players, questions, and combinations; expert players may see different combinations for the same question.Each player answers each question only once.Source: §5.3 · p. 6

6 RESULTS

The regression analysis finds that the most helpful interpretation differs by player expertise, while interpretations also change when players buzz. Examples show both evidence helping experts answer earlier and expert players rejecting a computer prediction despite supporting evidence.

6 RESULTS

  • The analysis uses separate expert and novice cohorts, with 30 experts answering 1,983 questions and 30 novices answering 600 after filtering.The results section first uses regression to assess accuracy, then examines buzzing behavior and cases where interpretations succeeded or failed to convince players.Source: §6 · p. 6

6.1 Regression Analysis

  • Experts benefit most from highlight, while novices benefit most from evidence; highlight plus evidence adds expert gain but no extra novice benefit, potentially due to information overload.The authors suggest experts use evidence more effectively, while question highlighting requires less multitasking.Source: §6.1 · p. 8
  • Experts have higher baseline accuracy, greater within-group accuracy variance, and greater sensitivity to question difficulty, but are less affected by interpretations.The baseline is accuracy without interpretation; the comparison is against the combined effect of all three interpretations.Source: Figure 3 · p. 6, §6.1 · p. 8

6.2 How Interpretations Change Player Behavior

  • Interpretations shift novices toward expert-like buzzing positions despite lower accuracy, suggesting novices trust the computer too much and play too aggressively for their skill level.Buzzing positions shift earlier with interpretations, and correct answers occur earlier; the earlier shift is especially marked for novices using highlights.Source: §6.2 · p. 8

6.3 Successes and Failures of Interpretations

  • With evidence, three experts answered the Civil Disobedience question correctly before 50% of its length, compared with an earliest correct answer at 72% without evidence.Twelve experts answered the question, eight correctly; the authors say evidence with highlights helped players infer Thoreau and judge the guess likely correct.Source: Figure 7 · p. 8, §6.3 · p. 8
  • Three experts rejected the computer’s correct Copernicus guess before 50%, although the evidence supported it; its different wording and a confusing “not” complicated the example.The evidence concerned Copernicus’s book preface, Ad Lectorem.Source: Figure 8 · p. 8, §6.3 · pp. 8–9

6.2 How Interpretations Change Player Behavior

  • Visualizations alone do not help novices distinguish useful from misleading assistance; they become too aggressive and trust computer predictions too much.The authors argue that visualizations should also communicate whether a prediction should be trusted.Source: §6.2 · p. 8

7 DISCUSSION

The discussion frames the evaluation as realistic but task-specific, then considers how its framework may transfer and identifies limitations in interface design and attributing performance changes. It also outlines finer-grained and adaptive evaluation as future directions.

7 DISCUSSION

  • The Quizbowl evaluation is realistic but task-specific, so its findings are grounded in a particular task setting.The discussion presents direct application to other settings as a topic, not as evidence that the results generalize unchanged.Source: §7 · p. 9

7.1 Forms and Methods of Interpretation

  • The framework can compare different interpretation methods within one form, including approximate interpretations for neural models where faithfulness matters.The paper’s study compares forms using one method per form; the same interface and regression analysis can be applied to methods within a form.Source: §7.1 · p. 9

7.2 Intrinsic and Extrinsic Evaluation

  • Extrinsic evaluation measures usefulness through a task, while intrinsic evaluation is more direct but can depend on a simpler benchmark model that performs similarly.The benchmark approach does not apply to the paper’s application-grounded setting; extrinsic evaluation also incorporates human trust.Source: §7.2 · p. 9

7.3 Generalizing to Other Tasks

  • The interface can be adapted to other text-classification tasks by presenting inputs incrementally, and simultaneous translation offers a related accuracy–delay trade-off.The authors note that Quizbowl’s characteristics make it uniquely suitable for their setting.Source: §7.3 · p. 9

7.4 Limitations

  • Fixed visualization placement avoids confusing players but creates uneven exposure, and a single display location may not suit every visualization.Input highlights should appear alongside the input, whereas evidence is best placed next to it for comparison.Source: §7.4 · p. 9
  • Changing visualization layouts may confuse users; the authors suggest warm-up questions or keeping layouts stable across questions to reduce context switches and cognitive load.The study used a tutorial, but the authors say this could be improved with warm-up questions.Source: §7.4 · p. 9
  • Performance improvements cannot be attributed to interpretation use because the study does not measure how much attention players give each visualization.The authors propose eye tracking as a way to measure that attention in future work.Source: §7.4 · p. 9

7.5 Future Work

  • Future work could assess specific highlighting choices and visualization attention, then use task performance to adapt interface elements through reinforcement learning.Proposed adaptations include changing font sizes, adding explicit agreement or disagreement buttons, and modifying highlighting strategies.Source: §7.5 · pp. 9–10

8 CONCLUSION

The paper evaluates interpretations in human–AI cooperation using Quizbowl and finds that experts and novices trust and use them differently. It concludes that user skill matters and that more intelligent, user-adapted combinations may further improve cooperation.

8 CONCLUSION

  • The evaluation uses a question-answering task derived from Quizbowl to study interpretation methods in human–AI cooperation.Source: §8 · p. 10
  • Experts and novices trust and use interpretations differently, making user skill important to a realistic evaluation of interpretability.Source: §8 · p. 10
  • Combining interpretations more intelligently and adapting them to users may further improve human–AI cooperation.Source: §8 · p. 10
Loading 1810.09648v3…