PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Diversity and Limits of Human Explanations

Chenhao Tan

arXiv:2106.11988v2cs.CLcs.AIcs.CY

TL;DR

NLP datasets increasingly collect human explanations, but it remains unclear whether they capture valid reasoning or can uniformly serve as ground truth. The paper synthesizes psychology perspectives, distinguishes explanation types, and examines how collection questions and human limits shape what explanations can provide.

  • Problem

    Human explanations are increasingly collected in NLP, but their validity as reasoning and their suitability as uniform ground truth remain open questions.

    Source: §1 · p. 1

  • Method

    The paper draws on psychology literature to classify NLP explanations and examine proxy questions and human cognitive limits.

    Source: §1 · p. 1, §2 · p. 2, §4 · p. 5

  • Results

    NLP explanations encompass proximal mechanisms, evidence, and procedures, while proxy questions vary and human prediction does not ensure valid explanations.

    Source: §1 · p. 1, §2 · p. 2, §3 · p. 3

  • Takeaways & Limitations

    Characterizing diverse human explanations before treating them as uniform labels can inform how they are collected and used in NLP.

    Source: §5 · p. 5

    Humans may provide explanations for incorrect labels or tasks they perform poorly on, and explanation can require deeper understanding than prediction.

    Source: §4 · p. 5

Abstract

from arXiv · show

A growing effort in NLP aims to build datasets of human explanations. However, the term explanation encompasses a broad range of notions, each with different properties and ramifications. Our goal is to provide an overview of diverse types of explanations and human limitations, and discuss implications for collecting and using explanations in NLP. Inspired by prior work in psychology and cognitive sciences, we group existing human explanations in NLP into three categories: proximal mechanism, evidence, and procedure. These three types differ in nature and have implications for the resultant explanations. For instance, procedure is not considered explanations in psychology and connects with a rich body of work on learning from instructions. The diversity of explanations is further evidenced by proxy questions that are needed for annotators to interpret and answer open-ended why questions. Finally, explanations may require different, often deeper, understandings than predictions, which casts doubt on whether humans can provide useful explanations in some tasks.

1 Introduction

The paper questions whether human explanations in NLP reliably capture valid reasoning or can serve uniformly as ground truth. It argues that explanation types and human cognitive limits matter when collecting and using these datasets.

  • The paper introduces psychology literature to challenge broad use of “explanation” and assumptions that human explanations provide valid reasoning for labels.Source: §1 · p. 1
  • Human explanations differ in what they answer, so they may not all serve the same goals, including use as ground-truth labels.Source: §1 · pp. 1–2
  • Proxy questions reveal that open-ended “why” prompts are underspecified, with annotators asked to identify different kinds of relevant words.Source: §1 · p. 1
  • Prediction does not entail explanation, and everyday explanations are incomplete accounts of deduction.Source: §1 · pp. 1–2

2 Types of Human Explanations in NLP

The paper groups NLP human explanations by the information they convey: proximal mechanisms, evidence, and procedures. These categories differ in explanatory content and in how they may be collected or used.

2 Types of Human Explanations in NLP

  • Proximal mechanisms describe how to infer a label from text, but typically provide partial rather than complete mechanisms.Source: §2 · p. 2
  • Evidence identifies relevant input tokens but does not explain how they lead to the label.Source: §2 · p. 2
  • Procedures give explicit, step-by-step rules grounded in input, but are distinct from mechanisms and apply only to some tasks.Source: §2 · p. 2
  • Proximal mechanisms may be difficult to collect and use for supervision or evaluation, while evidence may serve as additional supervision or ground truth.Source: §2 · p. 2
  • Procedures are instructions rather than explanations in psychology, and their study connects to learning from instructions.Source: §2 · pp. 2–3

3 Proxy Questions Used to Collect Human Explanations

Open-ended why questions often need proxy questions, whose wording shapes what annotators provide. The questions vary across tasks, and their effects on the collected explanations remain unclear.

3 Proxy Questions Used to Collect Human Explanations

  • Proxy questions are often needed to make open-ended why questions answerable, and different forms entail different properties of the collected explanations.The authors argue that token- and sentence-level annotations can produce substantially different forms.Source: §3 · pp. 3–4
  • For sentiment highlights, Zaidan et al. ask for the most important words that would tell someone to see the movie, while Sen et al. ask for all words reflecting sentiment.The questions differ both in asking for the “most important” versus “all” words and in their framing of the task.Source: §3 · p. 3
  • In personal attack detection, the negative class lacks textual evidence, so annotation questions almost exclusively target the positive class.The paper notes that this asymmetry may require special strategies for training models and evaluating machine rationales.Source: §3 · pp. 3–4
  • Document-query classification studies generally use similar sentence-selection questions, but some request rationales, “strong” reasons, or question-answer-sentence indices.The effects of differences in annotation processes and question phrasing remain unclear.Source: §3 · pp. 3–4

4 Can Humans Provide Explanations?

The paper argues that predicting a label does not ensure people can explain it: explanations may require greater depth, accurate reports of mental processes, and complete deductions that people may not provide.

4 Can Humans Provide Explanations?

  • Most human-explanation datasets assume people can predict labels and focus on emulation tasks, although gaming explanations are noted as an exception where people may choose sub-optimal actions.The paper distinguishes emulation, which models human intelligence, from discovery, where models may find patterns people do not recognize.Source: §4 · p. 5
  • Explanation requires more depth than prediction, so people may make common-sense judgments without being able to explain them in detail.The paper gives the example of explaining why one should not slap a stranger through a theory of morality.Source: §4 · p. 5
  • People's verbal reports of subjective mental processes can be highly inaccurate, and they may be unaware of biases that shaped their judgments.The paper cites admission decisions as an example of legitimate information being used to justify preferences based on illegitimate factors such as race.Source: §4 · p. 5
  • Human explanations are necessarily incomplete because people do not present every deductive step from axioms; in natural language inference, they may simply repeat the hypothesis.The paper says incomplete explanations can still be useful if their incompleteness is understood and characterized.Source: §4 · p. 5
  • Humans may provide explanations for incorrect labels or tasks they perform poorly on, making their explanations questionable as training supervision or evaluation groundtruths.The paper cautions against assuming people can supply valuable proximal mechanisms.Source: §4 · p. 5

5 Conclusion

The conclusion urges NLP researchers to study human explanations as complex phenomena and characterize them before treating them as uniform or optimizing leaderboards.

5 Conclusion

  • Human explanations are actively studied in psychology, cognitive science, and other social sciences.The authors frame NLP's growing interest as an opportunity to learn from these fields.Source: §5 · p. 5
  • The authors encourage NLP researchers to explore how humans approach explanations and contribute to understanding explanation processes.They expect this work to inform how human explanations are collected and used in NLP.Source: §5 · p. 5
  • The authors propose characterizing human explanations before assuming they are all equal or pursuing a leaderboard.Source: §5 · p. 5

A Proxy Questions for Proximal Mechanisms and Procedure

Studies collecting proximal mechanisms often use explicit why questions and structured guidelines, while procedural studies may decompose tasks differently and sometimes omit exact questions.

A Proxy Questions for Proximal Mechanisms and Procedure

B Detailed Proxy Questions

Proxy questions shape what annotators provide: they range from evidence selection to rationales, procedures, and question answering. The instructions also vary in how they define sufficient support and constrain annotation.

B Detailed Proxy Questions

  • For question answering, explanations count as procedures when they explicitly give step-by-step guides grounded in the input.The distinction from proximal mechanisms can be subtle.Source: B Detailed Proxy Questions · p. 8
  • Movie-review rationale prompts ask for words supporting a positive or negative recommendation, while advising annotators to mark convincing support without marking everything.Annotators were advised not to spend more than 2–3 minutes per claim.Source: B Detailed Proxy Questions · p. 9
  • Yang et al. (2018) ask five annotators per paragraph for 3–5 answerable questions requiring multiple sentences, plus answers and sentence indices.Source: B Detailed Proxy Questions · p. 10
  • For fact verification, workers first classify evidence text snippets as supporting, refuting, or stance-free, then select only sentences matching a non-neutral stance.Source: B Detailed Proxy Questions · p. 10
  • For long-answer identification, annotators select the earliest HTML element sufficient to infer an answer, or mark no answer when no allowed single element suffices.Allowed elements include paragraphs, tables, list items, or whole lists.Source: B Detailed Proxy Questions · p. 10
  • Evidence sets for claims should be minimal, and an abstract may contain multiple distinct sets that support or contradict the claim.Source: B Detailed Proxy Questions · p. 10
  • Relevance-assessment prompts request 2–3 webpage sentences supporting a decision, but permit a response that text did not help when no text or images informed it.Source: B Detailed Proxy Questions · p. 10

Related papers

Loading 2106.11988v2…