Source-linked AI summary
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing
Sarah Wiegreffe, Ana Marasović
TL;DR
EXNLP research lacks a comprehensive review of how human textual explanations are collected, despite dataset collection being an under-scrutinized part of the ML pipeline. This paper reviews 65 datasets across highlights, free-text, and structured explanations, analyzes collection methodologies, and recommends quality-control practices for future construction. It concludes that collection choices can affect downstream modeling and that systematic quality controls, including two-stage collection and judgment, are useful.
Problem
Dataset collection is an under-scrutinized part of the ML pipeline, while prior XAI surveys have not comprehensively reviewed datasets with textual explanations across tasks.
Method
The paper reviews 65 EXNLP datasets, categorizes them into highlights, free-text, and structured explanations, analyzes collection methodologies, and synthesizes recommendations.
Results
The review identifies collection discrepancies with downstream modeling effects and finds that two-stage collect-and-judge procedures improve dataset quality while enabling quality ratings.
Takeaways & Limitations
Future EXNLP dataset construction should apply task-appropriate quality controls, including teaching and testing the underlying task and judging collected explanations.
Takeaways & Limitations
Some EXNLP datasets have weak annotation constraints and quality controls, and certain datasets remain low-quality even after re-annotation.
Abstract
from arXiv · showhide
Explainable NLP (ExNLP) has increasingly focused on collecting human-annotated textual explanations. These explanations are used downstream in three ways: as data augmentation to improve performance on a predictive task, as supervision to train models to produce explanations for their predictions, and as a ground-truth to evaluate model-generated explanations. In this review, we identify 65 datasets with three predominant classes of textual explanations (highlights, free-text, and structured), organize the literature on annotating each type, identify strengths and shortcomings of existing collection methodologies, and give recommendations for collecting ExNLP datasets in the future.
1 Introduction
The review addresses the under-scrutinized collection of EXNLP datasets by surveying textual-explanation literature, existing datasets, and collection practices. It organizes explanations and collection concerns to support recommendations for future dataset construction.
- Motivation: Dataset collection is the most under-scrutinized part of the ML pipeline, motivating critical evaluation of EXNLP collection practices.Poor data quality is associated with downstream data cascades, reported by 92% of ML practitioners.
- Scope and contributions: The survey reviews textual-explanation collection literature, summarizes prior learning, and recommends practices for future EXNLP dataset construction.
- Scope and contributions: The paper overviews 65 existing EXNLP datasets and provides a live, community-contributed version of its dataset tables.
- Scope and contributions: It examines highlight-collection discrepancies, assumptions behind free-text explanations, structured explanations, and quality-control strategies.
2 Explainability Lexicon
EXNLP datasets contain three main explanation types: highlights, free-text explanations, and structured explanations. These types differ in how closely they follow the input, how expressive they are, and how much structure the collection process imposes.
- Highlights: Highlights select input words, phrases, or sentences to explain a prediction, with criteria including compactness, sufficiency, and comprehensiveness.
- Highlights: Collected highlight datasets vary greatly in how well they reflect the community’s compactness, sufficiency, and comprehensiveness criteria.
- Free-text explanations: Free-text explanations are unconstrained textual justifications that are generally more expressive and readable than highlights.
- Free-text explanations: Free-text explanations are useful when explaining reasoning requires information outside the given input sentence or document.
- Structured explanations: Structured explanations remain natural-language explanations but impose constraints such as required inference rules, with no single common definition.
3 Overview of Existing Datasets
The review catalogs EXNLP datasets across highlights, free-text, and structured explanations, documenting their collection procedures and recurring dataset patterns. Free-text explanations are popular, while structured explanations use task-specific forms such as fact chains and constrained text.
- Dataset scope: 65 datasets are organized into highlights, free-text explanations, and structured explanations, with collection details reported for each dataset.The review also classifies annotation as crowd, automatic, or expert, student, and author annotation.
- Highlights: Highlight granularity varies with the task, ranging from words, phrases, or sentences to one or more paragraphs in longer documents.The review excludes document-retrieval datasets without specified explanation locations and excludes the retracted BEERADVOCATE dataset.
- Free-text explanations: Free-text explanations are popular across textual and visual-textual tasks and usually contain no more than a few sentences per instance.LIAR-PLUS is an exception, using conclusion paragraphs from web-scraped fact-checking summaries.
- Structured explanations: Structured explanations use dataset-specific forms, commonly chains of facts or semi-structured text constrained by required input elements.WORLDTREE combines fact chains and semi-structured text into explanation graphs, while EQASC and EOBQA use automatically extracted two-fact chains validated through crowdsourcing.
- Structured explanations: Structured explanation datasets include semi-structured reading-comprehension explanations and fact-chain annotations for datasets such as NATURALQUESTIONS, SQUAD, and HOTPOTQA.
4 Link Between EXNLP Data, Modeling, and Evaluation Assumptions
EXNLP data collection choices are linked to modeling and evaluation assumptions, so dataset documentation must distinguish valid explanation properties from choices made for particular evaluations. The review identifies sufficiency as necessary for highlights, while comprehensiveness primarily supports faithfulness evaluation.
- Evaluation assumptions: Highlight evaluation distinguishes plausibility, which concerns human agreement, from faithfulness, which concerns correspondence with a model’s decision process.Human highlights are used to assess plausibility through overlap, whereas faithfulness evaluates whether highlighted content represents the model’s process.
- Highlights: Highlight-collection instructions commonly encourage sufficiency and compactness but not comprehensiveness, producing inconsistent assumptions across datasets.The review notes differing characterizations of MOVIEREVIEWS, EVIDENCEINFERENCE, FEVER, and E-SNLI highlights.
- Sufficiency: Human highlights may be insufficient for highly accurate models, raising concerns about using existing gold highlights as valid explanations.The review specifically identifies neutral E-SNLI pairs and no-attack WIKIATTACK examples as problematic cases.
- Highlights: Sufficiency is necessary for a valid highlight, but comprehensiveness is mainly a means to make faithfulness evaluation feasible.Non-comprehensive highlights can remain valid, yet their mismatch with comprehensive model highlights complicates plausibility evaluation.
- Documentation: Data-collection constraints can influence modeling assumptions, so researchers should document which explanation properties are required for validity and which serve evaluation purposes.The review gives self-contained explanations and comprehensive highlights as examples of constraints that should not automatically be treated as universal requirements.
5 Rise of Structured Explanations
Free-text explanations can be expressive but are difficult to control, while pilot studies can reveal when structured explanations better match how people explain a task. The survey recommends embracing natural structure and documenting its effects on modeling.
- Free-text explanations offer expressivity but can suffer from underspecification and inconsistency because quality control is difficult.
- E-SNLI and COS-E control quality partly by asking annotators to highlight input words before writing explanations, while filtering template-like responses.E-SNLI re-annotated 11% of explanations detected against 56 templates.
- Re-annotation did not eliminate label-specific templates in E-SNLI, and models trained on those explanations also generated template-like outputs.
- Pilot studies should examine how people define and generate explanations before free-text collection; naturally structured explanations should be incorporated into the annotation scheme.
- When pilots reveal no obvious structure, stronger quality control remains necessary because low-quality explanations can bottleneck high-quality models.The survey contrasts E-SNLI with COS-E, whose weaker constraints and controls have been judged to create serious annotation issues.
6 Increasing Explanation Quality
Explanation quality depends on task understanding, annotation procedures, and quality control, with two-stage collection offering a general strategy for reducing artifacts. Automatically collected datasets can be cost-effective but may require judging or editing to address quality tradeoffs.
- Two-stage collection: Collect-and-edit can reduce annotator bias, improve quality control, and potentially reduce dataset artifacts by having annotators revise collected items.
- Automatic collection: Automatically collected visual-textual datasets provide scalable data but exhibit quality tradeoffs that crowdsourced judging or editing can reveal and mitigate.
- Crowdworkers need training and testing on the underlying task and label set because label understanding is prerequisite to collecting or judging explanations.
- People may still struggle to explain deception detection and recidivism prediction after intensive training, limiting the usefulness of explanations for some tasks.
- Post-hoc explanation collection assumes annotators accept assigned labels, but that assumption is inaccurate for several NLP tasks.
- Collecting labels and explanations from the same annotators is ideal, while checker questions provide an alternative for assessing label agreement.
7 Increasing Explanation Diversity
Explanation diversity requires broader annotator participation, multiple explanations per instance, and attention to subjective variation in what counts as a correct explanation. Modeling only one plausible explanation can unfairly penalize valid alternatives.
- A small annotator pool can let models overfit annotator characteristics; E-SNLI reports an average of 860 explanations written per worker.
- Representative data collection should prevent a few workers from dominating, using participation checks and, where appropriate, demographic or annotator-graph information.
- Human explanations can vary with cognitive biases, social expectations, and sociocultural background, so a single ground truth may not capture valid diversity.
- If multiple explanations are plausible, retaining only one can unfairly penalize model explanations that do not match it; multiple explanations may also provide a useful learning signal.
- Contrastive explanations: Highlights may be insufficiently intelligible for complex tasks, and the same limitation may apply to contrastive input edits.
- Contrastive explanations: Contrastive explanations could answer why one prediction was made instead of another, but contrastive free-text and structured datasets were not known to exist in this review.
- Future collections should use large annotator pools, multiple annotations per instance, end-user needs, and inter-annotator agreement alongside explanation plausibility.
8 Conclusions
The survey reviews EXNLP datasets, identifies how collection choices can affect downstream modeling, and synthesizes recommendations for future data collection. It also responds to a rapidly expanding literature by promoting standardization and serving as a resource for further discussion.
- The paper reviews existing EXNLP datasets, highlights collection discrepancies with downstream modeling effects, and synthesizes recommendations for future data collection.
- Most reviewed work originated in the last 1–2 years, motivating efforts toward standardization, consistency, and continued discussion among current and future data collectors.
A Complementing Information
The appendices provide supplementary material on dataset licensing, collection, model assumptions, benchmark quality measures, and commonsense reasoning.
- Appendix B details dataset licenses across the surveyed datasets.
- Appendix C provides additional details about dataset collection.
- Figure 2 illustrates connections between assumptions in self-explanatory highlighting models.
- Tables 6–8 summarize quality measures and outcomes for E-SNLI, COS-E, and VCR.
- Appendix D discusses explanation and commonsense reasoning.
B Dataset Licenses
Licensing information is incomplete for many surveyed datasets, while the reported licenses range from permissive standards to dataset-specific, restricted, or unspecified terms.
- 33.96% of cited papers do not report a dataset license, while 45.61% use common permissive licenses.
- WIKIQA uses the Microsoft Research Data License Agreement, and MULTIRC uses a Research and Academic Use License.
- Hanselowski et al.’s data archive is under Copyright.
- COQA combines MSR-LA and RACE licenses with permissive BY-SA 4.0 and Apache 2.0 licenses.
- The TACRED-based portion of Wang et al.’s dataset cannot be distributed, and the remaining license is unspecified.
- BDD-X, VCR, VLEP, WORLDTREE V1, WORLDTREE V2, and ECQA each specify dataset- or project-specific licenses.
C Dataset Collection
The survey assembled its dataset inventory through expert seeding, literature searches, ongoing discovery, and public contributions to a community website.
- The authors used domain expertise and prior publications to construct a seed list of datasets.
- They expanded the list through new publications and preprints encountered during the year before submission.
- They searched the ACL Anthology for “explain,” “interpret,” “explanation,” and “rationale,” especially in proceedings from 2020 onward.
- A public website opened 3.5 months before submission, and all received dataset suggestions were integrated into the tables.
D Explanation and Commonsense Reasoning
The survey distinguishes task-based explanations from commonsense inferences about observed events, while acknowledging overlapping datasets and limits to universal explanation criteria.
- Scope: The survey focuses on textual explanations of human decisions, defined as task labels.
- Scope: Commonsense datasets explain observed events or phenomena, where the distinction between class label and explanation is not defined.
- Scope: Commonsense datasets generally produce (input, label) rather than (input, label, explanation), so they do not fit task-based explanation modeling goals.
- Overlap: SBIC, VLEP, and VCR span both categories by collecting task labels alongside explanations.
- Scope: The survey does not cover every relevant property of human-authored explanations and does not expect universal criteria for appropriate explanations.
- Highlighting assumptions: In supervised highlighting, human highlights are treated as gold truth, but this basic approach does not support faithfulness evaluation.
- Highlighting assumptions: Non-comprehensive human highlights hinder plausibility evaluation for unsupervised model highlights because the two highlight types do not match by design.