PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Faithfully Rationalize by Construction

Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, Byron C. Wallace

arXiv:2005.00115v1cs.CLcs.AIcs.LG

TL;DR

Neural text models make it difficult to determine which inputs informed a prediction, while prior discrete rationale methods complicate training. FRESH separates feature scoring, rationale extraction, and prediction so the final classifier sees only extracted snippets; empirically, it outperforms existing models, recovers most black-box performance, and produces rationales at least as plausible to humans.

  • Problem

    Contextualized neural text models make it difficult to identify which input tokens contributed to a specific prediction, while prior discrete rationale methods are difficult to train.

  • Method

    FRESH independently extracts rationales from binarized feature-importance scores and trains a prediction model exclusively on the extracted text.

  • Results

    FRESH outperforms existing models, recovers most of the original black-box model’s performance, and produces rationales at least as plausible to human users as comparable end-to-end methods.

  • Takeaways & Limitations

    Independent extraction and prediction provide faithful explanations by construction while allowing complex contextualized models and simplifying training.

  • Takeaways & Limitations

    The framework treats explainability as instance-specific and cannot currently prune or expand the selected token set after rationale extraction.

Abstract

from arXiv · show

In many settings it is important for one to be able to understand why a model made a particular prediction. In NLP this often entails extracting snippets of an input text `responsible for' corresponding model output; when such a snippet comprises tokens that indeed informed the model's prediction, it is a faithful explanation. In some settings, faithfulness may be critical to ensure transparency. Lei et al. (2016) proposed a model to produce faithful rationales for neural text classification by defining independent snippet extraction and prediction modules. However, the discrete selection over input tokens performed by this method complicates training, leading to high variance and requiring careful hyperparameter tuning. We propose a simpler variant of this approach that provides faithful explanations by construction. In our scheme, named FRESH, arbitrary feature importance scores (e.g., gradients from a trained model) are used to induce binary labels over token inputs, which an extractor can be trained to predict. An independent classifier module is then trained exclusively on snippets provided by the extractor; these snippets thus constitute faithful explanations, even if the classifier is arbitrarily complex. In both automatic and manual evaluations we find that variants of this simple framework yield predictive performance superior to `end-to-end' approaches, while being more general and easier to train. Code is available at https://github.com/successar/FRESH

1 Introduction

FRESH addresses the difficulty of identifying faithful explanations in contextualized neural text models by separating rationale extraction from prediction. It uses feature-importance scores to train extraction and lets a separate predictor operate only on extracted snippets.

  • 1 Introduction: Contextualized representations can encode information from many input tokens, making token-level importance scores misleading as explanations.Attention weights over contextualized embeddings may reflect unknown interactions among input tokens rather than individual-token importance.
  • 1 Introduction: Faithful explanations matter because they should reflect the information actually used for a prediction, including whether sensitive features influence decisions.The paper highlights faithfulness as distinct from explanations that merely appear acceptable to humans.
  • 1 Introduction: FRESH separates extraction and prediction modules so the predictor receives only selected inputs, making the rationale faithful by construction.This discrete selection permits an arbitrarily complex prediction network while constraining the text available to it.
  • 1 Introduction: Prior end-to-end rationale methods are difficult to train because discrete selection requires reinforcement learning or reparameterization, which can produce high variance and sensitivity to hyperparameters.Reparameterization methods may zero out token embeddings, potentially harming transformer training when lower layers are not fine-tuned.
  • 1 Introduction: Feature-importance scores from attention or gradient methods are discretized into rationales, and an independent network is trained exclusively on those extracted snippets.The scoring mechanisms need not themselves satisfy the faithfulness criterion.
  • 1 Introduction: FRESH rationales were also evaluated for human plausibility through readability judgments and human classification of instances based on the extracted text.The paper presents an example of extracted rationales from MultiRC in Figure 1.

2 Related Work

Related work distinguishes faithfulness from plausibility and develops rationale-based explanations through hard attention or human evaluation. These approaches establish the goals FRESH targets while exposing the training difficulty of jointly learned discrete selection.

  • 2 Related Work: Faithfulness concerns whether explanations reflect a model’s decision process, whereas plausibility concerns whether humans find them acceptable or useful.Prior work also frames rationales in terms of sufficiency, brevity, coherence, and comprehensiveness.
  • 2 Related Work: FRESH’s schematic separates scoring, extraction, and prediction, with the final classifier trained only on extracted rationales.The scoring model may use attention or gradients, and the extractor can use learned tagging or direct heuristics.
  • 2 Related Work: Existing NLP feature-attribution methods may produce heatmaps that are not robust or faithful explanations of predictions.This motivates scrutiny of token-level explanation methods beyond human acceptability.
  • 2 Related Work: Prior rationale systems use a generator to impose hard attention and pass selected inputs to a prediction model, sometimes adding an adversarial model.Joint training is difficult because hard attention is discrete and requires reinforcement learning.
  • 2 Related Work: Human evaluations test whether users can predict model results or prefer the rationales, while FRESH treats plausibility as a secondary aim.These studies assess human-facing usefulness rather than guaranteeing faithfulness.

3 Faithfulness through Discrete Rationale Selection

Prior work decomposes rationale extraction and prediction, but discrete token selection makes training difficult and potentially high-variance. The model uses a generator to select binary token masks and an encoder to predict from the selected inputs.

  • 3.1 End-to-End Rationale Extraction: Lei et al.’s model separates rationale extraction from prediction, using a generator and encoder as the starting point for FRESH.The generator samples binary token masks, and the encoder predicts from unmasked tokens.
  • 3.1 End-to-End Rationale Extraction: The generator samples a binary mask z_i over each document’s tokens, while the encoder predicts from the unmasked input.The mask is sampled from a Bernoulli distribution.
  • 3.1 End-to-End Rationale Extraction: Joint optimization is difficult because the objective requires marginalizing over all possible rationales z.Parameter estimation approximates this process by sampling rationales and averaging their gradients.
  • 3.1 End-to-End Rationale Extraction: REINFORCE-style estimation may have high variance because the large rationale state space is difficult to explore efficiently.The method samples from the generator during learning to estimate gradients.
  • 3.1 End-to-End Rationale Extraction: The loss combines prediction error with regularization encouraging rationale conciseness and contiguity.FRESH modifies the conciseness term so rationales are not penalized before exceeding a desired length d.

4 Faithful Rationale Extraction from Saliency tHresholding (FRESH)

FRESH avoids joint discrete-selection training by separating support scoring, rationale extraction, and prediction. The predictor sees only extracted snippets, making the resulting rationales faithful by construction while permitting arbitrary importance scores and prediction networks.

  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): FRESH decomposes prediction into independent support, rationale extractor, and classifier models.This removes the need to train the full rationale-selection system jointly.
  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): The support model predicts labels and supplies continuous token-importance scores, which the extractor converts into discrete rationales.The extractor may use a trained model or direct discretization heuristics.
  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): The classifier is trained and tested only on text provided by the extractor, so its predictions use only extracted snippets.This separation establishes faithfulness with respect to the classifier.
  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): FRESH can use arbitrary functions over support-model predictions to score input-token importance, including posthoc gradient methods and LIME.The scoring function need not itself faithfully identify features used by the support model.
  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): Even attention-based scores over contextualized representations can produce faithful final rationales because they only train the extractor, not the classifier’s input.The classifier receives only the snippets selected by the extractor.
  • 4 Faithful Rationale Extraction from Saliency tHresholding (FRESH): The framework may also reduce the text humans must inspect by presenting only short relevant snippets for classification.The paper identifies this as a possible efficiency gain in human document-classification settings.

5 FRESH Implementations

FRESH implementations use BERT-based support models, attention or gradient scores, and heuristics or trained extractors to convert continuous importance into rationales. The framework compares contiguous and top-k selection while leaving CRF-based sequential extraction for future work.

  • 5.1 Feature Scoring Methods: All models use BERT encoders or variants, with support models accepting one or two inputs depending on the classification task.The variants include RoBERTa and SciBERT, referred to collectively as BERT.
  • 5.1 Feature Scoring Methods: The evaluated feature-scoring mechanisms are self-attention scores and input gradients.Attention scores are derived from the [CLS] token’s penultimate-layer self-attention and averaged across heads.
  • 5.2 Discretizing Soft Scores: Continuous feature scores are mapped to discrete rationale labels using heuristic strategies.The mapping can be performed per instance or globally across a dataset.
  • 5.2 Discretizing Soft Scores: The contiguous heuristic selects the length-k span with the highest total importance score.It produces rationales consisting of one contiguous span.
  • 5.2 Discretizing Soft Scores: The top-k heuristic selects the k highest-scoring tokens regardless of whether they are contiguous.These outputs are called non-contiguous rationales.
  • 5.3 Extractor model: The extractor can directly discretize support scores or independently predict token labels with a parameterized model.An explicit extractor permits mixing heuristic pseudo-targets with direct rationale supervision.
  • 5.3 Extractor model: A CRF layer for tying sequential extractor predictions may improve performance, but the paper leaves this extension for future work.The limitation concerns an untested extractor architecture rather than the core FRESH decomposition.

6 Experimental Setup

The evaluation uses five English classification datasets and compares rationale-based models under fixed rationale lengths. It also examines variance, hyperparameter sensitivity, and rationale degeneration during training and inference.

  • Datasets: The experiments span five English text classification datasets covering sentiment, news categorization, biomedical evidence inference, movie reviews, and multiple-choice reading comprehension.Evidence Inference uses abstracts and MultiRC is converted to binary answer classification with supporting rationales.
  • Evaluation setup: The comparison reports mean Macro F1 on test sets, with minimum and maximum performance across random seeds; Full text is an upper-bound black-box baseline.Rationalized models use a percentage of each document, while Full text consumes the entire document without explanations.
  • Evaluation setup: Rationale lengths are set to the average human annotation length where available and otherwise fixed at k = 20%.The evaluation fixes rationale length to support comparisons across methods.
  • Training considerations: Lei et al.’s model requires extensive hyperparameter search for good performance, creating time and energy costs.Its conciseness and contiguity regularizer uses λ1 and λ2, and performance can be sensitive to their choices.
  • Training considerations: FRESH trains independent modules without additional hyperparameters, while incurring some energy cost from separately training those components.The paper quantifies Lei et al.’s hyperparameter-related variance and search compute in Appendix B.
  • Training considerations: REINFORCE-style training can exhibit high variance across random seeds, and competing rationale methods may degenerate to complete or empty text.Inference restricts rationales to the specified desired length k for comparability.

7 Quantitative Evaluation

FRESH is evaluated against end-to-end and differentiable rationale methods across datasets, rationale lengths, and supervision levels. It performs nearly as well as Full text with short rationales, generally outperforms Lei et al.’s method, and benefits selectively from direct rationale supervision.

  • Predictive performance: FRESH performs nearly as well as Full text while using only 10-30% of the original input text.This provides predictive performance with substantially shorter rationales.
  • Predictive performance: FRESH achieves better average performance than Lei et al.’s end-to-end method except on AGNews, where the models are comparable, and consistently outperforms Bastings et al.’s system.The comparison uses fixed rationale lengths across methods.
  • Rationale variants: [CLS] self-attention generally yields better results than gradients, except on MultiRC and Movies, while top-k discretization slightly outperforms the contiguous heuristic.The contiguous heuristic is expected to trade some performance for greater rationale coherence.
  • Robustness: FRESH exhibits lower variance across runs and does not require hyperparameter search.Figure 3 also evaluates performance at two target rationale lengths and reports standard deviations over five random seeds.
  • Rationale supervision: Mixing rationale-level supervision improves Evidence Inference performance by a small amount but does not improve MultiRC performance.The experiments vary explicit rationale supervision from 0% to 100%.
  • Rationale supervision: Direct supervision substantially improves Lei et al.’s performance on Evidence Inference but does not affect its MultiRC performance, while still leaving variance effects.FRESH’s parameterized extractor enables mixing direct supervision with heuristic targets.

8 Human Analysis

The human analysis evaluates whether extracted rationales support prediction and are readable. FRESH rationales were at least as plausible as comparable methods, while faithfulness and plausibility were not necessarily correlated.

  • 8.1 Experiments: The study evaluated rationale sufficiency through human label prediction, confidence ratings, and readability and understandability ratings.Annotators classified Movies and MultiRC examples, rated confidence on a 4-point scale, and rated readability on a 5-point scale.
  • 8.1 Experiments: 84% and 89% inter-annotator agreement was observed on Movies and MultiRC, respectively, using gold-label human rationales.The gold-label condition was used to validate the annotation interface; annotators also showed high accuracy, confidence, and readability.
  • 8.2 Results: Humans performed comparably to the trained model when predicting labels from model-extracted rationales alone.The evaluation compared rationales from human references, random selection, Lei et al. models, and FRESH models across 100 test instances.
  • 8.2 Results: Humans performed at least as well with FRESH rationales as with other methods and strongly preferred contiguous rationales.Confidence and readability were also high, suggesting that FRESH rationales were reasonably plausible to human users.
  • 8.2 Results: The results suggest that faithfulness and plausibility are not mutually exclusive, but they are not necessarily correlated.The study assessed plausibility as a separate property from the framework’s faithfulness guarantee.

9 Conclusions

The paper concludes that FRESH is a flexible, effective architecture that simplifies training while guaranteeing faithfulness through independent extraction and prediction modules. It reports strong predictive performance and human plausibility, while identifying limits on rationale refinement and selection explanations.

  • 9 Conclusions: FRESH is a simple, flexible, and effective method that works with any feature importance metric and often outperforms more complex rationalized models.The conclusion characterizes the method as easy to implement and train.
  • 9 Conclusions: Independent extraction and prediction modules ensure that explanations remain faithful regardless of the complexity of the component models.The predictor receives only the selected rationale, allowing contextualized models such as transformers to be used without sacrificing rationale-level explainability.
  • 9 Conclusions: FRESH avoids explicit rationale-level supervision such as REINFORCE and the reparameterization trick, greatly simplifying training.The approach instead separates the training regimes of the extractor and predictor.
  • 9 Conclusions: FRESH outperforms existing models, recovers most of the original black-box model’s performance, and produces rationales at least as plausible to human users as comparable end-to-end methods.These conclusions combine the reported predictive and human-evaluation outcomes.
  • 9 Conclusions: The framework cannot currently prune or expand the selected token set after rationale selection.This limitation constrains subsequent refinement of the final explanation.
  • 9 Conclusions: FRESH guarantees which document portion informed a classification but cannot readily explain why that specific rationale was selected.The paper also identifies understanding how the predictor uses the extracted rationale as future work.

B Hyperparameter sensitivity analysis

The hyperparameter sensitivity analysis reports that many Lei et al. searches fail to reach the desired rationale length, instead selecting the entire document or no tokens.

  • B Hyperparameter sensitivity analysis: Many hyperparameter searches fail to converge to the desired rationale length.The searches may select the entire document or degenerate by selecting no tokens.
  • B Hyperparameter sensitivity analysis: Figures 7, 8, and 9 report model accuracy across hyperparameter searches on three datasets.The analysis examines how search behavior relates to rationale length and model performance.

D Additional Dataset Details

The additional dataset details describe the reported dataset statistics, rationale-construction heuristics, and comparisons between global and instance-level extraction. Global and instance-level heuristics did not differ significantly in the reported test.

  • D Additional Dataset Details: Figures 6 through 9 cover training duration, validation macro F1, rationale length, and model performance across hyperparameter searches.The rationale-length and performance plots concern 20 searches for Lei et al. models.
  • D Additional Dataset Details: Table 5 reports mean and maximum document, query, and rationale lengths, along with dataset label distributions where available.Query and rationale lengths are not reported for SST and AgNews because they lack human rationale annotations.
  • D Additional Dataset Details: Global top-k selects a predetermined overall token proportion while ensuring each instance retains a minimum token proportion.The remaining token budget is filled using the highest-scoring tokens across the dataset.
  • D Additional Dataset Details: Global Contig. selects contiguous spans and greedily allocates additional tokens to maximize the increase in overall saliency weight.The procedure begins with minimally long spans and repeatedly adds the token with the largest marginal gain.
  • D Additional Dataset Details: No significant difference was found between global and instance-level rationale heuristics in the reported Welch’s t-test.Table 6 reports average F1 differences between the two extraction strategies for each dataset, saliency type, and rationale type combination.
Loading 2005.00115v1…