Source-linked AI summary
Pathologies of Neural Models Make Interpretations Difficult
Shi Feng, Eric Wallace, Alvin Grissom, Mohit Iyyer, Pedro Rodriguez, Jordan Boyd-Graber
TL;DR
The paper asks whether feature-attribution methods faithfully identify the words supporting neural predictions. It uses input reduction, including beam search, to remove supposedly unimportant words and then fine-tunes models to increase entropy on reduced examples. Reduced inputs often become nonsensical while preserving confident predictions, whereas entropy regularization improves interpretability without sacrificing accuracy on regular examples.
Problem
Existing perturbation- and gradient-based interpretations can identify influential features without revealing pathological behavior that emerges when supposedly unimportant words are removed.
Method
The paper applies input reduction with beam search and fine-tunes models to maximize entropy on reduced examples while retaining likelihood on regular examples.
Results
Input reduction often leaves one- or two-word examples that humans find nonsensical while models retain their original predictions with high confidence; entropy regularization improves interpretability without sacrificing regular-example accuracy.
Takeaways & Limitations
The findings connect interpretation pathologies to neural-model overconfidence and show that uncertainty on reduced inputs can make models more interpretable.
Takeaways & Limitations
Leave-one-out may fail to expose unreasonable behavior that appears only after several unimportant words have been removed.
Abstract
from arXiv · showhide
One way to interpret neural model predictions is to highlight the most important input features---for example, a heatmap visualization over the words in an input sentence. In existing interpretation methods for NLP, a word's importance is determined by either input perturbation---measuring the decrease in model confidence when that word is removed---or by the gradient with respect to that word. To understand the limitations of these methods, we use input reduction, which iteratively removes the least important word from the input. This exposes pathological behaviors of neural models: the remaining words appear nonsensical to humans and are not the ones determined as important by interpretation methods. As we confirm with human experiments, the reduced examples lack information to support the prediction of any label, but models still make the same predictions with high confidence. To explain these counterintuitive results, we draw connections to adversarial examples and confidence calibration: pathological behaviors reveal difficulties in interpreting neural models trained with maximum likelihood. To mitigate their deficiencies, we fine-tune the models by encouraging high entropy outputs on reduced examples. Fine-tuned models become more interpretable under input reduction without accuracy loss on regular examples.
1 Introduction
Existing interpretation methods highlight words deemed important by perturbation or gradients, but input reduction reveals that neural models can preserve confident predictions on inputs that are nonsensical to humans. Across several tasks, beam search produces extremely short reduced inputs without lowering confidence, exposing pathological behavior.
- Interpretation methods: Feature attribution estimates word importance through confidence changes after removal or through gradients, typically visualized as heatmaps.Leave-one-out treats a word as important when removing it significantly decreases model confidence; gradient methods use large positive gradients as evidence of influence.
- Input reduction: Input reduction iteratively removes supposedly unimportant words while maintaining the model’s prediction, so the remaining words should represent the model’s important features.This complementary approach studies model behavior on words assigned low importance rather than only highlighting high-importance words.
- Pathological behavior: Reduced inputs can become meaningless to humans while retaining the original prediction with high confidence, contradicting the expected alignment between importance scores and human interpretation.The paper characterizes these reduced examples as more like adversarial examples than explanations of the original prediction.
- Interpretive explanation: The resulting pathologies connect interpretation failures to adversarial-example behavior and the overconfidence of neural models.The paper uses these connections to explain why models remain highly confident on reduced inputs that lack sufficient information for human prediction.
- Empirical scope: Beam-search input reduction often shrinks examples to one or two words without lowering confidence across SQUAD, SNLI, and VQA.Crowdsourced experiments verify that the resulting reduced examples appear nonsensical to humans.
2 Input Reduction
Input reduction removes supposedly unimportant words while preserving the model’s prediction, revealing very short, nonsensical inputs that humans cannot confidently answer. Human experiments show substantial accuracy loss and near-random judgments on reduced examples, despite unchanged or similar model confidence.
- Interpretation pathology: The results create a mismatch between interpretation methods that identify reasonable important words and input reduction, which selects nonsensical remnants.The paper frames this mismatch as a puzzle about model confidence and feature selection.
- Input reduction: Input reduction iteratively removes the least important words until the model changes its prediction, while preserving the original prediction during reduction.Importance is based on confidence change after removing a word, with gradient-based approximation used for efficiency.
- Experimental setup: The study applies input reduction to SQUAD, SNLI, and VQA, reducing only the question or hypothesis while retaining the other task input.The retained inputs are the SQUAD context, SNLI premise, or VQA image.
- Human evaluation: Most reduced inputs are nonsensical to humans, who lose significant accuracy even though model predictions remain identical.The experiments sample correctly classified examples and compare human performance on original and reduced inputs.
- Reduction results: Beam search often reduces inputs to one or two words without lowering confidence on the original prediction.Confidence decreases slightly on SQUAD and SNLI and increases on VQA.
- Human evaluation: Reduced inputs appear almost random to humans, who show nearly equal preference between input-reduced and randomly shortened examples.This comparison tests whether the reduction preserves human-perceived information rather than merely shortening the input.
3 Making Sense of Reduced Inputs
Input reduction reveals that neural models can remain highly confident on reduced inputs that humans find meaningless, while interpretation signals themselves can shift sharply after small input changes.
- 3.1 Model Overconfidence: Neural models can assign high-confidence predictions to reduced inputs that humans cannot reasonably classify, turning valid examples into rubbish examples.The paper connects this behavior to overconfidence and unreliable uncertainty estimates, particularly on inputs outside the training distribution.
- 3.1 Model Overconfidence: Input reduction exposes model pathologies by removing words judged unimportant while preserving the original prediction and confidence.The resulting examples can resemble adversarial or rubbish examples rather than faithful explanations of the original prediction.
- 3.2 Second-order Sensitivity: Interpretation heatmaps can shift drastically after small input changes, even when the removed word was initially considered least important.This second-order sensitivity reflects high sensitivity of interpretations despite low first-order sensitivity to the removed feature.
- 3.1 Model Overconfidence: In the SQUAD reduction path, removing “Broncos” immediately makes the question unanswerable to humans, although the model remains correct and confident.The input gradient nevertheless identifies “Broncos” as the least important word in the original context.
- 3.2 Second-order Sensitivity: A heatmap can misleadingly suggest that a prediction is based on a weighted combination of words when the model composes representations nonlinearly.This limitation does not apply in the same way to models that actually use a weighted sum, such as a DAN.
4 Mitigating Model Pathologies
The paper mitigates overconfidence by fine-tuning models to maximize uncertainty on input-reduced examples while retaining likelihood on regular examples. This produces more reasonable reductions and preserves regular-example accuracy.
- 4.1 Regularization on Reduced Inputs: Entropy regularization fine-tunes models to maximize output uncertainty on reduced examples while maintaining log-likelihood on regular training examples.Reduced versions generated from training inputs form the negative example set used for regularization.
- 4.1 Regularization on Reduced Inputs: Model accuracy on regular validation examples remains largely unchanged after fine-tuning, with a slight increase for SQUAD.The table also reports increased reduced-example lengths across all three tasks.
- 4.2 Regularization Mitigates Pathologies: After entropy regularization, reduced inputs become more reasonable to humans and average reduced-example length increases across all tasks.In the SQUAD example, the reduced question changes from “did” to “spend Astor money on ?” after fine-tuning.
- 4.2 Regularization Mitigates Pathologies: Human accuracy increases across all three tasks, and humans prefer gradient-based reduced examples to randomly reduced examples after fine-tuning.These changes indicate that the reduced examples become more meaningful to humans and that model pathologies are mitigated.
- 4.2 Regularization Mitigates Pathologies: Randomly reduced examples matched for length improve neither regular-example accuracy nor interpretability under input reduction.This baseline supports using input reduction to generate negative examples for countering model overconfidence.
5 Discussion
The discussion situates input reduction among work on rubbish examples, adversarial robustness, and interpretation sensitivity, while reporting broader evidence that the observed pathologies transfer across tasks and architectures.
- 5 Discussion: Input reduction transforms valid NLP inputs into rubbish examples that remain close to the data distribution and also exhibit adversarial characteristics.Unlike image rubbish examples generated from pure noise, these examples arise through gradual removal from valid inputs.
- 5 Discussion: Incomplete user inputs can accidentally become adversarial by producing unreasonably high model confidence, extending the issue beyond deliberately constructed perturbations.The paper contrasts this behavior with common misspellings that disrupt neural machine translation models.
- 5 Discussion: Earlier training checkpoints show similar input-reduction behavior, suggesting that better maximum-likelihood convergence alone does not resolve the observed issues.The paper argues that training objectives must incorporate uncertainty estimation explicitly.
- 5 Discussion: Entropy regularization on reduced examples differs from adversarial training because it encourages uncertainty on rubbish inputs rather than robustness to adversarial perturbations.The approach also differs from post-hoc calibration methods that adjust the overall confidence distribution rather than specific reduced inputs.
- 5 Discussion: Input reduction produces similar pathological behavior across paired-input and single-input tasks, including SQUAD, SNLI, VQA, sentiment analysis, and Quizbowl.Short reduced questions and hypotheses can visibly lack the words humans need for prediction.
- 5 Discussion: 81.3% of reduced SNLI inputs generated with BIMPM triggered the same prediction in the Decomposable Attention Model.This cross-architecture transfer suggests that the reduced examples are not confined to the architecture that generated them.
6 Conclusion
The paper introduces input reduction to expose pathological neural-model behavior and connects these failures to overconfidence and sensitivity. Entropy-based fine-tuning improves interpretability without sacrificing accuracy on regular examples.
- Input reduction can reduce sentences to nonsensical one- or two-word inputs while models preserve their original predictions.Human accuracy degrades on reduced examples, whereas neural models maintain their original predictions.
- These pathologies reflect overconfidence and sensitivity to small input changes in neural models trained with maximum likelihood.Models may retain confidence on inputs that lack information for any label, while small perturbations can substantially alter interpretations.
- Input reduction can stress test model interpretability by exposing deeper problems that other interpretation methods may miss.
- Entropy regularization on reduced examples improves interpretability without sacrificing model accuracy on regular examples.
- The paper suggests future work on evaluating interpretation methods, estimating uncertainty and confidence, and moving beyond bag-of-word heatmaps.
A Model and Training Details
The experiments use established models for reading comprehension, textual entailment, and visual question answering. Training alternates maximum-likelihood updates on regular examples with maximum-entropy updates on reduced examples.
- Model configurations: The SQUAD experiments use DRQA’s Document Reader for reading comprehension.SQUAD inputs contain a context paragraph and question, and the task predicts an answer span.
- Model configurations: The SNLI experiments use the Bilateral Multi-Perspective Matching Model for textual entailment.The model matches premise and hypothesis representations across multiple perspectives before classification.
- Model configurations: The VQA experiments use Show, Ask, Attend and Answer with ResNet-152 image representations and an LSTM question representation.
- Entropy-regularized fine-tuning: Fine-tuning alternates maximum-likelihood updates on regular batches with maximum-entropy updates on randomly sampled reduced examples.Two separate Adam optimizers are used for the two terms.
- Entropy-regularized fine-tuning: The fine-tuning settings use learning rate 2e−4 and λ 1e−3 for SQUAD and SNLI, versus 1e−4 and 1e−4 for VQA.
B More Examples
The examples compare original inputs with reduced versions before and after entropy regularization across SQUAD, SNLI, and VQA. Reduction often leaves short, linguistically degraded inputs while the models still classify every shown example correctly.
- All displayed examples are correctly classified by the model before and after fine-tuning.
- SQUAD: Before fine-tuning, SQUAD questions are reduced to fragments such as “did” or “Who Broncos defeat.”The examples retain model predictions despite removing much of the original question.
- SQUAD: Entropy regularization changes some SQUAD reductions into longer, more informative fragments, such as “spend Astor money on ?”.
- SNLI: SNLI reductions preserve labels while often retaining only isolated or fragmentary words from the original premise-hypothesis pair.