Interpreting Language Models with Contrastive Explanations

• Neural language models remain difficult to understand because both language modeling and the architectures used for it are highly complex.Source: §1 · p. 1
• Standard explanations can overemphasize the token immediately preceding the prediction, even though language models must account for multiple linguistic features.For “Can you stop the dog from ____,” this attribution can identify syntactic evidence while obscuring other factors.Source: §1 · p. 1, Table 1 · p. 1
• Contrastive explanations identify why a language model predicts one token instead of another, such as why “barking” is preferred over “crying” or “walking.”The examples attribute semantic distinctions to “dog” and “stop,” respectively.Source: §1 · p. 2
• The paper extends three interpretability methods to produce contrastive explanations and tests them against non-contrastive counterparts.Source: §1 · p. 2
• The experiments ask whether contrastive explanations better identify evidence for linguistic phenomena, improve human simulation of model behavior, and expose coherent evidence patterns across word decisions.Source: §1 · p. 2

• Existing NLP explanation work has focused largely on classification, with relatively few studies explaining language-model predictions.The paper links this gap partly to the large output space of language models.Source: §2.1 · p. 2
• Contrastive explanations explain why a model predicts target yt rather than foil yf, whereas counterfactual explanations modify the input to favor yf instead of yt.Source: §2.2 · p. 2
• Contrastive and counterfactual explanations have been studied in model interpretability but were relatively new to NLP and had not yet been applied to language models.Source: §2.2 · p. 2
• The paper proposes contrastive explanations for language modeling, where both the number of input factors and the output space are larger than in many prior settings.Source: §2.2 · p. 2
• Because input-erasure counterfactuals can become intractable for long sequences and large foil spaces, the paper also develops gradient-based contrastive methods.Source: §2.2 · p. 2

• The paper extends three existing input-saliency methods to the contrastive setting and notes that they can also apply beyond language modeling, including machine translation.Source: §3 · p. 2

3.1. Gradient Norm

• Contrastive Gradient Norm measures how each input token increases the target probability while decreasing the foil probability.It uses the gradient of q(yt|x) − q(yf|x) and then takes an L1 norm.Source: §3.1 · p. 3

3.2. Gradient × Input

• Contrastive Gradient × Input combines the contrastive gradient with each input-token embedding, accounting for how strongly the token is expressed in the saliency score.Source: §3.2 · p. 3

3.3. Input Erasure

• Contrastive Input Erasure measures how removing an input token changes the model output, specifically whether the foil becomes more likely and the target less likely.Source: §3.3 · p. 3
• Erasure directly measures output changes after perturbation, whereas gradient methods approximate them; erasure is usually more computationally expensive because it requires runs for all input perturbations.Source: §3.3 · p. 3

4.2. Metrics

• Alignment is measured by the saliency assigned to known evidence, the number of tokens probed before finding it, and mean reciprocal rank.Dot Product sums saliency on evidence tokens; Probes Needed and MRR evaluate how highly the first evidence token is ranked.Source: §4.2 · p. 4

4.3. Results

• Contrastive explanations show higher alignment with BLiMP’s known linguistic evidence than non-contrastive explanations for both GPT-2 and GPT-Neo across metrics.They also outperform random vectors in most cases, whereas non-contrastive explanations do not consistently do so.Source: §4.3 · p. 4, Figure 1 · p. 5
• For most explanation methods, greater distance between known evidence and the target token corresponds to a larger MRR advantage for contrastive explanations.This indicates a particular advantage for decisions requiring longer-range context.Source: §4.3 · pp. 4–5, Figure 2 · p. 5

5.1. Study Setup

• The study gives users a GPT-2 input sequence, two next-token choices, and an explanation, then asks them to select the more likely model output and rate explanation usefulness.It compares Gradient × Input, Contrastive Gradient × Input, Erasure, and Contrastive Erasure, using 20 confusable word pairs.Source: §5.1 · p. 5
• The study recruits 10 graduate machine-learning students and balances word-pair direction and model correctness to reduce shortcut strategies.Annotator and word-pair variation are modeled with linear mixed-effects models.Source: §5.1 · p. 6, §5.2 · p. 6

5.2. Results

• Contrastive explanations significantly improve users’ accuracy in simulating GPT-2 outputs compared with non-contrastive explanations.Users have the lowest prediction accuracy without explanations, indicating that the evaluated explanation types generally help simulation.Source: §5.2 · p. 6, Table 3 · p. 6
• Users report contrastive explanations as useful significantly more often than non-contrastive explanations, with a particularly large gain for erasure-based explanations.Accuracy is higher on examples users deem useful than across all examples for every explanation method.Source: §5.2 · p. 6

6. What Context Do Models Use for Certain Decisions?

• The clustering methodology provides an aggregate analysis of linguistic distinctions and general properties of language-model decisions.It combines explanation vectors across sentences, clusters foils by similarity, and inspects mean cluster vectors.Source: §6 · p. 7, §6.1 · p. 8, §6.3 · p. 8
• Linguistically similar foils cluster together, including distinctions absent from nearest-neighbor word-embedding groups.This suggests models use similar input features for certain decisions.Source: §6.2 · p. 8

6.2. Foil Clusters

• Pronoun foils with different gender or number cluster together, whereas embedding neighbors mix genders, numbers, and grammatical roles.For a male pronoun target, the cluster containing “she” consists of female pronouns.Source: §6.2 · p. 8
• Animacy distinctions produce clusters of inanimate noun foils for animate noun targets, unlike embedding neighbors that mainly reflect lexical relatedness.The nearest neighbors of “fruit” are mostly singular and plural produce-related nouns.Source: §6.2 · p. 8
• Agreement distinctions organize foils by grammatical number and verb form, with plural or singular alternatives clustering against the target.Singular nouns contrast with plural noun clusters, while singular verbs cluster against plural verb targets.Source: §6.2 · p. 8, §6.3 · p. 8

6.3. Explanation Analysis Results

• Cluster explanations reveal the input evidence models use for lexical and grammatical decisions, including semantic words, associated verbs, gendered terms, and measurement cues.They also expose errors linked to gender-neutral or opposite-gender names and pronouns.Source: §6.3 · p. 8

7. Conclusion and Future Work

• The paper extends three input-saliency methods and introduces alignment, user-evaluation, and clustering-based methods for evaluating and exploring explanations.These methods respectively assess linguistic evidence, human model simulation, and aggregate decision patterns.Source: §7 · pp. 8–9
• Contrastive explanations are better aligned with evidence for major grammatical phenomena and enable better contrastive model simulatability for users.The paper also uses them to study decisions requiring similar evidence and characterize linguistic distinctions.Source: §7 · p. 9
• Overall, contrastive explanations provide a more intuitive and fine-grained interpretation of language models.This conclusion follows the paper’s alignment, simulatability, and clustering analyses.Source: §7 · p. 9
• Future work could apply contrastive explanations to other models and tasks, extend additional interpretability methods, and use the findings to improve models.These directions are presented as possibilities rather than established results.Source: §7 · p. 9

A.1. Extending Contrastive Explanations to NMT

• Contrastive explanations are extended to neural machine translation, where predictions depend on both the source sentence and partial translation.This setting shares language modeling’s interpretive complexity.Source: §A.1 · p. 10
• The contrastive gradient norm is computed over both encoder and decoder inputs to compare the probabilities of a target token and foil.The encoder input is the source sentence, while the decoder input is the partial translation.Source: §A.1 · p. 10
• The NMT examples use input × gradient to mark tokens that raise or lower the probability of each translation decision.Tokens with little influence are shown separately.Source: §A.1 · p. 11, Table 5 · p. 11

A.2. Qualitative Results

• The qualitative examples are drawn from five linguistic ambiguity types using MarianMT English-to-Romance translation models.The broader alignment analysis reports full GPT-2 and GPT-Neo scores in Table 6.Source: §A.2 · p. 11, B. Alignment of Contrastive… · p. 11
• Contrastive explanations identify gender evidence from the French antecedent and masculine adjective when translating English “it” as “il” rather than “elle”.Both explanation types mark the English antecedent “vase”, but only the contrastive comparison highlights the disambiguating French context.Source: §A.2 · p. 11
• For formality and verb-form decisions, contrastive explanations highlight partial-translation words such as “Vous”, “pouvez”, and “aimais” that match the target form.The corresponding non-contrastive explanations do not identify these same-form cues.Source: §A.2 · p. 11
• A contrastive explanation identifies the elided verb on the target side as making the correct French verb more likely than a different verb.This illustrates how the foil comparison isolates evidence for the selected translation.Source: §A.2 · p. 11
• For lexical cohesion, the contrastive explanation highlights “carnet” in the partial translation when distinguishing the notebook sense from a computer-related alternative.The non-contrastive explanation instead emphasizes “notebook” and the preceding target token.Source: §A.2 · p. 11

• Linear mixed-effects models quantify how explanation method affects user-study model-simulatability outcomes.The supplied passage identifies the dependent variables, intercepts, explanation-method effect, and p-values as model outputs.Source: D. Mixed Effects Models Results · p. 12

Parts of Speech

• Adjective, adposition, and adverb foils cluster by lexical or grammatical type, while associated verbs and semantically related words provide contrastive evidence.Adjective contrasts use semantically close inputs; adposition and adverb contrasts often rely on verbs associated with the target.Source: E. Analysis of Foil Clusters · p. 12

Determiners and Nouns

• Determiner and noun clusters reflect grammatical agreement, including possessive or demonstrative determiner distinctions and singular–plural inanimate noun contrasts.Demonstrative determiner–noun agreement is supported by similar evidence for disambiguating relevant foil nouns.Source: E. Analysis of Foil Clusters · p. 12

Numbers

• Number contrasts separate number words from non-number words and distinguish quantities such as “hundred” or “thousand” using enumeration and counting context.Inputs related to measurement, enumeration, or counting provide evidence for these number-generation decisions.Source: E. Analysis of Foil Clusters · pp. 12, 14

Pronouns

• Pronoun foil clusters capture gender-based distinctions, whereas the analysis finds no corresponding clusters for pronoun number.Same-gender proper nouns and gendered pronouns or determiners help distinguish a target pronoun, while repeated proper nouns can influence pronoun generation.Source: E. Analysis of Foil Clusters · p. 14

Verbs

• Verb foils with different forms cluster together, indicating shared evidence for subject–verb agreement; auxiliary or compound verbs help distinguish target forms.Auxiliaries support present-participle contrasts, while verbs in the same compound support infinitive contrasts.Source: E. Analysis of Foil Clusters · p. 14