PaperlayerView exact revision on arXiv

Source-linked AI summary

CausaLM: Causal Model Explanation Through Counterfactual Language Models

Amir Feder, Nadav Oved, Uri Shalit, Roi Reichart

arXiv:2005.13407v5cs.CLcs.AIcs.LG

TL;DR

Deep neural NLP models are difficult to interpret causally, particularly when correlated concepts and difficult-to-generate counterfactual texts obscure which concept affects predictions. CausaLM learns counterfactual language representations through adversarially trained encoders and uses them to estimate concept effects, while also supporting bias mitigation. The paper evaluates the approach across datasets and concept settings and identifies assumptions limiting interventions to switching concepts on or off.

  • Problem

    Existing explanations may detect correlations but cannot reliably distinguish whether a high-level language concept causes a model prediction, while generating controlled textual counterfactuals is difficult.

  • Method

    CausaLM fine-tunes a contextual language representation model with adversarial tasks derived from a causal graph to create representations unaffected by a concept of interest.

  • Results

    CausaLM estimates causal effects across four datasets and concept settings, and its counterfactual models can mitigate unwanted bias in classifier outcomes.

  • Takeaways & Limitations

    Representation-level interventions provide causal model explanations without requiring natural-language counterfactual generation and can support bias mitigation.

  • Takeaways & Limitations

    The estimator assumes concepts can only be switched on or off, so it cannot change a concept’s value, such as changing gender from female to male.

Abstract

from arXiv · show

Understanding predictions made by deep neural networks is notoriously difficult, but also crucial to their dissemination. As all machine learning based methods, they are as good as their training data, and can also capture unwanted biases. While there are tools that can help understand whether such biases exist, they do not distinguish between correlation and causation, and might be ill-suited for text-based models and for reasoning about high level language concepts. A key problem of estimating the causal effect of a concept of interest on a given model is that this estimation requires the generation of counterfactual examples, which is challenging with existing generation technology. To bridge that gap, we propose CausaLM, a framework for producing causal model explanations using counterfactual language representation models. Our approach is based on fine-tuning of deep contextualized embedding models with auxiliary adversarial tasks derived from the causal graph of the problem. Concretely, we show that by carefully choosing auxiliary adversarial pre-training tasks, language representation models such as BERT can effectively learn a counterfactual representation for a given concept of interest, and be used to estimate its true causal effect on model performance. A byproduct of our method is a language representation model that is unaffected by the tested concept, which can be useful in mitigating unwanted bias ingrained in the data.

1. Introduction

Deep neural NLP models are difficult to interpret at the level of meaningful linguistic concepts, while correlation-based explanations can confuse association with causation. CausaLM addresses this challenge by learning concept-counterfactual representations instead of generating counterfactual text.

  • Motivation: DNNs are difficult to interpret because their layered, nonlinear structure obscures how inputs affect outputs, especially for abstract linguistic concepts.Concepts such as topic and sentiment carry meaningful information but are not explicitly represented as ordinary input features.
  • Motivation: Correlation-based explanations cannot determine whether a political figure, adjectives, or both cause a classifier’s prediction.These concepts may be strongly correlated in the text, making local perturbation-based interpretations misleading.
  • Counterfactual challenge: Generating natural-language counterfactuals that change one concept while preserving others is intractable in many cases.Changing the discussed political figure while keeping other textual concepts unaffected is especially difficult.
  • CausaLM: CausaLM manipulates text representations rather than text by creating an encoder unaffected by a specified concept of interest.The method receives classifier training data and the concept whose causal effect is to be estimated.
  • Outcomes: The resulting counterfactual models can estimate causal concept effects and help mitigate unwanted bias in classifiers.The approach uses interventions before supervised task training and is not dependent on BERT’s specific implementation.

2. Previous Work

Prior NLP explanation methods largely measure correlations or inspect model behavior without directly estimating how linguistic concepts causally affect predictions. This work connects causal inference, counterfactual analysis, model explanation, and debiasing through representation-level interventions and interventional datasets.

  • Causal inference and NLP: Causal inference has had less impact in NLP because high-dimensional language does not fit existing causal frameworks easily.Applying causal inference to NLP is challenging, particularly when treatment variables are textual.
  • Causal inference and NLP: Existing NLP causal work uses text to answer causal questions, whereas this work uses causal inference to explain what affects an existing neural model’s decisions.The paper focuses on counterfactual predictions under changed textual conditions.
  • Causal model explanation: CausaLM intervenes on textual representations to answer causal questions about low- and high-level concepts without generating counterfactual examples.This distinguishes the approach from methods requiring manually constructed counterfactual queries.
  • Model interpretation: Structural and behavioral interpretation methods generally do not directly measure the effects of text structure or linguistic concepts on model outcomes.They often rely on token-level analysis and do not account for counterfactuals.
  • Debiasing: Correlation-based debiasing methods can overestimate a treatment concept’s causal effect when they fail to control for other concepts.The paper distinguishes causal model explanation from methods that only reveal associations between textual features and predictions.
  • Datasets and debiasing: The paper introduces datasets where bias can be measured by comparing predictions on examples with predictions on their counterfactuals.These datasets support direct comparison of methods for causal model explanation using interventional rather than observational measures.

3. Causal Model Explanation

The paper frames model explanation as causal inference: estimate how high-level concepts affect classifier predictions while accounting for confounding. CausaLM avoids difficult text counterfactual generation by intervening on language representations instead.

  • The framework treats a concept’s effect on classifier predictions as a causal inference problem rather than a purely associational explanation.It connects interventions on input concepts with causal model explanation for NLP classifiers.
  • Correlation-based explanation tools can misattribute predictions when concepts such as Adjectives and Political Figure are correlated or confounded.A confounding concept may affect both the concept of interest and the text or classifier output.
  • Direct text counterfactuals require changing one concept while holding other generating concepts fixed, but this is especially difficult for high-level language concepts.Changing a political figure can also alter topics, adjectives, and other textual choices.
  • The proposed alternative intervenes on the language representation, producing a counterfactual representation φC(X) that is unaffected by the selected concept.This bypasses the need to generate a new text and uses adversarial fine-tuning to alter the representation mechanism.
  • TReATE measures a concept’s effect on classifier probabilities through a representation intervention while controlling for potentially confounding concepts.The method is presented as a representation-based special case of the average treatment effect.
  • Representation intervention avoids generating additional examples and therefore does not depend on the quality of a text-generation process.The authors present comparison with the counterfactual representation as an approximation to comparison with a properly manipulated synthetic example.

4. Data

The paper constructs datasets to evaluate causal effect estimation for linguistic concepts, combining sentiment reviews with controlled adjective and topic variation and an enriched bias benchmark. Evaluation depends on explicit causal assumptions, while natural review data limits ground-truth counterfactual evaluation for topics.

  • Dataset and evaluation design: Causal effect identification requires a causal graph encoding assumptions about the data-generating process; the proposed evaluation includes four cases, three with ground-truth counterfactuals.Non-causal methods may find arbitrary correlations, whereas causal estimators can be compared with ground truth when counterfactual examples are available.
  • Product and movie reviews: The sentiment datasets combine product reviews from four domains with IMDB movie reviews, providing five domains for studying concept–label correlations.The domains are Books, DVD, Electronics, Kitchen Appliances, and Movies.
  • Product and movie reviews: Adjective counterfactuals are created by deleting words automatically tagged as adjectives from each review.For example, “It’s a lovely table” becomes “It’s a table”; the approach assumes adjective removal generally preserves grammaticality.
  • Product and movie reviews: Topics are modeled with 50 LDA topics, selecting a treated topic and a separate control topic for each domain based on relative domain association.Topic probabilities are also dichotomized at their median for the TC and CC tasks.
  • The enriched Equity Evaluation Corpus (EEEC): The enriched Equity Evaluation Corpus addresses the brevity of the original bias benchmark by creating more representative and natural training data.The original EEC contains 8,640 concise English sentences designed to probe racial and gender bias in mood-state predictions.

5. Tasks and Experiments

The experiments evaluate causal-effect estimation and counterfactual representations across sentiment and mood-state tasks involving Adjectives, Topics, Gender, and Race. They vary concept–label correlations, test forgetting and retention of concepts, and compare causal estimates with correlation-based baselines.

  • Experimental tasks: The experiments use Balanced, Gentle, and Aggressive datasets that differ in the correlation between the treated concept and the label.The Aggressive Adjective setting increases correlation by deleting the bottom half of sentences within positive reviews.
  • Experimental tasks: Four tasks examine forgetting Adjectives, Topics, Gender, and Race while controlling against forgetting potential confounders.The tasks cover sentiment classification and Profile of Mood States classification.
  • Research questions: The research questions test whether TReATE approximates ground-truth causal effects, whether BERT-CF forgets treated concepts, and whether it retains control concepts.A fourth question examines whether BERT-CF can help mitigate potential downstream classifier bias.
  • Adjectives: For Adjectives, the Is Masked Adjective objective predicts whether masked tokens are adjectives rather than predicting the masked words.Adjectives and an equal number of non-adjective words are masked to create a balanced token-level binary classification task.
  • Topics: For Topics, topic-model probabilities represent treated and control topics, denoted C0 and C1, within the counterfactual objectives.Topics are global concepts represented across reviews and can be trained without manual supervision.
  • Topics: Topics are confounders for one another by design because their probability distributions lie on a simplex.Decreasing one topic’s probability necessarily increases the probability of others, and the objectives are designed to circumvent these confounders.
  • Baselines: CONEXP uses passive conditional prediction differences, whereas TPR-GAP compares accuracy across concept-present and concept-absent conditions.TPR-GAP cannot be directly compared with ground-truth ATEgt or TReATE because it measures accuracy rather than class-probability distributions.

6. Results

Across experiments, TReATE(O, CF) closely estimates causal effects while counterfactual representations suppress treated concepts, preserve control concepts, and improve robustness to distribution shift. The analyses also examine baseline failures and multitask training dynamics.

  • Estimating TReATE: TReATE(O, CF) closely matches ATEgt(O) across experiments, including settings with different amounts of introduced bias.The estimator remains close to ground truth even when the true effect is near zero, such as Balanced Race.
  • Estimating TReATE: The CONEXP(O) baseline substantially underestimates causal effects in 7 of 9 experiments, while INLP substantially overestimates ATEgt(O).The authors suggest INLP may fail to preserve information about control concepts.
  • Estimating TReATE: Adjectives change sentiment classifiers’ output class probability distribution by 0.397 on average in the Balanced setting and 0.634 in the Aggressive setting.The aggressive-setting effect is 60% higher than the Balanced-setting effect.
  • Estimating TReATE: TReATE(O, CF) approximates ATEgt(O) with a maximal absolute error of 3.9% and an average absolute error of 2.6% for Gender and Race.For Adjectives, the gap remains 3% absolute across all three settings.
  • Analyzing the Counterfactual Model: BERT-CF substantially reduces treated-concept classification performance, with Gender and Adjectives only slightly above chance.BERT-O and BERT-MLM representations sometimes yield nearly 100% test-set accuracy on treated-concept classifiers.
  • Analyzing the Counterfactual Model: Control-concept accuracy changes by only 2−10% absolute when using BERT-CF instead of BERT-O, indicating related control information is largely retained.This supports the intended separation between forgetting treated concepts and preserving potentially confounding concepts.
  • Mitigating Bias: BERT-CF models trained under aggressive correlations outperform BERT-O and BERT-MLM models on balanced test sets when training correlations disappear.BERT-CF is reported as less affected by this distribution shift.
  • Analyzing the Stage 2 Multi-Task Training Scheme: TC tasks typically converge after 1−2 epochs, while adding a CC task initially increases TC loss but does not visibly affect MLM loss in the Adjectives treatment.The authors interpret this pattern as CC dampening, without overwhelming, the adversarial effect on MLM.

7. Discussion and Conclusion

The paper presents CausaLM as a causal framework for evaluating concept effects on DNN classifiers through counterfactual representations rather than generated text. Its validity depends on explicit causal assumptions, while dataset realism and global-concept analysis remain important limitations.

  • Contributions: CausaLM models data-generating processes with causal graphs and estimates concept effects using adversarially trained counterfactual representations.The method implements the do-operator through adversarial language-representation training, avoiding direct counterfactual text generation.
  • Contributions: The approach introduces four datasets, evaluation tools for counterfactual language representations, and experiments demonstrating the framework’s utility.Three datasets contain manually generated counterfactual examples that permit estimating true causal effects.
  • Assumptions: CausaLM requires explicit world-model assumptions and careful selection of treatment and control concepts to estimate a concept’s true effect.Different control concepts can yield different effect estimates, so sanity checks are used to assess whether controls were properly handled.
  • Assumptions: The representation model treats concepts as switchable on or off, so TReATE compares classifier output distributions with and without the concept rather than changing its value.This prevents interventions such as changing an example’s gender from female to male.
  • Limitations: Global concepts such as topics remain difficult to measure and analyze, leaving their effects on DNNs as an open challenge.The paper proposes a topic representation that shares information across sentences but identifies broader global-concept analysis as future work.
  • Limitations: Validation remains constrained because synthetic data lack natural language complexity and real-world counterfactuals can be inaccurate or require manual construction.The authors identify natural datasets with precise counterfactual examples as an important future need.

A. A Clinical Example

The clinical example shows why causal concept explanations matter when textual style may be entangled with patient condition, clinical decisions, and classifier outcomes. Intervening on style can also alter note structure and clinical content.

  • Motivation: Clinical prediction models require careful interpretation because reliance on doctor- or hospital-specific textual features could have devastating implications.The example concerns classifiers trained on clinical notes to predict clinically important factors.
  • Motivation: A classifier may fail on a concise note if it learned a doctor’s verbose writing style rather than the same clinical information.The example uses a lengthy, repetitive note that could be summarized without losing too much clinically relevant information.
  • Intervention challenge: Changing writing style is difficult because replacing or deleting long words can substantially alter a note’s structure and content.Word length is used as a proxy for writing style, but lengths also vary across note sections.
  • Confounding: A confounder can make doctor-specific writing style appear predictive when the doctor disproportionately treats patients with severe depression.Performance on that doctor’s notes may look promising while deployment risks patients’ health.
  • Confounding: Clinical recommendations may be difficult to disentangle from patient conditions when the condition causes the doctor’s treatment suggestion and the resulting note.This relationship motivates alternative causal graphs for the clinical setting.

B. Alternative Causal Graphs and Limitations

Alternative causal graphs expose practical limits on interventions when concepts inherit effects through clinical recommendations or when the recommendation itself determines the text. Clinical notes are discussed as an application, not evaluated experimentally.

  • Scope: Clinical notes are not included in the experiments because they were unavailable publicly, so the paper plans to create synthetic clinical data.
  • Alternative causal graphs: The clinical graphs model a patient condition causing a doctor’s recommendation, which then generates the clinical note.The alternatives differ in whether many conditions or only Depression cause the recommendation.
  • Limitations: In the graph where Anxiety, Depression, and Suicidal cause the recommendation, intervening on Depression also changes the recommendation and includes an indirect effect.Intervening on the recommendation itself would require a blank text under that graph.
  • Limitations: In the graph where only Depression affects the recommendation, the doctor’s counterfactual recommendation is unknown, preventing estimation of Depression’s or the recommendation’s causal effect.

C. The EEEC Dataset

The EEEC appendix documents dataset construction through generation templates and descriptive comparisons with the original EEC dataset. It also lists the experiment hyper-parameters.

  • Dataset construction: EEEC data are generated from templates, which are documented in Table 11.
  • Dataset comparison: Table 12 compares descriptive statistics for the original EEC and Enriched EEC datasets.The comparison illustrates the key modifications made in EEEC.
  • Experimental settings: Table 13 reports the hyper-parameters used in the experiments.

D. Experimental Pipeline and Hyper-parameters

Experiments apply a common pipeline across downstream classification and probing tasks and across Balanced, Gentle, and Aggressive dataset versions. The setup uses BERT-based representations, supervised classifiers, fixed optimization settings, and untuned hyperparameters.

  • Experimental Pipeline: The same experimental pipeline is used for Sentiment, POMS, TC, and CC tasks across Balanced, Gentle, and Aggressive dataset versions.The pipeline covers downstream classification and probing settings.
  • Experimental Pipeline: Stage 2 fine-tuning produces BERT-CF and BERT-MLM representations, followed by Stage 3 supervised training for the relevant downstream task.BERT-CF follows the intervention methodology, whereas BERT-MLM uses standard masked-language-model training.
  • Model and Classifier Setup: All experiments use case-sensitive BERT-base with 12 layers, 768-dimensional hidden vectors, 12 attention heads, and 110M parameters.The model was pretrained on BookCorpus and Wikipedia.
  • Model and Classifier Setup: The downstream classifier receives token representations from BERT’s top layer and its CLS token, while INLP uses the BERT-O top-layer CLS representation.INLP output is passed to its default logistic-regression classifier.
  • Optimization: Models use cross entropy with ADAM optimization, a 1e−3 learning rate, a 1e−8 fuzz factor, and no weight decay.The models and pipelines were developed with PyTorch using modified HuggingFace Transformers and PyTorch Lightning code.
  • Hyperparameters: Hyperparameters were not tuned because the extensive pipeline created many experiments across dataset versions and model variations.Table 13 reports the hyperparameters used across the experiments.
Loading 2005.13407v5…