PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, Stuart Shieber

arXiv:2004.12265v2cs.CL

TL;DR

Neural NLP interpretation methods often examine structure or behavior without connecting the two causally. The paper applies causal mediation analysis to Transformer language models, treating internal components as mediators of gender bias, and finds that bias effects are sparse, synergistic, and decomposable into direct and indirect effects.

  • Problem

    Existing neural NLP interpretation methods typically examine model structure or behavior separately, leaving unclear which internal components are causally implicated in predictions.

  • Method

    The paper uses causal mediation analysis with controlled interventions on inputs and internal components such as neurons and attention heads.

  • Results

    Gender bias effects are sparse, synergistic, and decomposable into direct effects from inputs and indirect effects through mediators.

  • Takeaways & Limitations

    The framework reveals internal mechanisms by which gender-bias effects flow from model inputs to outputs across Transformer language models.

  • Takeaways & Limitations

    The experimental design focuses on gender bias in a binary setup, and preliminary experiments found that chosen templates did not show learned gender-neutral references.

Abstract

from arXiv · show

Common methods for interpreting neural models in natural language processing typically examine either their structure or their behavior, but not both. We propose a methodology grounded in the theory of causal mediation analysis for interpreting which parts of a model are causally implicated in its behavior. It enables us to analyze the mechanisms by which information flows from input to output through various model components, known as mediators. We apply this methodology to analyze gender bias in pre-trained Transformer language models. We study the role of individual neurons and attention heads in mediating gender bias across three datasets designed to gauge a model's sensitivity to gender bias. Our mediation analysis reveals that gender bias effects are (i) sparse, concentrated in a small part of the network; (ii) synergistic, amplified or repressed by different components; and (iii) decomposable into effects flowing directly from the input and indirectly through the mediators.

1. Introduction

The paper presents causal mediation analysis as a structural-behavioral method for connecting neural NLP models’ internal components to their predictions. Applied to gender bias, it finds effects that are sparse, synergistic, and decomposable, with patterns extending across Transformer models.

  • Motivation: Structural analyses expose model components, whereas behavioral analyses assess predictions but typically do not connect them to internal mechanisms.Probing classifiers can be correlational rather than causally connected to predictions, while behavioral analyses generally lack internal explanations.
  • Method: Causal mediation analysis treats neurons and attention heads as mediators between model inputs and outputs.Controlled interventions on inputs and mediators reveal the causal role of specific components in model behavior.
  • Application: The framework analyzes grammatical gender bias in pre-trained Transformer language models using datasets designed to measure model sensitivity to bias.The study primarily examines several versions of GPT2 and evaluates gender-bias effects through different model components.
  • Findings: Gender bias is sparse, with much of its effect concentrated in relatively few model components.The analysis measures indirect effects flowing through different mediators to locate where gender-bias effects are captured.
  • Findings: Gender bias is synergistic: some components amplify or repress one another’s effects, while others operate independently and capture complementary aspects.The findings distinguish interacting components from components whose contributions are relatively independent.
  • Findings: The framework’s insights apply across three autoregressive models and, to a lesser extent, three masked language models.This suggests the analysis captures general causal-structure patterns in Transformer architectures rather than only model-specific characteristics.

2. Related Work

Prior NLP interpretation methods often separate structural analysis of representations from behavioral evaluation of predictions. This paper bridges those perspectives with causal mediation analysis for deep language models.

  • Structural and behavioral methods: Structural methods identify information in model components, but probing classifiers are not necessarily connected to the model’s task behavior.Probing classifiers predict linguistic properties from representations, measuring correlations that may not explain predictions.
  • Structural and behavioral methods: Behavioral methods directly evaluate predictions on curated examples but provide limited insight into the internal structure involved in producing them.Saliency methods identify important input features while typically ignoring the network’s internal structure.
  • Causal mediation: Causal mediation analysis bridges structural and behavioral interpretation by identifying internal components that are causally implicated in model behavior.The approach measures natural direct and indirect effects, making the role of internal components explicit.
  • Causal mediation: Unlike earlier mediation work on simple datasets and models, this framework targets deep language models and measures natural direct and indirect effects.The paper distinguishes its scope and effect decomposition from prior analyses limited to total and controlled direct effects.
  • Causal foundations: The framework views a neural network as a directed acyclic graph with the input as a common ancestor and the output as a common descendant.This formulation grounds the analysis strategy in causal-graph theory.

3. Methodology

The methodology treats internal model components as causal mediators between gender interventions on input text and biased language-model outputs, separating total, direct, and indirect effects. It applies this framework to gender-bias measurements while noting scope limits of the output construction and robustness across metrics.

  • Causal mediation analysis: Causal mediation analysis treats internal components such as neurons as mediators linking input interventions to model predictions.The input influences a mediator, which in turn affects the output, allowing component-level causal contributions to be measured.
  • Gender-bias measurement: The gender-bias measure compares the model’s relative probabilities for stereotypical and anti-stereotypical pronouns, with equality representing no bias.For The nurse said that, the comparison is between she and he; y(u) < 1 is stereotypical, y(u) > 1 is anti-stereotypical, and y(u) = 1 is unbiased.
  • Causal mediation analysis: The analysis compares an anti-stereotypical gender intervention with an unchanged sentence to measure its effect on a gender-bias response.The set-gender operation changes examples such as nurse to man or doctor to woman, while null leaves the sentence unchanged.
  • Causal mediation analysis: The total effect captures the change in bias between gendered and ambiguous readings, while direct and indirect effects separate pathways through a mediator.The direct effect holds the mediator at its original value; the indirect effect sets it to the value induced by the intervention while keeping the unit unchanged.
  • Model components as mediators: The framework evaluates how much a gender-edit effect flows through a specific neuron or attention component versus elsewhere in the model.The supplied methodology studies individual neurons and extends the mediator analysis to attention-level components.
  • Scope and robustness: The probability construction focuses on pronoun candidates for gender bias, although it could be extended to all output words.Broader outputs might reveal subtler effects on other stereotyped words, but would make gender-bias effects harder to identify concretely.
  • Scope and robustness: The reported findings are robust to the metric used to quantify effects.Alternate-metric analyses are provided for reference.

4. Experimental Details

The experiments use GPT2 and additional Transformer models, with neuron and attention interventions evaluated on profession and coreference datasets measuring gender bias.

  • Models: GPT2 is evaluated in small, medium, large, extra-large, and distilled variants, with additional experiments using Transformer-XL and masked language models.The setup tests whether findings extend across Transformer-based architectures.
  • Neuron interventions: Neuron interventions use profession templates instantiated with occupations rated for definitional and stereotypical gender associations.Actress is definitionally female, whereas nurse is stereotypically female.
  • Attention interventions: Attention interventions use 160/130 Winobias development/test examples and 44 Winogender examples, with full and total-effect-filtered datasets.Both datasets include external occupational gender-stereotype statistics.
  • Evaluation: Table 1 reports GPT2 model sizes and total gender-bias effects on Winobias, Winogender, and the Professions dataset.The table compares model variants across three evaluation datasets.

5. Results

Gender-bias effects are concentrated in particular components, interact differently across neurons and heads, and approximately decompose into direct and indirect pathways.

  • 5.2 Sparsity: Neuron effects are strongest in embeddings and the first hidden layer, with a smaller middle-layer increase and more even neutral-intervention distribution.The neutral case lacks a distinct sparse representation across layers.
  • 5.3 Synergism: Only 4% of neurons are needed to match the concurrent model effect, and most selected neurons are concentrated in embeddings and the first hidden layer.The concentration suggests neuron specialization, although qualitative analysis is difficult because there are many neurons.
  • 5.3 Synergism: Attention heads operate mainly independently and complementarily, with concurrent and summed indirect effects differing by 0.7% to 11.3%.Many heads contribute little or reduce the indirect effect, while the strongest heads capture distinct coreference aspects.
  • 5.3 Synergism: Neuron interventions combine synergistically: the summed independent indirect effects are much smaller than the concurrent intervention effect.By definition, concurrent intervention on all neurons has TE = NIE-all and NDE-all = 0.
  • 5.4 Decomposition of the Total Effect: In all but the smallest model, concurrent indirect effects exceed direct effects, indicating that attention heads mediate most of the bias effect.Other components, including word representations, still account for part of the total effect.
  • 5.4 Decomposition of the Total Effect: The total effect approximately decomposes as TE ≈ NDE + NIE for top heads and concurrent interventions despite the model’s nonlinearity.The decomposition is supported by the reported results but is not guaranteed generally without additional assumptions.
  • Summary: Gender-bias effects are sparse, synergistic, and decomposable into direct effects and indirect effects through model components.These findings summarize the mediation analysis across the examined components and datasets.

6. Discussion and Conclusion

The paper introduces causal mediation analysis as a structural-behavioral framework for interpreting neural NLP models and applies it to gender bias. It identifies bias as sparse, synergistic, and decomposable, while noting binary gender as a scope boundary.

  • Causal mediation analysis provides a structural-behavioral framework for interpreting how gender bias flows through components of neural NLP models.
  • Gender bias effects are sparse, synergistic, and decomposable into direct and indirect effects.
  • The experimental design focuses on gender bias in a binary setup, motivating extensions to gender-inclusive settings and other biases.
  • Mediation analysis may guide model selection and the design or assessment of debiasing interventions.
  • The framework uses interventions on trained models to compute causal effects for the same input under alternative interventions.
  • The authors note that causal-effect definitions could be refined and that alternative effect decompositions and path-specific effects are promising directions.

A.1 Professions Dataset

The professions dataset uses occupation templates and gender statistics to construct bias evaluations. Effects vary across dataset and filtering choices, with larger effects reported for stereotypically female professions.

  • The professions evaluation instantiates templates with occupations and verifies gender alignment using conditional probabilities of she and he.Across templates, p(she)/p(he) given woman ranges from 2.5 to 45.1, while p(he)/p(she) given man ranges from 3.0 to 55.4.
  • The dataset includes female, neutral, and male professions, with definitional words excluded from total-effect calculations.
  • Winobias and Winogender analyses exclude prompts that do not end in a pronoun, and Winobias uses Type 1 examples with alternate continuations.
  • Gender stereotypicality is assessed using U.S. Bureau of Labor Statistics statistics, with Winogender also incorporating text-based gender estimates.
  • Total-effect relationships between model and effect size remain relatively consistent across dataset variations, although magnitudes vary.
  • Total effects are much larger for stereotypically female than stereotypically male professions.The paper relates this difference to greater stereotypicality among the female-profession examples and societal narrowing of women’s professions.

Appendix C. Additional Attention Results

Additional attention analyses show that the strongest indirect effects tend to occur in middle layers and overlap across dataset variations. Direct and indirect effects together approximate total effects.

  • Attention heads with the largest indirect effects lie in the middle layers across additional GPT2 models.
  • The heads with the largest indirect effects show significant overlap across GPT2-small dataset variations.
  • Randomized GPT2-small weights provide a comparison for indirect effects against trained-model results.
  • For GPT2-small and GPT2-large on Winobias, the sum of direct and indirect effects approximates the total effect.
  • Attention visualizations compare heads across high-total-effect Winobias examples and across GPT2 models, distinguishing stereotypical and anti-stereotypical candidates.

Appendix D. Additional subset selection results

The appendix evaluates selecting small subsets of attention heads or neurons to approximate full-model bias effects. It compares TOP-K and GREEDY selection while addressing computational and optimization constraints.

  • Subset selection targets groups of heads or neurons that jointly reveal sparsity and their impact on gender bias.
  • Subset selection is an NP-hard combinatorial optimization problem, so the analysis uses algorithms motivated by submodular maximization.
  • TOP-K selects elements with the strongest individual effects, whereas GREEDY iteratively selects the largest marginal contributor.
  • Neuron subset selection uses TOP-K because sequential greedy selection is too computationally intensive.
  • Alternative adaptive or parallelized selection methods are identified as future options for neuron subset selection.

Appendix E. Proof that no-interaction in the difference NIE implies decomposition of the TE

Under the no-interaction assumption, the total effect decomposes into a natural direct effect and a natural indirect effect; empirically, the attention-intervention terms closely follow this relation.

  • Proof and assumption: No-interaction in the difference-scale NIE implies TE = NDE + NIE under the paper’s intervention scale.Both sides of Eq. 11 represent NIE contrasts under different fixed interventions on z, with the sentence u held constant.
  • Proof and assumption: Dividing Eq. 11 by ynull and taking expectations over u yields the normalized decomposition used in the analysis.The resulting terms are the normalized NDE and NIE for the set-gender and null interventions.
  • Empirical check: A linear fit across Winobias attention-head examples gives coefficient 1.04, intercept 0.00, and R2 = 0.78 for the two normalized sides.This supports approximate rather than necessarily exact equality after normalization.

Appendix F. Alternate Metrics

The appendix tests three alternate effect metrics across datasets and GPT2 variants, finding broadly consistent relative behavior despite metric-specific differences and scale caveats.

  • Metrics and evaluations: Four metrics are evaluated: the original metric, relative ℓ∞, normalized difference, and Total Variation distance.The alternate metrics use distance measures defined in Equations 10, 8, and 9.
  • Metrics and evaluations: Figures 25–27 apply all metrics to GPT2-small on filtered Winobias Dev, filtered Winogender Bergsma, and Professions datasets.These figures compare alternate metric behavior across the three evaluation datasets.
  • GPT2 results: Across GPT2 variants, alternate metrics produce robust overall relative behavior with only minor visible differences.The appendix notes that this consistency is not necessarily intuitive.
  • GPT2 results: Alternate metrics show less absolute sparsity but preserve relative sparsity, while larger GPT2 models retain more pronounced effects than smaller models.These patterns support the article’s conclusions across the alternate metric settings.
  • Metric caveat: TV-distance places effect values for different models and datasets on different absolute levels because it uses magnitude differences rather than ratios.Normalization to a constant typically restores consistency in the emerging relative behavior.

Appendix G. Attention intervention experiments with masked language models

The appendix adapts continuation scoring for masked language models using six scoring schemes and evaluates attention interventions across several model families. Results vary considerably by masked LM and scheme, but larger variants generally show larger total effects.

  • Scoring schemes: Masked language models lack a single obvious token-probability computation for multi-token continuations, unlike autoregressive models.The appendix therefore evaluates three masked-LM scoring schemes, with and without special tokens.
  • Scoring schemes: Schemes 1, 2, and 3 differ in which continuation tokens remain masked or are supplied when scoring each token.The corresponding primed schemes add cls and sep tokens.
  • Scoring schemes: Scheme 1 is chosen as closest to the autoregressive setting, whereas schemes 2 and 3 derive from prior literature.The appendix also defines exact copies of these schemes with special tokens included.
  • Masked-LM results: Figures 31–35 report head- and layer-level indirect effects for DistilBERT, BERT, and RoBERTa variants under all six scoring schemes.The evaluations use the filtered Winobias Dev dataset.
  • Masked-LM results: Results vary considerably across masked language models and scoring schemes, but larger variants of the same model generally have larger total effects.Tables 7 and 8 report total effects across datasets, models, and scoring schemes.
Loading 2004.12265v2…