PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Abstractions of Neural Networks

Atticus Geiger, Hanson Lu, Thomas Icard, Christopher Potts

arXiv:2106.02997v2cs.AIcs.LG

TL;DR

The paper addresses the lack of systematic causal explanations for neural network representations. It introduces causal abstraction analysis, which aligns neural representations with interpretable causal models and verifies the alignment through interchange interventions. Applied to MQNLI, the method finds that a high-performing BERT model partially realizes the natural logic structure, whereas the simpler baseline does not.

  • Problem

    Existing structural analysis methods reveal aspects of neural representations but provide limited systematic evidence about their causal roles in model behavior.

  • Method

    Causal abstraction analysis formulates an interpretable causal model, searches for an alignment with neural representations, and verifies shared causal properties using interchange interventions.

  • Results

    BERT achieves 88% on MQNLI and partially realizes the natural logic causal structure, whereas the LSTM-based model achieves 46% and does not show that structure.

  • Takeaways & Limitations

    The results demonstrate the feasibility of deriving interpretable causal explanations of neural network behavior and identifying compositional causal structure in BERT representations.

  • Takeaways & Limitations

    The MQNLI abstraction analysis is valid only on subsets of the input space where the abstraction relation holds, because experimental success is partial.

Abstract

from arXiv · show

Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model's causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.

1 Introduction

The paper introduces causal abstraction analysis as a systematic method for explaining neural network behavior, then applies it to MQNLI models. BERT performs well and partially realizes the natural logic causal structure, unlike the LSTM baseline.

  • Motivation: Structural analysis methods can reveal neural networks’ internal structure and complement behavioral evaluations, but existing tools have important limitations.The paper identifies systematic causal analysis as a challenge these methods have often been assumed unable to address.
  • Method: Causal abstraction analysis formulates a high-level causal model, aligns it with neural representations, and verifies the alignment using interchange interventions.The final verification tests whether aligned neural representations share the causal properties of the high-level variables.
  • Case study: BERT achieves 88% accuracy on MQNLI, whereas the LSTM-based model achieves 46%.The BERT model uses a standard NLI-classifier structure, while the result had previously required highly customized task-specific models.
  • Case study: The BERT model partially realizes the natural logic causal structure, while the LSTM model does not.The analysis searches over alignments and performs interchange interventions to test this hypothesis.
  • Relation to prior methods: Causal abstraction analysis systematically aggregates causal properties to evaluate precise hypotheses about abstract causal structure, unlike probing alone.The paper also compares the methodology with integrated gradients, which it characterizes as measuring causal properties.

2 Related Work

The paper distinguishes causal abstraction analysis from probing and attribution methods. Its motivating examples show why information encoded in a representation need not be causally involved in model behavior.

  • Probing: Probes determine what neural representations encode but cannot directly measure their causal properties.The paper cites prior arguments that probes provide limited even indirect evidence about causal properties.
  • Probing: A representation can perfectly encode information while having no causal role in the network’s output.In the addition example, xW1 encodes i + j and xW3 encodes k, but only xW2 contributes to the output.
  • Attribution methods: Integrated gradients can be given an unambiguous causal interpretation for a specific input relative to a baseline.Its components measure average limiting causal effects along the straight line from the baseline vector to the input.
  • Causal abstraction: Causal abstraction analysis measures and aggregates causal properties to evaluate a precise hypothesis about abstract causal structure.The method is presented as complementary to attribution methods while extending beyond their individual causal measurements.
  • Causal abstraction: Constructive abstraction relates high-level and low-level causal models through systematic correspondences between interventions on aligned variables.Low-level variables can be partitioned so each high-level variable corresponds to a partition cell.

3 Causal Abstraction Analysis of Neural Networks

Causal abstraction analysis tests whether neural representations implement an interpretable causal computation. The procedure combines hypothesis formation, alignment search, and interchange-intervention verification.

  • Formulating a hypothesis: The motivating network computes x + y + z from vector representations of three integers.The analysis seeks an informative causal explanation of this network’s behavior.
  • Formulating a hypothesis: The hypothesis aligns an intermediate representation with x + y, another with z, and the output with their sum.This corresponds to a symbolic computation in which the first two inputs are added before the third is incorporated.
  • Alignment search: Alignment search maps input embeddings and the output neuron to model variables while testing possible locations for intermediate variables.Each candidate alignment specifies where the network stores and uses the hypothesized high-level values.
  • Interchange interventions: Interchange interventions replace a base-input representation with the corresponding representation from a source input and compare counterfactual behavior.The same intervention logic is applied to the neural network and the high-level causal model.
  • Verification: If intervention outputs match for all source and base inputs, the corresponding high-level intervention has an equivalent neural intervention.Matching claims for all aligned variables establish the constructive abstraction relation.
  • Interpretation: The method identifies both causally impactful information content and the abstract causal structure through which representations are composed.For the arithmetic example, aligned representations encode high-level variables and play parallel causal roles in producing the output.

4 The Natural Language Inference Task and Models

MQNLI is a difficult, tree-structured natural language inference benchmark whose labeling process supplies a candidate causal model. A pretrained BERT classifier performs well on its hardest generalization task, unlike the BiLSTM.

  • Dataset: MQNLI contains templatically generated NLI examples with complex interactions among quantifiers, negation, and modifiers.Empty-string tokens ensure perfect alignments between premises, hypotheses, and examples.
  • Dataset: MQNLI labels are generated by a tree-structured implementation of natural logic, with train/dev/test splits that vary in difficulty.In the hardest setting, the training set is the minimal set needed for a simple symbolic model to solve development and test examples perfectly.
  • Dataset: The dataset supports causal-structure analysis because its labeling algorithm suggests a causal model for the data-generating process.Its difficult examples provide opportunities to test whether neural models implement variants of that model.
  • Models: The evaluated models are a randomly initialized bidirectional LSTM and a BERT classifier fine-tuned on the MQNLI training set.Both models predict relations for aligned phrase pairs using the final representation above the [CLS] token.
  • Results: BERT achieves high accuracy on the hardest fair generalization task, while the BiLSTM fails to learn it.Without pretraining, BERT achieves approximately 49.1%, and prior models above 50% were task-specific tree-structured systems.
  • Analysis motivation: The study tests whether BERT’s performance reflects an approximation of the tree-like data-generating structure in its internal layers.Causal abstraction analysis is used to evaluate this hypothesis directly.

5 A Case Study in Structural Neural Network Analysis

The case study tests whether natural-logic causal models are abstractions of neural NLI models by searching alignments and validating them with interchange interventions. BERT partially realizes the natural-logic structure, with strongest evidence for the NPObj representation, while probes identify information more broadly than interventions.

  • Hypotheses and alignment search: The analysis formulates natural-logic causal hypotheses and searches possible alignments between their variables and hidden representations in BERT and LSTM NLI models.The search considers layers and token-based locations, including representations above lexical and phrasal nodes.
  • Interchange interventions: Interchange interventions test whether replacing a neural representation produces the same causal result as intervening on its aligned natural-logic variable.A pair is counted as impactful when the intervention changes the output label, and successful pairs match the target causal-model result.
  • NPObj alignment: 383 of 1000 sampled examples support the strongest NPObj alignment, associating NPObj with the fourth-layer representation above the AdjPObj token.Under this alignment, the representation captures the relation between the premise and hypothesis object noun phrases and contributes to the final prediction.
  • Model comparison: The BERT model realizes the natural-logic causal abstraction on significant subsets for almost all tested nodes, whereas the corresponding subsets are much smaller for BiLSTM.Alternative NPObj hypotheses that add or remove lexical items produce smaller clique sizes, supporting the selected causal structure.
  • Comparison with probing and attribution: Probing finds relation information in nearly all considered locations, but interventions identify only a small number of causally relevant locations.This makes interchange interventions more discriminating for causal characterization than probing in the NPObj analysis.

6 Conclusion

The paper introduces causal abstraction analysis for interpretable causal explanations of neural network behavior and demonstrates it on MQNLI. The analysis finds that BERT partially realizes the natural logic model’s causal structure, unlike weaker baseline models.

  • Causal abstraction analysis formulates an interpretable causal model, aligns it with neural representations, and experimentally verifies matching causal properties.The method is grounded in a formal theory of causal abstraction.
  • In the MQNLI case study, the method tested whether neural models recapitulate the causal structure of the natural logic data-generating model.MQNLI provides a challenging dataset whose labeling process suggests an interpretable causal model.
  • A state-of-the-art BERT-based model partially realizes the natural logic causal structure, whereas lower-performing baseline models fail to do so.The conclusion presents this as evidence that the methodology can work in practice.

A.1 Dataset Description

MQNLI uses structured sentence templates with quantified subjects and objects, lexical categories, and optional modifiers. Its examples are designed to support natural-logic composition and include difficult generalization splits.

  • MQNLI sentences follow the template QS AdjS NS Neg Adv V QO AdjO NO.The template includes subject and object quantifiers, adjectives, nouns, negation, an adverb, and a verb.
  • NS and NO are nouns, V is a verb, AdjS and AdjO are adjectives, and Adv is an adverb, with 100 words in each category.Neg is “does not”; quantifiers are every, not every, some, or no, and several modifiers may be empty.
  • Non-identical non-empty lexical items in matching positions are semantically unrelated, making identical items equivalent and different items independent.This construction makes the lexical portion of the learning task trivial.
  • MQNLI labels are generated by a tree-structured natural-logic model that compositionally computes phrase relations.The hardest split is designed to test generalization beyond memorization.

B.2 Dataset Augmentation with Labeled Subphrases

The hard MQNLI task augments each full-sentence example with labeled subphrase examples corresponding to intermediate causal-model nodes. This augmentation is crucial for BERT’s performance on the required generalization task.

  • Each training example is expanded with one labeled subphrase example for every intermediate CNatLog node.The subphrase label is the relation computed by CNatLog, and its label space is disjoint from full-sentence labels.
  • For BERT, subphrase pairs retain their original sentence positions while other positions are padded and masked.The BiLSTM instead aligns subphrase pairs to the left with a separator between the two parts.
  • 88.25% to 55.42%: removing augmented examples decreased BERT’s dev set accuracy.This ablation compares BERT trained with and without the labeled-subphrase augmentation.
  • 88.50% to 54.51%: removing augmented examples decreased BERT’s test set accuracy.The result supports the importance of augmentation for the hard MQNLI generalization task.
  • The experiments analyze 14 intermediate high-level nodes across 11 BERT layers, excluding the final layer.Each node uses a fixed set of hand-specified intervention locations.

C.3 Extended Probe Analysis

Extended probing results are interpreted alongside intervention results across open-class words, multi-word phrases, and closed-class words. Probe selectivity depends strongly on the difficulty of controlling for memorization.

  • Open-class words: For aligned single open-class words, intervention accuracy is near 100% at most locations, while probe selectivity is high mainly in later BERT layers.Lower layers contain more word-identity information, allowing probes to memorize input pairs.
  • Multi-word subphrases: For aligned multi-word subphrases, control probes achieve near-random performance, so selectivity and accuracy differ by the random baseline.This pattern appears for NPSubj, VP, NPObj, QPObj, and NegP.
  • Closed-class words: For aligned closed-class words, probing has near-zero selectivity because the control-function domain is small enough for trivial identity memorization.The category includes quantifiers and negation.

D Probing and Intervention Heatmaps

Figures 5–7 organize probing and interchange-intervention results by BERT layer and token position for high-level natural-logic nodes. Figure 8 additionally compares integrated-gradients attributions for differing and matched positions.

  • Probing and intervention heatmaps: Figures 5–7 cover probing and interchange-intervention results for high-level subject, object, verbal, adjectival, adverbial, quantifier, and negation nodes.The reported nodes include NPObj, NObj, AdjObj, VP, V, Adv, QObj, QPObj, Neg, NegP, NPSubj, NSubj, AdjSubj, and QSubj.
  • Heatmap layout: Vertical axes denote BERT layers, while horizontal axes denote token positions of hidden representations.
  • Intervention and clique measures: Intervention success rates count experiments that changed the output label, whereas clique sizes are percentages of all examples.
  • Integrated gradients: Figure 8 compares integrated-gradients values at the unique differing position with values at a randomly selected matched position.The two populations differ according to a Wilcoxon signed-rank test with p < 0.00001.

E Integrated Gradients

Integrated gradients provide attribution values with a causal interpretation, while causal abstraction formalizes mappings between low-level and high-level causal models. Constructive abstraction further decomposes this mapping into component functions over low-level variable partitions.

  • Integrated gradients: For examples differing at exactly one aligned position, the analysis compares that position with a randomly selected position elsewhere.
  • Integrated gradients: The differing positions have positive attribution on average, and the differing and matched populations are statistically distinct.The caption reports a Wilcoxon signed-rank test with p < 0.00001.
  • Causal abstraction: A τ-abstraction maps low-level variable settings to high-level settings and requires surjectivity plus agreement under corresponding interventions.The induced intervention map is denoted ωτ.
  • Constructive abstraction: A constructive τ-abstraction additionally partitions low-level variables so each high-level variable has a component mapping from its associated partition.

G Causal Abstraction Analysis of C+

The paper defines a causal model for a simple addition network and proves that it is a constructive abstraction of the corresponding neural network. The proof checks surjectivity, intervention correspondence, and matched causal evolution.

  • Formal definition of C+: The addition causal model assigns input variables X, Y, Z, and W domains N9, with intermediate variables S1 and S2 having domains N18 and N27.
  • Constructive mapping: The abstraction mapping partitions neural variables into cells for inputs, hidden units, and output, using one-hot decoding for inputs and identity mappings for selected internal variables.
  • Proof conditions: The proof establishes that the mapping τ and induced intervention mapping ωτ are surjective onto the high-level states and admissible interventions.
  • Proof conditions: The two causal processes produce corresponding outcomes under every admissible intervention, completing the proof that C+ is an abstraction of N+.

H Causal Abstraction Analysis of CNatLog

The paper constructs a natural-logic causal model and a neural causal model for MQNLI, then defines an alignment whose successful interchange interventions establish the natural-logic model as an abstraction of the neural models.

  • Natural-logic causal model: The natural-logic model represents lexical and compositional relations using seven natural-logic relations and functions including REL, COMP, and PROJ.Its variables cover subject and object nouns, adjectives, verbs, adverbs, quantifiers, negation, and intermediate composition nodes.
  • Neural causal model: The neural causal model treats representations as a grid across layers and token positions, followed by a three-class prediction from the final [CLS] representation.The LSTM has n = 2 rows and BERT has n = 12; m is the number of tokenized MQNLI tokens.
  • Alignment: For each non-leaf natural-logic node, the abstraction maps token-vector sequences to input variables, selected representations to intermediate values, and logits to output predictions.The component functions are T for input variables, QL for an intermediate variable, and P for the three-way output.
  • Intervention domain: The admissible neural interventions are restricted because the component mappings T, QL, and P are partial functions.
  • Proof of abstraction: The proof verifies surjectivity, intervention correspondence, and equality of causal outcomes, concluding that CNatLog is an abstraction of CNNLI.The final causal-process condition is supported by successful intervention experiments.
  • Proof of abstraction: The paper states that the abstraction result holds for each non-leaf node in CNatLog when the corresponding representation location satisfies the intervention condition.
Loading 2106.02997v2…