Inducing Causal Structure for Interpretable Neural Networks

1. Introduction

• Symbolic causal knowledge may compensate for data gaps or provide useful inductive biases while models continue learning from data.The paper asks how to bring such knowledge into trained models without abandoning data-driven learning.Source: §1 · p. 1
• IIT aligns causal-model variables with neural representations and trains the neural model to match the causal model’s counterfactual behavior under aligned interchange interventions.Its objectives are differentiable, and zero loss guarantees the target causal model is a causal abstraction of the neural network.Source: §1 · p. 1
• The method extends causal abstraction analysis from passively studying models to optimizing them to acquire specific causal structures.This creates an interplay between model analysis and model improvement.Source: §1 · p. 1
• The evaluation covers MNIST-PVR vision, ReaSCAN grounded language understanding, and MQNLI natural language inference.Each case aligns high-level causal variables with low-level neural representations to define IIT objectives.Source: §1 · p. 1
• IIT outperforms multi-task objectives and data augmentation on systematic generalization benchmarks and yields more interpretable models.Interpretability is assessed by how well models realize the target causal models.Source: §1 · pp. 1–2

Probes

• Probes reveal encoded information, but probed information is not guaranteed to play a causal role in network behavior (Ravichander et al., 2020; Elazar et al., 2020; Geiger et al., 2021; 2020).The paper distinguishes encoding evidence from causal relevance.Source: §2 · p. 2

Feature Attribution

• Feature-attribution methods can measure causal properties, but they passively study trained networks rather than actively improving them.Erion et al. (2021) is noted as a path from attribution to improved optimization.Source: §2 · p. 2

Intervention-Based Analyses

• Intervention-based analyses change input or internal representations to study effects, and IIT extends these methods to optimization.Their finely controlled interventions provide a toolkit for understanding causal structure.Source: §2 · p. 2

Multi-Task Training

• Multi-task training jointly trains a model on multiple tasks, and the paper uses causal-model-defined objectives as a comparison point for IIT (Crawshaw, 2020).Unlike standard supervised probing, multi-task training allows the target model’s parameters to change during the probing process.Source: §2 · p. 2

Data Augmentation

• Data augmentation generates additional examples from existing ones, whereas IIT interchanges internal representations rather than only creating inputs (Perez & Wang, 2017; Kaushik et al., 2019).The paper uses causal-model-generated data augmentation as another comparison point.Source: §2 · p. 2

• An interchange intervention runs the model on a base and source input, then replaces the base run’s aligned internal state with the state obtained from the source.The intervention’s output is the model’s output for the base input with that internal-state substitution.Source: §3 · p. 3
• A causal abstraction requires aligned interventions in the high- and low-level models to have the same effects across input pairs.The alignment maps high-level intermediate variables to non-overlapping subsets of low-level variables.Source: §3 · p. 3
• A causal model that is an abstraction of a network serves as a faithful interpretation of how it achieves its behavior, rather than a story about its reasoning.The aligned high-level variables and their parenthood relation help interpret neural representations and how they are composed.Source: §3 · p. 3
• Interchange intervention accuracy measures the percentage of aligned interventions producing the same output; exhaustive evaluation may be impractical because the input-pair space can be enormous.In that case, the measure is estimated using randomly sampled input pairs.Source: §3 · p. 3
• IIT compares the causal model’s counterfactual output with the neural network’s intervened prediction, then combines that loss with other objectives and updates the network.The intervention joins computation graphs from the base and source runs, so backpropagation branches at the intervention site and updates weights from both runs.Source: §3 · pp. 3–5
• Despite perfect Boolean-conjunction accuracy, the example network’s interchange intervention accuracy is 81.25%; after one IIT update, it reaches 1 and the causal model becomes an abstraction of the network.The initial network matches the causal model on all four inputs but disagrees on some aligned interventions.Source: §3 · p. 4, Figure 1 · p. 4
• When the IIT loss is zero, the causal model and neural network are in a causal abstraction relationship, although the reverse does not necessarily hold.The example illustrates the asymmetry: its network becomes an abstraction after an IIT update even though the initial network already had perfect behavioral accuracy.Source: Figure 1 · p. 4, §3 · p. 5

Task and Symbolic Causal Structure

• The task uses four MNIST digits in a grid; the top-left digit points to one of the other three, and the causal model abstracts away digit recognition to represent pointer reasoning.The systematic-generalization split removes training cases with specified digit values in the non-pointer positions.Source: §4 · p. 5

Alignments

• Neural representations are aligned to quadrants of the first max-pooling layer, with each quadrant directly above its corresponding input; alignments closer to the classifier head were ineffective.The authors attribute the placement sensitivity as likely due to convolutional locality.Source: §4 · p. 5

Typed Interchange Intervention Training

• For typed IIT, the four intermediate causal variables and their aligned neural representations share value spaces, enabling interventions between different variables.The method also compares against multi-task prediction of intermediate variables and data augmentation that swaps input quadrants.Source: §4 · pp. 5–6

Results

• 96.01% behavioral accuracy and 96.64% interchange intervention accuracy are the best reported MNIST-PVR test results, achieved by combining IIT with multi-task training.IIT alone reaches 93.93% behavioral accuracy and 94.85% interchange intervention accuracy; standard and multi-task-only models fail to generalize behaviorally (0%).Source: §4 · pp. 6–7
• IIT reaches 92.00% test interchange intervention accuracy with data augmentation, but still underperforms IIT; augmentation alone remains below 90.90% behavioral performance.Standard training has 88.80% training interchange intervention accuracy but only 20.60% on test data.Source: §4 · pp. 6–7

Typed Interchange Intervention Training

• Typed IIT is crucial for generalization: without it, behavioral and interchange intervention accuracy plummets on test data.The best model's causal model CPVR is described as a near-perfect abstraction of its neural network.Source: §4 · p. 7

Symbolic Causal Structure

• The causal solver parses target attributes, locates the target and agent, computes their horizontal and vertical distances, then produces an action sequence.The experiment uses the simplest ReaSCAN command structure, excluding relative clauses.Source: §5 · pp. 7–8

Systematic Generalization

• The four test splits probe unseen compositions of command attributes and output actions, including novel color, size, direction, and length.The test sets assess zero-shot generalization to unseen combinations of concepts seen during training.Source: §5 · p. 8

Alignments

• The model aligns target-property variables to chunks of the noun representation and position-difference variables to chunks of the decoder's initial hidden representation.The target-property representation is split into three 25-dimensional chunks, and the 100-dimensional initial hidden representation into two 50-dimensional chunks.Source: §5 · p. 8

Results

• IIT and multi-task training together produce the best behavioral generalization across all four ReaSCAN test splits.IIT alone outperforms multi-task training on novel sizes and lengths, and performs similarly on novel colors and lengths.Source: §5 · p. 8
• The combined IIT and multi-task model also achieves the strongest interchange intervention accuracy by wide margins across all four splits.Without IIT, standard and multi-task models show non-zero accuracy only on the easier novel-color and novel-size splits; IIT also gains traction on novel direction and length.Source: §5 · p. 8

Symbolic Causal Structure

• The causal submodel computes the relation between aligned quantified verb phrases, then uses it to infer the relation between the sentences.The quantified verb phrase comprises the adverb, verb, and quantified object noun phrase.Source: §6 · p. 9

Systematic Generalization

• The MQNLI train-test split is designed to be difficult while remaining solvable by a compositional memorization-based model.The task labels sentence relations as entailment, contradiction, or neutral.Source: §6 · pp. 8–9

Results

• Data augmentation gives excellent or perfect behavioral accuracy but low interchange intervention accuracy, so behavioral success alone does not establish an interpretable solution.The authors state IIT is needed to produce an interpretable solution.Source: §6 · p. 9
• IIT on the quantified-phrase causal model solves MQNLI with near-perfect behavioral and IIT accuracy (approximately 100%).With alignment at BERT layer 6 or later, multi-task training further improves the best models.Source: §6 · p. 9
• Aligning the quantified-phrase variable to BERT's first few layers yields lower accuracy; layer 6 and later yield near-perfect behavioral and intervention accuracy.The evaluation reports results on the evaluation set.Source: §6 · p. 9

7. Conclusion

• Across all three case studies, IIT models perform best on challenging behavioral evaluations and conform best to the task-specific high-level causal models.The settings are MNIST-PVR, ReaSCAN, and MQNLI.Source: §7 · p. 9
• The paper applies IIT to vision, grounded language understanding, and natural language inference.The case studies use MNIST-PVR, ReaSCAN, and MQNLI, respectively.Source: §7 · p. 9
• IIT combines with multi-task objectives, which further strengthen the results.Source: §7 · p. 9

A. Zero Loss Entails Causal Abstraction

• If the non-negative loss is zero and zero loss implies x = κ(y), then the causal model is a causal abstraction of the neural network.Source: A. Zero Loss Entails Causal Abstraction · p. 13
• A zero sum of the loss terms makes each addend zero.Source: A. Zero Loss Entails Causal Abstraction · p. 13
• The loss function’s zero-loss property yields the condition for abstraction in Eqn. 2.Source: A. Zero Loss Entails Causal Abstraction · p. 13

B. ReaSCAN

• Novel-color and novel-size splits reuse one training set with separate test sets, while novel-length generation restricts commands to “walk to” and excludes adverbs.Novel-color and novel-size generation allows three verbs and four adverbs; novel direction and novel length restrict verbs to “walk to” and exclude adverbs.Source: B. ReaSCAN · p. 13
• ReaSCAN generation has known artifacts, while the symbolic causal structure and variable mappings are non-unique and mapping choice substantially affects performance and generalizability.The formulation of base/source example pairs also affects IIT performance, a question left for future research.Source: B. ReaSCAN · p. 13
• Training uses Adam with a decaying learning rate for 100,000 epochs, selects checkpoints on a 2,000-example development set, and repeats experiments across three random seeds.Training takes about one day on a GeForce RTX 2080 Ti; dataset generation takes approximately 30 minutes on a multi-CPU cluster.Source: B. ReaSCAN · p. 13
• ReaSCAN tests zero-shot generalization on unseen composites through novel color, novel size, novel direction, and novel action-sequence-length splits.Source: B. ReaSCAN · p. 14
• The attribute splits hold out unseen combinations such as “yellow square” and small-cylinder commands; the direction split holds out southwest targets, and the length split tests sequences of 11–13 actions.Source: B. ReaSCAN · p. 14
• The ReaSCAN model uses a CNN for grid worlds, a bidirectional LSTM for commands, and an LSTM decoder with cross-modal attention.Source: B. ReaSCAN · p. 14

C. Experiment Details of Multiply Quantified Natural Language Inference (MQNLI)

• The MQNLI dataset uses 500K training, 60K development, and 10K test examples, with complete sentences and QPObj labels for IIT and multi-task objectives.Source: C. Experiment Details of Multiply… · p. 14
• The BERT model is the uncased-base variant with 12 layers and hidden size 768; the QPObj probe is a one-layer softmax classifier with a 32-dimensional factorization.Source: C. Experiment Details of Multiply… · p. 14
• Training pairs two MQNLI samples, replacing the first example’s quantified verb phrase with the second’s; IIT uses the first as base and the second as intervention source.The experiments use 50,000 base examples and 20 intervention sources per base example, with 10 intended to produce impactful interventions.Source: C. Experiment Details of Multiply… · p. 14
• All settings use the same 50,000 × 20 pairs per epoch, and each added objective receives weight 1.0 alongside the base objective.Source: C. Experiment Details of Multiply… · p. 14