Source-linked AI summary
Invariant Rationalization
Shiyu Chang, Yang Zhang, Mo Yu, Tommi S. Jaakkola
TL;DR
MMI-based rationalization can select spurious correlations rather than features underlying the input-output relationship. The paper introduces game-theoretic invariant rationalization, which constrains rationales across environments, and reports better removal of false correlations, generalization, and human alignment.
Problem
MMI may select features that correlate with outputs in training data but do not explain the underlying relationship when their effects change at test time.
Method
INVRAT uses a rationale generator and environment-agnostic and environment-aware predictors to enforce invariant rationales through a game-theoretic framework.
Results
INVRAT theoretically solves invariant rationalization and empirically removes false correlations, generalizes to unknown test environments, and better aligns explanations with human judgments.
Takeaways & Limitations
Invariant rationales can distinguish valid explanations from features that are merely predictive because their relationships with outputs vary across environments.
Takeaways & Limitations
The analysis assumes p(Y |X1) remains the same across environments because X1 is the true cause of Y.
Abstract
from arXiv · showhide
Selective rationalization improves neural network interpretability by identifying a small subset of input features -- the rationale -- that best explains or supports the prediction. A typical rationalization criterion, i.e. maximum mutual information (MMI), finds the rationale that maximizes the prediction performance based only on the rationale. However, MMI can be problematic because it picks up spurious correlations between the input features and the output. Instead, we introduce a game-theoretic invariant rationalization criterion where the rationales are constrained to enable the same predictor to be optimal across different environments. We show both theoretically and empirically that the proposed rationales can rule out spurious correlations, generalize better to different test scenarios, and align better with human judgments. Our data and code are available.
1. Introduction
Selective rationalization seeks compact input subsets that support predictions, but maximum mutual information can mistake correlated features for explanations. INVRAT instead imposes invariance across environments and is reported to improve spurious-correlation removal, generalization, and alignment with human judgments.
- Selective rationalization identifies a small input subset that suffices to yield the same outcome and can serve as a justification for predictions.
- MMI selects a rationale that maximizes prediction accuracy or mutual information between the selected subset and the output.
- MMI may select spurious correlations that reflect model behavior but fail to explain the underlying input-output relationship under changed test conditions.
- Invariant rationalization approximates causal-feature selection by requiring predictors based on causal features to remain optimal across environments.
- INVRAT uses a rationale generator, environment-agnostic predictor, and environment-aware predictor to close their performance gap while preserving predictive accuracy.
- INVRAT theoretically solves the invariant rationalization problem and empirically removes false correlations, generalizes to unknown environments, and better aligns with human judgments.
2. Preliminaries: MMI and Its Limitation
MMI can select features that predict an output without being its valid explanation because correlated, downstream, or unrelated variables may have equal predictive power. The section formalizes this limitation through a probabilistic model and toy construction.
- MMI finds a masked rationale that maximizes mutual information with the output under sparsity and continuity constraints.
- The probabilistic model divides input X into X1, X2, and X3, representing a cause of Y, a variable influenced by Y, and a variable without direct connection to Y.
- X1 is the valid explanation, whereas X2 and X3 are invalid explanations despite all three potentially being highly predictive of Y.
- The toy construction sets priors so that X1, X2, and X3 have exactly the same predictive power, leaving MMI no basis to favor X1.
- In the beer-review example, smell corresponds to X1, the overall review to X2, and the palate review to X3, which can remain predictive through correlations.
3. Adversarial Invariant Rationalization
INVRAT enforces invariant rationales by requiring predictions from the rationale to remain independent of the environment, using a game between rationale generation and environment-aware and environment-agnostic predictors. The framework is designed to exclude spurious correlations and support worst-case generalization to unknown environments.
- 3.1. Invariant Rationalization: A small change in environment priors exposes non-invariant features: q(Y |X2) and q(Y |X3) differ from their original conditional relationships, whereas q(Y |X1) remains unchanged.The reported values are qY |X2(1|1) ≈0.926, qY |X2(0|0) ≈0.867, qY |X3(1|1) ≈0.912, and qY |X3(0|0) ≈0.883.
- 3.1. Invariant Rationalization: Invariant rationalization maximizes I(Y ; Z) while constraining Y ⊥E | Z, so the rationale screens out features whose predictive relationships vary across environments.The environment variable changes feature priors, while the conditional relationship between the true cause and Y is assumed stable.
- 3.2. The INVRAT Framework: The generator cooperates with the environment-agnostic predictor but adversarially challenges the environment-aware predictor, with optimization performed by alternating gradient descent/ascent.This game approximates the constrained optimization in the invariant rationalization objective.
- 3.2. The INVRAT Framework: INVRAT implements the constraint as a three-player game involving a rationale generator, an environment-agnostic predictor, and an environment-aware predictor.Both predictors use the rationale to predict Y; only the environment-aware predictor additionally receives E.
- 3.4. Invariance and Generalizability: Theoretical analysis shows that the invariant rationale minimizes risk under the most adverse unknown test environment, with Z = X1 achieving the minimax saddle point under stated assumptions.The analysis assumes sufficient training data, an environment-agnostic predictor with sufficient representation power, and convergence to a global optimum.
- 3.5. Incorporating Sparsity and Continuity Constraints: Sparsity and continuity constrain the mask so its number of selected positions is bounded and its selected positions are contiguous.These constraints are incorporated alongside the invariant rationalization objective.
4. Experiments
INVRAT is evaluated on synthetic IMDB data with controlled punctuation correlations and multi-aspect beer reviews with correlated ratings and human annotations. Across these settings, it excludes false correlations and produces rationales that better match human judgments than the baselines.
- Datasets: The experiments use synthetic IMDB data with environment-dependent punctuation correlations and multi-aspect beer reviews with correlated aspect ratings.The beer dataset also includes sentence-level annotations for about 1,000 reviews, enabling automatic rationale-precision evaluation.
- Baselines and implementation: INVRAT is compared with RNP, an MMI-based generator-predictor, and 3PLAYER, which adversarially trains against a complement predictor.The implementations use matched settings, including bidirectional GRUs and fixed GloVe embeddings.
- Synthetic IMDB results: 78.24% of RNP testing samples select injected punctuation, whereas INVRAT selects none and achieves about 15% higher test performance.3PLAYER is excluded because the dataset construction makes it fail by including all punctuation as rationales.
- Beer review results: INVRAT consistently surpasses both baselines in human-annotation alignment across most rationale lengths and aspects.The evaluation reports precision, recall, and F1 for highlight lengths of 10, 20, and 30.
- Beer review results: INVRAT improves F1 by more than 20 absolute percentage points for the appearance aspect because the baselines fail to distinguish aspect-specific clues.The generated rationales are also visualized for multiple beer-review aspects.
- Subjective evaluation: In subjective evaluation, participants identify the target aspect from INVRAT rationales most accurately for all tested rationale lengths.The study uses 400 held-out examples per aspect and tests rationale lengths of 10, 20, and 30.
5. Related Work
The paper relates invariant rationalization to selective rationalization, broader self-explaining models, and research on dataset-specific biases.
- Selective rationalization: Selective rationalization uses a generator-predictor framework to select input spans that support predictions, commonly optimizing mutual information with labels.The related work positions this framework as a cooperative game.
- Self-explaining models beyond selective rationalization: Other interpretability approaches compose predefined modules or estimate feature importance using gradients and local perturbations.Module-based methods are limited by their restriction to predefined programs.
- Learning with biases: The work also connects to studies showing that neural models can fit dataset-specific biases, such as word overlap or negation cues in natural language inference.These examples motivate examining spurious patterns learned by neural models.
6. Conclusion
The paper proposes game-theoretic invariant rationalization to constrain rationale-conditioned predictions across environments and rule out spurious words. Theory and evaluations on synthetic and multi-aspect datasets show favorable rationale-generation performance and better generalization to shifted test distributions.
- Conclusion: INVRAT constrains the probability of the output conditional on rationales to be the same across multiple environments.Its three players competitively rule out words strongly correlated with the output.
- Conclusion: The authors theoretically show that the game-theoretic framework drives solutions toward better generalization when test distributions differ from training distributions.The conclusion reports both synthetic and multi-aspect sentiment evaluations.
A. Proof To Theorem 1
The proof partitions variables into invariant and non-invariant components, then selects priors under which non-invariant variables are uninformative about Y. It shows that only invariant features remain predictive and that optimizing over the relevant priors preserves the inequality for every π1.
- Proof To Theorem 1: The proof partitions any variable set Z into invariant variables ZI and non-invariant variables ZV.
- Proof To Theorem 1: The constructed priors make all non-invariant variables uninformative about Y, and this choice is feasible for the test adversary.
- Proof To Theorem 1: Under these priors, non-invariant features are not predictive of Y, whereas invariant features are predictive.
- Proof To Theorem 1: The proof uses the invariant-variable identity p(Y | X1, E) = p(Y | X1) to justify the final equality in the loss expression.
- Proof To Theorem 1: Because p(Y | X1, ea) does not depend on π2 and π3, optimizing over those priors yields the stated relation.
- Proof To Theorem 1: Since the inequality holds for every π1, taking the maximum over π1 preserves it.