Source-linked AI summary
A causal framework for explaining the predictions of black-box sequence-to-sequence models
David Alvarez-Melis, Tommi S. Jaakkola
TL;DR
Black-box structured predictors are difficult to interpret, especially when both inputs and outputs are combinatorial objects. The paper introduces SOCRAT, which uses VAE-generated perturbations, causal token graphs, and partitioning to explain specific predictions. Across sequence-generation tasks, the method recovers known dependencies and yields coherent insights into translation systems and their biases.
Problem
Complex structured predictors can be effective yet remain difficult to interpret, while existing prediction explanations largely target scalar or categorical outputs.
Method
SOCRAT queries a black-box model with VAE-generated input perturbations, infers causal relationships between input-output tokens, and selects relevant explaining components.
Results
The method recovers known dependencies, resembles neural machine translation attention scores, and provides coherent insights into translation behavior and biases.
Takeaways & Limitations
The framework provides a partial view into black-box structured systems and suggests that explanations can support questioning predictions and improving existing systems.
Takeaways & Limitations
The study focuses on sequence-to-sequence tasks and uses VAE-based sampling for perturbations, although other domains and perturbation approaches are possible.
Abstract
from arXiv · showhide
We interpret the predictions of any black-box structured input-structured output model around a specific input-output pair. Our method returns an "explanation" consisting of groups of input-output tokens that are causally related. These dependencies are inferred by querying the black-box model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to select the most relevant components. We focus the general approach on sequence-to-sequence problems, adopting a variational autoencoder to yield meaningful input perturbations. We test our method across several NLP sequence generation tasks.
1 Introduction
The paper introduces a model-agnostic framework for explaining structured black-box predictions through causally related input-output tokens. It uses VAE-generated perturbations and evaluates the resulting explanations across sequence-to-sequence tasks.
- Motivation: Complex NLP predictors achieve strong results but remain largely uninterpretable, motivating methods that explain individual predictions.The motivation includes trust, error analysis, model refinement, and bias detection.
- Framework: The framework explains structured black-box predictions using sets of input and output tokens that are causally related.It requires only oracle access and summarizes model operation rather than replacing the model with a simpler predictor.
- Evaluation: The method recovers known dependencies, resembles neural machine translation attention scores, and provides insights into translation behavior and biases.Experiments include recovery of a grapheme-to-phoneme dictionary and qualitative comparison with attention scores.
- Contribution: The authors propose a general framework for explaining structured black-box models.The framework is evaluated on multiple sequence-to-sequence prediction tasks.
- Method: A variational autoencoder generates semantically related perturbations of sequential inputs for causal analysis.The perturbation model changes elements and their order while targeting preservation of semantic content.
2 Related Work
The related work spans interpretable modeling and post-hoc prediction explanations, while this paper extends local, model-agnostic explanation to structured inputs and outputs.
- Interpretability research: Interpretability research spans multiple fields, with goals, methods, and formalisms that are often non-overlapping.The literature includes medical applications and methods using activations, gradients, or influence propagation.
- Prior approaches: Prior approaches explain scalar or categorical predictions using input fragments or locally faithful sparse linear models.Lei et al. incorporate explanation generation into learning, while Ribeiro et al. explain black-box classifiers post hoc.
- Contribution: This framework extends local, model-agnostic explanation to structured inputs and outputs, including variable-size combinatorial objects.The structured setting requires rethinking what an explanation should contain.
- Contribution: The paper addresses the challenge of generating meaningful perturbations for structured objects in sequence-to-sequence learning.This perturbation problem is identified as a key difference from earlier model-agnostic classifier explanations.
3 Interpreting structured prediction
The paper formulates structured prediction explanations as local, model-agnostic causal relationships between input and output tokens. SOCRAT infers these relationships from perturbations and selects coherent explaining components.
- 3 Interpreting structured prediction: Structured predictions vary in size and complexity, so explanations must decide which output parts to explain and should decompose into cognitive chunks.The desired explanation size should grow with the input and output, while each component remains bounded in size.
- Formal setup: The framework represents a black-box mapping from feature-set inputs to feature-set outputs, including sequences, graphs, and images.Input and output elements are called units or tokens, although the formulation applies to other combinatorial objects.
- Formal setup: For a given input-output pair, the method seeks a locally faithful and model-agnostic explanation of output tokens in terms of input tokens.It models token influence with a weighted bipartite graph whose edges connect input tokens to output tokens.
- Explaining components: An explanation is a collection of subgraph components in which selected input elements are likely causes of selected output elements.The components provide potentially several justifications for parts of the output relative to parts of the input.
- SOCRAT: SOCRAT combines local perturbation, causal inference, and selection steps to partition and retain the most relevant associations.The framework is named the structured-output causal rationalizer.
- Alignment models: Global alignment models require substantial parallel data and may have varying coverage of the local context, whereas this approach explains the black-box mapping locally.The distinction concerns local behavior around the current input rather than a globally trained alignment model.
4 Building blocks
The method builds interpretable explanations by perturbing structured inputs, estimating token-level dependencies, and selecting robust graph partitions. For sequence inputs, a VAE generates semantically similar variations, while uncertainty-aware partitioning identifies relevant explanation components.
- Perturbation Model: A variational autoencoder generates semantically similar input perturbations with possible changes in elements and order.Perturbations are introduced in a continuous latent representation rather than directly on structured inputs.
- Perturbation Model: Repeatedly decoded samples are passed through the black-box model to obtain perturbed input-output pairs exercising its local behavior.The resulting pairs provide the data used for causal analysis around the original input.
- Causal Inference: For each output token, logistic regression uses simultaneous input-token presence indicators to estimate dependency coefficients and their uncertainty.The method produces coefficients θij and uncertainty estimates for all original input-output token pairs.
- Explanation Selection: The dense bipartite dependency graph is partitioned into disjoint subsets to minimize the weight of omitted edges and yield interpretable explanation components.When the input and output are small, all resulting partitions can be returned as explanation chunks.
- Explanation Selection: Robust partitioning incorporates interval-valued edge-weight estimates by minimizing worst-case cut values through a mixed-integer programming formulation.The robustness parameter Γ controls the number of deviations from mean edge values considered in the partition.
5 Experimental Framework
The experiments evaluate SOCRAT from recovering simple character-to-phoneme dependencies to explaining machine translation, dialogue predictions, and translation biases. Across these settings, the method produces structured explanations that can be compared with alignments or attention and used to inspect model behavior.
- Experimental setup: The sentence experiments use a pretrained variational autoencoder for perturbations and compare robust partitioning with coclustering and biclustering.The robust formulation incorporates uncertainty, whereas the classical alternatives do not.
- 5.2 Recovering simple mappings: The CMU Dictionary experiment tests whether SOCRAT can infer character-to-phoneme dependencies from a known grapheme-to-phoneme mapping.The dataset contains about 130K word-to-phoneme pairs, with perturbations drawn from words within edit distance 2.
- 5.2 Recovering simple mappings: SOCRAT’s uncertainty-clustering version achieves an alignment error rate only ten points above an oracle alignment system trained on the full ARPAbet dictionary.Evaluation uses 100 randomly selected dictionary pairs with manually labeled alignments; Figure 2 reports results across perturbation counts and five repetitions.
- 5.3 Machine Translation: For English-to-German translation, explanations for Azure, NMT, and human outputs show mostly consistent clustering around two input phrases, with the subject chunk receiving the highest cut value.The human system has overall higher dependency coefficients than the automatic systems, potentially because its translations were produced in context.
- 5.3 Machine Translation: The NMT comparison clusters its dense attention matrix before comparing it with SOCRAT’s dependency estimates and explanation graph.Attention is shown for qualitative comparison because it is not directly comparable to the dependency scores without normalization and partitioning.
- 5.4 A (mediocre) dialogue system: In the dialogue experiment, a seemingly good prediction is explained as being influenced mainly by the tokens “What” and “you,” exposing the limited basis of the output.The model was intentionally constrained and produced mostly short, repetitive responses despite some seemingly meaningful predictions.
- 5.5 Bias detection in parallel corpora: The biased-translation experiments use SOCRAT to inspect artificial and industrial translation behavior, including gender associations between source tokens and gender-marked French outputs.The industrial system shows stronger cross-word associations in gendered examples than in a plural example without grammatical-gender choice.
6 Discussion
The framework produces coherent explanations for structured-data predictions and suggests potential for improving systems. Although demonstrated on sequence-to-sequence tasks, it can generalize to settings where inputs and outputs are feature sets.
- The framework produces reasonable, coherent, and often insightful explanations for structured-data predictions.
- The machine translation results provide a partial view into the inner workings of black-box systems.
- SOCRAT can generalize beyond sequence-to-sequence tasks when inputs and outputs are expressed as sets of features.
- The method uses VAE-based sampling for perturbations, but other approaches may be suitable depending on the domain or data.
A Formulation of graph partitioning with uncertainty
The uncertain bipartite graph formulation partitions both vertex sets into bounded subsets while modeling uncertain edge contributions through a robust objective. Binary variables encode memberships and cross-partition edges, and dualization yields a linear formulation solvable as a mixed integer program.
- The problem partitions U and V into K subsets each, with subset cardinalities constrained to specified ranges.
- Binary indicators encode vertex membership, while y_ij indicates whether paired vertices belong to different corresponding subsets.
- Constraints enforce unique subset membership, subset-size bounds, and the definition of y_ij.
- The robust formulation incorporates edge uncertainty represented by intervals θ_ij ± ˆθ_ij through an additional objective term.
- The robustness parameter Γ ranges from 0 to |V| and trades greater attention to edge variance against a more complex optimization problem.
- Dualizing the uncertainty term returns the objective to a linear form, producing a mixed integer programming problem solvable with specialized packages such as GUROBI.
B Details on optimization and training
Optimization uses approximate MIP solutions because exact optimality can be prohibitive for large graphs. Relaxing the solver gap tolerance substantially reduces runtime with little observed change in solutions.
- Relaxing Gurobi’s absolute gap tolerance from 10^-12 to 10^-4 decreased solve time by orders of magnitude with little minimal change in solutions.The method also imposes a two-minute runtime limit.
C Details on the variational autoencoder
The perturbation model is a variational autoencoder trained on English WMT14 sentences and used to sample local sentence variations. Its architecture and training schedule specify the latent representation and annealing procedures used in the experiments.
- The shared VAE uses three-layer GRU encoder and decoder networks connected by a stacked three-layer variational autoencoder.
- Table 3 presents English VAE samples around two WMT14 input sentences for increasing α.
- The VAE is trained on a 10M-sentence English subset of WMT14 with KLD and variance annealing for 50 epochs.
- PERTURB encodes an input sentence, samples N latent representations from a Gaussian distribution, decodes them, and returns the resulting sentence variations.
- The scaling parameter α controls locality by scaling posterior variance, with larger values producing samples farther from the mean encoding.
D Black-box system specifications
The machine translation evaluation used three systems: Azure’s MT Service, a neural sequence-to-sequence model, and a human translator. The neural system was trained on WMT15 English-German data, while the human translated perturbed English sentences without additional context.
- The machine translation task evaluated Azure’s MT Service, a neural sequence-to-sequence system, and a human translator.These three systems are introduced for the task in Section 5.3.
- Azure’s system used REST API calls to Microsoft’s Translator Text service through Azure cloud services.
- The neural system used attention, OpenNMT, and WMT15 English-German training data, with two layers, hidden state dimension 500, and 13 training epochs.A pretrained model was obtained from the OpenNMT model repository.
- The human translator was a native German speaker fluent in English who translated perturbed English sentences in one pass without additional instructions or context.For sentences not directly translatable as written, the translator rendered them word-to-word as far as possible.