PaperlayerView exact revision on arXiv

Source-linked AI summary

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel S. Weld

arXiv:2101.00288v2cs.CL

TL;DR

Existing counterfactual generation is costly or limited in perturbation scope, despite counterfactuals’ value for NLP analysis and training. Polyjuice fine-tunes GPT-2 on paired sentences with controls over perturbation types and locations, producing realistic variants for multiple applications. It supports training, evaluation, explanation, and error analysis, with around 70% less annotation effort than creating counterfactuals manually.

  • Problem

    Prior counterfactual generators require costly manual creation or support only limited perturbation relationships, restricting counterfactuals available for NLP analysis and training.

  • Method

    Polyjuice fine-tunes GPT-2 on sentence pairs to generate application-agnostic counterfactuals with control codes and blanking that specify perturbation types and locations.

  • Results

    Polyjuice supports training, evaluation, explanation, and error analysis, achieving around 70% less annotation effort than manual counterfactual creation.

  • Takeaways & Limitations

    A single generator paired with task-specific selection heuristics supports diverse downstream counterfactual applications across tasks and domains.

  • Takeaways & Limitations

    Polyjuice generation is not exhaustive, may favor some perturbation patterns unpredictably, and can miss complex or context-dependent changes.

Abstract

from arXiv · show

While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.

1 Introduction

Polyjuice separates general-purpose counterfactual generation from downstream selection, addressing costly and narrow prior approaches. A GPT-2-based generator controls perturbation types and locations, supporting training, evaluation, explanation, and error analysis.

  • Counterfactual reasoning supports causality assessments relevant to NLP model evaluation, error analysis, and explanation.
  • Polyjuice generates an application-agnostic set of counterfactuals before application-specific methods select useful subsets.This separates generation from downstream use and permits the same pool to support multiple applications.
  • The model fine-tunes GPT-2 on sentence pairs and uses control codes plus fill-in-the-blank structures to target perturbation types and locations.Intrinsic evaluation reports counterfactuals that are fluent, diverse, and close to the original input.
  • Around 70% less annotation effort supports training and evaluation, while Polyjuice also augments explanations and enables systematic counterfactual error analysis.The paper describes a single model supporting diverse downstream applications through simple selection heuristics.

2 General-Purpose Counterfactuals

Polyjuice defines counterfactual generation as controllable conditional text generation: it produces close, fluent, diverse variants from paired data, then filters and evaluates them using intrinsic criteria.

  • 2.1 Definition and Desiderata: A counterfactual generator maps an input x to a set of variants with different relationships, whose applicability guides downstream selection.The same pair can instantiate multiple relationships, such as negation and task-dependent label flipping.
  • 2.1 Definition and Desiderata: Useful counterfactuals should remain close to x while being fluent, grammatically correct, and semantically meaningful.Closeness favors minimal changes that establish an effect, while fluency represents probable counterfactual text.
  • 2.2 Conditional Counterfactual Generation: Polyjuice conditions GPT-2 on the original input, optional control codes, and blank locations to control perturbation type, granularity, and placement.Users can provide only x or progressively specify the code and blanks for different degrees of control.
  • 2.2 Conditional Counterfactual Generation: The model is trained on six sentence-pair datasets plus naturally occurring close pairs from CommonGen, Natural Questions, and SQuAD.Pairs are converted into prompts and assigned control codes using part-of-speech tags, dependency trees, and semantic-change heuristics.
  • 2.2 Conditional Counterfactual Generation: 657,144 prompts are obtained from 186,451 sentence pairs by varying blanking over changed tokens, parsing structures, merged changes, and entire sentences.
  • 2.2 Conditional Counterfactual Generation: A fluency filter removes counterfactuals whose GPT-2 log-probability decreases by more than 10 points relative to x.Stricter filtering may benefit fully automated uses at the cost of diversity.
  • 2.3 Intrinsic Evaluation: Polyjuice stays closer to the original than non-finetuned GPT-2 and T5, while producing greater diversity than RoBERTa.T5 is more diverse but less close, whereas RoBERTa is restricted to word substitution.
  • 2.3 Intrinsic Evaluation: Control codes improve the success rate for desired perturbation types by as much as 42% for patterns such as negation and insert.

3 Counterfactual Evaluation & Training

Polyjuice counterfactuals support evaluation and training across sentiment, NLI, and QQP by letting annotators label generated examples rather than create them. They improve generalization and challenge-set performance with substantially lower annotation effort, while targeted perturbations are important for some tasks.

  • 3.1 Evaluation with Contrast Sets: Polyjuice counterfactuals create contrast sets with performance gaps similar to expert-created sets, without requiring expert annotators.These sets pair each instance with a nearby counterfactual having a different groundtruth label.
  • 3.2 Training with Counterfactuals: Polyjuice augmentation consistently improves or maintains out-of-domain and challenge-set accuracy across sentiment, NLI, and QQP.The largest gains occur on challenge, contrast, or sufficiently out-of-domain datasets.
  • 3.2 Training with Counterfactuals: 11 out of 27 QQP Checklist tests improve, while 2 out of 27 worsen; sentiment improves on 5 out of 15 tests and worsens on 1.These results are reported with a low counterfactual-to-original data ratio, below 10% for NLI and QQP.
  • 3.2 Training with Counterfactuals: Polyjuice counterfactuals are more effective than equivalent additional non-counterfactual data for training across varied tasks.For NLI, they are as effective as or more effective than manually created counterfactuals, with less implementation and annotation effort.
  • 3.3 Discussion: Human labeling replaces counterfactual creation: NLI labeling took a median 10 seconds per example and approximately 36 seconds after quality validation.Manual NLI counterfactual creation previously averaged approximately 2 minutes per example before quality validation.
  • 3.3 Discussion: Polyjuice may miss complex or context-dependent changes, so manual creation and machine generation can be complementary.The authors suggest targeted perturbations and hybrid human-machine workflows as ways to address Polyjuice’s lack of domain knowledge.

4 Counterfactual Explanations

Polyjuice counterfactual explanations complement token-importance methods by making model behavior concrete and surfacing surprising effects. Automatically selected surprising examples reveal behavior users miss even after inspecting explanations and generating counterfactuals themselves.

  • 4 Counterfactual Explanations: Token-importance scores can be too abstract to convey how individual words affect predictions.Concrete counterfactual examples provide a more direct view of a token’s behavioral significance.
  • 4 Counterfactual Explanations: The proposed hybrid explanation displays feature attributions alongside selected Polyjuice counterfactuals that violate attribution-based expectations.Selection targets large prediction changes with low importance scores and small changes with high importance scores.
  • 4 Counterfactual Explanations: 45% ± 6% was the human simulation accuracy for Polyjuice-surprise examples, only slightly above random guessing.This indicates that the examples remained surprising after users saw explanations and created their own counterfactuals.
  • 4 Counterfactual Explanations: Polyjuice-surprise counterfactuals were judged likely native-speaker text 95% of the time, compared with 85% for Expert-surprise.They were also closest to the original examples on both syntactic tree edit and Levenshtein distance measures.
  • 4 Counterfactual Explanations: Participants concentrated 84% of perturbed tokens in the top 15% of attribution weights, often overlooking how other tokens affected predictions.Participants also made mistakes when predicting model behavior for tokens they had inspected.
  • 4 Counterfactual Explanations: Counterfactual explanations reveal information users often miss beyond feature attributions and manual exploration.The paper reports that Polyjuice examples were more surprising and fluent than Expert-surprise examples despite being generated automatically.

5 Interactive Analysis

Polyjuice supports interactive counterfactual error analysis by generating multiple controlled perturbations per instance, exposing model behaviors that manual analysis may miss. In NLI, this reveals unstable responses to negation and quantifier changes.

  • Interactive Analysis: Polyjuice generates multiple counterfactuals per instance and supports controlled perturbations for systematic counterfactual error analysis.The authors use RoBERTa on NLI as a case study, while describing the techniques as generally applicable.
  • Negation Analysis: Manual analysis may miss distinctions among different negation forms, including a puzzling inconsistency between “not” and “n’t”.The first two generated counterfactuals expose this failure mode in a random Neutral instance.
  • Quantifier Analysis: The analysis uncovered unstable and erroneous quantifier behaviors that warranted further exploration and potential mitigation.Figure 6 extends the analysis by perturbing the subject and producing erroneous predictions for different quantifiers that should be Neutral.
  • Negation Analysis: 92.8% of correctly predicted Neutral instances flipped to Contradiction when a determiner was replaced with “no”, compared with ≈43% for “not” or “n’t”.These patterns were measured across n = 895 instances using generated negation counterfactuals.
  • Interactive Analysis: Control codes and blank locations enable analyses that would be non-trivial to perform manually or with masked language models.The authors also identify interactive test creation and controlled data augmentation as related uses.

6 Related Work

Polyjuice differs from prior counterfactual methods by separating general-purpose generation from application-specific selection and by producing multiple counterfactuals for analysis and explanation.

  • Training and Evaluation: Human-generated counterfactuals can achieve similar or better results at lower cost when people label Polyjuice generations instead of creating examples from scratch.This motivates mixtures of manual and semi-automated generation.
  • Analysis: Polyjuice enhances expert analysis by generating multiple counterfactuals that may be overlooked and providing abstractions for new analyses.The related-work comparison refers to the paper’s Section 5 results.
  • Scope of Counterfactual Generation: Prior automated generators typically target narrower relationships, such as preserving semantics while changing predictions or changing predictions without semantic constraints.Polyjuice instead supports a mix of label-preserving and label-flipping counterfactuals.

7 Conclusion and Future Work

Polyjuice is a controllable, general-purpose counterfactual generator that supports diverse downstream NLP applications. The authors identify biases in its controls and training data, uneven gains, and risks in human-generator interaction as future-work priorities.

  • Conclusion: Polyjuice generates fluent and diverse counterfactuals while controlling perturbation types and locations.Task-specific selection heuristics support data augmentation, contrast sets, explanations, and error analysis across domains.
  • Limitations: Polyjuice’s pre-defined control codes are not exhaustive, and its paired training datasets make some perturbations more likely than others.The authors observe that negative-sentiment words can be slightly more likely than positive ones in some contexts.
  • Future Work: Naturally occurring counterfactual collection and control without a-priori codes remain future research directions.These directions address limitations in the current generator’s control design and training data.
  • Limitations: Improvements from counterfactual augmentation are consistent across tasks but less significant on some datasets.The authors suggest that already-diverse data or subtle perturbations may limit the value of augmentation.
  • Human-Generator Interaction: Human-Polyjuice collaboration requires richer interaction mechanisms because users may over-rely on generations and inherit their uneven perturbation coverage.The paper calls for interaction designs that support rather than hinder human performance.

Ethical Considerations

The paper combines paired and naturally occurring sentence datasets to train and evaluate Polyjuice, measuring its control, closeness, diversity, and fluency. Ethical considerations include participant compensation and risks from human reliance on generated counterfactuals.

  • Participant Studies: The study compensated MTurk workers $2.5 for ≈15 minutes of labeling and graduate students $20 for a one-hour user study, with IRB approval.The reported payments were above the U.S. federal minimum wage.
  • Compute: Polyjuice was finetuned from GPT-2 rather than trained from scratch, with approximately 8 hours of finetuning compute.The finetuning experiments used RoBERTa on smaller datasets.
  • Training Data: The training corpus combines six paired sentence datasets with naturally occurring pairs from CommonGen, Natural Questions, and SQuAD.Naturally occurring pairs were filtered by edit distance to preserve closeness while increasing diversity.
  • Training Data: The component datasets cover distinct perturbation phenomena, including lexical exchanges, paraphrases, shuffling, restructuring, negation, and syntactic changes.WinoGrande supports lexical exchanges; ParaNMT paraphrases; PAWS shuffle and restructure; HANS rarer structural changes; CAD contributes especially to negation.
  • Evaluation: The evaluation compares Polyjuice with RoBERTa, T5, and original GPT-2 using self-BLEU for diversity and normalized word-level Levenshtein and syntactic tree-edit distances for closeness.The generators were run on 300 sentences, with Polyjuice enumerating control codes over the same blank placements as RoBERTa and T5.
  • Evaluation: Polyjuice achieves a balance between diversity and closeness relative to the evaluated alternatives.The authors report that Table 2 supports this balance, while noting that two concurrent systems were not open-sourced for comparison.

A.2.2 Controllability

Polyjuice's control codes improve targeted counterfactual generation over comparable baselines, while generation can fail when codes conflict with blanks or salient patterns dominate. The evaluation also documents annotation procedures and downstream dataset trends.

  • Controllability: 26%±13% average control-success increase across codes, ranging from 6% for quantifier to 42% for negation.Success rose from 5% to 47% for negation; non-finetuned T5 had 33% lower success on average.
  • Controllability: Polyjuice outperformed its uncontrolled GPT-2 counterpart on controllability, while non-finetuned T5 achieved less control.The comparison used 300 prompts per control code and counted success when one of three generated counterfactuals matched the desired code.
  • Failure cases: Control codes can fail when they conflict with the blank, lack an applicable pattern, or compete with salient perturbations.For example, shuffle does not apply to “the movie is good,” and quantifier perturbations can dominate regardless of the requested code.
  • Annotation: Crowdworkers labeled three counterfactuals per example across 20 actual rounds and two gold rounds, with filtering based on timing and gold-label accuracy.The median annotation time was around 15 minutes, and participants were recruited from qualified US-based MTurk workers.
  • Downstream trends: Counterfactual augmentation remained useful across SemEval as training size varied, but too many counterfactuals appeared harmful for Amzbook.Figure 8 compares augmentation with m = 2k counterfactuals against the corresponding m-baseline.
  • Annotation: κ = 0.77 agreement was obtained on a 100-example Sentiment subset, with 88% raw labeling agreement.The subset contained 100 counterfactuals on 100 original instances.

B.2 Training Details & m/n Ratios, for §3.2

The training experiments varied original and counterfactual data amounts while controlling optimization and validation procedures. Counterfactuals generally helped, but excessive augmentation could reduce performance when it displaced too much original data.

  • Training details: Each (m, n) setting used three training-data samples averaged over four random seeds, with model selection by validation accuracy.Training used 20 epochs, dropout 0.1, batch size 16, and task-specific learning rates of 1e-5 or 2e-5.
  • m/n ratios: Counterfactual augmentation remained effective on most Sentiment datasets but hurt Amzbook when counterfactuals formed a large proportion of training data.Yelp showed a similar but milder trend.
  • m/n ratios: The authors suspect that replacing too much original data reduces data diversity and thereby decreases model performance.This interpretation is presented as a suspicion rather than a directly established mechanism.
  • m/n ratios: Augmenting n = 1.7k NLI examples with m = 6.6k counterfactuals did not improve model generalization accuracy.The passage attributes this observation to Huang et al. (2020).

C.1 Selection Methods

The explanation selection method compares SHAP-based expected prediction changes with changes observed across Polyjuice counterfactuals. It then surfaces tokens and examples where model behavior most sharply diverges from attribution-based expectations.

  • Expected versus actual change: SHAP importance s(t) defines the expected prediction change when perturbing token t.For example, s(t=depression) = 0.276.
  • Expected versus actual change: Df(t, x) is the weighted average prediction-probability change over counterfactuals affecting t, with each edit weighted by the inverse number of edited words.The grouping Gt contains counterfactuals whose edited-word sets include t.
  • Expected versus actual change: The SHAP weight smooths the observed change estimate, penalizing outliers when measuring the gap between expectation and reality.The gap is represented as ∆Df(t, x) = Df(t, x) − H[Df(t, x)].
  • Abnormal tokens: The method identifies tokens with low SHAP but large observed effects, and tokens with high SHAP but usually intact predictions.These are selected as tL and tU using the largest positive or negative expectation–reality gaps.
  • Counterfactual selection: It chooses extreme counterfactuals within the abnormal-token groups using prediction change and aggregated SHAP weights of changed tokens.The selected examples serve as concrete counterfactual explanations.
  • Study interface: The participant interface showed a reference example and query box before displaying explanation tasks after task start or ten queries.Participants completed a one-hour study and received $20.

D.1 Additional Case Study: Quantifiers

The quantifier case study probes entailment instances containing numbers and reveals that the NLI model responds inconsistently to quantifier modifiers. The analysis uses representative perturbation patterns to organize these counterfactuals.

  • Additional case study: Quantifiers: The case study slices entailment data for hypotheses containing numbers, then perturbs their quantifiers.This targets model behavior around numerical quantification.
  • Additional case study: Quantifiers: Figure 3's explanation task provides the interface context for the paper's counterfactual-based analysis.The supplied figure caption identifies it as a sample explanation task for §4.
  • Additional case study: Quantifiers: The NLI model cannot perform the actual counting when the exact number is missing from the premise.Figure 10 isolates this counting behavior in the quantifier analysis.
  • Additional case study: Quantifiers: Adding “only” or “exactly” flips predictions to neutral 90% of the time, whereas adding “more than” changes predictions 52.6% of the time.Adding “at least” leaves the result as entailment in the reported examples.
  • Representative perturbation templates: Representative perturbation discovery extracts templates from changed spans and selects them through weighted set coverage.The weighting penalizes templates that are too abstract or concentrated on a single original example.
  • Representative perturbation templates: The template-selection objective maximizes counterfactual coverage while minimizing aggregate template weight.A greedy algorithm selects the subset of templates.
Loading 2101.00288v2…