PaperlayerView exact revision on arXiv

Source-linked AI summary

Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen

arXiv:2004.11999v1cs.CL

TL;DR

BERT can score highly on standard NLI data while relying on inference heuristics that ignore syntax. The paper augments MNLI with syntactically transformed examples, finding that subject-object inversion raises HANS word-overlap accuracy from 0.28 to 0.73 and generalizes to other constructions. The authors conclude that augmentation is a simple way to mitigate such heuristics, while noting inconsistent effects of augmentation size and limited comparability with modified-model approaches.

  • Problem

    BERT fine-tuned on MNLI performs well on its standard test set but shows little sensitivity to syntax on HANS.

  • Method

    The authors augment MNLI with syntactically transformed examples, including subject-object inversion, and fine-tune BERT on the resulting datasets.

  • Results

    HANS word-overlap non-entailment accuracy increased from 0.28 without augmentation to 0.73 with the large inversion-based augmentation set.

  • Takeaways & Limitations

    A small inversion-based augmentation set generalized to additional constructions and supported the Missed Connection Hypothesis.

  • Takeaways & Limitations

    Augmentation size had moderate and inconsistent effects across HANS subcases, so denser sampling may be needed to clarify its role.

Abstract

from arXiv · show

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model's limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT's accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.

1 Introduction

Standard NLI performance can conceal weak syntactic reasoning: BERT performs well on MNLI yet often ignores word order on HANS. The paper tests whether targeted syntactic augmentation can connect BERT’s existing representations to inference and generalize across constructions.

  • 1 Introduction: High accuracy on in-distribution tests does not necessarily indicate task mastery or robustness to linguistic perturbations.Models may drop sharply on related datasets or become overly sensitive to irrelevant input changes.
  • 1 Introduction: BERT fine-tuned on MNLI achieves high MNLI test accuracy but often wrongly treats reversed subject-object roles as entailment.For example, it judged “The lawyer saw the actor” to entail “The actor saw the lawyer.”
  • 1 Introduction: The Representational Inadequacy and Missed Connection hypotheses offer competing explanations for BERT’s weak HANS performance.The former posits missing syntactic information; the latter posits that BERT has the information but does not apply it for inference.
  • 1 Introduction: The Missed Connection Hypothesis predicts that a small amount of augmentation from one construction will improve performance on other constructions.The Representational Inadequacy Hypothesis instead predicts larger augmentation sets and little cross-construction generalization.

2 Background

HANS diagnoses whether NLI models rely on shortcuts that succeed on MNLI but fail when syntax determines the label. The paper focuses on lexical overlap while testing whether augmentation generalizes to subsequence and constituent heuristics.

  • 2 Background: HANS is a template-generated challenge set designed to test three syntactic heuristics in NLI models.These are lexical overlap, subsequence, and constituent heuristics.
  • 2 Background: The lexical overlap heuristic predicts entailment whenever all hypothesis words appear in the premise.This shortcut often works in MNLI but fails when word order changes the meaning.
  • 2 Background: Crowdsourced MNLI construction may have encouraged lexical overlap because workers generated hypotheses from premises while minimizing effort.As a result, the shortcut often made correct predictions in MNLI despite being generally invalid.
  • 2 Background: HANS also tests subsequence and constituent heuristics, which treat structural subsets of premises as entailed.The experiments examine whether training for lexical-overlap counterexamples transfers to these other cases.
  • 2 Background: Prior augmentation often improved the targeted case without generalizing, suggesting that models could overfit to augmentation examples.This motivates testing cross-construction generalization rather than only in-domain improvement.

3 Generating Augmentation Data

The authors generate syntactic augmentation data from MNLI using inversion and passivization, varying how premises and hypotheses are constructed and using sets far smaller than MNLI.

  • 3 Generating Augmentation Data: Augmentation examples are generated from MNLI with two transformations: subject-object inversion and passivization.Inversion swaps the source sentence’s subject and object.
  • 3 Generating Augmentation Data: The ORIGINAL PREMISE strategy preserves the MNLI premise while transforming its hypothesis.The TRANSFORMED HYPOTHESIS strategy instead uses the original hypothesis as the new premise and the transformed hypothesis as the new hypothesis.
  • 3 Generating Augmentation Data: Augmentation sets contain 101, 405, or 1215 examples, compared with 297k examples in MNLI training.All augmentation sets are substantially smaller than the original training corpus.
  • 3 Generating Augmentation Data: The generated examples were not engineered for naturalness, and their labels were somewhat noisy.For instance, inversion can produce awkward sentences and does not always change entailment to neutral.
  • 3 Generating Augmentation Data: A random-shuffling condition tests whether syntactically uninformed examples can teach that word order provides no reliable inference.Both premises and hypotheses are shuffled and paired with random labels.

4 Experimental setup

The experiments separately add augmentation sets to MNLI, fine-tune BERT across random seeds, and evaluate on HANS and MNLI development data using a standardized non-entailment label.

  • 4 Experimental setup: Each augmentation set is added separately to MNLI before fine-tuning BERT.The most successful inversion-plus-transformed-hypothesis condition receives 15 runs per augmentation size; other combinations receive five.
  • 4 Experimental setup: HANS evaluation merges the model’s neutral and contradiction outputs into a single non-entailment label.This follows the evaluation procedure used by McCoy et al.
  • 4 Experimental setup: The study compares separate inversion and passivization conditions with combined datasets and label-specific passivization variants.An unaugmented BERT baseline is estimated with 100 runs.
  • 4 Experimental setup: Accuracy is reported on HANS and the MNLI development set, without tuning parameters on that development set.All discussed comparisons are reported as significant at p < 0.01 using two-sided t-tests.

5 Results

Syntactic augmentation substantially improved BERT’s robustness on HANS while preserving MNLI performance, with subject/object inversion generalizing to several constructions. The results also show that some cases remain harder and that augmentation strategies differ in effectiveness.

  • Overall performance: MNLI accuracy remained at the unaugmented baseline of 0.84 with augmentation sets of up to 1215 examples.This suggests the intervention did not harm overall MNLI performance.
  • Lexical-overlap cases: 0.28 to 0.73: large inversion with a transformed hypothesis improved HANS accuracy on non-entailment word-overlap cases.The strategy was the strongest augmentation method by a large margin.
  • Lexical-overlap cases: Lexical-overlap accuracy became similar for cases where the heuristic was correct and incorrect, suggesting the intervention prevented heuristic adoption.The strategy also reduced accuracy on cases where lexical overlap gives the correct prediction.
  • Augmentation strategies: Random shuffling did not improve over baseline, while inversion outperformed passivization, indicating that syntactically informed transformations were important.On passive HANS examples, large inversion reached 0.13 versus 0.01 for large passivization.
  • Generalization: 1.00 accuracy on the HANS subject/object-swap category with medium and large augmentation showed transfer across vocabulary and some syntactic differences.Small augmentation reached only 0.53, suggesting that 101 examples were insufficient for this category.
  • Generalization: Inversion generalized to prepositional-phrase cases, improving accuracy from 0.41 to 0.89, but subsequence cases improved more moderately.NP/S cases improved from 0.02 to 0.50, while further subsequence gains may require targeted augmentation; augmentation size effects were moderate and inconsistent across subcases.

6 Discussion

Subject/object inversion augmentation generalized from HANS to additional syntactic constructions, supporting the Missed Connection Hypothesis, but generalization remained incomplete for passive sentences. The approach can generate premise/hypothesis pairs from any corpus, though cross-domain augmentation may harm MNLI performance.

  • 6 Discussion: Subject/object inversion augmentation generalized from HANS to relative clauses and prepositional phrases, rather than merely inoculating BERT against the challenge set.This supports the Missed Connection Hypothesis and suggests induced abstract syntactic sensitivity.
  • 6 Discussion: BERT still performed poorly on passive sentences after inversion augmentation, supporting Representational Inadequacy for that construction.The authors suggest passives may require substantially more construction-specific augmentation examples.
  • 6 Discussion: The best-performing strategy generated premise/hypothesis pairs from single source sentences, so it did not require an NLI corpus.This makes augmentation possible from arbitrary corpora, although different-domain sentences may hurt MNLI performance.
  • 6 Discussion: The authors conclude that data augmentation is a simple and effective interim strategy for mitigating known inference heuristics in BERT-like models.They also identify stronger syntactic inductive biases and more diverse training sets as desirable longer-term directions.

A.1 Fine-tuning details

The experiments used bert-base-uncased, fine-tuned on MNLI with a trainable linear classifier over the final-layer CLS embedding. Models were trained for three epochs with reshuffled example order.

  • A.1 Fine-tuning details: All experiments used bert-base-uncased.The model was pretrained before MNLI fine-tuning.
  • A.1 Fine-tuning details: Fine-tuning trained a linear classifier on the CLS token’s final-layer embedding while updating BERT’s parameters.This follows the stated standard fine-tuning procedure.
  • A.1 Fine-tuning details: All models were trained for three epochs, with training examples reshuffled for each model.These details applied across the reported fine-tuning runs.

A.2 Generating augmentation examples

Augmentation examples were generated from transitive MNLI sentences using subject/object inversion and passivization, with several premise–hypothesis construction strategies. Sets were size-controlled for comparison.

  • A.2 Generating augmentation examples: The augmentation strategies used subject/object inversion and passivization, which respectively swap grammatical roles and form passive sentences.Inversion generally changes sentence meaning, whereas passivization alone preserves it.
  • A.2 Generating augmentation examples: Inversion with the original premise generated (p, INV(h), ↛) from entailment sources, discarding non-entailment sources.INV switches the source sentence’s subject and object.
  • A.2 Generating augmentation examples: Transformed-hypothesis inversion discarded the premise and generated (h, INV(h), ↛) from any source example.This strategy used the source hypothesis as the new premise.
  • A.2 Generating augmentation examples: Original-premise passivization generated (p, PASS(h)) with the same label, because PASS preserves the source sentence’s meaning.The source premise was retained while the hypothesis was passivized.
  • A.2 Generating augmentation examples: Transformed-hypothesis passivization generated both entailment and non-entailment pairs using h, PASS(h), and PASS(INV(h)).The premise was discarded in this strategy.
  • A.2 Generating augmentation examples: Source sentences were transitive MNLI examples selected by constituency parses, excluding TELEPHONE and restricting pronouns, copular or possessive verbs.Verb tense was preserved and agreement was modified when necessary.
  • A.2 Generating augmentation examples: The largest augmentation set contained 1215 examples for every strategy, while a Medium set sampled 405 cases.Equal maximum sizes were used for fair comparison, even when some strategies could generate more examples.
  • A.2 Generating augmentation examples: The Combined dataset concatenated inversion and passivization examples before randomly discarding half to match the other datasets’ size.The resulting numbers of inversion and passivization examples were not exactly equal.

A.3 Detailed Results

The detailed-results tables organize augmentation strategies, HANS heuristic-specific accuracy, training-set sizes, architectures, and subcase outcomes. They focus especially on subject/object inversion with a transformed hypothesis across multiple HANS breakdowns.

  • A.3 Detailed Results: Table A.2 reports mean accuracy on MNLI and HANS non-entailment cases diagnosing lexical overlap, subsequence, and constituent heuristics.The listed HANS cases use non-entailment as the correct label.
  • A.3 Detailed Results: The experiments compare small, medium, and large augmentation sets containing 101, 405, and 1215 examples, respectively.These sizes are used in the HANS accuracy tables and architecture comparisons.
  • A.3 Detailed Results: Table A.3 isolates subject/object inversion with a transformed hypothesis and compares unaugmented BERT against models trained with each augmentation size.The comparison concerns HANS accuracy after fine-tuning on MNLI with or without this augmentation.
  • A.3 Detailed Results: Table A.4 breaks overall HANS accuracy down by diagnostic heuristic and gold label across architectures and training methods.The heuristic abbreviations are lexical overlap, subsequence, and constituent; most systems use BERT as the base model.
  • A.3 Detailed Results: Tables A.5–A.7 examine subject/object inversion with a transformed hypothesis across HANS subcases for lexical-overlap, subsequence, and constituent heuristics.Each table compares unaugmented training with small, medium, and large augmentation; the tables distinguish entailment and non-entailment cases where specified.
Loading 2004.11999v1…