Source-linked AI summary
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, Weizhu Chen
TL;DR
Fine-tuning large pretrained models can damage generalizable representations, while adversarial training addresses this with costly perturbation computation. Cutoff instead stochastically erases input information, uses Jensen-Shannon consistency across restricted views, and performs competitively or better across NLU and translation tasks with lower overhead.
Problem
Fine-tuning pretrained models on limited task-specific data can degrade their representation generalizability, while adversarial regularization requires costly perturbation-direction computation.
Method
Cutoff stochastically erases tokens, embedding dimensions, or spans to create partial input views and aligns their predictions with Jensen-Shannon Divergence consistency loss.
Results
Cutoff performs on par or better than competitive adversarial approaches on GLUE and machine translation while requiring less computation, including 37.6 BLEU on IWSLT2014 German-to-English.
Takeaways & Limitations
Cutoff provides an efficient data-augmentation alternative to adversarial training across natural language understanding and generation tasks.
Takeaways & Limitations
Cutoff assumes augmentations remain label-preserving while being sufficiently diverse from the original input.
Abstract
from arXiv · showhide
Adversarial training has been shown effective at endowing the learned representations with stronger generalization ability. However, it typically requires expensive computation to determine the direction of the injected perturbations. In this paper, we introduce a set of simple yet effective data augmentation strategies dubbed cutoff, where part of the information within an input sentence is erased to yield its restricted views (during the fine-tuning stage). Notably, this process relies merely on stochastic sampling and thus adds little computational overhead. A Jensen-Shannon Divergence consistency loss is further utilized to incorporate these augmented samples into the training objective in a principled manner. To verify the effectiveness of the proposed strategies, we apply cutoff to both natural language understanding and generation problems. On the GLUE benchmark, it is demonstrated that cutoff, in spite of its simplicity, performs on par or better than several competitive adversarial-based approaches. We further extend cutoff to machine translation and observe significant gains in BLEU scores (based upon the Transformer Base model). Moreover, cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.
1 Introduction
Cutoff addresses degraded fine-tuned representations by generating restricted input views through stochastic erasure and enforcing prediction consistency. Across GLUE and machine translation, it matches or outperforms adversarial approaches with lower computational overhead.
- 1 Introduction: Fine-tuning pretrained models can degrade representation generalizability, while adversarial regularization improves robustness but requires extra backward passes and memory.Cutoff is motivated as a lower-overhead alternative to this adversarial-training trade-off.
- 1 Introduction: Cutoff generates restricted views by erasing tokens, embedding dimensions, or contiguous spans at the input-embedding layer.These structured partial views are produced stochastically during fine-tuning.
- 1 Introduction: Jensen-Shannon Divergence consistency loss matches predictions across multiple partial views, complementing the usual cross-entropy objective.The loss supports consensus across more than two views without requiring adversarial perturbation directions.
- 1 Introduction: On GLUE, cutoff consistently outperforms competitive adversarial-based methods while requiring much less computation.The evaluation uses RoBERTa, and the approach is reported to extend readily to other pretrained models.
- 1 Introduction: Cutoff outperforms adversarial training on WMT2014 English-to-German and IWSLT2014 German-to-English, reaching 37.6 BLEU with Transformer Base on IWSLT2014 German-to-English.The paper reports state-of-the-art test performance on the latter dataset.
2 Related Work
Cutoff builds on multi-view consistency and adversarial training while differing in where views are generated and how prediction agreement is enforced. It operates directly on input embeddings without auxiliary model components.
- 2 Related Work: Multi-view learning supplies the broader idea of producing distinct views and encouraging consistent predictions, while cutoff’s views can overlap.This differentiates cutoff from approaches whose feature subsets are entirely independent.
- 2 Related Work: Unlike latent-representation methods such as CVT, cutoff creates partial views at the input-embedding layer, making it more model-agnostic.The comparison also distinguishes cutoff’s supervised setting from CVT’s semi-supervised setting.
- 2 Related Work: Cutoff requires no auxiliary prediction module, whereas CVT adds one during training.Span cutoff therefore leaves the original model architecture unchanged.
- 2 Related Work: Jensen-Shannon Divergence matches predictions across multiple cutoff views more naturally and stably than CVT’s multiple KL-divergence terms.This consistency objective is presented as more efficient for aggregating varied views.
3 Proposed Approach
The proposed Cutoff approach creates structured, label-preserving partial views by erasing input embeddings and trains models to make consistent predictions across them. It combines token, feature, and span masking with a Jensen-Shannon consistency objective while requiring less computation than adversarial training.
- 3.1 Motivation: The method is motivated by multi-view learning: agreement among predictions on restricted views can improve generalization and reduce error rates.The views should preserve labels while remaining diverse enough to enrich observations of the data space.
- 3.2 Constructing Partial Views: Cutoff constructs partial views by erasing structured portions of an input embedding matrix, rather than randomly zeroing individual elements as Dropout does.It can remove whole token embeddings, embedding dimensions across tokens, or contiguous spans.
- 3.2 Constructing Partial Views: Span cutoff removes a randomly positioned contiguous span of length l = floor(α × L), producing harder examples that retain the original sentence label.The remaining text is intended to encourage reliance on multiple semantic features rather than only the most salient ones.
- 3.3 Incorporating Augmented Samples: Training combines cross-entropy on original and augmented samples with a Jensen-Shannon divergence loss that matches predictions across all partial views.The loss averages predictions across the original and cutoff samples before comparing each prediction with that average.
- 3.5 Computational Complexity: Cutoff uses one backward pass and two forward passes, compared with 1 + T forward and backward passes for adversarial methods requiring T ascent steps.Because T ≥ 1, the approach has lower computational overhead than the cited adversarial methods.
4 Experimental Setup
The experiments evaluate Cutoff across GLUE natural language understanding tasks and compare it with adversarial and other augmentation baselines using RoBERTa models.
- Datasets: GLUE covers inference, entailment, paraphrase, sentiment, similarity, and acceptability tasks, using task-appropriate evaluation metrics.Accuracy is used for most datasets, with Spearman and Matthews correlations for STS and CoLA, respectively.
- Models and benchmark: The study uses both RoBERTa-base and RoBERTa-large as testbed models for the Cutoff strategies.
- Training details: Training uses Adam with selected learning rates, warmup and linear decay, weight decay, and up to ten epochs.The cutoff loss coefficients are selected on the validation set.
- Baselines: Comparisons include adversarial methods such as PGD, FreeAT, FreeLB, and ALUM, alongside back translation and other language augmentation strategies.The adversarial baselines are described as more computationally intensive than Cutoff.
5 Experimental Results
Across GLUE and machine translation, Cutoff is competitive with or superior to adversarial and other augmentation methods, while its consistency loss and removal ratio affect performance.
- GLUE Benchmark Evaluation: Cutoff methods consistently outperform ALUM on RoBERTa-base GLUE evaluations while being more computationally efficient, with span cutoff strongest on most datasets.Table 1 reports the GLUE dev-set comparisons for RoBERTa-base and RoBERTa-large.
- GLUE Benchmark Evaluation: Cutoff is competitive or better than adversarial approaches with RoBERTa-large and matches or exceeds back translation on 6 of 8 NLU tasks.
- Application to Machine Translation: On WMT2014 English-to-German, token cutoff achieves the best BLEU score among several Transformer Base baselines and outperforms adversarial training.The JS divergence objective provides further gains beyond the standard cross-entropy objective.
- The effect of JS divergence loss: Adding JS divergence consistently improves performance over cross-entropy alone, with β=1 producing the best MNLI result.The ablation evaluates span cutoff with accuracy on the MNLI dev set.
- The effect of Cutoff ratios: The strongest cutoff ratios are 0.15 for token cutoff, 0.2 for feature cutoff, and 0.1 for span cutoff.Ratios that are too large tend to yield smaller improvements because label preservation may fail under larger perturbations.
6 Conclusion
Cutoff provides stochastic partial views as an efficient replacement for adversarial-training augmentation. Across natural language understanding and machine translation, it improves results and reaches state-of-the-art performance on IWSLT2014 German-English.
- 37.6 test BLEU yielded a state-of-the-art result on IWSLT2014 German-English when cutoff was combined with the proposed JS divergence loss.
- Cutoff produced significant gains across natural language understanding and machine translation experiments.
- Cutoff performed on par with or better than several adversarial-training baselines while taking a fraction of the training time.