PaperlayerView exact revision on arXiv

Source-linked AI summary

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Nathan Ng, Kyunghyun Cho, Marzyeh Ghassemi

arXiv:2009.10195v2cs.CLcs.LGstat.ML

TL;DR

Out-of-domain robustness is difficult because training distributions may not cover desired test distributions, while natural-language augmentation must preserve the data manifold. SSMBA uses corruption and reconstruction functions, implemented with masked language models, to generate manifold-based synthetic examples; across 3 tasks and 9 datasets, it consistently outperforms baselines and other augmentation methods on in-domain and OOD data.

  • Problem

    Training distributions may not cover desired test distributions, and generating natural-language examples that remain on the underlying data manifold is difficult.

  • Method

    SSMBA generates synthetic examples by corrupting inputs off the data manifold and reconstructing them with a learned reconstruction function, using masked language models for natural language.

  • Results

    Across 3 tasks and 9 datasets, SSMBA consistently outperforms baseline models and other augmentation methods on in-domain and OOD data, including 1.8% OOD accuracy improvement on MNLI.

  • Takeaways & Limitations

    SSMBA provides task-agnostic, dataset-agnostic synthetic augmentation with reported gains on sentiment analysis, natural language inference, and machine translation.

  • Takeaways & Limitations

    The approach assumes inputs concentrate around an underlying lower-dimensional data manifold, which defines the neighborhood used for augmentation.

Abstract

from arXiv · show

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English.

1 Introduction

Training distributions may not cover desired test distributions, making robustness to unseen and out-of-domain examples a central challenge. SSMBA addresses this challenge by generating manifold-neighborhood examples through corruption and reconstruction, and improves performance across diverse NLP tasks.

  • Training and test distributions can differ because of biased data collection or distribution drift, motivating out-of-domain robustness.
  • Data augmentation synthetically generates training examples to improve out-of-domain robustness.
  • SSMBA perturbs examples off the data manifold and reconstructs them back onto it to generate synthetic examples in difficult-to-characterize domains.
  • The method is applicable to any supervised task, requires no task-specific knowledge, and avoids class- or dataset-specific fine-tuning.
  • SSMBA is evaluated across 3 diverse tasks, 9 datasets, and 4 model types, outperforming baseline models and other augmentation methods on in-domain and out-of-domain data.

2 Background and Related Work

The paper frames augmentation as sampling near training examples while seeking neighborhoods on the data manifold. SSMBA uses denoising autoencoders and masked language models to corrupt and reconstruct text for synthetic example generation.

  • Existing augmentation methods: Existing NLP augmentation methods use task-specific, word-level, back-translation, embedding-based, contextual, or generative-model approaches to create synthetic examples.Many rely on human-designed heuristics or task and dataset knowledge.
  • Augmentation framework: Vicinal Risk Minimization enlarges training support by sampling near existing examples, typically using dataset-dependent heuristics to define each vicinity.Computer-vision examples include scale, color, translation, and rotation augmentation.
  • Manifold assumption: The manifold assumption defines a training example’s useful vicinity as the portion of its neighborhood lying on a lower-dimensional data manifold.This assumption has also been used for robustness and adversarial-example generation.
  • Denoising autoencoders: A denoising autoencoder learns to reconstruct clean inputs from stochastic corruptions, and repeated corruption-reconstruction can sample along the manifold.As training data increases, the generated-sample distribution is described as approaching the true data-generating distribution.
  • Language-model reconstruction: SSMBA applies masked language models as denoising autoencoders by corrupting input tokens and reconstructing them from their left and right context.The process is illustrated as applying MLM corruption and then reconstructing the corrupted sentence.

3 SSMBA: Self-Supervised Manifold Based Augmentation

SSMBA generates augmented inputs by corrupting examples off the manifold and reconstructing them back onto it. It can preserve labels or use a teacher model, then adds repeated synthetic examples to the original training set.

  • Algorithm: The algorithm trains a model on the original dataset, generates augmented inputs and outputs, forms Daug, and augments D with Daug.The pseudocode explicitly defines the augmented dataset as reconstructed examples paired with their outputs.
  • Manifold-based generation: SSMBA assumes inputs concentrate around a lower-dimensional manifold M and samples corrupted inputs x′ from q(x′|x) that leave M.A reconstruction function r then produces samples intended to lie on M.
  • Manifold-based generation: The method reconstructs each corrupted input with r, producing perturbed examples that can be repeated multiple times for every original training example.These reconstructed examples form the augmented input set.
  • Label generation: Corresponding augmented outputs can preserve the original label or be generated by a teacher model trained on the original dataset.The teacher option addresses cases where manifold neighbors may cross decision boundaries on sensitive tasks.
  • Implementation: SSMBA uses the MLM training corruption function and a pretrained BERT reconstruction model without task-specific knowledge or dataset-specific fine-tuning.The method requires only a corruption function q and reconstruction function r and applies to supervised natural-language tasks.

4 Datasets

The evaluation spans 9 datasets across sentiment analysis, natural language inference, and machine translation, using metadata or related datasets to create domain splits. The benchmarks include both in-domain and out-of-domain settings with varying domain shifts.

  • Evaluation scope: The study evaluates 9 datasets: 4 sentiment-analysis, 2 natural-language-inference, and 3 machine-translation datasets.Datasets use metadata-based or related-dataset domain separations.
  • Sentiment analysis: Amazon review benchmarks predict 1-to-5-star ratings, with AR-Full spanning 10 large categories and AR-Clothing dividing one category into subcategories.AR-Clothing therefore has less domain shift than AR-Full.
  • Sentiment analysis: The Movies dataset pairs SST2 movie-review excerpts with full-length IMDb reviews, and the task is binary sentiment prediction.The two datasets provide related but distinct review domains.
  • Natural language inference: MNLI trains on five genres and tests across all ten, using validation data as the test set because labeled test data are unavailable.The procedure samples 2,000 examples from each training set for validation.
  • Natural language inference: ANLI contains adversarially designed NLI examples organized into three difficulty levels treated as separate textual domains.The dataset was designed so state-of-the-art models fail to classify examples correctly.
  • Machine translation: Machine-translation evaluation covers German-to-English and German-to-Romansh, including in-domain IWSLT14 and OOD OPUS data across highly dissimilar domains.The German-to-Romansh setting is practically relevant because Romansh is low-resource, with an estimated 40,000 native speakers.

5 Experimental Setup

The experiments cover sentiment analysis, natural language inference, and machine translation using multiple model types and augmentation baselines. Models are trained across domains, evaluated on in-domain and out-of-domain performance, and averaged across random seeds and test domains.

  • Sentiment analysis uses LSTM and CNN models, NLI uses fine-tuned RoBERTaBASE models, and translation uses transformers.
  • MLM corruption is used for all experiments, with tuned corruption percentages and fixed 80% masked, 10% random, and 10% unmasked operations.
  • Hyperparameters, including corruption percentage and augmented-example labelling, are selected using in-domain validation performance.
  • The sentiment and NLI experiments compare against EDA, CBERT, and UDA augmentation methods.
  • Translation experiments compare against word dropout, RAML, and SwitchOut without requiring additional target-side monolingual data.
  • Models use 10, 5, or 3 random seeds depending on architecture, then are evaluated across domains with performance averaged across seeds and test domains.

6 Results

Across sentiment analysis, natural language inference, and machine translation, SSMBA generally outperforms baseline models and competing augmentation methods on in-domain and out-of-domain evaluations, with some task-specific trade-offs.

  • Sentiment Analysis: SSMBA retains OOD benefits under substantial domain shifts, including Amazon reviews with differing domains and Movies data with large sentence-length differences.On AR-Full, RNNs gain 1.1% OOD accuracy over baseline models; on Movies, SSMBA beats other augmentation methods in OOD performance despite no RNN ID improvement.
  • Natural Language Inference: 1.8% OOD accuracy improvement on MNLI exceeds the best baseline’s 0.3% improvement, while SSMBA also improves ID accuracy by 1.4%.Other baseline methods either hurt both metrics or provide negligible improvements.
  • Natural Language Inference: On ANLI, SSMBA maintains baseline OOD accuracy while improving ID accuracy by 6%.Other augmentation methods improve ID accuracy less and degrade OOD accuracy.
  • Machine Translation: Almost 1.5 BLEU improvement on IWSLT14 de→en outperforms all baseline and comparison models.Other augmentation methods yield much smaller improvements or degrade performance.
  • Machine Translation: On OPUS, SSMBA gives a small OOD BLEU boost but degrades ID performance, whereas on de→rm it improves OOD BLEU by 2.4 points and ID BLEU by 0.4 points.The OPUS setting uses highly specialized training language that differs substantially from other domains and the learned MLM manifold.

7 Analysis and Discussion

The analysis examines how dataset size, reconstruction models, corruption, sampling, augmentation quantity, and label generation affect SSMBA. Results indicate robustness to several design choices, while corruption sensitivity and supervision quality depend on the task and model.

  • Experimental setup: Ablations focus on the Baby domain of AR-Clothing, using CNN models unless otherwise stated; results are not comparable with Table 2.The analysis uses one domain because of dataset size, limited OOD domains, and domain-shift characteristics.
  • Reconstruction model: SSMBA evaluates reconstruction models spanning 82M, 125M, and 355M parameters to test whether model expressiveness affects generated-example quality.The compared models are DistilRoBERTa, RoBERTaBASE, and RoBERTaLARGE.
  • Corruption amount: 45% corruption maximizes OOD performance, while very large corruption amounts degrade performance; augmented models still outperform unaugmented models at 95% corruption.On the more input-sensitive NLI task, large noise amounts can reduce performance below baselines.
  • Sampling method: Unrestricted sampling produces the greatest OOD-accuracy increase, while top-k sampling methods perform similarly.The result suggests that SSMBA benefits from exploring the manifold without sampling restrictions.
  • Augmentation amount: Increasing augmentation improves OOD accuracy and reduces variance across trained models.The comparison uses multiple augmentation amounts and averages results across repeated datasets and models.
  • Label generation: Soft and hard labels from low-accuracy supervision models degrade performance, whereas sufficiently accurate supervision makes soft labels outperform the other labeling methods.The accuracy threshold varies with dataset and task difficulty; in ANLI, even poor supervision models improved downstream accuracy.

8 Conclusion

The conclusion presents SSMBA as a broadly applicable augmentation method for difficult-to-characterize manifolds and reports gains across NLP tasks and robustness settings. Its analysis finds robustness to several configuration choices, while future work targets additional manifolds and tasks.

  • Conclusion: SSMBA generates synthetic data without task-specific knowledge or dataset-specific fine-tuning and applies across supervised tasks.The method is demonstrated on sentiment analysis, natural language inference, and machine translation.
  • Conclusion: 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 de→en are reported gains.The conclusion also reports OOD improvements in most analyzed settings.
  • Conclusion: SSMBA is reported as robust to initial dataset size, reconstruction model choice, and corruption amount.Future work includes applying SSMBA to the target-side manifold in structured prediction and to other language tasks and settings.

A Datasets

The paper provides dataset statistics and details in a table and makes all task data splits available for download.

  • Datasets: Full dataset statistics and details are provided in Table 7.
  • Datasets: All data splits for all tasks can be downloaded from the linked Box location.
  • Datasets: The dataset-download link continues across the passage text and ends with the Box resource identifier.

B Data Preprocessing

Sentiment-analysis and NLI data share preprocessing based on GPT-2-style tokenization, BPE encoding, label dictionaries, binarization, and end-to-end model training.

  • Tokenization: All sentiment-analysis and NLI data are tokenized with a GPT-2-style tokenizer and fairseq BPE vocabulary containing 50,263 types.
  • Encoding: Labels are encoded with a dictionary containing as many types as there are classes, and inputs and labels are binarized for training.
  • Model training: Models share the vocabulary but randomly initialize embeddings and train the entire model end-to-end.

C Model Architecture and Training Hyperparameters

The study evaluates LSTM, CNN, RoBERTa, and Transformer models trained in fairseq, with architecture-specific dimensions, optimization schedules, and compute allocations. SSMBA hyperparameters are selected using in-domain validation, and statistical significance is assessed with task-appropriate tests.

  • Training and evaluation: Training runs use fairseq with T4 GPUs, while SSMBA settings are selected by in-domain validation and significance tests vary by task.Wilcoxon ranked-sum tests are used for sentiment and NLI, whereas machine translation uses paired bootstrap resampling.
  • LSTM: LSTM models use a single 512-node layer and a 512-dimensional MLP classifier, with 28M parameters and dropout regularization.Input and output encoder dropout are 0.3, while classifier dropout is 0.1.
  • CNN: CNN models use three convolutions with kernel sizes 3, 4, and 5, producing 27M-parameter classifiers from max-pooled representations.Each convolution has 256 output channels, and the pooled outputs form a 768-dimensional representation before classification.
  • RoBERTa: RoBERTa models fine-tune a pretrained RoBERTaBASE model with a 512-dimensional hidden classifier and 125M parameters.Training uses learning rate 1e−5, two warmup epochs, and inverse square root decay.
  • Transformer: Transformer models use 512-dimensional embeddings, 1024-dimensional feed-forward layers, four encoder heads, and six encoder and decoder layers.The models contain 52M parameters and use label smoothing of 0.1, dropout of 0.3, and weight decay of 0.0001.
Loading 2009.10195v2…