Source-linked AI summary
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations
Sosuke Kobayashi
TL;DR
Text classifiers can overfit, while preparing large annotated datasets is time-consuming. The paper introduces contextual augmentation with context-predicted substitutions and label conditioning, and reports improved neural classifiers across six text-classification tasks.
Problem
Neural NLP models can overfit, and generalization depends on training-data size and quality, while preparing large annotated datasets is time-consuming.
Method
Contextual augmentation replaces words with varied words predicted by a bidirectional language model and uses label conditioning to preserve compatibility with annotated labels.
Results
The method improved neural classifiers across six text-classification tasks, outperforming synonym-based augmentation; label conditioning achieved the best accuracies on average.
Takeaways & Limitations
Contextual augmentation provides a generally applicable, task-knowledge-independent method for classification across domains.
Takeaways & Limitations
The improvement from contextual augmentation is sometimes marginal.
Abstract
from arXiv · showhide
We propose a novel data augmentation for labeled sentences called contextual augmentation. We assume an invariance that sentences are natural even if the words in the sentences are replaced with other words with paradigmatic relations. We stochastically replace words with other words that are predicted by a bi-directional language model at the word positions. Words predicted according to a context are numerous but appropriate for the augmentation of the original words. Furthermore, we retrofit a language model with a label-conditional architecture, which allows the model to augment sentences without breaking the label-compatibility. Through the experiments for six various different text classification tasks, we demonstrate that the proposed method improves classifiers based on the convolutional or recurrent neural networks.
1 Introduction
Contextual augmentation addresses limited NLP augmentation by replacing words with context-predicted alternatives, while conditioning predictions on labels to preserve compatibility.
- Motivation: NLP augmentation is limited because universal, automatically applicable transformations that preserve data quality are difficult to obtain.Synonym-based methods also provide only a small set of substitutions.
- Proposed approach: Contextual augmentation replaces words with alternatives predicted by a bidirectional language model from surrounding context.The method uses context-predicted words with paradigmatic relations rather than relying only on synonyms.
- Label compatibility: A label-conditional language-model architecture is introduced to prevent replacements from becoming incompatible with sentence annotations.The architecture allows augmentation while retaining label compatibility.
- Reported outcome: Experiments show contextual augmentation improves classifiers based on recurrent or convolutional neural networks across varied classification tasks.The paper evaluates the approach on six text classification tasks.
2 Proposed Method
The method broadens substitutions beyond synonyms by sampling words predicted from bidirectional context, then conditions those predictions on labels to avoid contradictory augmentations.
- 2.1 Motivation: Synonym-based augmentation cannot produce numerous varied patterns because available synonyms are limited.The paper motivates broader substitutions using the sentence “the actors are fantastic.”
- 2.2 Word Prediction based on Context: Contextual augmentation uses words predicted by a language model from the context surrounding the word being replaced.These alternatives may have paradigmatic relations without being semantically similar to the original word.
- 2.2 Word Prediction based on Context: A bidirectional LSTM-RNN encodes words on both sides of a target and produces a vocabulary probability distribution for that position.The leftward and rightward outputs are concatenated before feed-forward prediction.
- 2.2 Word Prediction based on Context: Substitutes are sampled from a temperature-adjusted probability distribution during model training rather than selected conclusively as top-K words.Temperature controls augmentation strength, ranging from near-uniform sampling to highest-probability predictions.
- Limitations of prior augmentation: Synonym-based augmentation additionally requires word-sense disambiguation or rules for selecting appropriate synsets.This requirement is identified as a limitation of the synonym-based approach.
- Label-conditional augmentation: A label-conditional language model predicts pτ(·|y, S\{wi}) so replacements do not reverse information associated with sentence labels.The label embedding is concatenated with a feed-forward hidden layer alongside contextual information.
3 Experiment
The experiments compare augmentation methods across six classification tasks and two neural architectures, finding stronger performance from contextual and label-conditional augmentation than from synonym-based augmentation.
- 3.1 Settings: Six text classification tasks and two neural model types are used to test combinations of three augmentation methods.The tasks include SST5, SST2, Subjectivity, MPQA, RT, and TREC; the models are LSTM-RNN and CNN classifiers.
- 3.1 Settings: The tested methods are synonym-based augmentation, contextual augmentation, and contextual augmentation with a label-conditional architecture.Contextual augmentation uses a bidirectional LSTM language model pretrained on WikiText-103.
- 3.2 Results: Contextual augmentation improves performance across datasets from different domains more significantly than synonym-based augmentation.Table 1 reports accuracies with and without augmentation.
- 3.2 Results: The label-conditional architecture boosts average performance and achieves the best accuracies among the evaluated methods.The reported accuracies are averaged over eight models trained from different seeds.
- 3.2 Results: With positive labels, predictions for “fantastic” include positive expressions, whereas negative labels produce negative alternatives.Predictions for unrelated words such as “actors” remain relatively similar across labels.
4 Related Work
Prior NLP augmentation methods used synonym lists, task-specific rules, generative models, and language models. This work is distinguished by jointly using bidirectional context and label conditioning for classification augmentation.
- Prior NLP augmentation used synonym lists, grammar induction, task-specific heuristic rules, and neural generative models.
- Latent Words LM selected substitute words using top-K scores from fixed-length contexts.
- The authors use a bidirectional LSTM language model that captures variable-length contexts from both directions jointly.
- Label-conditional fill-in-the-blank contexts are used for data augmentation, alongside broader applications of context encoding with bidirectional language models.
5 Conclusion
The paper concludes that contextual augmentation uses language-model predictions and label conditioning to generate varied, label-compatible training examples. It improves neural classifiers across classification settings, although gains are sometimes marginal.
- Contextual augmentation uses numerous words predicted by a bidirectional language model, with label conditioning added to preserve compatibility with original labels.
- Experiments found that the method produced label-compatible words and improved neural classifiers more than synonym-based augmentation.
- The method is independent of task-specific knowledge or rules and is intended for classification tasks across various domains.
- Improvement from the method is sometimes marginal, motivating future comparison and combination with other generalization methods.