Source-linked AI summary
Augmenting Data with Mixup for Sentence Classification: An Empirical Study
Hongyu Guo, Yongyi Mao, Richong Zhang
TL;DR
The paper addresses the limited evidence on applying Mixup to NLP and its effectiveness for sentence classification. It introduces word- and sentence-embedding interpolation and finds accuracy improvements across CNN and LSTM models, with domain-independent regularization.
Problem
How Mixup can be applied to NLP tasks and whether it improves sentence classification remains insufficiently investigated.
Method
The paper applies Mixup to sentence classification by interpolating either word embeddings or encoder-produced sentence embeddings, using mixed samples for training.
Results
Accuracy improves for CNN and LSTM sentence-classification models across benchmark datasets, with relative gains exceeding 3.3% for CNN on SST-1 and MR and reaching 5.22% for LSTM on SST-1.
Takeaways & Limitations
Embedding interpolation provides an effective, domain-independent regularizer for sentence-classification overfitting avoidance without relying on human knowledge for text transformations.
Takeaways & Limitations
The experiments focus on the Random-dataset setting because RandomTune has the largest parameter count and requires strong regularization to avoid overfitting.
Abstract
from arXiv · showhide
Mixup, a recent proposed data augmentation method through linearly interpolating inputs and modeling targets of random samples, has demonstrated its capability of significantly improving the predictive accuracy of the state-of-the-art networks for image classification. However, how this technique can be applied to and what is its effectiveness on natural language processing (NLP) tasks have not been investigated. In this paper, we propose two strategies for the adaption of Mixup on sentence classification: one performs interpolation on word embeddings and another on sentence embeddings. We conduct experiments to evaluate our methods using several benchmark datasets. Our studies show that such interpolation strategies serve as an effective, domain independent data augmentation approach for sentence classification, and can result in significant accuracy improvement for both CNN and LSTM models.
1 Introduction
Deep learning models need substantial training data, but annotated NLP data are costly and difficult to augment safely. This paper studies Mixup interpolation for sentence classification using word and sentence embeddings.
- Motivation: Large parameterized deep learning models require substantial data to avoid overfitting and improve generalization.Collecting large annotated datasets is time-consuming and expensive.
- Motivation: Automatic data augmentation generates synthetic samples as additional training data for regularizing learning models.The approach has been used successfully in computer vision and speech recognition.
- Motivation: NLP augmentation is difficult because small word changes can dramatically alter sentence meaning, unlike many image transformations.Existing approaches often use synonym replacements from ontologies or word similarity.
- Research aim: The paper asks how Mixup applies to NLP and evaluates its effectiveness for sentence classification.It proposes interpolation in word-embedding and sentence-embedding spaces.
- Contribution: Word- and sentence-embedding interpolation improve accuracy for both CNN and LSTM sentence classifiers.The strategies are described as domain independent, human-knowledge-free, and computationally inexpensive.
2 Data Augmentation through Sentence Interpolation
The paper adapts Mixup to sentence classification by interpolating either word embeddings or encoder-produced sentence embeddings before classification. The resulting synthetic examples are trained with mixed targets and a classification loss.
- Mixup background: Mixup creates synthetic samples by linearly interpolating pairs of inputs and their one-hot class targets.The mixed samples are used for supervised training, with pairs efficiently formed within a mini-batch.
- Sentence classification: Standard CNN and LSTM classifiers encode word sequences into sentence embeddings before a softmax predicts target classes.This architecture provides two locations for applying interpolation.
- Model variants: Figure 1 contrasts wordMixup and senMixup, highlighting the added interpolation components relative to the standard classifier.wordMixup operates on word embeddings, whereas senMixup operates before the softmax on the final hidden representation.
- wordMixup: wordMixup zero-pads sentences and interpolates every dimension of corresponding word embeddings across a sentence pair.The resulting synthetic sample and mixed label are used for training.
- senMixup: senMixup encodes two sentences with a CNN or LSTM, then linearly interpolates each dimension of their same-sized sentence embeddings.The mixed embedding is passed to a softmax and trained with multiclass cross-entropy.
3 Experiment
The experiments evaluate wordMixup and senMixup across five sentence-classification benchmarks, CNN and LSTM encoders, and four embedding settings. Results generally show improved accuracy and regularization, with important exceptions for SST-2 when embeddings are fixed.
- Experimental setup: The study evaluates wordMixup and senMixup with CNN and LSTM encoders under RandomTune, RandomFix, PretrainTune, and PretrainFix embedding settings.RandomTune uses randomly initialized trainable embeddings, while the other settings vary initialization and whether embeddings are trainable.
- Regularization effect: Training-loss curves indicate that wordMixup and senMixup maintain nonzero, relatively higher training loss, preserving training signal and reducing overfitting to individual examples.The authors attribute the higher loss to the larger space of mixed samples; baseline CNN training loss quickly reaches zero.
- LSTM results: With LSTM encoders, the largest relative improvements are 4.62% on TREC and 5.22% on SST-1.On Subj, wordMixup outperforms senMixup by 1.2%; the methods perform comparably on the other four datasets, and both improve SST-2 by about 2%.
4 Conclusion and Future Work
The study finds that wordMixup and senMixup improve accuracy for both CNN and LSTM sentence classifiers and act as domain-independent regularizers. Future work will examine newer Mixup variants and why interpolation works.
- wordMixup and senMixup improve accuracy for both CNN and LSTM sentence classification models.
- The interpolation strategies serve as domain-independent regularizers for avoiding overfitting in sentence classification.
- Future work will investigate Manifold Mixup and AdaMixup for sentence classification.
- The authors also plan to study what mixed sentences look like and why interpolation works for sentence classification.