Source-linked AI summary
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
Jiaao Chen, Zichao Yang, Diyi Yang
TL;DR
Limited labeled data makes text classifiers vulnerable to overfitting, while unlabeled data is cheaper to obtain. MixText combines TMix hidden-space interpolation with label guessing for unlabeled examples, and experiments show strong gains across benchmark settings, especially with extremely limited supervision.
Problem
Limited labeled data can cause supervised text models to overfit, motivating classifiers that use both labeled and unlabeled data.
Method
MixText guesses labels for unlabeled data and applies TMix to interpolate labeled, unlabeled, and augmented text in hidden space.
Results
MixText consistently achieved the best performance across four benchmark datasets and significantly outperformed baseline methods, especially with extremely limited labeled data.
Takeaways & Limitations
Hidden-space interpolation and label guessing provide a semi-supervised text-classification framework that uses unlabeled data alongside labeled examples.
Takeaways & Limitations
The paper evaluates TMix for text classification and leaves other applications, including sequential labeling, for future work.
Abstract
from arXiv · showhide
This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data.By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.
1 Introduction
Limited labeled data can cause supervised text classifiers to overfit, motivating methods that use cheaper-to-collect unlabeled data. MixText addresses this setting with TMix augmentation and label guessing, and performs especially well under extremely limited supervision.
- Motivation: Limited labeled data makes supervised deep learning models prone to over-fitting.Unlabeled data is generally easier and cheaper to collect than labeled data.
- Approach: TMix interpolates two text instances in hidden space to create augmented training samples.MixText first guesses low-entropy labels for unlabeled data, then applies TMix to labeled and unlabeled examples.
- Approach: MixText explicitly models relationships between labeled and unlabeled samples through hidden-space interpolation.The method encourages models to behave linearly between training examples while using information from unlabeled sentences.
- Results: Experiments on four benchmark datasets found that MixText significantly outperformed baseline methods, especially with extremely limited labeled data.The comparisons included state-of-the-art semi-supervised methods and models pretrained on large amounts of unlabeled data.
2 Related Work
Related work spans latent-variable reconstruction, confidence-based self-training, consistency training, and large-scale pretraining. Interpolation methods have succeeded for images but remain challenging for text because text inputs are discrete and structurally complex.
- Pretraining: Pretraining and fine-tuning can achieve strong NLP performance even with small amounts of labeled data.Common pretraining objectives include language modeling and masked-language modeling.
- Semi-supervised learning: Semi-supervised text classification methods include VAEs, confident self-training, consistency training, and large-scale pretraining.These approaches respectively reconstruct sentences, exploit confident unlabeled predictions, use perturbations or augmentations, and leverage unlabeled corpora.
- Interpolation methods: Interpolation-based regularizers combine inputs and labels to create virtual training data and have achieved strong results in image tasks.Variants interpolate in input space, combine interpolation with cutoff, or mix hidden representations.
- Interpolation methods: Interpolation techniques had not been explored in NLP because text inputs are discrete rather than continuous image-like values.Text structure further complicates direct interpolation in the input space.
- Text augmentation: Text augmentation is difficult because text has complex syntactic and semantic structures.Existing approaches include synonym replacement, random insertion, random swap, random deletion, and paraphrasing.
3 TMix
TMix extends Mixup to text by interpolating hidden representations at an intermediate encoder layer, then propagating the mixed representation through upper layers. The method mixes labels for classification and trains a classifier on the resulting virtual examples.
- Mixup background: Mixup creates virtual training samples by linearly interpolating two labeled examples and their one-hot labels.Its direct extension to text is difficult because discrete tokens cannot be interpolated feasibly.
- TMix design: TMix overcomes discrete-token interpolation by mixing textual hidden representations inside a multilayer encoder.For an encoder with L layers, the hidden representation is mixed at layer m, where m ∈ [0, L].
- TMix design: TMix computes two samples’ lower-layer representations separately, mixes them at layer m, and feeds the result through the upper layers.The operation produces a final mixed hidden representation for downstream prediction.
- Text classification: TMix applies the same interpolation principle to labels and uses the mixed representation-label pair as a virtual training example.In text classification, the classifier minimizes KL-divergence between mixed labels and predicted probabilities.
- Layer selection: TMix samples its interpolation layer from M = {7, 9, 12}, selected to contain both syntactic and semantic information in BERT.The paper also evaluates how performance changes with different mix-layer sets.
- Text classification: The classifier receives TMix’s mixed representation and jointly trains with the encoder parameters.The implementation uses a two-layer MLP that returns a probability vector.
4 Semi-supervised MixText
MixText combines label guessing, text augmentation, and TMix to use labeled, unlabeled, and augmented data in semi-supervised training. Its objective combines supervised, consistency, and entropy-minimization losses.
- Framework: MixText uses TMix on labeled and unlabeled data, treating guessed-label unlabeled examples as additional labeled data.The framework also applies additional augmentation to generate more training samples under extremely limited supervision.
- Data Augmentation: Back translation generates diverse paraphrases of each unlabeled example using different intermediate languages and random sampling.These augmentations are used to generate labels for unlabeled data.
- Label Guessing: Labels for an unlabeled example and its augmentations are generated by a weighted average of the model’s predictions.Separate weights control contributions from the original example and augmentations, while sharpening can make the generated label one-hot as T approaches zero.
- TMix Training: The labeled, unlabeled, and augmented sets are merged into one training set, with each augmented sample sharing its original unlabeled example’s generated label.TMix then interpolates randomly sampled pairs across labeled and unlabeled categories.
- Training Objective: MixText uses supervised loss for labeled-dominated samples, consistency loss for unlabeled-dominated samples, and entropy minimization for uncertain unlabeled predictions.Entropy is minimized when prediction entropy exceeds the margin γ.
5.1 Dataset and Pre-processing
The experiments use four English text-classification benchmarks with original test sets and sampled training, unlabeled, and development splits. German and Russian back translations provide unlabeled-data augmentations.
- Datasets: Experiments cover AG News, DBpedia, Yahoo! Answers, and IMDB text-classification benchmarks.The original test sets were retained, while training data were sampled to form unlabeled and development sets.
- Pre-processing: German and Russian serve as intermediate languages for back translation, using FairSeq2 with random sampling temperature 0.9.The resulting paraphrases provide augmented versions of unlabeled examples.
5.2 Baselines
MixText is compared with VAMPIRE, BERT, and UDA, covering variational pretraining, supervised pretrained classification, and consistency-based augmentation.
- Baselines: VAMPIRE pretrains a unigram document model as a variational autoencoder on in-domain unlabeled data and uses its internal states for classification.
- Baselines: BERT uses a pretrained BERT-base-uncased encoder, average pooling, and a two-layer MLP fine-tuned for classification.
- Baselines: UDA uses the same BERT-based encoder and unlabeled augmentations, predicting labels from original unlabeled data and applying consistency loss.Its implementation uses the same softmax-sharpening temperature as MixText.
5.3 Model Settings
The experiments use a BERT-based encoder with average pooling and an MLP, compare multiple supervision levels, and distinguish methods by their use of unlabeled data.
- Architecture: The model uses a BERT-based-uncased tokenizer and encoder, average pooling, and a two-layer MLP with 128 hidden units and tanh activation.Inputs are truncated to 256 tokens; learning rates are 1e-5 for BERT and 1e-3 for the MLP.
- Training Data: TMix uses only labeled data, whereas MixText uses both labeled and unlabeled data during training.MixText follows UDA’s broader training settings while applying two augmentations per unlabeled example.
- Data Splits: The dataset split reports unlabeled, development, and test counts per class.
- Evaluation: Table 2 reports test accuracy averaged over three runs for models trained with 10, 200, or 2500 labeled examples per class.VAMPIRE, BERT, and TMix omit unlabeled data, while UDA and MixText use it.
5.4 Results
Across four text-classification datasets, MixText achieved the strongest performance, particularly with extremely limited labeled data, while more unlabeled data improved accuracy and TMix/MixText reduced overfitting.
- Accuracy improved from 69.5% to 74.1% on AG News with 10 labeled data per class when using TMix instead of BERT.
- MixText consistently achieved the best performance across four datasets by combining unlabeled data, TMix, and weighted-average label guessing.
- With 10 labeled data per class, increasing unlabeled data from 0 to 10000 produced much higher accuracy on AG News and Yahoo! Answer.
- BERT’s development loss increased around epoch 10, whereas UDA, TMix, and MixText showed more stable trends and lower loss.TMix also helped address overfitting without extra data.
5.5 Ablation Studies
Ablation studies show that TMix performance depends on the selected hidden layers and that every major MixText component contributes, with unlabeled data and TMix having the largest effects.
- Different Mix Layer Set in TMix: TMix achieved its best performance with mixup layers {7, 9, 12}, which capture substantial syntactic and semantic information.
- Different Mix Layer Set in TMix: Mixing at the input and lower layers {0, 1, 2} did not improve performance, while the {3,4} subset containing mainly surface information was weaker.
- Remove Different Parts from MixText: Removing any MixText component reduced performance, with the largest decrease occurring after removing unlabeled data.
- Remove Different Parts from MixText: Removing TMix reduced performance more than replacing weighted-average prediction for unlabeled data, indicating TMix’s largest impact beyond unlabeled data.
6 Conclusion
The paper presents MixText and TMix for semi-supervised text classification, reporting better accuracy and more stable loss trends than pre-training, fine-tuning, and other state-of-the-art semi-supervised methods. It identifies broader NLP tasks and real-world limited-label settings as future directions.
- MixText combines semi-supervised learning with TMix, an interpolation-based augmentation and regularization technique for text classification.
- Across four benchmark datasets, TMix and MixText showed better test accuracy and more stable loss trends than current pre-training, fine-tuning, and semi-supervised methods.
- Future work will examine MixText on sequential labeling and other real-world scenarios with limited labeled data.