Source-linked AI summary
Adversarial Training Methods for Semi-Supervised Text Classification
Takeru Miyato, Andrew M. Dai, Ian Goodfellow
TL;DR
The paper addresses the mismatch between adversarial perturbations and sparse, discrete one-hot text inputs. It applies adversarial and virtual adversarial perturbations to word embeddings in recurrent text classifiers, achieving strong benchmark performance and improving embedding quality. The results also indicate a limitation when unlabeled-example regularization overwhelms supervised learning.
Problem
Adversarial and virtual adversarial training require perturbing many real-valued inputs, which is inappropriate for sparse, high-dimensional one-hot word representations.
Method
The method applies adversarial and virtual adversarial perturbations to continuous word embeddings within a recurrent neural network instead of to discrete word inputs.
Results
The approach achieves state-of-the-art performance across multiple semi-supervised text classification tasks and improves learned word embeddings over baseline methods.
Takeaways & Limitations
Adversarial and virtual adversarial training provide effective regularization for sequence models on text classification tasks, with benefits for both classification and embedding quality.
Takeaways & Limitations
On some data, virtual adversarial loss on unlabeled examples overwhelmed supervised loss, causing the model to prioritize perturbation robustness over correct answers.
Abstract
from arXiv · showhide
Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting. Code is available at https://github.com/tensorflow/models/tree/master/research/adversarial_text.
1 INTRODUCTION
The paper adapts adversarial and virtual adversarial training to text by perturbing continuous word embeddings rather than discrete one-hot inputs. It uses this approach to regularize text classifiers and reports strong performance across semi-supervised tasks, alongside improved embeddings.
- Virtual adversarial training regularizes unlabeled examples by matching the model’s output distribution on original and adversarially perturbed inputs.Unlike adversarial training, this objective does not require the actual label, enabling semi-supervised learning.
- Text inputs are discrete, high-dimensional one-hot vectors, so infinitesimal adversarial perturbations are defined on continuous word embeddings instead.This changes the training perturbation target from the discrete input representation to the embedding layer.
- The proposed method is intended exclusively as regularization that stabilizes the text classifier, not as defense against an input-level adversary.
- With neural language-model pretraining, the approach achieves state-of-the-art performance on multiple semi-supervised sentiment and topic classification tasks.The reported result required optimizing only one additional hyperparameter, ǫ, controlling perturbation size.
- Adversarial and virtual adversarial training improve word embeddings over baseline methods in qualitative model analysis.
2 MODEL
The model maps discrete words to continuous embeddings and processes them with an LSTM text classifier. During adversarial training, bounded perturbations are applied to normalized embeddings, with bidirectional variants also evaluated.
- A vocabulary embedding matrix maps each discrete word, plus an end-of-sequence token, to a continuous D-dimensional vector.The resulting embedding sequence is fed to the recurrent classifier at each time step.
- The base text classifier is an LSTM-based neural network that receives the embedding corresponding to each word in the sequence.
- A bidirectional variant adds an LSTM over the reversed sequence and predicts from the concatenated outputs at both sequence ends.
- Adversarial and virtual adversarial training perturb the embeddings with bounded-norm changes rather than perturbing the discrete word inputs.The method uses normalized embeddings to prevent the model from making bounded perturbations insignificant through arbitrarily large embedding norms.
3 ADVERSARIAL AND VIRTUAL ADVERSARIAL TRAINING
Adversarial training regularizes classifiers by training against approximately worst-case perturbations, while virtual adversarial training smooths predictions without requiring labels. For text, the paper applies these perturbations to continuous word embeddings rather than sparse one-hot inputs.
- Adversarial training: Adversarial training adds a loss term that trains classifiers to withstand approximately worst-case input perturbations.Exact minimization is generally intractable, so the perturbation is approximated by linearizing the model loss and using backpropagation.
- Virtual adversarial training: Virtual adversarial training smooths the classifier along directions to which its current predictions are most sensitive.Its loss uses KL divergence between model output distributions and requires only inputs, not true labels, enabling semi-supervised learning.
- Text adaptation: For text classification, perturbations are applied to concatenated normalized word embeddings because one-hot inputs are discrete and high-dimensional.The paper defines the adversarial perturbation on the embedding sequence and constrains its norm.
- Text adaptation: The text model trains against embedding perturbations while replacing embeddings with normalized versions to prevent large embedding norms from making perturbations insignificant.The adversarial loss is minimized together with the negative log-likelihood using stochastic gradient descent.
- Scope: The proposed approach is presented as a regularizer that stabilizes the text classifier, not as a defense against an adversary.Virtual adversarial training uses both labeled and unlabeled examples in the text setting.
4 EXPERIMENTAL SETTINGS
Experiments evaluate the method across five text datasets spanning sentiment and topic classification, with supervised and semi-supervised settings. Models use recurrent language-model pretraining and compare adversarial methods under validation-selected regularization settings.
- Datasets: Five datasets cover sentiment classification on IMDB, Elec, and Rotten Tomatoes, plus topic classification on DBpedia and RCV1.DBpedia has no additional unlabeled examples and is evaluated only in the supervised setting.
- Datasets: Rotten Tomatoes uses a 90% training and 10% test split repeated with five random seeds, with unlabeled Amazon Reviews data added for semi-supervised experiments.The dataset itself does not provide separate test or unlabeled examples.
- Model preparation: Recurrent language-model pretraining initialized the word embeddings and LSTM weights, and was effective on classification performance across all tested datasets.The reported Section 5 results therefore use this pretraining.
- Model preparation: The experiments use unidirectional and bidirectional LSTM classifiers, with the bidirectional model tested on IMDB, Elec, and RCV1.The bidirectional model converged more slowly and was trained for 15,000 steps.
- Optimization: Validation tuning selected shared architecture settings and method-specific embedding dropout rates and adversarial norm constraints ǫ.The method-specific hyperparameters were optimized using validation performance.
5 RESULTS
Adversarial and virtual adversarial training improve performance across several text-classification settings, reduce overfitting during IMDB training, and produce embedding neighborhoods more aligned with sentiment semantics. Results vary by dataset, with virtual adversarial training underperforming on Rotten Tomatoes when unlabeled-loss effects dominate.
- IMDB performance and model analysis: Virtual adversarial training maintained low IMDB test negative log likelihood while baseline and adversarial training began to overfit later in training.The same tendency appeared in adversarial and virtual adversarial losses.
- IMDB performance and model analysis: 7.39% error rate was achieved by the embedding-dropout baseline on IMDB, while virtual adversarial training reached the 5.91% state-of-the-art error rate.The result matched the state of the art despite using a unidirectional rather than bidirectional LSTM.
- IMDB performance and model analysis: Every adversarial training method outperformed every random perturbation method on IMDB.The control experiments replaced adversarial perturbations with scaled-norm Gaussian random perturbations.
- IMDB performance and model analysis: Adversarial training moved ‘bad’ from ‘good’s’ nearest-neighbor list to rank 19, and virtual adversarial training moved it to rank 21.Cosine distances increased from 0.361 for the baseline and 0.377 for random perturbation to 0.463 and 0.464, respectively.
- IMDB performance and model analysis: 0.159–0.331 cosine distance for adversarial methods was smaller than the baseline and random methods’ 0.244–0.399 range for ‘great’ neighbors.The weaker positive word ‘good’ also moved from third to fifteenth nearest neighbor after virtual adversarial training.
- Elec, RCV1, and Rotten Tomatoes: The proposed method improved over baselines and achieved state-of-the-art performance on Elec and RCV1, while combined adversarial costs nearly matched the state of the art on Rotten Tomatoes.Only virtual adversarial training performed worse than the baseline on Rotten Tomatoes; the authors speculate that few, short labeled sentences allowed unlabeled loss to overwhelm supervised loss.
6 RELATED WORKS
The paper situates adversarial training among regularization and semi-supervised methods, extending input-space robustness ideas to text through continuous word embeddings. Compared with alternatives, the approach uses a straightforward, low-hyperparameter formulation and complements pretrained text representations.
- Regularization methods: Adversarial and virtual adversarial training outperformed random perturbations in the authors’ experiments and previous work.The comparison is framed as regularization against overfitting.
- Semi-supervised learning: Generative semi-supervised models require numerous hyperparameters, whereas adversarial methods require only one with a straightforward robust-optimization interpretation.The comparison concerns model complexity and interpretability of the training objective.
- Connections to SVMs: Adversarial methods and transductive SVMs both push decision boundaries away from examples, but impose margins in input space versus kernel-defined feature space.The paper distinguishes the spaces in which the respective margins are enforced.
- Text classification methods: View-embedding methods and the proposed method are complementary, because the proposed method improved performance from a recurrent pretrained language model.The cited comparison links pretrained representations with adversarial regularization.
7 CONCLUSION
Across text classification experiments, adversarial and virtual adversarial training provided strong regularization and matched or exceeded state-of-the-art performance. The methods also improved word-embedding quality, suggesting applicability beyond classification.
- Conclusion: Adversarial and virtual adversarial training had good regularization performance in sequence models on text classification tasks.This conclusion is stated across the authors’ experiments.
- Conclusion: On all datasets, the proposed method exceeded or was on par with state-of-the-art performance.The conclusion summarizes the cross-dataset classification result.
- Conclusion: Adversarial and virtual adversarial training improved both classification performance and the quality of word embeddings.The paper reports this as a joint experimental finding.
- Conclusion: The authors suggest applying the method to machine translation, distributed word or paragraph representations, question answering, video, and speech.These are proposed future or broader sequential-task applications.