Source-linked AI summary
Generative and Discriminative Text Classification with Recurrent Neural Networks
Dani Yogatama, Chris Dyer, Wang Ling, Phil Blunsom
TL;DR
The paper asks whether generative LSTM classifiers can address discriminative neural networks' data demands and poor adaptation under distribution shifts. It compares discriminative and generative LSTM text classifiers across standard, continual, and zero-shot settings, finding faster generative learning in small-data regimes and stronger suitability for shifting-data settings despite higher asymptotic error. Generative models also incur substantially greater training and inference costs because of vocabulary-wide softmax computation.
Problem
Discriminative neural networks need abundant data and generalize poorly when labels, domains, or tasks shift, motivating generative models for sample complexity and adaptation.
Method
The paper empirically compares discriminative and generative LSTM-based text classification models, including shared-parameter and independent generative variants, in standard, continual, and zero-shot settings.
Results
Generative models perform better in the small-data regime and are more suitable for continual and zero-shot settings, while discriminative models achieve lower asymptotic error rates.
Takeaways & Limitations
Generative models can be preferable when labeled data are scarce or the learning setting changes, despite their higher asymptotic error relative to discriminative models.
Takeaways & Limitations
Generative models are much slower to train and infer because word prediction requires softmax computation over vocabularies often containing hundreds of thousands of words.
Abstract
from arXiv · showhide
We empirically characterize the performance of discriminative and generative LSTM models for text classification. We find that although RNN-based generative models are more powerful than their bag-of-words ancestors (e.g., they account for conditional dependencies across words in a document), they have higher asymptotic error rates than discriminatively trained RNN models. However we also find that generative models approach their asymptotic error rate more rapidly than their discriminative counterparts---the same pattern that Ng & Jordan (2001) proved holds for linear classification models that make more naive conditional independence assumptions. Building on this finding, we hypothesize that RNN-based generative classification models will be more robust to shifts in the data distribution. This hypothesis is confirmed in a series of experiments in zero-shot and continual learning settings that show that generative models substantially outperform discriminative models.
1 Introduction
The paper compares discriminative and generative LSTM text classifiers, asking whether generative models improve sample efficiency and robustness to shifting data distributions. Generative models approach their asymptotic error faster, while experiments examine their behavior in continual and zero-shot learning.
- Motivation: Discriminative neural networks work well with abundant, stable data but require substantial training data and generalize poorly under distribution shifts.The paper considers shifts involving new labels, domains, and tasks.
- Approach: The study compares discriminative and generative text-classification variants built from a common LSTM architecture.The generative model produces documents word by word conditioned on a learned class embedding.
- Main finding: Discriminative models achieve lower asymptotic error, but generative models approach their asymptotic error rate more rapidly.This reproduces the pattern established theoretically for linear generative and discriminative classifiers despite the nonlinear models' greater expressive power.
- Continual learning: Continual learning introduces labels sequentially while restricting learning to examples from newly introduced classes.The paper investigates whether generative models are better suited to this setting than discriminative models vulnerable to catastrophic forgetting.
- Zero-shot learning: In zero-shot learning, the paper compares mapping documents into a fixed semantic label space with generating documents from points in that space.The authors report substantial benefits for generative models.
2 Models
The paper defines discriminative classifiers through p(y | x) and generative classifiers through p(x | y)p(y), then implements both with LSTM-based architectures. Its generative variants either share parameters across classes or use independent class-specific models.
- Training objectives: Discriminative models maximize the conditional probability of labels given documents, whereas generative models maximize the joint probability under p(x | y)p(y).Generative prediction uses Bayes' rule to compute p(y | x).
- Discriminative model: The discriminative model encodes documents with a peephole LSTM and averages hidden representations before classification.A softmax layer predicts labels from the averaged document representation.
- Discriminative model: The discriminative model uses a simple hidden-state average because it performed better than the last state and costs less than attention for long documents.The model is trained to maximize the conditional probability of the label given the document.
- Generative models: The Shared LSTM generative model factorizes p(x | y) into sequential word predictions conditioned on prior words and a label embedding.It shares word embeddings, LSTM parameters, and vocabulary-softmax parameters across classes.
- Generative models: Independent LSTMs remove parameter sharing by giving every class its own word embedding, LSTM, and softmax parameters.This increases the number of parameters but allows training to be parallelized across classes.
- Generative models: Both generative models train the LSTM to maximize joint probability and predict labels by maximizing p(x | y)p(y) using empirical class frequencies.Compared with the discriminative model, they add word-prediction parameters U.
3 Experiments
Experiments compare generative and discriminative LSTM classifiers across standard, small-data, continual-learning, and zero-shot settings. Generative models trail discriminative models asymptotically but show advantages with limited data and shifting or unseen classes.
- Datasets and models: The evaluation uses six standard text-classification datasets spanning news, sentiment, Wikipedia, and question-answer categorization, with 5,000 training examples held out for development.Models are also compared with prior-work baselines and several generative baselines.
- Standard and sample-complexity results: Discriminative LSTMs achieve lower asymptotic error than generative models, matching the pattern established for generative and discriminative linear classifiers.The recurrent generative models are more expressive because they model dependencies among words.
- Standard and sample-complexity results: Generative LSTM models outperform naïve Bayes, Kneser–Ney Bayes, and naïve Bayes neural-network baselines, indicating benefits from modeling word dependencies.Shared and independent generative LSTMs perform roughly similarly.
- Standard and sample-complexity results: With 5, 20, 100, or 1000 examples per class, the shared generative LSTM outperforms the discriminative model in almost every small-data case except AG News.It also remains stronger than naïve Bayes and Kneser–Ney Bayes in this regime.
- Continual learning: In continual learning, models receive classes sequentially and cannot retrain on previously seen classes; discriminative LSTMs suffer catastrophic forgetting in this setup.The experiments examine whether generative models adapt better when new classes appear.
- Zero-shot learning: The experiments also test zero-shot learning with semantically meaningful labels and hidden classes, comparing discriminative mapping with generative modeling in the label space.Generative models show substantial benefits, while performance declines when two classes are hidden because hidden-class predictions add noise.
4 Discussion
The discussion examines document likelihood as a possible signal for detecting unknown classes and highlights the computational cost of generative models.
- Computational complexity: Generative training and inference are substantially slower because its softmax is computed over the entire vocabulary.For datasets with 115,000 and 1,395,000 examples, generative training took approximately two hours and two days, versus 20 minutes and six hours discriminatively.
- Computational complexity: Approximate softmax methods can accelerate generative models, but discriminative models remain much faster.The cited approximations include hierarchical softmax, noise contrastive estimation, sampled softmax, and one-vs-each approximation.
- Data likelihood: Generative models compute document probability by marginalizing over classes, whereas discriminative models lack an explicit p(x) model.The generative probability is p(x) = Σ_y∈Y p(x | y)p(y); obtaining p(x) discriminatively would require separate language-model training.
- Data likelihood: In an AG News experiment, the generative LSTM was trained using examples from only three of four labels and evaluated on all test documents.Figure 4 reports test-data log likelihood when training examples from class 0 or class 1 are excluded.
5 Conclusion
The paper finds that generative LSTM classifiers learn effectively with limited data and are more suitable for continual and zero-shot settings than discriminative models. In these settings, they can achieve performance comparable to generative models trained on full datasets in the standard setting.
- Conclusion: Generative models are better than discriminative models in the small-data regime, extending Ng and Jordan’s results from linear to nonlinear models.The authors describe this transfer of generalization behavior as notable because the neural models are more complex.
- Conclusion: Generative models are more suitable than discriminative models in continual and zero-shot learning settings.The conclusion reports this pattern across the collection of continual- and zero-shot-learning results.
- Conclusion: Generative models achieved performance comparable to generative models trained on full datasets in the standard setting.This comparison is stated for the continual and zero-shot settings.