Source-linked AI summary
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
Adji B. Dieng, Chong Wang, Jianfeng Gao, John Paisley
TL;DR
Existing language models capture local sequence structure but struggle with long-range semantic dependencies, while topic models capture global semantics without word order. TopicRNN combines an RNN with latent topics, learning both components end-to-end; it improves contextual-RNN word prediction and achieves a 6.28% IMDB sentiment-classification error rate.
Problem
Language models need local syntactic structure and global semantic coherence, but existing RNNs and topic models do not explicitly capture both properties together.
Method
TopicRNN combines an RNN for sequential dependencies with latent topics that provide additive contextual bias, using stop-word indicators and joint variational learning.
Results
TopicRNN achieves lower perplexity than the contextual RNN baseline on Penn TreeBank and reports a 6.28% error rate for IMDB sentiment classification.
Takeaways & Limitations
TopicRNN serves as an unsupervised document feature extractor and produces sensible topics and text, offering an alternative to document models such as LDA.
Takeaways & Limitations
Topic models generally ignore word order, limiting their performance on general-purpose language modeling, and they model stop words poorly.
Abstract
from arXiv · showhide
In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of $6.28\%$. This is comparable to the state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.
1 INTRODUCTION
Language models must capture local syntax and global semantic coherence, but existing approaches struggle to model both explicitly. TopicRNN addresses this gap by combining recurrent sequence modeling with latent-topic context learned end-to-end.
- Correct syntax is primarily local because prediction often depends on a few preceding words and their order.
- Semantic coherence is more global because understanding a sentence or document can require many preceding words, whose ordering matters less.
- Fixed context windows limit traditional n-gram and feed-forward models, while RNNs can still struggle to explicitly model syntax and semantic coherence together.
- TopicRNN captures long-range semantic dependencies through latent topics that provide context to an RNN language model.
- Unlike contextual RNN baselines using pre-trained topic features, TopicRNN learns its topic context and RNN parameters jointly end-to-end.
- The paper evaluates TopicRNN on Penn TreeBank word prediction and IMDB sentiment classification, and outlines the remaining sections covering model details and experiments.
2 BACKGROUND
RNN language models extend beyond fixed context windows but face practical difficulties with long-range dependencies. Probabilistic topic models capture global semantic coherence while ignoring word order, motivating complementary modeling approaches.
- Language modeling assigns probabilities to word sequences and supports applications including speech recognition and machine translation.
- Traditional n-gram and feed-forward neural models impose fixed-size context windows through Markov assumptions.
- RNN language models condition each word on all preceding words through a hidden state, avoiding the fixed-window assumption.
- A footnote states that Ghosh et al. (2016) did not publish Penn TreeBank results and that the authors did not find its code online.
- In practice, large RNNs can encounter optimization and overfitting difficulties, making better long-range dependency modeling an open challenge.
- Probabilistic topic models represent documents through latent topics that capture global semantic coherency and groups of co-occurring words.
- LDA uses K underlying topics, each a distribution over a fixed vocabulary, and generates document words through latent topic proportions.
2. For the tth word in the document,
Topic models capture global semantics by ignoring word order, but this limits general-purpose word prediction and leaves stop words poorly modeled. TopicRNN addresses stop-word handling with explicit indicators.
- Marginalizing each latent topic assignment yields the probability of a document word sequence through matrix factorization and integration over θ.
- Most topic models ignore word order, which helps capture global semantic information but limits their performance on general-purpose language modeling.
- Higher-order topic models quickly become intractable, despite proposals such as bigram topic models.
- Stop words are difficult for topic models because they carry little semantic meaning and occur frequently across documents.
- Figure 1 encodes stop-word status with binary indicators, showing stop words in red and content words in blue.
3 THE TOPICRNN MODEL
TopicRNN combines latent topics with an RNN to model global semantic dependencies alongside local sequence dynamics. It uses variational inference and a topic-based output bias to separate global semantics from local language dynamics.
- TopicRNN model: TopicRNN uses latent topics to capture global semantic dependencies while the RNN models local sequence dynamics.The model is generative and combines topic information with recurrent language modeling.
- Topic-conditioned output: The stop-word indicator controls whether the topic vector contributes to the output distribution.For non-stop words, the topic vector adds a bias based on its interaction with the latent word vector.
- Topic-conditioned output: TopicRNN separates global semantics from local dynamics by excluding stop words from topics while retaining them in the RNN hidden states.The topic vector is used as an output bias rather than being passed into the recurrent hidden state.
- Inference: The model learns its parameters jointly by maximizing a variational evidence lower bound because direct marginal-likelihood optimization is intractable.The variational distribution is produced by a feed-forward ReLU inference network that projects document term frequencies into a K-dimensional latent space.
- Prediction: During prediction, TopicRNN estimates the topic vector from the current document context and updates that estimate as non-stop words are observed.A sliding window is used to reduce the cost of updating the variational distribution after each word prediction.
- Model complexity: TopicRNN has complexity O(H × H + H × (C + K) + Wc), adding inference-network parameters Wc to the recurrent model terms.The contextual RNN comparison omits pre-training parameters, which may exceed this additional cost.
4 EXPERIMENTS
Experiments evaluate TopicRNN and its recurrent variants on Penn TreeBank word prediction, text generation, and IMDB sentiment analysis. The results show lower perplexity than the previous contextual-RNN baseline and a 6.28% IMDB classification error rate, while generated features cluster reviews by sentiment.
- Experimental setup: TopicRNN, TopicLSTM, and TopicGRU were evaluated on Penn TreeBank word prediction and IMDB sentiment analysis.The experiments used RNN, LSTM, and GRU cells within the topic-enhanced framework.
- Word prediction: A TopicGRU with 100 neurons achieved per-word perplexity of 112.4 versus 115.9 for stacked LSTMs with 200 neurons each.The table compares TopicRNN variants and counterparts across 10-, 100-, and 300-neuron networks.
- Word prediction: Per-word perplexity decreased with larger networks, contextual RNNs outperformed non-contextual RNNs, and TopicRNN improved on Mikolov and Zweig’s reported baseline.For fair comparison, topic vectors were estimated from preceding batches using a sliding window.
- Text generation: Generated text was conditioned on topics inferred from seed documents, with examples reported for models trained on PTB and IMDB data.The IMDB-generated text was described as encoding a negative sentiment.
- Sentiment analysis: TopicRNN feature vectors were clustered with K-Means and visualized using two-dimensional PCA, showing positive and negative reviews as green and red clusters.The experiment used a sample of 10,000 IMDB reviews.
5 DISCUSSION AND FUTURE WORK
The paper concludes that TopicRNN combines local syntactic and global semantic dependencies through jointly learned latent-topic context. It reports competitive PTB perplexity and IMDB sentiment error, sensible topics and text, and proposes broader applications and dynamic stop-word discovery as future work.
- Discussion: TopicRNN combines RNNs and latent topics to capture local syntactic and global semantic dependencies between words.Latent topics provide contextual bias to the RNN, learned jointly by maximizing the variational-inference evidence lower bound.
- Discussion: TopicRNN yields competitive per-word perplexity on Penn TreeBank and a competitive sentiment-classification error rate on IMDB 100K.The conclusion reports these outcomes without giving additional numerical values.
- Future work: TopicRNN also generates sensible topics and text, supporting its use for modeling document context.The paper identifies dynamic stop-word discovery and dialog modeling as future directions.
A.1 DIMENSION OF THE PARAMETERS OF THE MODEL:
This section defines the main TopicRNN dimensions and points to a table listing the dimensions of its parameters.
- C denotes the vocabulary size, including stop words.
- H denotes the number of hidden units in the RNN.
- K denotes the number of topics, while E denotes the inference-network hidden-layer dimension.
- Table 5 lists the dimensions of the TopicRNN model parameters, ignoring biases.
A.2 DOCUMENTS USED TO INFER THE DISTRIBUTIONS ON FIGURE 2
The appendix identifies document text associated with the left, middle, and right portions of Figure 2, preserving their word sequences as examples.
- The left document begins with market conditions after the China crisis and discusses Japanese financial activity, Hong Kong exports, and U.S. demand.
- The middle document describes a man diagnosed with AIDS, followed by testing of his associates and family.
- The right document concerns bridge replacement and related changes affecting locations including Hartford and Peninsula, Ohio.
A.3 MORE GENERATED TEXT FROM THE MODEL:
This section presents additional text generated by TopicRNN trained separately on the PTB and IMDB datasets, using the stated experimental settings.
- PTB-generated text: The PTB generation uses 50 neurons and 100 topics.The generated examples include financial language involving bonds, funds, rates, and markets.
- PTB-generated text: One PTB sample discusses treasury-related funds, institutions, investors, and compounding.
- PTB-generated text: Another PTB sample discusses stock-exchange bonds, shares, yields, and long-term funds.
- IMDB-generated text: The IMDB generation uses the same settings as the sentiment-analysis experiment and produces movie-review-like text with mixed evaluative language.
A.4 TOPICS FROM IMDB:
The IMDB sentiment-analysis experiment produces 200 topics, all centered on movies because every review concerns movies.
- The TopicRNN model yields 200 topics from the IMDB sentiment-analysis data.
- All displayed topics revolve around movies, as expected because the dataset consists entirely of movie reviews.