Source-linked AI summary
Modeling Documents with Deep Boltzmann Machines
Nitish Srivastava, Ruslan R Salakhutdinov, Geoffrey E. Hinton
TL;DR
Document topic models must provide useful latent representations while balancing flexible priors against tractable training and inference. The paper introduces a parameter-tied two-hidden-layer DBM that can be pretrained efficiently and achieves stronger modeling and feature-extraction results than several baselines. Its pretraining procedure is a practical initialization rather than a general likelihood maximizer.
Problem
Topic models need latent document representations, but flexible two-layer DBMs are more expensive to train and infer than efficient RBMs.
Method
The paper introduces an Over-Replicated Softmax two-hidden-layer DBM using tied parameters, efficient pretraining, and mean-field and MCMC-based approximate learning.
Results
The model improves on standard Replicated Softmax and outperforms LDA and DocNADE on document retrieval and classification tasks.
Takeaways & Limitations
Flexible priors can improve latent document features without increasing the number of trainable parameters, while training remains comparable to RBM-style CD.
Takeaways & Limitations
The tied-weight pretraining procedure does not generally maximize the model likelihood, although it provides a useful initialization for generative fine-tuning.
Abstract
from arXiv · showhide
We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a state initialization scheme that aids inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.
1 Introduction
The paper develops topic representations for document classification and retrieval by combining the efficient inference of RBMs with the more flexible priors of DBMs. Its Over-Replicated Softmax model is designed to improve performance while remaining efficient to train.
- Topic modeling discovers latent structure that represents document information for classification and retrieval.
- LDA and related directed models are widely used, but exact inference is difficult despite efficient approximate inference schemes.
- Replicated Softmax models provide exact, efficient latent-topic inference, but training remains difficult and sensitive to hyperparameter selection.
- RBMs train efficiently and infer hidden states exactly, whereas two-layer DBMs offer more flexible priors at the cost of more expensive training and inference.
- The Over-Replicated Softmax model combines efficient training and fast approximate inference with added flexibility in the prior over latent representations.
- The model improves on standard Replicated Softmax performance and outperforms LDA and DocNADE on classification and retrieval tasks.
2 Replicated Softmax Model
The Replicated Softmax model represents documents with shared-weight softmax visible units connected to binary hidden topic features. Its exact inference is efficient, but exact maximum-likelihood learning is intractable and therefore uses Contrastive Divergence.
- The model is a family of RBMs with softmax visible variables and binary hidden topic features.The visible matrix V has N word positions and K dictionary values, while h contains F binary hidden features.
- The energy function uses visible-hidden interactions and visible and hidden biases to define the model over document states.The parameters are θ = {W, a, b}, with W as interaction terms and a and b as hidden and visible biases.
- Each document receives as many softmax units as words, with shared weights connecting all units to the binary hidden layer.
- The hidden biases are scaled by document length so hidden units behave sensibly across documents of different lengths.The conditional distributions use softmax and logistic functions.
- The model can equivalently be viewed as one multinomial visible unit sampled N times.
- Exact maximum-likelihood learning is intractable because partition-function derivatives require time exponential in min{D, F}.In practice, the model is trained approximately with Contrastive Divergence.
3 Over-Replicated Softmax Model
The Over-Replicated Softmax model is a two-hidden-layer DBM with tied parameters that adds an implicit prior over latent topics while retaining the Replicated Softmax model’s parameter count. Efficient mean-field learning, pretraining, and inference procedures make the model practical for variable-length documents.
- Model structure: The model uses N visible softmax units, a binary topic layer h(1), and M hidden softmax units H(2), with shared and tied weights.The visible units represent document words, while H(2) forms an additional unobserved hidden layer.
- Model structure: The model can be viewed as a family of document-specific DBMs whose visible-layer size varies with document length while the second layer remains fixed.The number of visible softmax units equals the document’s word count N, whereas M is fixed across documents.
- Model properties: The second hidden layer changes the marginal distribution over V and supplies an additional implicit prior over h(1) without increasing the number of trainable parameters.The prior over h(1) combines distributions associated with the V–h(1) and h(1)–H(2) RBMs.
- Model properties: The relative influence of the data and the second-layer prior scales with N and M, so the prior dominates for N ≪ M and weakens for N ≫ M.When M = N, the marginal distribution over h(1) is the product of two identical distributions; the second layer performs half the modeling work compared with the first layer.
- Learning and inference: Learning uses a variational lower bound with a fully factorized mean-field posterior and MCMC-based stochastic approximation for model expectations.Mean-field inference estimates data-dependent expectations, while alternating Gibbs sampling and stochastic parameter updates approximate model expectations.
4 Experiments
Experiments on 20 Newsgroups and Reuters RCV1-v2 evaluate the Over-Replicated Softmax model as a generative model and feature extractor. It improves perplexity, retrieval—especially for short documents—and classification relative to comparison models.
- Experimental setup: Experiments use 20 Newsgroups and Reuters RCV1-v2 to evaluate generative modeling, document retrieval, and document classification.The datasets contain single-label and multilabel documents, respectively.
- Perplexity: Perplexity decreases from 1081 to 1060 on Reuters and from 965 to 958 on 20 Newsgroups versus Replicated Softmax.All models use 128 latent topics; the reported perplexities are upper bounds because the log-probability estimate is a lower bound.
- Document Retrieval: The Over-Replicated Softmax DBM outperforms other models on both retrieval datasets, particularly when retrieving the top few documents.Documents are ranked using cosine distance between representations based on conditional posterior distributions over hidden features.
- Document Retrieval: Retrieval gains are largest for short documents, which are frequent in both datasets and comprise 50% of 20 Newsgroups documents below 35 words.The fixed M prior has a stronger effect on short documents, while other models often perform poorly for them.
- Document Classification: For 512 topics on 20 Newsgroups, accuracy rises from 64.2% for LDA and 67.7% for Replicated Softmax to 69.4% for Over-Replicated Softmax.On Reuters, Mean Average Precision reaches 0.453 versus 0.427 for DocNADE and 0.421 for Replicated Softmax.
- Document Classification: Classification performance is similar across models for long documents, whereas the Over-Replicated Softmax model performs well on short documents.This document-length pattern parallels the retrieval results.
5 Conclusion
The Over-Replicated Softmax model provides a flexible prior without increasing trainable parameters, while enabling fast RBM-like training and strong document features. Variants that adapt the hidden softmax units to document length further improve over standard baselines.
- The model defines a flexible prior over latent topic features without increasing the number of trainable parameters.It adds only a minor increase in training algorithm complexity.
- Fast approximate training makes the Deep Boltzmann Machine trainable with contrastive divergence like an RBM.
- Its document features outperform standard Replicated Softmax and LDA features and are comparable to DocNADE across network sizes.
- Setting M to depend on document length offers alternatives that improve over standard Replicated Softmax, LDA, and DocNADE.One option uses M = cN; another uses M = Nmax − N.
- Choosing M = Nmax − N keeps the total number of replicated softmax units fixed, yielding one architecture and a single partition function.