Source-linked AI summary
Nested Hierarchical Dirichlet Processes
John Paisley, Chong Wang, David M. Blei, Michael I. Jordan
TL;DR
The nCRP restricts each document to a single tree path, limiting how documents combine topics and encouraging topic duplication. The paper develops the nHDP, which gives words document-specific path distributions over a shared tree, and derives scalable inference methods. Experiments on millions of New York Times and Wikipedia documents show that the approach can learn meaningful topic hierarchies while supporting large-scale inference.
Problem
The nCRP restricts each document to one path, limiting available topics and causing similar topics to appear in multiple parts of the tree.
Method
The nHDP uses nested document-specific Dirichlet-process path distributions over a shared tree and stochastic variational inference for scalable learning.
Results
The nHDP improved predictive performance over LDA and HDP on The New York Times and learned meaningful topic hierarchies on very large data sets.
Takeaways & Limitations
Allowing documents to borrow thematic content from different parts of a shared tree supports hierarchical topic modeling for massive text collections.
Takeaways & Limitations
The nCRP comparison is constrained by shallow truncations, where each document has only three topics for its thematic content.
Abstract
from arXiv · showhide
We develop a nested hierarchical Dirichlet process (nHDP) for hierarchical topic modeling. The nHDP is a generalization of the nested Chinese restaurant process (nCRP) that allows each word to follow its own path to a topic node according to a document-specific distribution on a shared tree. This alleviates the rigid, single-path formulation of the nCRP, allowing a document to more easily express thematic borrowings as a random effect. We derive a stochastic variational inference algorithm for the model, in addition to a greedy subtree selection method for each document, which allows for efficient inference using massive collections of text documents. We demonstrate our algorithm on 1.8 million documents from The New York Times and 3.3 million documents from Wikipedia.
1 INTRODUCTION
Hierarchical topic models represent text with trees whose topics become more specific down each path, but the nCRP restricts each document to one path. The nHDP replaces this rigidity with document-specific path distributions and scalable inference for massive corpora.
- Hierarchical topic models learn tree structures in which topics become increasingly specific from the root downward.
- The nCRP restricts each document to one path, limiting its available topics and creating duplicated topics across the tree.With shallow trees, a document has too few topics; with deep trees, inference involves too many nodes.
- The nHDP lets each word follow its own path under a document-specific distribution on a shared tree.This allows cross-thematic borrowing while preserving separate general topic areas in different subtrees.
- Stochastic variational inference enables hierarchical topic modeling on very large corpora, including millions of documents.The paper evaluates the approach on New York Times and Wikipedia collections.
RIC PRIORS FOR TOPIC MODELS
The nHDP is built from Bayesian nonparametric priors reviewed for posterior inference of hierarchical topic models.
- The paper reviews the Dirichlet process, nested Chinese restaurant process, and hierarchical Dirichlet process as foundations for the nHDP.
2.1 Dirichlet processes
The Dirichlet process provides a Bayesian nonparametric mixture prior that can learn an appropriate number of shared traits, with constructive representations supporting inference.
- Dirichlet process mixtures partition data by shared statistical traits while learning the number of traits and their parameters.
- The Dirichlet process generates a discrete random distribution over mixture parameters, even when its base measure is nonatomic.
- The Chinese restaurant process integrates out the Dirichlet process measure and induces dependence among observations through clustering.
- Stick-breaking constructs the random measure explicitly, preserving independence among observation-specific parameters for mean-field variational inference.
2.2 Nested Chinese restaurant processes
The nested Chinese restaurant process extends the Chinese restaurant process into a tree, where documents select paths and topics along those paths generate text. Its single-path restriction limits document-level thematic coverage and can duplicate topics across the tree.
- The nCRP recursively selects tables and transitions to nested restaurants, generating potentially infinite sequences of parameters.
- An nCRP tree gives each parent infinitely many children, with child selection based on prior sibling counts or a new-child probability proportional to α.
- A document selects one path through the shared tree, and an additional document-level stick-breaking process distributes probability over topics on that path.
- Issues with the nCRP: The single-path structure can leave shallow-tree documents with only three topics and cause similar topics to recur in multiple tree locations.Deeper trees become difficult to learn because their node count grows exponentially.
- Issues with the nCRP: The nHDP addresses this restriction by allowing words to choose paths under document-specific distributions rather than selecting one document-level path.
2.3 Hierarchical Dirichlet processes
The hierarchical Dirichlet process shares discrete atoms across groups while allowing each group to have its own distribution, providing a nonparametric topic-modeling extension of LDA.
- The HDP is a multi-level Dirichlet process whose discrete shared atoms let different groups reuse topics while retaining distinct probability distributions.Its base distribution is itself drawn from a Dirichlet process, so the shared atoms are learned rather than fixed in advance.
- The HDP has been used as a nonparametric extension of LDA for topic modeling.
- A two-level stick-breaking construction provides an explicit HDP representation for inference.The construction first samples a global distribution and then group-specific distributions in the same way.
- Because the global distribution is discrete, atoms can repeat across draws, and the resulting independent random variables simplify variational inference.
3 NESTED HIERARCHICAL DIRICHLET PRO-
The nHDP replaces the nCRP’s document-level single path with word-specific paths and document-specific distributions over a shared tree, while retaining thematic coherence through shared path probabilities.
- 3.1 Constructing a distribution on paths: The nHDP gives each document access to the entire shared tree while learning document-specific topic distributions that remain thematically coherent.Its design aims to preserve a dominant thematic path while allowing off-shoots for additional topics.
- 3.1 Constructing a distribution on paths: Each word follows its own path to a topic, and each document has its own distribution on paths in the shared tree.These are the two major changes from the nCRP formulation.
- 3.2 Generating a document: The nHDP generates a document by first distributing probability over paths and then selecting a terminating topic node for each word.
- 3.1 Constructing a distribution on paths: The model constructs a document-specific tree with the same nodes as the global tree but document-dependent path probabilities.
- 3.2 Generating a document: At each node, a document-specific beta switch determines whether a word stops at that topic or continues farther down the tree.
- 3.2 Generating a document: The resulting construction reuses paths across words while permitting off-shoots, and it recovers the nCRP when β →0.The shared switch variables create overlap among paths in a subtree.
4 STOCHASTIC VARIATIONAL INFERENCE FOR THE NESTED HDP
The paper develops stochastic variational inference for the nHDP by processing document subsets, updating local variables, and taking natural-gradient steps on global parameters, with greedy subtree selection to reduce per-document computation.
- 4.1 Stochastic variational inference: Stochastic variational inference targets massive corpora by separating local document variables from global variables shared across the dataset.It processes smaller groups of documents, updates their local variables, and then updates global variables rather than using batch inference.
- 4.1 Variational inference: Mean-field variational inference approximates the posterior with a factorized distribution and optimizes a variational objective over its parameters.
- 4.1 Variational inference: For conjugate exponential-family models, variational inference uses a simple optimization procedure based on sufficient statistics and a conjugate variational distribution.
- 4.1.2 A stochastic extension: Stochastic optimization forms a noisy objective gradient from a random data subset and optimizes the full objective in expectation.Equal-probability subsets make the subset objective an unbiased representation of the full objective.
- 4.1.2 A stochastic extension: Global variational parameters are updated by blending previous values with subset-derived information along a preconditioned natural-gradient direction.The preconditioning matrix can be chosen as the inverse Fisher information, yielding weighted sufficient-statistic updates.
- 4.2 Variational inference for the nHDP: The nHDP algorithm alternates word allocation and topic updates, then collects sufficient statistics for natural-gradient updates of global variables.
- 4.2 Variational inference for the nHDP: A greedy variational-objective procedure selects a small document-specific subtree, reducing the number of topics considered during inference.Words are restricted to the selected subtree, and indicators for topics outside it are fixed to zero.
5 EXPERIMENTS
Experiments compare the nHDP with the nCRP, LDA, and HDP in batch and stochastic settings, including massive New York Times and Wikipedia corpora. Results show improved predictive performance, broader per-document tree usage, meaningful hierarchical structure, and robustness to several parameter choices.
- Batch comparison: The batch nHDP outperforms the variational nCRP on all three data sets and outperforms Gibbs-sampling nCRP on the two larger data sets.The reported benefit of per-document distributions over the full tree appears to increase with corpus and document size.
- Stochastic inference for large corpora: 1.8 million New York Times articles and roughly 2.7 million Wikipedia pages are used to evaluate stochastic inference on massive corpora.The New York Times articles average 254 words from an 8,000-word vocabulary; Wikipedia documents average 164 words.
- The New York Times: On The New York Times, predictive performance improves with more documents processed, and nHDP performance improves over stochastic LDA and HDP.The evaluation uses average predictive log likelihood on held-out words as a function of documents processed.
- The New York Times: Roughly 12 topics are used per New York Times document despite a three-level tree, compared with three topics available per document under the nCRP.The nHDP therefore gives each document access to a substantially broader portion of the learned tree.
- The New York Times: The learned tree organizes topics with increasing specificity, including sports branching into sports and teams and foreign affairs into subregions or issues.A document combining sports and foreign affairs can allocate words to both subtrees under the nHDP, whereas the nCRP requires one path.
- Wikipedia and sensitivity analysis: Wikipedia shows the same pattern: nHDP improves over LDA and HDP, uses more of the tree than nCRP, and ultimately uses only a small subset of 1,220 initial nodes.Sensitivity analyses report robustness to the base Dirichlet parameter λ0, global DP parameter α, and reasonable ranges of β and (γ1, γ2).
6 CONCLUSION
The nHDP extends the nCRP by allowing observations to follow document-specific paths through a shared topic tree. Its stochastic inference method scales to very large datasets while learning meaningful topic hierarchies.
- The nHDP lets each observation follow its own path to a topic in a shared tree.
- Document-specific access to the entire tree enables borrowing thematic content from different parts of the hierarchy.
- The model uses a nested hierarchy of Dirichlet processes to sample document-specific path distributions for the shared tree.
- A stochastic variational inference algorithm makes the nHDP scalable to very large datasets.
- Comparisons with stochastic LDA and HDP showed that nHDP can learn meaningful topic hierarchies.