Source-linked AI summary
Syntactic Topic Models
Jordan Boyd-Graber, David M. Blei
TL;DR
Existing models fix the number of latent components, while language exhibits both document-level thematic and local syntactic structure. The STM combines these influences in a Bayesian nonparametric model, using variational inference, and generally achieves lower perplexity than syntax-only and topic-only alternatives.
Problem
Choosing the number of latent topics or syntactic states a priori can be difficult because existing models assume it is fixed.
Method
The STM models words in dependency-parsed documents using both document-level and parse-tree-level topic variables, with Bayesian nonparametric distributions and variational posterior inference.
Results
STM had lower perplexity overall than HDP and the infinite tree, while capturing topical themes and parts of speech in synthetic data.
Takeaways & Limitations
The model jointly represents thematic and syntactic influences, providing more structural information than either topic or syntax models individually.
Takeaways & Limitations
The model does not itself model parse structure, so richer extensions would be needed to use diverse corpora without relying on an external parser.
Abstract
from arXiv · showhide
The syntactic topic model (STM) is a Bayesian nonparametric model of language that discovers latent distributions of words (topics) that are both semantically and syntactically coherent. The STM models dependency parsed corpora where sentences are grouped into documents. It assumes that each word is drawn from a latent topic chosen by combining document-level features and the local syntactic context. Each document has a distribution over latent topics, as in topic models, which provides the semantic consistency. Each element in the dependency parse tree also has a distribution over the topics of its children, as in latent-state syntax models, which provides the syntactic consistency. These distributions are convolved so that the topic of each word is likely under both its document and syntactic context. We derive a fast posterior inference algorithm based on variational methods. We report qualitative and quantitative studies on both synthetic data and hand-parsed documents. We show that the STM is a more predictive model of language than current models based only on syntax or only on topics.
1. Background: Topics and Syntax
Topic models capture document-level thematic patterns, while syntax models capture patterns tied to sentence structure. The STM builds on both perspectives and Bayesian nonparametric methods to represent latent components whose number can be determined by the data.
- Probabilistic topic models: LDA models documents as mixtures of topics, with each topic distributing probability over terms and each document drawing topic proportions.Posterior inference recovers corpus-wide word co-occurrence patterns and document-specific mixtures of topics.
- Probabilistic topic models: LDA captures semantic correlations but ignores syntactic connections, whereas FTIC provides a syntactic complement by conditioning words on sentence structure.The two models share latent-component generation but differ in whether component weights are shared within documents or parse-tree contexts.
- Probabilistic syntax models: FTIC models parsed sentences with latent syntactic states whose word distributions and transitions follow the dependency-tree structure.Its components reflect how words occur within parse trees and correlate with part-of-speech labels.
- Bayesian nonparametric methods: Dirichlet processes and stick-breaking distributions provide Bayesian nonparametric priors with an unbounded number of components determined by the data.The stick-breaking concentration parameter controls how peaked or spread the component weights are, while the Dirichlet-process scaling parameter controls closeness to the base distribution.
2. The Syntactic Topic Model
The STM combines document-level thematic distributions with parse-tree-level syntactic distributions to model topics that are coherent in both contexts. It uses Bayesian nonparametrics and variational inference to learn latent structure from dependency-parsed documents.
- Model definition: The STM models words in documents of dependency parse trees, using observed words and tree structures while leaving topic assignments latent.It does not model the tree structure itself or use syntactic labels.
- Model definition: Each word’s topic is sampled from the normalized product of document-level and parse-tree-level topic weights.The document distribution enforces thematic consistency, while the parse-tree distribution enforces syntactic consistency.
- Generative process: Topics have word distributions and syntactic transition distributions, while documents have topic weights and the corpus has top-level weights.Transitions govern topics assigned to children of parents with a given topic.
- Nonparametric structure: The Bayesian nonparametric construction lets the data determine how many topics are used instead of fixing their number in advance.Stick-breaking and Dirichlet-process machinery provide the unbounded topic framework.
- Model relationships: The posterior jointly explains document-level and syntactic patterns, while limiting assumptions recover models that remove either tree or document influence.Fixing π to ones yields the HDP; fixing θ to ones yields the ITIC.
- Posterior inference: Exact posterior computation is intractable, so the model uses a truncated variational distribution to approximate the posterior and log likelihood.The approximation truncates the possible number of topics and supports parameter learning.
3. Experiments
Experiments on synthetic and Treebank data show that STM recovers both thematic and syntactic structure, while achieving lower perplexity overall than comparison models.
- Topics Learned from Synthetic Data: Syntactic models recover parts of speech but miss thematic distinctions, whereas HDP recovers themes but conflates parts of speech and misses their connections.The comparison uses synthetic documents with shared syntax and document-specific thematic distributions.
- Topics Learned from Synthetic Data: STM recovers both thematic themes and parts of speech in synthetic data, including latent-class interconnections identified by the infinite tree model.It separates thematic and syntactic divisions as complementary axes, although prepositions are grouped with nouns.
- Qualitative Description of Topics learned by the STM from Hand-annotated Data: STM learns specialized noun topics alongside broadly reused functional topics in the Treebank corpus.Examples include media, company-associated individuals, stock-price concepts, pronouns, possessives, and general verbs.
- Qualitative Description of Topics learned by the STM from Hand-annotated Data: Topics used in nearly every document appear to serve functional roles, while sparsely used topics represent specialized semantic content.This pattern is observed in per-document topic distributions with α = 0.1.
- Quantitative Results on Synthetic and Hand-annotated Data: On real data, STM has lower perplexity overall than HDP and the infinite tree, combining strong predictions for functional and content categories.HDP performs well on nouns, while STM and the infinite tree perform well on verbs; STM remains competitive with HDP on content words and predicts prepositions well.
4. Conclusion
The STM jointly learns thematic and syntactic influences, supporting richer structural representations than either model alone. The framework also suggests domain adaptation, while several extensions remain for more accurate parsing.
- The STM simultaneously models thematic and syntactic influences, capturing more structure than either model individually.
- The model illustrates how a mixture model can provide two simultaneous explanations for latent-class selection.
- Variational inference in the STM can support distributions beyond the discrete observations used here.
- A richer syntactic model could learn parse structure directly, allowing larger and more diverse corpora without externally supplied trees.
- Removing child-independence assumptions could improve approximation of state-of-the-art syntax models, especially for verbs with many children.
- The framework could adapt parsers to different domains while sharing information across them through document or corpus context.
Appendix A: Document Likelihood Bound
The appendix expands the likelihood lower bound by evaluating variational expectations. The STM’s renormalized product of document and syntactic distributions requires an auxiliary variational parameter, while other terms resemble LDA.
- The appendix explicitly computes expectations in the variational likelihood lower bound introduced earlier.
- Each word’s topic is sampled from the renormalized point-wise product of two multinomial distributions rather than one multinomial.
- The derivation introduces one variational parameter ω_n per word through a Taylor approximation to handle the renormalization’s expected log sum.
- Dirichlet expectations introduce the digamma function Ψ, the derivative of the logarithm of the gamma function.
- The remaining document likelihood terms are similar to LDA’s objective, including expectations over θ, z, and w distributions.
Appendix B: Document-specific Variational Updates
Document-specific variational updates handle the coupled document and syntactic distributions. Because their normalizer prevents explicit optimization, the updates use slack variables, derivatives, and numerical maximization.
- The appendix derives updates for all document-specific variational parameters except φ_n, which follows Equation 3.
- A slack term ω_n accounts for the fact that the point-wise product of π_k and θ_d may not sum to one.
- Coupling π and θ makes the normalizer prevent explicit solutions for γ and ν, so γ_d is optimized using componentwise partial derivatives.
Appendix C: Global Updates
The global variational updates expand the remaining likelihood terms and recover parameters of the stick-breaking representation. Coupled normalizers require implicit differentiation and conjugate-gradient optimization.
- The global update section expands likelihood terms not covered by Equation A.2, including the variational expectation involving β* and the GEM prior.
- Stick-breaking weights are recovered by dividing each β*_z by the sum of subsequent indices, using T_z as the remaining mass.
- The total likelihood bound combines the global contribution with the document-level bound L_d.
- The π–θ normalizer prevents explicit optimization of ν_i, so each ν_i is maximized individually with conjugate-gradient optimization after enforcing non-negativity.
- The final global variational parameter is β*, the variational estimate of the top-level weights β.
- Implicit differentiation is also required for the stick-breaking proportions T_z, including the final implicitly defined β_K.