Source-linked AI summary
Hierarchical relational models for document networks
Jonathan Chang, David M. Blei
TL;DR
Document-network modeling must account for both node attributes and uncertain relational structure. The RTM jointly models document words and binary links, outperforming alternatives in link and word prediction while supporting link suggestion and soft spatial clusterings.
Problem
Document networks require modeling document attributes alongside links whose absence may not reliably indicate that a relationship is absent.
Method
The RTM is a hierarchical model that combines LDA-style topic distributions over document words with binary link probabilities conditioned on document contents.
Results
The three RTM variants outperform all alternatives for link and word prediction; on Cora, predictive rank improves nearly 40% over baseline, while top-20 precision improves 80% over LDA + Regression.
Takeaways & Limitations
Jointly incorporating link and node information supports generalization to new documents, automatic link suggestions, and soft clusterings of geographically tagged data.
Takeaways & Limitations
The bag-of-words representation discards internal document structure such as syntax, motivating future RTM extensions for richer structure.
Abstract
from arXiv · showhide
We develop the relational topic model (RTM), a hierarchical model of both network structure and node attributes. We focus on document networks, where the attributes of each document are its words, that is, discrete observations taken from a fixed vocabulary. For each pair of documents, the RTM models their link as a binary random variable that is conditioned on their contents. The model can be used to summarize a network of documents, predict links between them, and predict words within them. We derive efficient inference and estimation algorithms based on variational methods that take advantage of sparsity and scale with the number of links. We evaluate the predictive performance of the RTM for large networks of scientific abstracts, web documents, and geographically tagged news.
2.1. Modeling assumptions.
The RTM combines LDA-style mixed-membership document topics with a probabilistic model of binary links, so the same latent representations explain words and connections.
- Modeling assumptions.: Each document has a Dirichlet-distributed topic-proportion vector, and each word is generated from a topic assignment followed by that topic’s word distribution.This reuses LDA’s mixed-membership assumptions for document attributes.
- Modeling assumptions.: Unlike traditional mixture models, mixed-membership topics allow each document to exhibit multiple latent components through its topic-proportion vector.The topic assignments for a document are collected in zd and participate in link modeling.
- Modeling assumptions.: For every document pair, the RTM generates a binary link variable conditioned on the topic assignments used to generate both documents.This statistically connects document content to network structure.
- Modeling assumptions.: The model parameters include topic-word multinomial vectors β1:K, a Dirichlet parameter α, and a binary link-probability function ψ.Observed documents are represented by discrete words from a fixed vocabulary, with binary variables representing links.
- Modeling assumptions.: The full graphical model contains observed words for all D documents and D2 link variables for every possible document connection.Figure 2 shows only a two-document segment, while plates indicate replication across the corpus.
2.2. Link probability function.
The RTM links documents through functions of their latent topic assignments, enforcing that the topics explaining words also explain links. It compares four link-probability forms with distinct similarity-response shapes.
- Link probability function.: The link probability function depends on the latent topic assignments zd and zd′ rather than only on document-level topic proportions.This ties the topics generating document words directly to the topics used for link prediction.
- Link probability function.: The model explores four link probability functions, including logistic, exponential, probit, and one-sided Gaussian forms.The functions differ in how link probability responds to latent-document similarity.
- Link probability function.: The logistic function ψσ models each binary link with regression coefficients η and intercept ν applied to Hadamard-product covariates.The element-wise product captures similarity between the documents’ hidden topic representations.
- Link probability function.: The logistic and probit functions are similarly sigmoidal, whereas the exponential remains steep at high similarity and the Gaussian has a one-sided form.The Gaussian function instead uses a weighted squared Euclidean difference and is constrained to return probabilities between zero and one.
- Link probability function.: Figure 3 plots link probability against the inner product of latent topic assignments, with all functions parameterized to share the same endpoints.The comparison isolates differences in the shapes of the probability curves.
3.1. Inference.
Inference approximates the RTM posterior with a fully factorized variational distribution and optimizes its ELBO using coordinate ascent. Treating unobserved nonlinks as hidden variables preserves ambiguous link semantics and makes computation scale with observed links.
- Inference.: The RTM uses variational inference because exact posterior inference is intractable for this hierarchical Bayesian model.Variational parameters are fit so their distribution approximates the true posterior under relative-entropy minimization.
- Inference.: The fully factorized variational family treats topic proportions and all topic assignments as independent, with γ and Φ as variational parameters.The expectation of each topic assignment is represented by the corresponding multinomial parameter φd,n.
- Inference.: Inference maximizes the evidence lower bound, whose first term incorporates document connections and distinguishes RTM inference from LDA.The ELBO includes expected link log-likelihood, topic-assignment terms, and entropy.
- Inference.: Only observed links are modeled, leaving nonlinks unobserved when their absence cannot be interpreted as evidence of no connection.This is motivated by settings such as social networks, where unrecorded relationships may still exist.
- Inference.: Inference complexity scales linearly with the number of observed links rather than the number of document pairs.Because unobserved link variables are leaves, they can be removed from computation; this matters when networks are sparse.
- Inference.: Coordinate ascent optimizes the ELBO with respect to γ and Φ, while link information nudges documents’ latent topic assignments toward those of their neighbors.The magnitude of this influence depends on similarity-related quantities and the chosen link function.
3.2. Parameter estimation.
The RTM estimates topic and link-function parameters with variational expectation-maximization, alternating ELBO optimization over the variational distribution and model parameters.
- Parameter estimation.: Parameter estimation seeks maximum-likelihood estimates for the topic vectors β1:K and link-function parameters η and ν.The likelihood problem is approximated because direct optimization is intractable.
- Parameter estimation.: Variational expectation-maximization alternates optimization of the ELBO over the variational distribution and the model parameters.This replaces the exact posterior computation in ordinary EM with variational inference.
- Parameter estimation.: The topic-parameter update uses expected sufficient statistics under the variational distribution and otherwise matches the variational EM update for LDA.Pseudocount smoothing is applied to topic-word estimates to help prevent overfitting.
- Parameter estimation.: Gradient-based optimization estimates the logistic link-function parameters η and ν using an approximation to the objective gradient.The procedure and updates for other link functions are given in the appendix.
- Parameter estimation.: Without negative observations yd1,d2 = 0, the link-function gradients cannot directly optimize the parameters.The authors address this constraint with a regularization penalty.
3.3. Prediction.
The RTM supports link prediction from words and word prediction from links, including for documents absent from training. Its evaluation uses held-out documents and predictive probabilities or ranks.
- The RTM defines two predictive tasks: estimating links from a new document’s words and estimating words from a new document’s links.
- Link prediction evaluates how probable a new document’s links are given its words and training data, using a variational approximation to an intractable posterior.
- A variant of link prediction ranks candidate links for a new set of documents using predictive probabilities proportional to equation (3.9).
- Word prediction estimates the words of a new document from its links using the same variational-posterior approximation technique.
- The model generates predictive distributions for words given links, links given words, or mixtures thereof, unlike models that treat link endpoints as training-data indices.
- The empirical study includes Cora, WebKB, PNAS, and geographically tagged local-news data, with preprocessing and summary statistics reported for the evaluated datasets.
4.1. Evaluating the predictive distribution.
The evaluation compares three RTM link functions with alternative models on held-out link and word prediction across document networks. RTMs outperform the alternatives on both tasks, with especially strong link-prediction gains on Cora.
- The evaluation compares logistic, exponential, and probit RTMs with Pairwise Link-LDA and LDA + Regression.
- Held-out evaluation asks how probable new documents’ links are given words and how probable their words are given links, reporting predictive rank versus topic count.
- For all datasets, the three RTM variants outperform every alternative model in link prediction.
- Nearly 40% improvement over baseline and 25% over LDA + Regression are observed for RTM link prediction on Cora.
- RTM link-prediction performance generally improves with more topics, whereas Pairwise Link-LDA worsens as topic count increases.
- The three RTM variants also outperform all alternatives in word prediction because link information influences their predictive word distributions.
4.2. Automatic link suggestion.
RTM is evaluated as a tool for suggesting citations from document text, using Cora examples and corpus-level precision comparisons against LDA + Regression. It produces more accurate and more specifically related suggestions.
- Table 2 compares the top eight link predictions from RTM and LDA + Regression for two Cora documents fitted with 10 topics.Boldfaced titles are documents actually cited by or citing each document.
- RTM finds 3 connected documents for each example, compared with 1 and 0 for LDA + Regression, respectively.
- 80%: RTM improves precision over LDA + Regression among the first 20 retrieved documents across the corpus.Twenty documents is presented as a reasonable number for a user to examine.
- RTM’s suggestions are more specifically related to the input, whereas LDA + Regression often retrieves only indirectly related documents.For the examples, RTM emphasizes convergence and stationary behavior in Monte Carlo papers and genetic algorithms rather than genomics.
- RTM distinguishes topics with similar vocabularies but different citation structures, while LDA + Regression conflates them when relying only on words.
4.3. Modeling spatial data.
On geographically linked local-news documents, RTM combines text with spatial connectivity to produce geographically coherent topics and region-specific word clusters, unlike LDA’s diffuse topics.
- Modeling spatial data: RTM combines local-news text with geographical adjacency to model spatially connected documents.The LocalNews corpus contains one document per state, with links determined by geographical adjacency.
- Topic distributions: LDA produces geographically diffuse topic assignments, whereas RTM concentrates topic mass in coherent regions.For example, RTM concentrates one topic in Southern states and another in Northeastern states, while LDA spreads corresponding topics across noncontiguous regions.
- Topic words: RTM’s topic assignments yield words that characterize geographically localized news stories more specifically than LDA’s broad cross-state themes.RTM identifies California-related terms such as “measure,” “protesters,” and “appeals,” and Northeastern terms such as “manslaughter,” “developer,” and “route.”
- Regional interpretation: RTM provides an exploratory tool for clusters driven jointly by word co-occurrence and geographic proximity.Its soft regional clusterings correspond to broad areas such as the South and Northeast, while placing Virginia between the MidAtlantic and South.
APPENDIX A: DERIVATION OF COORDINATE ASCENT UPDATES
Variational inference optimizes the evidence lower bound over γ and Φ, then derives coordinate updates by collecting terms, imposing normalization, and differentiating.
- Inference optimizes variational parameters γ and Φ by maximizing the evidence lower bound L.The expectations in L are expanded before deriving updates.
- Terms involving γ_d are collected after isolating the document-pair contribution L_d1,d2, which is independent of γ.
- Differentiating the collected γ terms and setting the derivatives to zero yields the coordinate update for γ.
- The update for φ_d,n is derived from its associated terms, with a Lagrange multiplier enforcing normalization.
- The φ_d,n gradient depends on the selected link probability function; for ψ_N, the expectation is expanded using word independence under the variational distribution.
APPENDIX B: DERIVATION OF PARAMETER ESTIMATES
Parameter estimation maximizes the variational objective over topic and link parameters, using constrained updates and regularization to maintain valid link probabilities.
- The model estimates topic parameters β and link parameters η,ν by maximizing the variational objective L.
- β is optimized with a Lagrange multiplier enforcing normalization, while ν scales the Gaussian distribution to match observed-link frequency and preserve admissible probabilities.
- Because all gradients are positive in the one-class estimation problem, unchecked parameter estimates diverge; regularization addresses this issue.
- ℓ2 regularization penalizes L with λ∥η∥2 and has a Bayesian interpretation as a Gaussian prior on η.
- An alternative regularization injects ρ observations with link variable y = 0 and associates them with document similarity.
- The approximation for ψ_e is constrained so its link functions remain probabilities and yields explicit parameter updates.
- Combining the likelihood gradient with the penalty produces updates guaranteed to satisfy 0 ≤ψ_e ≤1.