Source-linked AI summary
Low-dimensional Embeddings for Interpretable Anchor-based Topic Inference
Moontae Lee, David Mimno
TL;DR
Greedy anchor selection can yield poor, hard-to-interpret topics, while anchor-word inference depends on choosing good anchors. The paper finds exact convex hulls in visualizable low-dimensional embeddings; t-SNE gives the best overall anchor selection performance, with higher-frequency terms and clearer, more interpretable anchors. The approach also lets users adjust topic granularity by changing projection dimension.
Problem
Greedy anchor selection can produce eccentric words and nearly identical topics, while anchor-based inference depends on selecting good anchors.
Source: §1 · pp. 1–2Method
The method projects word co-occurrence statistics into low-dimensional spaces and selects exact convex-hull vertices as anchors.
Source: §1 · p. 2Results
t-SNE provides the best overall anchor selection performance, consistently choosing higher-frequency terms that yield greater clarity and interpretability.
Source: §5 · p. 9Takeaways & Limitations
Changing projection dimension lets users choose topic granularity, without claiming that one topic count is correct for a corpus.
Source: §5 · p. 9Held-out probability is sensitive to smoothing parameters and should be used alongside other topic-quality metrics.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
The anchor words algorithm performs provably efficient topic model inference by finding an approximate convex hull in a high-dimensional word co-occurrence space. However, the existing greedy algorithm often selects poor anchor words, reducing topic quality and interpretability. Rather than finding an approximate convex hull in a high-dimensional space, we propose to find an exact convex hull in a visualizable 2- or 3-dimensional space. Such low-dimensional embeddings both improve topics and clearly show users why the algorithm selects certain words.
1 Introduction
The paper replaces high-dimensional approximate-hull anchor selection with exact hulls in visualizable low-dimensional embeddings, aiming for more salient, interpretable anchors. This addresses greedy selections that can harm topic quality and make labels less useful.
1 Introduction
- Anchor words support separable topic inference, but their usefulness depends on selecting good anchors; the algorithm reconstructs non-anchor co-occurrence patterns from anchor patterns.Each anchor serves as a pivot for disambiguating its topic distribution.Source: §1 · p. 1
- In the Yelp t-SNE illustration, hull vertices are anchor words and interior points are non-anchor words.Source: Figure 1 · p. 1
- Greedy selection can choose infrequent, idiosyncratic words early, leading to poor topic recovery and many nearly identical topics.The algorithm selects the farthest point from the current subspace at each step.Source: §1 · pp. 1–2
- Greedy extreme words may be distinctive but have low probability in their topics, limiting their usefulness as topic labels.Good summary terms should be both distinctive to a topic and probable within it.Source: §1 · p. 2
- The method finds exact convex hulls in visualizable 2D or 3D projections, producing topics described as as good or better than greedy anchors and providing a visual explanation for word selection.The projected input matrix is up to 3600 times narrower than the original for the training corpora.Source: §1 · p. 2
2 Related Work
The related work situates anchor inference within LDA and recovery from word co-occurrence statistics. It highlights the separability assumption, limits of unconstrained subset-selection alternatives, and prior uses of low-dimensional visualization.
2 Related Work
- LDA estimates topic-word and document-topic distributions and latent topic assignments from an observed word-document matrix.Source: §2 · p. 2
- Anchor-word inference recovers word-topic and topic-topic matrices using the empirical word-word matrix as an approximation to its expectation.Source: §2 · p. 2
- The anchor method assumes each topic has a word occurring with non-negligible probability only in that topic, and its overall performance depends heavily on anchor quality.The algorithm first selects a user-specified number of anchors, then recovers the matrices based on them.Source: §2 · p. 2
- Subset-selection algorithms did not change anchor selection in the authors’ experiments, largely because they produce unconstrained bases rather than nonnegative convex bases.Source: §2 · pp. 2–3
- Prior work used t-SNE to display word embeddings and topic spaces, but the authors knew of no prior use of it as a fundamental component of topic inference.Source: §2 · p. 3
3 Low-dimensional Embeddings
The method reduces word co-occurrence representations to low-dimensional spaces, then selects exact convex-hull vertices as anchors. PCA and t-SNE provide different projection properties, while projection dimension controls topic granularity and supports visualization.
3 Low-dimensional Embeddings
- Because the word co-occurrence matrix scales quadratically with vocabulary size, the method uses low-dimensional projections rather than roughly 1,000-dimensional random projections that still require approximate hull selection.Source: §3 · p. 3
- PCA linearly captures the largest variances along orthogonal axes, whereas t-SNE nonlinearly preserves local structure by aligning low-dimensional pairwise probabilities with high-dimensional similarities.Source: §3 · p. 3
- The method selects exact vertices of the convex hull in the projected space as anchors; the 2D figures are illustrative, and 3D projections perform better in practice.Source: §3 · p. 3
- Projection dimension determines the number of hull vertices and thus lets users tune topic granularity while visualizing relationships between anchors and other vocabulary words.The examples have 21 vertices in 2D and 69 in 3D.Source: §3 · p. 4
4 Experimental Results
Across three real datasets, low-dimensional embeddings produce more distinctive topics and more salient anchor words than the greedy method, while maintaining comparable held-out likelihood and semantic coherence. The experiments compare anchor-selection methods quantitatively and qualitatively, with t-SNE performing especially well on topic specificity and held-out probability.
4 Experimental Results
- On the political-blog corpus, 86% of t-SNE topics have a unique top word, versus 66% for greedy, which assigns 17 topics to obama.Only one t-SNE topic has obama as its most probable word.Source: §4 · p. 4
- Experiments use Yelp reviews, political blogs, and NYT articles, vary anchor selection while keeping topic-word recovery the same, and run five t-SNE initializations per dataset.The number of selected anchors varies across initializations.Source: §4 · p. 4, §4.3 · p. 8, §4.2 · p. 8
4.1 Anchor-word Selection
- For NYT, t-SNE and PCA select more general anchors such as police, business, and court than greedy’s cavalry, al-sadr, and yiddish, yielding more specific topics.The paper attributes the greedy topics’ similarity to their closeness to the corpus unigram distribution.Source: §4.1 · pp. 5–6
4.2 Quantitative Results
- t-SNE has slightly better recovery error than greedy, with all methods improving substantially after the first 10–15 anchor words.Recovery results for the original, unprojected matrix are much worse than for the other algorithms.Source: §4.2 · p. 6
- Across topic counts, t-SNE and PCA have consistently lower normalized entropy and better specificity and dissimilarity than greedy, whose topics stay closer to the corpus distribution.With few anchors, the topic distributions are close to the corpus distribution and similar to one another.Source: §4.2 · pp. 6–7
- t-SNE anchor words remain relatively more probable within their topics as topic counts increase, while greedy anchors become more prominent only as the number of topics rises.The paper measures anchor-word prominence using hard and soft rank.Source: §4.2 · pp. 7–8
- The t-SNE method appears to do best on held-out probability, particularly for NYT, while greedy coherence is higher but comes with more overly general or repetitive topics.Held-out probability is sensitive to smoothing parameters and is intended to be considered alongside other topic-quality metrics.Source: Figure 7 · p. 7, §4.2 · p. 8
4.3 Qualitative Results
- In qualitative comparisons, greedy topics corresponding to t-SNE topics usually have higher topic-order numbers, and greedy anchor words are often tens of words down the rank list.PCA and t-SNE are often consistent in anchor selection, while PCA can suffer from crowding in lower-dimensional projection.Source: §4.3 · pp. 8–9
5 Conclusion
Low-dimensional embeddings address anchor quality, yielding salient anchors and improved topic-model properties while making anchor selections visually explainable. They also make topic count a projection-dimension choice, not a uniquely correct corpus value.
5 Conclusion
- Inference scales with vocabulary rather than corpus size, but adding documents increases co-occurrence-matrix construction time and anchor quality remains crucial.Source: §5 · p. 9
- Low-dimensional co-occurrence embeddings produce distinctive, probable anchors, and models trained with them have better quantitative and qualitative properties along various metrics.Source: §5 · p. 9
- Topic count depends on projection dimension, letting users choose granularity without claiming one dimension is correct for a corpus.Source: §5 · p. 9
- t-SNE provides the best overall anchor selection, consistently choosing higher-frequency terms for greater clarity and interpretability; PCA is less effective on held-out probability for sufficiently large corpora.Source: §5 · p. 9
- Visualizable co-occurrence spaces can make anchor choices clear, support effective topic labels, build user confidence, and enable semisupervised topic reconstruction environments.Source: §5 · p. 9