Source-linked AI summary
Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec
Christopher E Moody
TL;DR
The paper addresses the gap between dense word representations and interpretable document-level topic mixtures. It introduces lda2vec, which jointly trains word, topic, and document vectors using sparse Dirichlet-constrained memberships. The model produces coherent topics and learns semantic token relationships across the evaluated corpora.
Problem
Dense word vectors capture token-level regularities, while topic models provide interpretable document representations; combining these capabilities remains a central modeling goal.
Method
lda2vec extends SGNS by jointly training word, topic, and document vectors, with document-topic proportions constrained to be non-negative and sum to unity.
Results
lda2vec yields high mean topic coherences on Twenty Newsgroups and discovers salient Hacker News topics while learning linear relationships between words.
Takeaways & Limitations
The model provides sparse, interpretable document-to-topic proportions while preserving semantic regularities in learned word vectors.
Abstract
from arXiv · showhide
Distributed dense word vectors have been shown to be effective at capturing token-level semantic and syntactic regularities in language, while topic models can form interpretable representations over documents. In this work, we describe lda2vec, a model that learns dense word vectors jointly with Dirichlet-distributed latent document-level mixtures of topic vectors. In contrast to continuous dense document representations, this formulation produces sparse, interpretable document mixtures through a non-negative simplex constraint. Our method is simple to incorporate into existing automatic differentiation frameworks and allows for unsupervised document representations geared for use by scientists while simultaneously learning word vectors and the linear relationships between them.
1 Introduction
Topic models offer interpretable sparse document representations, whereas neural approaches provide dense semantic vectors. lda2vec combines these strengths while making model development practical within automatic differentiation frameworks.
- Topic models organize documents into interpretable themes and can reveal word clusters and temporal patterns.
- Neural document representations model documents as dense vectors, while probabilistic topic models use sparse mixed-membership representations.
- Automatic differentiation lets practitioners focus on model design rather than deriving customized approximations for each probabilistic topic model.
- Traditional topic models do not exploit distributed word representations that capture semantic regularities between tokens.
- Existing word-level representations support information retrieval and parsing, motivating their use in document-level abstractions.
- lda2vec extends distributed word representations with interpretable document representations and supports model inference and extension within automatic differentiation.
2 Model
lda2vec modifies SGNS by combining local word representations with document-wide vectors, while constraining documents to sparse mixtures of shared topic vectors. It jointly learns interpretable document memberships and word embeddings, with topic coherence evaluated across model settings.
- Model objective: lda2vec modifies the SGNS objective to use document-wide feature vectors while learning continuous document weights over topic vectors.The loss combines SGNS with a Dirichlet-likelihood term over document weights.
- Word-context training: Pivot-target pairs come from words cooccurring within a moving window, with five tokens before and after each pivot in the experiments.The same dense word vectors are used for pivot and target representations.
- Word-context training: Negative sampling uses u^β, with β = 3/4 and 15 negative samples by default, slightly emphasizing infrequent words relative to the unigram distribution.The model learns word vectors conditional on context rather than modeling overall token popularity through softmax.
- Document representations: The context vector sums a document vector and pivot word vector, joining both spaces while capturing document-wide and local inter-word relationships.Dropout is applied to both unnormalized vectors to prevent co-adaptation.
- Document mixtures: Document vectors are mixtures of shared topic vectors whose non-negative weights sum to one, so memberships can be interpreted as percentages.A softmax maps latent vectors onto the simplex, while topic and word vectors share dimensionality and space.
- Sparse memberships: A low-concentration Dirichlet likelihood encourages sparse document memberships; α = n^-1 and λ = 200 work well in the reported experiments.Without the sparsity term, probability mass spreads across topics and the topic basis becomes incoherent.
- Evaluation: Topic coherence is evaluated with Palmetto’s Cv measure, while token-to-word similarity is evaluated using 3COSMUL.Cv averages NPMI for submitted topic-word pairs within a sliding window on an external corpus.
3 Experiments
Experiments on Twenty Newsgroups and Hacker News show that lda2vec discovers interpretable topics while learning corpus-specific word similarities and linear semantic relationships.
- 3.1 Twenty Newsgroups: On Twenty Newsgroups, the best topic coherences were achieved with n = 20 topics and negative sampling power β = 0.75.The evaluation varied topic counts n ∈20, 30, 40, 50 and β ∈0.75, 1.0.
- 3.1 Twenty Newsgroups: The discovered Twenty Newsgroups topics aligned qualitatively with Space, Encryption, X Windows, and Middle East themes.Topic vectors were interpreted through their most similar words, including astronomical terms, encryption terms, and words associated with the named newsgroups.
- 3.2 Hacker News Comments corpus: The Hacker News corpus contained 75 million tokens, 66 thousand documents, and 110 thousand unique tokens after noun-phrase preprocessing.The preprocessing preserved phrases such as community policing measure and prominent figures such as Steve Jobs as single tokens.
- 3.2 Hacker News Comments corpus: Using 40 topics and 256 hidden units, lda2vec recovered Hacker News themes including housing, search engines, Bitcoin, compensation, and technological hardware.The example topics included housing supply and house prices, DDG and Bing, Bitcoin and Mt. Gox, equity and vesting, and devices such as Surface Pro and Mac Pro.
- 3.2 Hacker News Comments corpus: Hacker News word similarities captured corpus-specific relations, such as fonts near Comic Sans and computer-science terms near Functional Programming.Similarities also connected San Francisco with American cities and Bay Area locations.
- 3.2 Hacker News Comments corpus: Lda2vec learned linear token relationships covering locations, currencies, server-client roles, leadership, machine-learning fields, political figures, nationalities, companies, and hardware.Examples included Silicon Valley with California and technology, Bitcoin as a digital currency, and Kindle with Surface Pro as tablets.