Source-linked AI summary
Sentence-Level Topic Modeling with Lexical and Contextual Evidence
Federica Granese, Serena Villata, Charles Bouveyron
TL;DR
Topic models face a granularity gap: word-level assignments lack context, while document-level assignments do not localize topics within multi-topic documents. SentenceETM assigns topics to sentences by combining lexical and contextual evidence. It achieves the best ARI and AMI in almost all settings and reveals differences in debate topic emphasis and within-turn diversity.
Problem
Word-level models lack contextual evidence, while document-level models do not localize topics within documents; prior sentence-level models still use one evidence type for assignment.
Source: §1 · pp. 1–2Method
Results
SentenceETM achieves the best ARI and AMI in almost all settings and remains robust to a smaller sentence encoder, unlike SentenceLDA.
Source: §5 · p. 8Takeaways & Limitations
In a U.S. presidential debate, sentence-level topics reveal differences in candidates’ topic emphasis and within-turn topic diversity.
Source: §5 · p. 8Inference uses an amortized diagonal-covariance Gaussian approximation to the intractable posterior.
Source: §2.2 · p. 4
Abstract
from arXiv · showhide
Topic modeling faces a fundamental granularity dilemma: probabilistic topic models assign topics at the word level, whereas transformer-based methods operate at the document level. The latter overlooks that real-world documents typically cover multiple topics, whereas the former relies on individual words, which rarely provide sufficient context to identify them. We introduce SentenceETM, a neural topic model that assigns a topic to each sentence by jointly using lexical and contextual evidence. Each topic is described both by a distribution over the vocabulary, defined through word embeddings, and by a Gaussian over sentence embeddings concentrated on a topic-specific low-dimensional subspace. The topic of a sentence generates both its words and its embedding and is marginalized exactly, so that its posterior combines the two sources of evidence. Document proportions are inferred with amortized variational inference, while the Gaussian parameters are re-estimated in closed form. Experiments show that SentenceETM consistently improves sentence-level topic assignment over state-of-the-art approaches, and a case study of a presidential debate shows how sentence-level topics reveal how candidates handle the same subjects.
1 INTRODUCTION
Word-level models lack sentence context, while document-level models cannot localize topics within documents; prior sentence-level models still assign topics using only one evidence type. SentenceETM combines lexical and contextual evidence at sentence level, alongside a synthetic evaluation protocol and a debate case study.
1 INTRODUCTION
- Word-level models cannot use document context, whereas document-level models use lexical evidence only after topic assignment and cannot localize topics within documents.Prior sentence-level models also relied on a single evidence type for assignment.Source: §1 · pp. 1–2
- SentenceETM assigns each sentence a topic using both its words and contextual embedding, with topic-specific word distributions and low-dimensional Gaussian regions for sentence embeddings.The sentence topic is marginalized exactly, combining the two evidence sources in its posterior.Source: §1 · p. 2
- Synthetic corpora provide known sentence topics and document mixtures controlled by construction for evaluating sentence-level topic models.Sentence-level assignments also reveal how topics are distributed through a text.Source: §1 · p. 2
- An interchangeable sentence encoder adapts SentenceETM to different domains without modifying its generative model, as illustrated on a televised presidential debate.The sentence-level representation supports examining candidates’ topic distributions within debate segments and how often they introduce topics during turns.Source: §1 · p. 2
1.1 Related works
- Earlier sentence-level models assigned topics to sentences while retaining topic–word distributions, but contextual methods used lexical evidence only to describe topics.SentenceLDA and Schneider (2023) use contextual information for assignment; document-level contextualized models combine evidence but return one topic per document.Source: §1.1 · p. 2
2 THE PROPOSED APPROACH
SentenceETM assigns topics to sentences by combining each sentence’s words and embedding with document-level topic proportions. It models topic-word distributions and topic-specific low-dimensional Gaussian regions, then infers document proportions variationally and updates contextual parameters in closed form.
2 THE PROPOSED APPROACH
- SentenceETM combines document-level topic proportions with lexical and contextual sentence evidence to form a posterior topic assignment.A topic contributes when it explains both the sentence embedding and its words.Source: §2 · p. 2
2.1 Generative model
- Each topic has a vocabulary distribution defined through word embeddings and a Gaussian region concentrated on a topic-specific low-dimensional subspace.The subspace covariance avoids requiring a full E × E covariance matrix.Source: §2.1 · p. 3
Generative process.
- A sentence draws one topic from its document’s proportions, and that topic generates both its normalized embedding and its bag of words.Sentence topics are marginalized exactly; the only latent variable requiring posterior approximation is the document-level variable.Source: Generative process · p. 3
2.2 Inference model
- Document-level proportions are inferred with an amortized variational distribution, while contextual parameters are re-estimated in closed form from sentence-topic posteriors.The inference network conditions on document words; contextual evidence enters through the likelihood.Source: §2.2 · p. 4
3 EXPERIMENTAL SETTING
The evaluation creates synthetic documents with sentence-level topic labels and controlled document mixtures, then compares SentenceETM with word-, document-, and sentence-level alternatives. It also applies the model to a 2020 presidential debate organized into six discussion segments.
3 EXPERIMENTAL SETTING
- Synthetic evaluation corpora provide known sentence-level topic labels and document mixtures controlled by construction; labels are withheld during unsupervised training.Prior evaluations based on document representations did not directly assess how topics are distributed across sentences.Source: §3 · p. 4
- Each synthetic document has 20 sentences, with 12 from its dominant topic and the remaining eight split among other topics; the study uses BBC News, 20 Newsgroups, and Wikipedia.It evaluates K = 2, 3, 5, and 9 for 20 Newsgroups and Wikipedia, and K = 2, 3, and 5 for BBC News.Source: §3 · p. 5
- The debate case study uses the longest ElecDeb60to20 debate: the Biden–Trump debate of October 29, 2020, divided into six segments.The dataset comprises 44 televised U.S. presidential and vice-presidential debates from 1960–2020.Source: §3 · p. 5
- The comparison includes LDA and ETM at word level, BERTopic at document level, and senLDA and SentenceLDA at sentence level.Because LDA, ETM, and BERTopic do not directly output sentence topics, the evaluation derives sentence assignments from word majorities or document clusters.Source: §3 · p. 5
4 EXPERIMENTAL RESULTS
SentenceETM’s synthetic-corpus results favor combining lexical and contextual evidence, while the debate case study illustrates how sentence-level assignments distinguish topic emphasis and within-turn diversity across candidates and segments.
4 EXPERIMENTAL RESULTS
- A randomly selected synthetic BBC News document shows topic assignments changing with sentence content, though the model misassigns a sentence with no topic-specific information.The paper also presents the latent-space structure for a randomly chosen Wikipedia run; additional sensitivity and computational-cost analyses are in the appendices.Source: §4 · pp. 5–6, Figure 4 · p. 6, §4.1 · p. 6
4.1 Results on synthetic corpora
- For BBC News, ARI is about 0.62–0.66 and stable for c ≥ 0.03, with best results around c = 0.1 across tested subspace dimensions.The gain over lexical-only evidence is about +0.33 ARI; the gain over contextual-only evidence ranges from about +0.08 at h_k = 4 to about +0.01 at h_k = 64.Source: §4.1 · p. 6, Figure 3 · p. 6
- SentenceETM outperforms all baselines across all 31 synthetic class subsets in ARI and AMI, with median gains of about 0.03–0.2.Gains are largest at small K and decrease as K grows, as the strongest baseline, typically SentenceLDA, becomes more competitive.Source: Figure 5 · p. 7, §4.1 · p. 7
- Across almost all settings, SentenceETM achieves the best ARI and AMI, while document-level baselines remain near zero across the three synthetic datasets.The document-level methods become only marginally informative as K grows because the dominant topic stands out more clearly.Source: §4.1 · p. 6, Figure 6 · p. 7
4.2 Results on ElecDeb60to20
- In the debate, the largest topic-use differences within segments show distinct candidate emphases on race, the economy, election integrity, and COVID-19.For example, Biden’s share is higher for racism toward African Americans (−0.12), while Trump’s is higher for lockdowns on the economy (+0.14).Source: §4.2 · p. 7, Figure 7 · p. 8
- Across debate turns, the new-topic index is almost identical overall: 0.45 for Trump and 0.43 for Biden, or about two sentences in five.Differences are larger across segments than between speakers; COVID-19 is least cohesive for both (+0.15 Trump, +0.26 Biden).Source: §4.2 · p. 8, Figure 8 · p. 8
5 CONCLUSION
The conclusion presents SentenceETM and its synthetic evaluation protocol, reporting strong sentence-level assignment results and a debate application. It also identifies future extensions and discloses the role of generative AI in the work.
5 CONCLUSION
- SentenceETM combines topic–word distributions with topic-specific low-dimensional subspaces, and achieves the best ARI and AMI in almost all settings.The conclusion also reports robustness to a smaller sentence encoder, unlike SentenceLDA.Source: §5 · p. 8
- Future work includes modeling sentence order and topic transitions, and complementing synthetic labels with human annotations..Source: §5 · p. 8
AI use statement
- Generative AI assisted with implementation, notebooks, integration of existing methods, results interpretation, and debate-topic labels; authors reviewed assisted material and take responsibility for the final content.The authors state that AI did not generate synthetic data, formulate hypotheses or novel mathematical claims, or clean datasets.Source: AI use statement · pp. 8–9
A PROOFS
This appendix supports Section 2 with proofs and summarizes the generative and inference procedures. It also derives the contextual likelihood used by the model.
A PROOFS
- The appendix provides proofs supporting Section 2 and algorithms for the generative and inference procedures.Source: A PROOFS · p. 11
A.1 ELBO decomposition
- The ELBO decomposition is established by taking expectations under q and regrouping terms, including the Kullback–Leibler divergence.Source: §A.1 · p. 11
A.2 Derivation of Equation (4)
- Equation (4) follows by substituting into the objective and applying expectation linearity and the definition of the Kullback–Leibler divergence.Source: §A.2 · p. 11
A.3 Computing the contextual likelihood
- The contextual likelihood separates embedding deviation within a topic subspace from distance away from that subspace, with cost O(Eh_k) per sentence and topic and no E × E matrix.The within-subspace and off-subspace distances are weighted by the topic-direction variances and noise variance, respectively; variance parameters are lower-bounded to prevent degenerate topics.Source: §A.3 · p. 12
A.4 Generative process
- The generative procedure defines topic-word distributions and topic Gaussians, then draws document proportions and a topic, embedding, and words for each sentence.Source: §A.4 · p. 12
A.5 Inference process
- Inference initializes contextual parameters from k-means assignments, optimizes the variational and lexical parameters by gradient steps, and re-estimates contextual parameters in closed form.Sentence topics are returned using the maximum-responsibility assignment.Source: §A.5 · p. 13
B.1 Synthetic dataset construction pipeline
The appendix describes how synthetic sentence-labeled corpora are assembled from existing topic-labeled collections and how sentences are represented for modeling. It also specifies the debate corpus and its segment topics.
B.1 Synthetic dataset construction pipeline
- Synthetic corpora draw source documents from BBC News, 20 Newsgroups, and Wikipedia, selecting K topic labels for each run.When K equals the maximum available topics, repetitions vary the corpus seed rather than the selected classes.Source: §B.1 · pp. 13–14, Table 1 · p. 14, Table 2 · p. 15
- Each synthetic document combines sentences from one dominant topic and K − 1 non-dominant topics into K contiguous topic blocks.The dominant topic is sampled uniformly; sentences retain their source order and are drawn without replacement within an article.Source: §B.1 · p. 14
B.1.1 Preprocessing
- Preprocessing represents each document as a bag of words and sentence sequence, pairing sentence-level word counts with embeddings from frozen encoders.The lexical vocabulary retains 15,000 frequent words after filtering; embeddings are computed once and cached.Source: §B.1.1 · pp. 14, 16
B.2 ElecDeb60to20
- The case is the first 2020 presidential debate between President Trump and former Vice President Biden, organized into six roughly 15-minute segments, each beginning with a two-minute candidate answer before open discussion.Source: §B.2 · p. 16
- The six debate segments cover the Supreme Court and healthcare, COVID-19, the economy, race, climate change, and election integrity.Source: §B.2 · p. 16
B.3 Model training
The appendix reports SentenceETM’s training setup, its sentence-topic evaluation metrics, and computational costs. Its training time is lower than SentenceLDA’s but higher than document-level models and senLDA.
B.3 Model training
- SentenceETM uses cached embeddings from a frozen encoder and trains for 300 epochs with Adam, while topic Gaussians are re-estimated in closed form after each epoch.The setup uses batches of 64 documents, topic subspace dimension h_k = 32, and contextual-evidence weight c = 0.5.Source: §B.3 · p. 16
B.4 Metrics
- Sentence-topic assignments are evaluated against known sentence labels using Adjusted Mutual Information and Adjusted Rand Index.Because every sentence has a known topic, these metrics evaluate the inferred quantity directly rather than using topic-quality proxies.Source: §B.4 · p. 17
B.5 Computational cost
- The closed-form Gaussian update requires O(SKE^2) operations to accumulate covariances and O(KE^3) for eigendecompositions.The update is independent of vocabulary size and grows linearly with the number of sentences and topics.Source: §B.5 · p. 17
- SentenceETM trains in 340 to 391 seconds per run on average over K, compared with 1717 to 3118 seconds for SentenceLDA.It is slower than document-level models and senLDA, and its cost grows linearly with K.Source: §B.5 · p. 17
- After training, assigning a topic to a new sentence requires one inference-network pass and evaluation of K Gaussian densities.Source: §B.5 · p. 17
C.1 Numerical results
SentenceETM outperforms SenClu across the reported synthetic settings, although SenClu’s results vary substantially across runs. Visualizations show that SentenceETM’s assignments form more distinct embedding-space regions than those of BERTopic and ETM, with some topic-quality weaknesses remaining.
C.1 Numerical results
- SentenceETM outperforms SenClu in every reported setting, while SenClu’s standard deviation reaches 0.32 ARI on Wikipedia.The authors attribute SenClu’s run-to-run variation to random initialization of its topic centroids.Source: §C.1 · p. 17
C.2 Structure of the latent space
- SentenceETM’s assigned sentences largely form distinct embedding-space regions, and topic top words identify corresponding classes; BERTopic and ETM assignments are more mixed.The visualizations project sentence embeddings onto their first two principal components and list each topic’s three most probable words.Source: §C.2 · p. 17, Figure 10 · p. 18
- Some SentenceETM topics remain weak: the 20 Newsgroups Atheism topic is dominated by conversational words, while the X window topic has uninformative words and few sentences.The authors note that these observations are consistent with the dataset’s lower scores.Source: §C.2 · p. 18
C.3 Sensitivity to the sentence encoder
SentenceETM is relatively robust to replacing all-mpnet-base-v2 with all-MiniLM-L6-v2, whereas SentenceLDA degrades sharply. The authors suggest their different topic representations may explain the contrast.
C.3 Sensitivity to the sentence encoder
- SentenceETM’s ARI and AMI drops are at most 0.05 except on BBC News at K = 5, where they are 0.19 ARI and 0.15 AMI.At that exception, SentenceETM’s ARI falls slightly below senLDA’s.Source: §C.3 · p. 18
- SentenceLDA’s ARI falls to at most 0.16 in every setting, and its AMI decreases by up to 0.39.On BBC News at K = 2, its ARI drops from 0.52 to 0.02; SentenceLDA’s ARI is below senLDA’s in every smaller-encoder setting.Source: §C.3 · p. 19
- With the same smaller embeddings, SenClu loses at most 0.06 ARI and improves in most settings, while encoder-independent senLDA, ETM, and LDA results remain unchanged.The authors propose that SentenceETM’s covariance constraint to a few principal directions plus isotropic noise may make it less sensitive to directions carrying little topical information.Source: §C.3 · p. 19
C.4 Sensitivity to document length
Across these analyses, SentenceETM remains strongest across tested document lengths, while learned topics are coherent and diverse and sentence-level analysis reveals both debate themes and interpretable label disagreements.
C.4 Sensitivity to document length
- ARI ranges from 0.46 to 0.49 and AMI from 0.54 to 0.56: SentenceETM leads at every tested document length, with differences within one standard deviation.The Wikipedia experiment varies sentences per document from 20 to 80 while keeping each corpus near 14,000 sentences.Source: §C.4 · pp. 19, 21, Figure 11 · p. 19
C.5 Results on original corpora
- On original corpora, SentenceETM combines consistently high topic coherence with high diversity across BBC News, 20 Newsgroups, and Wikipedia; diversity is near one and less discriminating.The learned topics come from synthetic-corpus runs; top-word coherence is evaluated against the corresponding original corpora without retraining.Source: §C.5 · p. 21, Figure 12 · p. 21
C.6 Examples of misassigned sentences
- Misassignments cluster around low-information sentences, sentences whose content differs from inherited document labels, and overlapping categories.The examples include cases where predicted topics better match sentence content than the source-document label, highlighting noise from propagating those labels to sentences.Source: §C.6 · p. 22, Table 8 · p. 23
C.7 Top-words in ElecDeb60to20
- The debate model separates specific themes within moderator segments and also groups interjections, accusations, and achievement claims as register-focused topics; two topics lack clear interpretation.For example, it distinguishes preventive measures from lockdowns within the COVID-19 segment.Source: §C.7 · p. 22, Table 9 · p. 24
20 Newsgroups
- A 20 Newsgroups example assigns an alt.atheism post about Soviet and French tanks to talk.politics.mideast, illustrating a mismatch between sentence content and its source label.Source: §C.6 · p. 22, §20 · p. 23