Source-linked AI summary
Comment-level Topic Drift Analysis in the Reddit Corpus
Steven Morse, Daniel Runfola, Trenton W. Ford
TL;DR
Existing topic-analysis pipelines rarely measure whether topics move meaningfully through embedding space over time. This paper traces comment-level topic trajectories across 12.7 billion Reddit comments and finds directional drift in contentious domains, while music and sports remain comparatively stable.
Problem
Most topic-analysis pipelines track keywords or list topics without measuring meaningful movement through embedding space over time.
Method
The framework embeds comments, clusters monthly, aligns related clusters into topic paths, and tests observed trajectories against a null model.
Results
Topic movement reflects measurable semantic evolution: political, social-identity, and religious topics drift directionally, whereas sports and music remain comparatively stable.
Takeaways & Limitations
Embedding-space topic trajectories provide quantitative evidence that public discourse reorganizes through changing associations and emerging conceptual alignments.
Takeaways & Limitations
The minimally filtered Reddit corpus may contain increasing automated or low-quality content, and Reddit’s distinctive discourse conventions limit generalization.
Abstract
from arXiv · showhide
We present a novel application of embedding-based dynamic topic modeling techniques to detect and quantify topic drift at the comment level in a massive corpus. By leveraging pretrained language models to generate contextualized semantic embeddings for short text, we analyzed 12.7 billion Reddit comments spanning 2006 to 2022. Using unsupervised methods on these embeddings, we identify dynamically evolving topic clusters over time. Our primary contribution is a methodology for analysis of semantic drift and discourse evolution in the embedding space itself. We also demonstrate modifications to existing methods that enable this analysis at scale, and we propose and demonstrate a null model comparison test to filter spurious dynamics. Key findings suggest that politically and socially contentious topics exhibit significant directional drift in embedding space, with inter-topic distances changing systematically over time beyond what the null model can explain, whereas domains such as music and sports remain comparatively stable.
1 Introduction
This work treats comment-level topical discourse change as trajectories in embedding space, using time-local clustering, cross-time alignment, and a null model to distinguish genuine drift from noise. It addresses limitations of existing topic-dynamics methods by enabling scalable geometric analysis of topic drift at sentence or comment level.
- Motivation: Time-resolved geometric modeling is necessary because contextualized topic semantics shift with events and norms.Without such a model, those changes cannot be measured or forecast.
- Approach: The study models topical discourse change as trajectories in embedding space and asks when observed movement exceeds stochastic variation.The framework embeds comments, clusters within monthly windows, aligns similar clusters across time, and tests topic paths against a null model.
- Contribution: The contribution is a scalable pipeline for studying drift at the sentence or comment level in an embedded space and at large scale.The approach performs clustering directly in the original embedding space within monthly windows before reducing dimensionality and aligning centroids.
- Motivation: Existing topic-dynamics methods emphasize keyword or coherence summaries rather than embedding-space geometry and lack a formal test for drift.Global clustering can mask topic drift, while local clustering and alignment methods do not provide a formal test.
- Contribution: The method introduces a likelihood-ratio test based on a random walk null model to filter genuine topic drift from noise.The framework provides quantitative measures of topic trajectories and displacement.
2 Data & Methodology
The study analyzes 12.7 billion Reddit comments from 2006–2022 in 204 monthly windows using fixed contextual embeddings, within-window clustering, and temporal topic alignment. It then measures topic trajectories in embedding space and tests whether observed drift is stable rather than an artifact of large-scale clustering.
- Data: Comments exclude top-level submissions, multimedia, and deleted-account authors, while retaining very short replies and truncating each comment to at most 128 tokens.Retaining short replies preserves conversational structure and matches the embedding model’s context window through truncation.
- Data: 12.7 billion comments spanning 204 monthly windows form the filtered Reddit corpus, with over 9 billion comments in the most recent five years.The recent five-year period contains more than 160× the first five years, which had fewer than 60 million comments.
- Embedding: Each comment is embedded with the fixed pretrained all-MiniLM-L6-v2 transformer into a d = 384-dimensional vector representation of contextual semantic similarity.The same transformer is used across the entire dataset, producing timestamped embedded mini-documents.
- Topic alignment: Topic centroids are aligned across adjacent periods by embedding-space proximity, enabling analysis of absolute and relative topic drift trajectories.Alignment is required because period-specific clusterings do not preserve cluster identities or invariant topic representations.
- Topic clustering: K-means clustering is applied independently within each calendar month, using k = 50 clusters to represent topics without imposing a global topic geometry.Independent clustering permits different topic clusters across windows, while k-means scales in the full embedding space.
- Validation: A bootstrapped permutation test checks that observed drift is stable, not caused by batched large-scale clustering, and that resampling recovers the same aligned groups with only small outliers.The secondary test is described in Appendix B.
3 Results
Embedding-based clustering yields coherent, semantically interpretable topic groups and reveals that discourse evolves through both within-topic semantic drift and systematic changes in inter-topic relationships. Politically oriented and other contentious domains show stronger directional movement, whereas music, sports, and related topics remain comparatively stable.
- Topic coherence and structure: Monthly embedding clusters recover coherent domains and substructure, including politics, computing, film, dating, travel, science, and political subtopics.The method also isolates short-response clusters such as “ha” and “link.”
- Topic coherence and structure: Embedding proximity reflects semantic relatedness, grouping arts topics and economic discussions while exposing subtler links among therapy, food, and loneliness.
- Within-topic drift: Many topic groups move through embedding space over time, with politically oriented topics showing pronounced movement relative to the stability of “news.”
- Within-topic drift: Volatile domains such as politics and conspiracy theory exhibit the largest total displacement, while food and music remain nearly stationary.The displacement measure is computed in the full embedding space with d = 384.
- Within-topic drift: Significant directional drift distinguishes domains such as government, science, and religion from more stable music, sports, and humor groups.The significant groups have corresponding p-value < 0.01, while the test separates trajectories that reject versus fail to reject zero-mean movement.
- Inter-topic dynamics: Semantic change also alters the relative configuration of topics, as racism converges toward several social discussions, streaming approaches music and technology, and political topics drift toward new contextual regions.These patterns indicate structural evolution in the organization of discourse, not only change within individual topics.
4 Discussion
The discussion presents embedding-space trajectories as a quantitative framework for measuring comment-level topic drift and discourse reorganization. It finds selective directional and relational change in contentious domains, while identifying corpus, clustering, and modeling limitations for future work.
- Methodological contribution: Embedding-based topic modeling extends prior sentence- and word-level approaches by tracing comment-level motion within topics and their relative semantic configuration over time.The framework uses contextualized embeddings and aligned unsupervised clusters to measure shifts directly in embedding space.
- Implications: Temporal geometry has practical implications because stationary-topic assumptions in retrieval, moderation, and classifier calibration can degrade as concepts acquire new associations.Examples include racism-related topics converging toward police, religion, and women, and streaming converging toward music, video, and phones.
- Empirical findings: Directional drift is selective: contentious and political, social-identity, and religious topics move directionally, whereas sports and music remain comparatively stable.The analysis characterizes contentious domains by high Λk and low pk, while stable domains show near-zero displacement.
- Empirical findings: Inter-topic distances change systematically, indicating that discourse evolves through semantic-network reconfiguration rather than independent topic shifts.The geometry also reveals concepts gravitating toward new associations and domains decoupling over time.
- Limitations and future work: Minimal filtering maximizes corpus coverage but admits automated and low-quality content whose increasing prevalence may add residual noise to inferred semantic dynamics.Applying the framework to scientific communication, news archives, and professional forums could test generalization across linguistic communities.
- Limitations and future work: Nondeterministic clustering can affect topic alignment and trajectory estimates, motivating null-model comparison and bootstrapping for statistical validation and robustness assessment.Future refinements include stronger embeddings, improved clustering and alignment, multilingual representations, and hybrid probabilistic models.
Appendix A Cluster selection
The appendix evaluates cluster-count selection using average within-cluster sum-of-squares across Reddit time periods and selects k = 50. Consistent inertia across periods supports using one cluster count throughout the analysis.
- Metric: Average within-cluster sum-of-squares (WCSS), also termed inertia, measures the average distance between each datapoint and its assigned centroid.The metric is evaluated for each time period t ∈ T.
- Across-time evaluation: 204 time periods from 2006-2022 were evaluated at k = 30, 50, 70, 90, with average inertia remaining remarkably consistent across periods.Consistency was especially strong for all time periods 2009 and on.
- Final choice: The pipeline therefore used k = 50 for all time periods rather than varying k separately by period.The choice was supported by the across-period consistency and the single-period elbow analysis.
- Cluster-count selection: A sharper decrease in inertia occurred for 1 ≤k ≤50, followed by diminishing returns after k ≈50, motivating the elbow-criterion selection of k = 50.The larger k grid was evaluated for a single time period.
Appendix B Cluster validation
Permutation testing shows that the pipeline recovers stable aligned cluster groups despite batching-induced stochasticity. Large-scale batching reduces, rather than eliminates, clustering variation, while the random-walk test filters the resulting noise.
- Deterministic validation: R = 20, k = 50, and G = 50 yielded identical aligned centroids under a single batch and fixed initialization, with zero groups having p-values below 0.05.This deterministic validation establishes the expected exact recovery when the full time window can be processed as one batch.
- Large-scale batching: Around 2015, time windows grow from hundreds of thousands to tens of millions of embeddings, making the deterministic single-batch approach prohibitive.The authors therefore adopt M = 1 million embeddings for large-scale permutation testing.
- Noise filtering and limitations: The authors conclude that batching-induced randomness is minimized by reasonable batch sizes and can be filtered using the random-walk likelihood-ratio test.They also note that Aligned-UMAP introduces additional potential noise and is prohibitive at the corpus scale.
Appendix C Random Walk & LR test
The appendix tests whether observed topic-centroid displacement reflects systematic drift by comparing ordered step vectors against a zero-mean random-walk null. It stabilizes the likelihood-ratio calculation with PCA and evaluates each topic group using permutation replicates.
- Topic-centroid steps are defined as displacement vectors ∆t = ct+1 − ct between consecutive time points.Each centroid ct lies in Rd, with T ≥ 2.
- The test evaluates H0: µ = 0 against H1: µ ≠ 0, where nonzero mean step vectors represent systematic drift.Steps are modeled as i.i.d. Gaussian vectors with unknown mean and positive-definite covariance.
- r = 45 PCA components explaining > 90% of variance are used to stabilize covariance estimation before computing the likelihood-ratio statistic.The statistic is computed in the projected space using the projected empirical mean and covariance.
- B = 10,000 shuffled time-order replicates form a permutation null for each topic group’s observed likelihood-ratio statistic.The shuffling is performed within each topic group using its projected steps.