PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Topic Modeling for Cross-Corpus Temporal Analysis

Ruoxuan Li, Bruce Kogut

arXiv:2608.23284v1cs.CL

TL;DR

Cross-corpus comparison is difficult because independently learned dynamic topics require post-hoc alignment and may not preserve stable topic correspondence. The paper learns a shared dynamic backbone over merged corpora, then applies corpus-specific residual adaptation around it. Across three corpora spanning 97 years, this approach improves corpus-specific fit while preserving same-index alignment, reaching 97.5 ± 0.7% trajectory Retrieval@1 versus 17.9 ± 1.1% for full fine-tuning.

  • Problem

    Independent topic models do not guarantee stable topic correspondence across corpora and time, complicating reliable cross-corpus comparison.

  • Method

    The framework learns a common dynamic topic backbone over merged corpora and applies source-specific residual adaptation around the shared backbone.

  • Results

    97.5 ± 0.7% trajectory Retrieval@1 versus 17.9 ± 1.1% for full fine-tuning, while residual adaptation improves corpus-specific fit and preserves same-index alignment.

  • Takeaways & Limitations

    Shared topic coordinates support more stable over-time cross-corpus comparisons while retaining corpus-specific lexical variation.

  • Takeaways & Limitations

    Runtime comparisons are restricted to two hardware-consistent A100 runs, leaving broader topic-count and hardware scaling for future work.

Abstract

from arXiv · show

Dynamic Embedded Topic Models (D-ETM) provide an interpretable framework for modeling temporal semantic evolution, but cross-corpus comparison remains difficult because topics are often learned independently and aligned only after training, a process that does not guarantee stable topic correspondence across corpora and time. To address this problem, we propose a D-ETM framework that first learns a common dynamic topic space over a merged multi-corpus collection, which we call the shared backbone, then introduces corpus-specific residual adaptation around the frozen backbone without creating separate latent topic spaces. This design preserves a shared topic index for cross-corpus comparison while allowing each corpus to specialize lexically. We evaluate the framework on three temporally structured corpora spanning 97 years: the Corpus of Historical American English, Harvard Business Review, and International Labour Review. Residual adaptation improves corpus-specific fit relative to the shared backbone while preserving the same-index cross-corpus topic trajectories, achieving substantially stronger alignment than full fine-tuning from the same backbone, with $97.5 \pm 0.7\%$ versus $17.9 \pm 1.1\%$ trajectory Retrieval@1, as well as stronger alignment than independent training with post-hoc Hungarian matching. These results suggest that incorporating topic alignment into the model can support more stable over-time cross-corpus comparisons while retaining corpus-specific lexical variation.

1 Introduction

D-ETM supports interpretable temporal semantic analysis, but independently learned topics require difficult post-hoc alignment for cross-corpus comparison. The shared-backbone framework learns common topic coordinates first, then enables corpus-specific adaptation while preserving correspondence.

  • D-ETM models time-specific topic-word distributions with embeddings and smooth topic trajectories, supporting interpretable analysis of long-term semantic evolution.
  • Independent corpus models do not guarantee corresponding topic indices, making explicit post-hoc alignment necessary for cross-corpus comparison.
  • The proposed framework learns a common semantic coordinate system over a merged multi-corpus collection before introducing source-specific residual adaptation.
  • Residual adaptation lets corpora specialize lexically while preserving same-index topic correspondence across corpora.

2 Related Work

Dynamic topic modeling represents evolving themes, while cross-corpus alignment is commonly handled after independent training. The paper motivates evaluating topic identity directly across corpora and time, while excluding approaches not designed for explicit temporal trajectories and source-specific drift.

  • 2.1 Dynamic Topic Modeling: Dynamic topic models extend static topic modeling by allowing topic-word distributions to evolve across ordered time slices.
  • 2.2 Cross-Corpus Topic Alignment: A common cross-corpus strategy trains models independently, aligns their topics post hoc, and compares the resulting temporal trajectories.
  • 2.2 Cross-Corpus Topic Alignment: Contextualized topic models and BERTopic are outside this work’s scope because standard formulations lack explicit temporal trajectories or source-specific drift parameters.
  • 2.3 Topic Model Evaluation: Predictive fit and semantic coherence assess model performance, but they do not directly test whether topic indices provide comparable semantic coordinates across corpora.
  • 2.3 Topic Model Evaluation: The framework therefore introduces trajectory-level alignment metrics to evaluate whether topic identities remain comparable in the cross-corpus dynamic setting.

3 Preliminaries

The paper defines cross-corpus dynamic topic modeling as tracking themes over shared time while preserving topic identity across sources. Its D-ETM preliminaries represent time-specific topics through embeddings and encourage smooth trajectories.

  • 3.1 Problem Definition: The task models topic evolution across corpora that differ in domain and style but share a temporal span.
  • 3.1 Problem Definition: Topic identity should persist across time and source while allowing each corpus to express the same topic through source-specific vocabulary.
  • 3.2 Dynamic Embedded Topic Model: D-ETM represents topics in a continuous embedding space and learns time-specific topic embeddings for each topic and time slice.
  • 3.2 Dynamic Embedded Topic Model: A random-walk prior encourages each topic embedding trajectory to evolve smoothly over ordered time slices.
  • 3.2 Dynamic Embedded Topic Model: Combining a topic embedding with shared word embeddings produces a time-specific probability distribution over the vocabulary.

4 Methodology

The framework learns a shared dynamic topic backbone across merged corpora, then adapts each corpus through residual offsets while preserving same-index topic correspondence.

  • Shared Dynamic Topic Backbone: Stage 1 trains one D-ETM on the merged multi-corpus collection to learn a shared dynamic topic trajectory.The backbone provides a common semantic coordinate system before corpus-specific adaptation.
  • Shared Dynamic Topic Backbone: Source-aware weighting gives each corpus equal total weight in the Stage 1 objective.This weighting is intended to prevent larger corpora from dominating backbone training.
  • Source-Specific Adaptation: Stage 2 models each corpus as a residual perturbation around the shared backbone, allowing lexical specialization without separate topic spaces.The source-adapted topic embedding combines the frozen backbone with source-specific residual offsets.
  • Source-Specific Adaptation: Anchor and temporal-smoothness penalties constrain residual magnitude and discourage abrupt changes across adjacent time slices.The anchor penalty preserves topic identity, while smoothness encourages specialization to evolve gradually.
  • Source-Specific Adaptation: The shared word embeddings and backbone topic embeddings remain frozen while residual offsets and inference networks are trained.Residuals are initialized at zero, so each source begins from the shared backbone.

5 Experiments

Experiments compare three temporally aligned corpora from distinct linguistic domains over 20 five-year bins spanning 1922–2019.

  • Datasets: The evaluation uses Harvard Business Review, International Labour Review, and Corpus of Historical American English.HBR covers business discourse, ILR labor and social policy, and COHA broader historical American English.
  • Temporal Coverage: The corpora share the period 1922–2019 and are grouped into 20 non-overlapping five-year time slices.This creates a common temporal structure for cross-corpus comparison.
  • Preprocessing: COHA is restricted to Magazine and News genres and downsampled at ρCOHA = 0.3 to bring its effective size closer to HBR and ILR.The sampling procedure preserves COHA’s temporal distribution while reducing its size imbalance.

5.2 Setup

The setup compares five training conditions using fixed topic and temporal granularities, with post-hoc trajectory matching for independently trained models.

  • Training Conditions: Five training conditions separate vocabulary choice, joint training, residual adaptation, and full fine-tuning.The conditions are summarized in Table 2.
  • Common Configuration: All models use K = 20 topics and non-overlapping five-year bins as a common operating point.The fixed configuration supports comparison across training regimes.
  • Residual Adaptation: SB-RA freezes shared embeddings and backbone topics while training source-specific residuals and inference networks for 20 adaptation epochs.SB-RA is initialized from the SB-Joint checkpoint.
  • Post-Hoc Alignment: Independent models use post-hoc matching over full topic trajectories rather than matching separately within each time slice.Trajectory costs are computed with Jensen–Shannon divergence and matched one-to-one using the Hungarian algorithm.

5.3 Evaluation Metrics

Evaluation measures predictive fit, topic quality, and alignment of complete cross-corpus topic trajectories using complementary divergence- and index-based metrics.

  • Evaluation Dimensions: The evaluation covers predictive fit, full-reference topic quality, and cross-corpus trajectory alignment.Predictive fit and topic quality follow standard topic-modeling practice, while alignment metrics target the dynamic cross-corpus setting.
  • Predictive Fit and Topic Quality: Perplexity, topic diversity, topic coherence, and topic quality assess predictive performance and topic interpretability.Topic coherence uses pairwise NPMI over high-probability topic words, with −1 for zero document co-occurrence.
  • Trajectory Representation: Each trajectory is represented as a probability distribution over word–time pairs with equal weight for every time bin.This representation spans the T × V word–time space.
  • Trajectory Distances: The trajectory-level JSD matrix measures distances between full trajectories for topic pairs across two sources.Its entries compare topic k in one source with topic ℓ in another.
  • Alignment Metrics: Same-index JSD, trajectory margin, Retrieval@1, and Hungarian-matched JSD capture complementary alignment properties rather than independent evidence.Retrieval@1 tests whether the nearest trajectory retains the same index, while Hungarian-matched JSD evaluates the best post-hoc one-to-one cost.

5.4 Experiment Analysis

Residual adaptation improves source-specific fit while preserving cross-corpus topic alignment, outperforming full fine-tuning and independent post-hoc matching on trajectory correspondence. Alignment remains robust across topic counts, though increasing K raises training time and GPU memory.

  • Predictive fit: SB-RA improves over SB-Joint across all three sources and improves over independent baselines on HBR and ILR, but incurs a predictive-fit cost on COHA.SB-FT achieves the lowest perplexity on all three sources, while SB-RA remains broadly comparable to independently trained models.
  • Topic quality: SB-Joint has the highest merged-reference NPMI coherence, while SB-RA improves NPMI-based TC and TQ over independent baselines on COHA and HBR.SB-RA yields negative NPMI-based coherence on ILR, although its C_V coherence remains within the baseline range.
  • Cross-corpus trajectory alignment and ablations: 97.5 ± 0.7% Retrieval@1 shows SB-RA preserves inherited topic correspondence, with same-index and Hungarian trajectory JSD both 0.169 ± 0.001.The positive trajectory margin is +0.166 ± 0.002, indicating inherited indices already realize the optimal one-to-one matching.
  • Cross-corpus trajectory alignment and ablations: 17.9 ± 1.1% Retrieval@1 for SB-FT is substantially below SB-RA, despite SB-FT achieving stronger predictive fit and topic quality.Removing both SB-RA regularizers yields 33.8 ± 4.3% Retrieval@1, still above SB-FT.
  • K-sensitivity: SB-RA's alignment advantage over SB-FT persists at K∈{10, 20, 30}, with lower same-index trajectory JSD, positive margins, and higher Retrieval@1.SB-FT yields negative margins throughout these topic-count settings.
  • Runtime analysis: Increasing the topic count from K=10 to K=30 increases SB-Joint and SB-RA training time and peak allocated GPU memory.The runtime analysis uses sensitivity runs executed on NVIDIA A100-SXM4-80GB GPUs.

5.5 Case Study

The case study examines whether the shared backbone yields interpretable historical trajectories and how residual adaptation expresses them through corpus-specific vocabulary. It also evaluates alignment robustness, vocabulary overlap, and computational cost.

  • Case-study design: SB-RA preserves same-index topic trajectories while allowing source-specific adaptation, supporting qualitative inspection of temporal semantic change.The shared backbone provides a common coordinate system, while residuals permit corpus-local lexical specialization.
  • Vocabulary differentiation: Top-30 overlap measures how closely each source-specific topic remains to the shared backbone and counts additional corpus-local vocabulary.The overlap is computed for each source, topic, and time bin.
  • Vocabulary differentiation: COHA has the highest average vocabulary overlap with the shared backbone, consistent with its broad general-domain coverage.The passage also notes that merged-corpus training may favor the corpus with broader lexical coverage.
  • Historical interpretability: The shared backbone yields a macroeconomic Topic 0 trajectory spanning industrial crisis, corporate and welfare expansion, workplace reorganization, and digital transformation.Topic 15 and Topic 16 provide complementary managerial and labor-protection perspectives on this broader economic transformation.
  • Corpus-local expression: Topic 0 is realized differently across corpora: HBR emphasizes business and markets, while ILR emphasizes wages and labor.The analysis focuses on Topic 0 because its shared historical axis is broadly interpretable across all three corpora.
  • Corpus-local expression: Figure 2 plots source-specific probabilities for depression, internet, consumer, wage, and labor across five-year bins.These words illustrate both diachronic change and corpus-specific lexical realization.

6 Conclusion and Future Work

The paper introduces a shared-backbone D-ETM that learns common dynamic topics and adapts each corpus with residuals. Across three 97-year corpora, this improves corpus-specific fit while preserving alignment more effectively than full fine-tuning.

  • Conclusion: The framework learns a common dynamic topic backbone over a merged multi-corpus collection, then applies source-specific residual adaptation.This keeps a shared topic coordinate system while allowing corpus-specific specialization.
  • Conclusion: 97.5 ± 0.7% trajectory Retrieval@1 for SB-RA exceeds 17.9 ± 1.1% for full fine-tuning from the same backbone.The comparison concerns same-index trajectory alignment.
  • Conclusion: Across three temporally structured corpora spanning 97 years, residual adaptation improves corpus-specific fit while largely preserving the shared topic coordinate system.The ablation results indicate that both residual regularizers contribute to preserving same-index trajectories.
  • Future work: The qualitative analysis finds historically meaningful backbone trajectories alongside corpus-specific lexical differentiation.Future work includes human assessments of topic interpretability and evaluation on additional corpora.

GenAI Usage Disclosure

The authors disclose using GenAI tools for language editing, formatting, experimental planning, and code development and debugging.

  • Disclosure: GenAI tools assisted with manuscript editing, formatting, experimental planning, and code development and debugging.The authors state that all experimental outputs, analyses, citations, and reported results were verified by them.
Loading 2608.23284v1…