PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Agnostic Neural Topic Modeling with Contextual Token-Level Semantic Graph Representation

Seung-Won Seo, Won Ik Cho, Yongmin Yoo

arXiv:2608.16269v1cs.CLcs.LG

TL;DR

PLM-powered topic models can lose topic quality on specialized domains because domain-specific terms occupy poorly differentiated embedding regions. DARTOPIC inserts a learnable token-level semantic graph between frozen PLM embeddings and topic inference, and across general, biomedical, and legal benchmarks it improves topic coherence and document clustering while remaining robust and efficient without encoder fine-tuning.

  • Problem

    PLM-powered topic models can suffer degraded topic coherence on specialized domains because domain-specific terms map to poorly differentiated embedding regions.

  • Method

    DARTOPIC constructs token-level semantic graphs from frozen PLM embeddings and jointly optimizes a GNN graph layer with variational topic inference.

  • Results

    DARTOPIC achieves the best NPMI and TQ across general, biomedical, and legal domains using frozen PLMs, with robust PLM performance, improved document clustering, and lower runtime than fine-tuning alternatives.

  • Takeaways & Limitations

    Corpus-learned token-level graph representations can improve cross-domain topic quality without domain-specific pre-training or encoder fine-tuning.

  • Takeaways & Limitations

    Evaluation is limited to English corpora and relies mainly on standard automatic topic-coherence metrics, leaving multilingual and human-evaluation validation for future work.

Abstract

from arXiv · show

Recent advances in neural topic models with pre-trained language models (PLMs) have achieved strong performance by leveraging general-domain pre-training, yet their topic interpretability often degrades on specialized corpora. This limitation primarily stems from the geometry of the embedding space, where domain-specific terms unseen during pre-training collapse into an indistinguishable region, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clus- tering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning based alternatives.

1 Introduction

PLM-powered topic models perform strongly but can produce incoherent topics in specialized domains because domain-specific terms collapse into poorly differentiated embedding regions. DARTOPIC addresses this limitation with a learnable token-level semantic graph layer that refines frozen PLM embeddings jointly with topic inference.

  • Motivation: PLM-based methods replace bag-of-words inputs with dense contextual document embeddings and outperform classical probabilistic and earlier VAE-based topic models.Traditional bag-of-words assumptions disregard word order and miss semantic similarity between lexically distinct but contextually related terms.
  • Problem: In biomedical and legal domains, rarely seen domain-specific terms collapse into narrow embedding regions, causing representation degeneration and incoherent topic inference.The resulting geometry makes semantically distinct concepts difficult to distinguish.
  • Limitations: Existing remedies are incomplete because domain-specific PLMs require large-scale pretraining, while word-level graph models use fixed, context-independent node features.These approaches therefore address only partial aspects of domain-shifted topic modeling.
  • Approach: DARTOPIC constructs a dynamically learned token-level semantic graph from frozen PLM embeddings and jointly trains a GNN encoder with topic inference.Token-level construction preserves corpus-specific contextual structure while avoiding encoder fine-tuning.
  • Contributions: DARTOPIC produces coherent biomedical topics without domain-specific pre-training or fine-tuning and is designed to bridge domain gaps across diverse PLMs and domains.The introduction highlights topics organized around oxidative stress, respiration, and carbon metabolism, alongside robust topic quality across various PLMs.

2 Related Works

Neural topic modeling progressed from VAE-based inference with bag-of-words and static embeddings to contextual PLM and graph-based representations. However, these approaches remain tied to fixed pretrained knowledge, motivating DARTOPIC’s corpus-adaptive token-level graph refinement.

  • Neural Topic Models: Early neural topic models used VAE-based amortized inference, while bag-of-words representations limited topic coherence and static embeddings partially mitigated this limitation.These approaches include the VAE frameworks of Miao et al. (2016) and Srivastava and Sutton (2017), plus GloVe-based semantic guidance.
  • Contextual PLM Models: Contextual PLM models introduced dense embeddings, clustering, optimal transport alignment, and mutual information maximization, achieving strong performance on general-domain benchmarks.CombinedTM, BERTopic, FASTopic, and NeuroMax exemplify these developments.
  • Domain Limitations: PLM-powered topic models degrade in specialized biomedical and legal domains because pretraining coverage diverges from target-domain distributions.Domain-specific PLMs can partially address this gap but require costly large-scale pretraining, while PVTM uses prefix tuning to mitigate domain shift.
  • Graph-Based Topic Models: Graph-based topic models enrich structural signals through corpus-level word co-occurrence graphs, while newer methods combine GNNs with pretrained or dependency-based word representations.Examples include GINopic, CGTM, and Meta-CETM, alongside earlier short-text graph models addressing sparse token co-occurrences.
  • DARTOPIC: DARTOPIC addresses fixed pretrained representations by initializing token-level semantic graphs from a frozen PLM and jointly refining them with topic inference on target data.This design learns corpus-adaptive representations without relying on domain-specific pre-training.

3 Proposed Methodology

DARTOPIC is a lightweight domain-agnostic neural topic model that represents documents as token-level semantic graphs built from PLM embeddings. A GCN encodes document-specific token relations, and a VAE-based topic module is trained jointly with it for unsupervised topic inference.

  • Token-Level Graph Representation: DARTOPIC represents each document as a token-level graph whose nodes are tokens and whose edges capture within-document token relationships.This design targets heterogeneous domains where training-corpus context and corpus-level statistics may yield inadequate word representations.
  • Token-Level Graph Representation: The framework constructs an undirected weighted graph from PLM token embeddings, connecting token pairs according to cosine similarity above a sparsity-controlling threshold.The PLM supplies pretrained semantic knowledge while the threshold τ controls graph sparsity.
  • Graph Encoder: A two-layer GCN propagates information across the token graph to produce node embeddings that combine token semantics with document-specific structural relations.Mean pooling over contextualized token nodes yields a fixed-length document representation for topic inference.
  • Overall Framework: DARTOPIC combines a token-graph encoder with a compact VAE-based topic inference module in a lightweight architecture.The model avoids jointly inputting dense contextual representations with sparse TF-IDF or BoW features.
  • Topic Inference: The VAE maps graph-derived document representations to Gaussian latent topic variables, samples them with reparameterization, and transforms them into document-topic distributions.The latent variable z is converted to θ through a softmax function.
  • Overall Training Objective: The GNN encoder and VAE topic model are trained jointly end-to-end using reconstruction likelihood, with KL regularization from a standard Gaussian prior.The reconstruction target is the observed document word-count vector, and the decoder generates a reconstructed word distribution.

4 Experiments

Experiments evaluate DARTOPIC across general, biomedical, and legal benchmarks using topic-quality and document-clustering metrics. Results indicate consistently strong topic-word performance across domains and strong clustering performance on 20NG and BioASQ, especially BioASQ.

  • Datasets: Experiments use 20NewsGroup, BioASQ, and BillSum, representing general, biomedical, and legal domains.Processed dataset statistics are reported in Appendix A, Table 7.
  • Evaluation: Evaluation combines NPMI, NPMI-In, TU, and Topic Quality with document-clustering Purity and normalized mutual information.NPMI-In targets coherence on domain-specific texts, while clustering evaluates document-topic distributions.
  • Topic-Word Distribution Quality: DARTOPIC achieves consistently strong topic-word distribution performance across all three datasets despite substantially different document lengths.Average document lengths are 48.02 for 20NG, 7.44 for BioASQ, and 76.28 for Bills.
  • Topic-Word Distribution Quality: DARTOPIC captures topic structure across short and long documents, while PVTM performs strongly through parameter-efficient fine-tuning and NeuroMax trades topic diversity for weaker topic quality.The passage contrasts PVTM’s domain adaptation with NeuroMax’s embedding clustering regularization.
  • Doc-Topic Distribution Quality: DARTOPIC performs strongly on document clustering for 20NG and BioASQ, with a clearer advantage on BioASQ.Bills is omitted because it lacks document labels; documents are assigned to their highest-probability topic.

5 Analysis

DARTOPIC maintains stable topic-modeling performance across PLMs of different scales and domain specificity, while often improving runtime efficiency over PVTM, especially on longer documents. These results support decoupling topic quality from PLM capacity through frozen-PLM token-level graph learning without encoder fine-tuning.

  • Robustness to PLM Choice: DARTOPIC achieves consistently strong, stable performance across lightweight, large, and domain-specific PLMs, with only limited variation.The evaluated encoders include MiniLM, RoBERTa-large, Qwen-0.6B, and BioBERT.
  • Robustness to PLM Choice: Existing PLM-based models show greater sensitivity to PLM choice: some favor MiniLM, whereas FASTopic and ZeroShotTM benefit substantially from BioBERT.CombinedTM, CGTM, and NeuroMax often outperform their BioBERT variants with MiniLM, while FASTopic and ZeroShotTM show the opposite pattern.
  • Efficiency without Fine-Tuning: DARTOPIC is faster than PVTM on 20NG and Bills while maintaining comparable efficiency on BioASQ.Both per-epoch training time and full-corpus inference time were compared on a single NVIDIA H100 GPU.
  • Decoupling Topic Quality from PLM Capacity: DARTOPIC achieves the best NPMI and TQ across all three domains using only a frozen PLM, indicating that corpus-learned token-level graphs can replace domain-specific pre-training.The analysis identifies this as decoupling topic quality from PLM capacity.

6 Conclusion

DARTOPIC decouples topic quality from PLM pre-training coverage by inserting a corpus-learned token-level semantic graph between frozen embeddings and variational topic inference. Across general, biomedical, and legal domains, it improves topic coherence and document clustering while remaining robust to PLM choice and faster than fine-tuning alternatives.

  • Conclusion: DARTOPIC interposes a corpus-learned token-level semantic graph between frozen PLM embeddings and variational topic inference.This decouples topic quality from PLM pre-training coverage.
  • Conclusion: Experiments across general, biomedical, and legal domains show consistent improvements in topic coherence and document clustering.The framework also demonstrates robustness to PLM choice.
  • Conclusion: A jointly optimized graph layer can reconstruct domain-specific geometric structure from target-corpus evidence alone when the frozen encoder lacks it.This result supports a broader principle for adapting frozen encoders without relying on their pre-training coverage.

Limitations · A Benchmark Datasets

The paper reports strong cross-domain effectiveness but identifies English-only evaluation as a limitation. It also describes benchmark datasets 20NewsGroup, BioASQ, and Bills, whose preprocessed statistics appear in Table 7.

  • Limitations: DARTOPIC consistently improves topic quality and document representations across heterogeneous domains.The framework is characterized as an efficient, small-scale PLM-powered design.
  • Limitations: The experiments are limited to English corpora.The limitation is noted despite gains across general, biomedical, and legal benchmarks.
  • Limitations: The framework relies on token-level semantic graph construction.
  • A Benchmark Datasets: The benchmark-dataset section covers 20NewsGroup.
  • A Benchmark Datasets: The benchmark-dataset section covers BioASQ.
  • A Benchmark Datasets: The benchmark-dataset section covers Bills.The passage introduces these datasets as benchmarks and continues the list in the following passage.
  • A Benchmark Datasets: Preprocessed dataset statistics are presented in Table 7.

B Implementation Details and Hyperparameter Settings · C Evaluation Metrics

The experiments use a unified PyTorch and GPU environment with shared PLM settings, while DARTOPIC adjusts its graph threshold by dataset. Evaluation combines topic coherence, diversity, overall quality, and document-clustering agreement metrics.

  • B Implementation Details and Hyperparameter Settings: All models were implemented in PyTorch 2.10.0 and trained on a single NVIDIA H100 GPU under the same environment for fair comparison.The unified setup also uses all-MiniLM-L6-v25 as the PLM for all methods and Adam for optimization.
  • B Implementation Details and Hyperparameter Settings: DARTOPIC sets the edge-weight threshold τ to 0.2 for BioASQ and 20NG, and 0.3 for Bills.
  • C Evaluation Metrics: NPMI evaluates topic coherence by averaging pairwise top-word co-occurrence scores across words and then across all topics.The metric is implemented using TopMost and is based on reference-corpus word probabilities.
  • C Evaluation Metrics: Topic uniqueness evaluates diversity from decoder topic-word rankings, with higher TU indicating fewer shared top words and lower redundancy.TU ranges from 1/K to 1 and is averaged over all topics.
  • C Evaluation Metrics: Overall topic quality TQ is calculated as the product of topic coherence and diversity values.
  • C Evaluation Metrics: Document-topic representations are evaluated through clustering using Purity and normalized mutual information against ground-truth document labels.Higher Purity reflects class-dominated clusters, while higher NMI indicates stronger consistency with true labels.

D Analysis of Graph Construction Methods

DARTOPIC compares token-level n-hop and semantic graph construction, finding that their relative effectiveness depends on document length. N-hop graphs suit short BioASQ documents, while semantic graphs better support longer 20NG documents.

  • Graph construction strategies: DARTOPIC evaluates token-level n-hop graphs and token-level semantic graphs as alternative graph construction strategies.The comparison is conducted within the proposed DARTOPIC framework.
  • Graph construction strategies: The n-hop graph performs better on BioASQ, whose documents are relatively short.This result indicates that local structural relations are useful for short texts.
  • Graph construction strategies: The semantic graph performs better on 20NG, whose documents are relatively long.This result suggests that broader semantic connections become more important for longer texts.

E Hyperparameter Analysis

DARTopic’s graph threshold τ is evaluated across three datasets, with the best topic quality generally occurring at τ values between 0.2 and 0.3. The selected values are τ = 0.2 for 20NG and BioASQ and τ = 0.3 for Bills, while the optimal range remains similar across document lengths.

  • E Hyperparameter Analysis: τ controls the semantic similarity threshold used to construct DARTopic’s token-level semantic graph.The analysis evaluates τ ∈ {0.1, 0.2, 0.3, 0.4} on 20NG, BioASQ, and Bills.
  • E Hyperparameter Analysis: The best topic quality is generally achieved when τ is set between 0.2 and 0.3.20NG reaches its highest TQ at τ = 0.2, while Bills reaches its best TQ at τ = 0.3; BioASQ is stable in this range.
  • E Hyperparameter Analysis: DARTopic uses τ = 0.2 for 20NG and BioASQ and τ = 0.3 for Bills in the main experiments.Both too small and too large τ values can be less effective.
  • E Hyperparameter Analysis: The optimal τ range remains similar across datasets despite differences in document length.This tendency differs from n-hop graph construction methods, whose performance can be highly affected by document length.
  • E Hyperparameter Analysis: Token-level semantic graph construction is less sensitive to document length and less dependent on dataset-specific hyperparameters.The conclusion is drawn from the similar optimal τ range across BioASQ, 20NG, and Bills.

F Qualitative Evaluation · G Effect of Contextual Embeddings · H Use of AI Assistants

DARTopic produces more interpretable, domain-specific biomedical topics than baselines by preserving contextual token semantics, while AI assistants were limited to writing and submission support. Its qualitative advantage reflects reduced mixing of general-purpose and specialized terms.

  • F Qualitative Evaluation: Baseline topics mix biomedical terms with general words such as “efficiency”, “food”, and “prediction”, reducing interpretability.These general words may be frequent in BioASQ but are less informative for explaining biomedical topics.
  • F Qualitative Evaluation: DARTopic generates coherent, domain-specific topic words including “replication”, “leukemic”, “rna”, “adenovirus”, and “messenger”.The selected topics contain biomedical-related keywords such as “acid”, “dna”, “rna”, and “cell”.
  • F Qualitative Evaluation: These specialized terms provide clearer semantic explanations and demonstrate improved interpretability on the BioASQ biomedical corpus.Table 10 reports representative topic word lists, with blue highlighting general or less domain-specific terms.
  • G Effect of Contextual Embeddings: Word2Vec uses context-independent embeddings whose edge weights reflect pre-trained global word similarity rather than document-level semantic relationships.This is the same word-level design adopted by GINopic.
  • G Effect of Contextual Embeddings: GINopic falls substantially behind DARTOPIC across all three domains, whereas SBERT contextual token embeddings vary with co-occurring words.The contextual embeddings allow the same token to receive different representations depending on its document context.
  • H Use of AI Assistants: AI assistants supported limited writing, language editing, and submission-material preparation, while the authors performed and checked all technical work.The authors retained responsibility for methodology, implementation, experimental results, and final verification.
Loading 2608.16269v1…