PaperlayerView exact revision on arXiv

Source-linked AI summary

TopicMerge: A Divide and Conquer Approach to Topic Modeling

Matthew Raffel, Lizhong Chen, Sai Munikoti, Sameera Horawalavithana

arXiv:2610.19323v1cs.CLcs.LG

TL;DR

LLM topic models offer strong clustering performance but can be costly and lack a global view, while embedding methods are more scalable. TopicMerge combines efficient clustering with contextual LLM-guided merging and reports competitive or state-of-the-art benchmark performance with over 92% fewer prompts than document-level models.

  • Problem

    LLM topic models face high computational costs and poor scalability, and their local processing lacks a global view of corpus themes.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    TopicMerge oversegments documents into fine-grained clusters, labels them, then uses context-aware LLM arbitration and relabeling to merge clusters hierarchically.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    Over 92% fewer LLM prompts than document-level models, with state-of-the-art or highly competitive performance across P1, ARI, and NMI on Bills and Wiki.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    TopicMerge makes LLM-enhanced topic modeling practical for real-world datasets by combining high-quality topics with reduced prompt requirements.

    Source: §5 · p. 9

    Performance depends on initial embedding quality, which can be less discriminative for very short texts; the closed-source LLM also raises contamination and reproducibility concerns.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advancements in topic modeling have split into two main paths: efficient embedding-based methods that offer scalability but are often outperformed on standard clustering metrics, and Large Language Model (LLM) approaches that achieve state-of-the-art clustering performance but at the cost of prohibitive computational expense and poor scalability. This paper introduces TopicMerge, a novel multi-stage pipeline that bridges these approaches to achieve both state-of-the-art clustering performance and computational efficiency. TopicMerge first employs an efficient clustering pipeline to deliberately over-segment a document corpus into a large set of fine-grained, semantically pure micro-clusters, which are then given initial semantic labels. It then introduces a novel, iterative merging algorithm where an LLM, guided by local neighborhood context, arbitrates on consolidating candidate pairs. These approved merges are then selectively executed and semantically relabeled by the LLM, hierarchically building broader, more coherent topics. Extensive experiments on the Bills and Wiki datasets show that TopicMerge achieves state-of-the-art clustering performance while reducing required LLM calls by over 92% compared to document-level methods, offering a practical and scalable solution.

1 Introduction

TopicMerge addresses the cost and limited corpus-wide perspective of LLM topic models by combining efficient clustering with cluster-level LLM reasoning. It reports state-of-the-art clustering performance while reducing LLM prompts by over 92% compared with document-level approaches.

1 Introduction

  • LLM topic models face high computational costs and poor scalability, while their local processing lacks a global view of corpus themes.The paper frames the challenge as leveraging LLM reasoning without inheriting these limitations.Source: §1 · p. 1
  • TopicMerge combines global structural awareness from clustering with local semantic reasoning from an LLM in a three-stage hybrid pipeline.It deliberately oversegments documents into fine-grained micro-clusters, then labels and hierarchically merges them.Source: §1 · pp. 1–2
  • Its merging algorithm uses the LLM to arbitrate candidate cluster pairs and summarize approved merges at the cluster level.This avoids per-document analysis while supporting nuanced semantic decisions.Source: §1 · p. 2
  • The experiments evaluate clustering quality with P1, ARI, and NMI, prompt counts, and ablations of contextual clusters and fine-grained initial clusters.Baselines span LDA, BERTopic, TopicGPT, and LiSA.Source: §1 · p. 2
  • Over 92% fewer LLM prompts than document-level LLM approaches, while achieving state-of-the-art clustering performance on widely used benchmark datasets.The evaluation uses Bills and Wiki and compares against traditional, embedding-based, and LLM-based baselines.Source: §1 · p. 2

2 Related Works

The related work traces topic modeling from bag-of-words methods to embedding clustering and LLM-based approaches. It positions TopicMerge as combining clustering’s global awareness with targeted LLM reasoning to address the scope-efficiency tradeoff.

2.1 Traditional Topic Modeling

  • LDA and NMF rely on bag-of-words representations, which limit their ability to capture word order and semantic context and can weaken topic coherence.LDA represents documents as mixtures of latent topics; NMF decomposes a document-term matrix.Source: §2.1 · p. 2

2.2 Embedding-based Topic Modeling

  • Embedding-based methods cluster contextualized document vectors, but geometric clustering can produce generic topics despite coherent groups.Examples include Top2Vec and BERTopic; FASTopic targets speed and stability.Source: §2.2 · pp. 2–3

2.3 Topic Modeling with Large Language Models

  • LLM approaches use models for topic discovery or as components in clustering and refinement pipelines, with prior work reporting state-of-the-art results.Examples include TopicGPT, PromptTopic, LISA, k-LLMmeans, EDTM, and ClusterLLM.Source: §2.3 · p. 3
  • Existing LLM-centric methods trade scope for efficiency: local judgments lack corpus-wide context, while broader or more complex querying can be prohibitively costly.TopicMerge is positioned as combining clustering’s global awareness with targeted local LLM reasoning.Source: §2.3 · p. 3

3 Methods

TopicMerge first forms fine-grained, semantically pure clusters, labels them, and then iteratively consolidates them into broader topics with LLM guidance. The merging process narrows candidate comparisons, uses contextual arbitration and purity-based selection, and relabels each merged cluster.

3 Methods

  • TopicMerge combines traditional clustering with LLM semantic judgments, using fine-grained, pure micro-clusters to make merging inputs clearer.The pipeline is designed to reduce LLM calls by operating at the cluster level.Source: §3 · p. 3

3.1 Initial Clustering Stage

  • Initial clustering deliberately produces many pure clusters so later labeling and merging are less influenced by noisy clusters.The method embeds documents, reduces dimensionality, then clusters; its experiments use agglomerative clustering and set the cluster count above a silhouette-based lower bound.Source: §3.1 · pp. 3–4

3.2 Initial Labeling Stage

  • Initial labeling uses diverse FPS-sampled documents to prompt an LLM for a concise cluster label and a detailed description.Example topic labels help constrain breadth and avoid labels that are too generic or too narrow.Source: §3.2 · p. 4

3.3 LLM-Guided Cluster Merging Stage

  • The merging stage iteratively converts fine-grained clusters into broader, semantically coherent topics, with the LLM acting as arbiter and label synthesizer.Each iteration has four phases: candidate generation, contextual arbitration, purity-based selection, and merge execution with relabeling.Source: §3.3 · p. 5

3.3.1 Efficient Candidate Generation

  • An augmented reciprocal-nearest-neighbor graph generates sparse candidate pairs, adding minimum-cost bridge edges to connect graph components.This reduces the search space while covering all components.Source: §3.3.1 · p. 5

3.3.2 Context-Aware LLM Arbitration

  • The LLM judges candidate pairs using nearby and distant cluster context plus example topics, with two-fold agreement and cached decisions.Prompts use cluster labels and descriptions rather than raw text.Source: §3.3.2 · p. 5

3.3.3 Purity Ranking and Set Selection

  • Approved pairs are ranked by a local silhouette-based purity score, then a greedy maximal independent set prevents any cluster from merging twice in one iteration.The selection avoids cascading merges within a pass.Source: §3.3.3 · pp. 5–6

3.3.4 Merge Execution and Relabeling

  • Selected merges are executed with Union-Find, and each new super-cluster is relabeled using source labels, descriptions, sampled data, and example topics.The cycle repeats with the smaller cluster set until an iteration approves no new merges.Source: §3.3.4 · p. 6

4 Experiments

TopicMerge is evaluated on Bills and Wiki against traditional, embedding-based, and LLM-based baselines using clustering-quality metrics and LLM prompt counts. It achieves top or highly competitive clustering results, uses over 92% fewer prompts than document-level LLM approaches, and ablations examine context and initial cluster granularity.

4.1.1 Dataset

  • Bills contains 8,024 summaries from the 110th–114th U.S. Congresses, and Wiki contains 15,242 articles; they have 21 and 15 high-level labels, respectively.Both datasets are manually annotated.Source: §4.1.1 · p. 6

4.1.2 Evaluation Metrics

  • Evaluation measures cluster alignment with labels using purity (P), P1, Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI).The metrics provide complementary perspectives on cluster quality.Source: §4.1.2 · p. 6

4.1.3 Baselines

  • The nine baselines span traditional LDA and NMF, neural or embedding-based CTM, BERTopic, and FASTopic, and LLM-based EDTM, PromptTopic, TopicGPT, and LiSA.Source: §4.1.3 · p. 6

4.1.4 Hyperparameters

  • Experiments use 256-dimensional Gemini embeddings reduced to 64 dimensions with PaCMAP, then cosine complete-linkage clustering into 80 initial clusters.Labeling and merging primarily use Gemini 2.5 Flash Lite at temperature 0; main results also evaluate Gemini 3.1 Flash Lite.Source: §4.1.4 · p. 7

4.2 Clustering Results

  • TopicMerge is top-performing on Wiki, with ARI 0.71, NMI up to 0.74, and P1 0.73; on Bills it has the best NMI (0.53) and ARI (0.42), with P1 0.57.The results are reported across both benchmark datasets and compared with the listed baselines.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 LLM Prompt Scaling and Efficiency

  • 645 prompts on Bills and 562 on Wiki represent a reduction of over 92% versus document-level LLM approaches.LiSA may require over 8,000 prompts for Bills and over 15,000 for Wiki; these counts scale with dataset size.Source: §4.3 · p. 7, Table 2 · p. 7

4.4.1 Context Cluster Ablation

  • On Wiki, removing context clusters drops ARI from 0.71 to 0.63, and the same performance-drop trend holds on Bills.The ablation compares TopicMerge with and without context clusters.Source: §4.4.1 · p. 8, Table 3 · p. 8

4.4.2 Cluster Count and Purity Ablation

  • On Wiki, starting with 80 clusters at purity 0.88 yields final ARI 0.71, while 30 clusters at purity 0.79 yields ARI 0.50.On Bills, performance is stable at high initial cluster counts (ARI 0.42–0.43) before dropping at lower granularities; the figure contrasts fragmented initial assignments with coherent final topics.Source: Table 4 · p. 8, §4.4.2 · p. 8, Figure 2 · p. 8

5 Conclusion

TopicMerge combines efficient clustering with LLM semantic reasoning to address the cost and scalability constraints of direct LLM topic modeling. On Bills and Wiki, it achieves state-of-the-art or highly competitive clustering performance while reducing LLM prompts by over 92% versus document-level models.

5 Conclusion

  • Direct LLM topic modeling has set a new state of the art but remains constrained by high computational costs and limited context windows.These constraints impede scalability.Source: §5 · p. 8
  • TopicMerge achieves state-of-the-art or highly competitive P1, ARI, and NMI performance on Bills and Wiki while using over 92% fewer LLM prompts than document-level models.The pipeline assigns corpus organization to efficient clustering and reserves the LLM for semantic arbitration.Source: §5 · p. 9

Limitations

TopicMerge depends on initial embedding quality, which may be less discriminative for very short texts. Its closed-source LLM also raises potential contamination and reproducibility concerns.

Limitations

  • Initial embedding quality constrains TopicMerge, particularly for corpora of very short texts where embeddings may be less discriminative.Specialized short-text embedding techniques are proposed as a way to expand applicability.Source: Limitations · p. 9
  • Reliance on a closed-source LLM introduces potential benchmark data contamination, stochastic outputs, and challenges for full reproducibility.The paper suggests open-source models as future work to improve reproducibility and mitigate contamination risks.Source: Limitations · p. 9

Ethical Considerations

TopicMerge's reliance on LLMs carries the potential for inherited societal bias. Such bias may appear in generated topic labels and merge decisions, while model opacity complicates auditing and mitigation.

Ethical Considerations

  • The underlying LLMs are trained on web-scale corpora containing societal biases, creating potential for inherited bias in TopicMerge.The paper identifies possible stereotypical or unfair representations involving demographic groups and sensitive social issues.Source: Ethical Considerations · p. 9
  • Inherited bias may manifest in generated topic labels and merge decisions as stereotypical or unfair representations of sensitive topics.The text specifically mentions demographic groups and sensitive social issues.Source: Ethical Considerations · p. 9
  • LLM opacity makes bias sources difficult to audit and complicates mitigation.Source: Ethical Considerations · p. 9

A Evaluation Metrics

Evaluation compares predicted cluster assignments with ground-truth classes using external metrics, for which higher values indicate better clustering performance.

A Evaluation Metrics

  • Evaluation aligns predicted cluster assignments Y with ground-truth classes X using external metrics.X contains ground-truth classes and Y contains predicted assignments.Source: A Evaluation Metrics · p. 11
  • Higher values on the stated external metrics indicate better clustering performance.Source: A Evaluation Metrics · p. 11

A.1 Purity and P1

The section defines purity-based and information-based measures for comparing predicted clusters with ground-truth labels, including adjustments for cluster size and chance agreement.

A.1 Purity and P1

  • Purity measures cluster homogeneity by taking the most frequent ground-truth class in each predicted cluster and weighting its proportion by cluster size.The overall score is a weighted average across clusters.Source: §A.1 · p. 11
  • P1 balances purity with inverse purity so clusters are both homogeneous and comprehensive.It addresses purity’s failure to penalize many small, pure clusters.Source: §A.1 · p. 11

A.2 Adjusted Rand Index

  • ARI compares observed pairwise agreement with its expected value under random assignment, correcting the Rand Index for chance agreement.The Rand Index counts document pairs treated consistently in both clusterings; ARI ranges from -1 to 1, with 1 indicating perfect agreement.Source: §A.2 · p. 11

A.3 Normalized Mutual Information

  • NMI normalizes mutual information between ground-truth labels and predicted clusters by their average entropy, producing scores from 0 to 1.The normalization makes the metric robust to differences in the number of clusters.Source: §A.3 · p. 11

B Artifacts, Licensing, and Intended Use

B.1 Datasets

The paper uses the Bills and Wiki benchmarks and documents the models, software, baseline provenance, and prompts used in its experiments.

B.1 Datasets

  • Bills and Wiki are public academic benchmarks used to evaluate clustering performance, and the authors describe their use as consistent with the datasets’ intended research purpose.The passage also characterizes these established corpora as posing minimal risk of private identifying information or significant offensive content.Source: §B.1 · p. 11

B.2 Models and Software

  • Gemini models provide embeddings and LLM reasoning, with Gemini 2.5 Flash Lite used for experiments, prompt measurements, and ablations unless noted otherwise.The models are proprietary and accessed through an API.Source: §B.2 · p. 12
  • The implementation uses PaCMAP for dimensionality reduction, scikit-learn Agglomerative Clustering for initial micro-clusters, and DSPy to manage labeling and merging prompts.The passage identifies their respective licenses as Apache-2.0, BSD-3-Clause, and MIT.Source: §B.2 · p. 12

C Prompts

D Topic Labels

TopicMerge’s generated labels are more granular than the Bills and Wiki ground-truth categories, reflecting the model’s emphasis on fine-grained semantic purity during merging.

D Topic Labels

  • TopicMerge’s labels are more granular than the Bills and Wiki ground-truth categories, reflecting the LLM’s tendency to prioritize fine-grained semantic purity during merging.The authors identify controlling abstraction, such as through few-shot examples or different merging heuristics, as future work.Source: D Topic Labels · p. 12

E Exploring Greedy Ground-Truth Guided Merging with TopicMerge

A ground-truth-guided oracle experiment shows a substantial performance improvement over other methods, while the authors identify greedy merging and initial clustering as remaining opportunities for improvement.

E Exploring Greedy Ground-Truth Guided Merging with TopicMerge

  • The ground-truth-guided oracle substantially outperforms all other methods, indicating a high performance ceiling for TopicMerge’s core pipeline.The experiment greedily merges clusters sharing the same majority ground-truth class; the table compares this oracle with baselines and marks the best and second-best standard-method results.Source: E Exploring Greedy Ground-Truth Guided… · p. 12, Table 6 · p. 16
  • The oracle is not a theoretical maximum because its merging heuristic is greedy and the experiment does not refine the initial cluster creation stage.The authors cite non-greedy merging and improvements to embeddings, dimensionality reduction, or clustering as possible refinements.Source: E Exploring Greedy Ground-Truth Guided… · p. 12
  • The merge prompt approves pairs for the same or naturally unified topic, redundant splits, or subtopics at the target granularity, while rejecting different domains or noise-diluting merges.It instructs the model to keep clusters separate whenever there is reasonable doubt.Source: E Exploring Greedy Ground-Truth Guided… · p. 13
  • The labeling prompts require concise, specific, unified topic labels grounded in representative texts and constrained by dataset examples of the target granularity.The label description should define the shared topic scope without tangential themes or specific document examples.Source: E Exploring Greedy Ground-Truth Guided… · p. 14

Related papers

Loading 2610.19323v1…