PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment-Level Agentic Topic Modeling for Improved Data Exploration and Resource Efficiency

Myeongjun Erik Jang, Antonios Georgiadis, Sae Young Moon, Fran Silavong

arXiv:2609.31460v1cs.AI

TL;DR

LLM topic assignment can miss the contributions of multiple topics within a document and is costly at document scale. SELATM generates topics from clustered text segments and iteratively refines them with agentic feedback; experiments report strong performance across public and business datasets.

  • Problem

    LLM-based topic assignment typically assigns one topic per document, cannot precisely quantify multiple topics’ contributions, and has call costs of O(N × L).

    Source: §1 · p. 1

  • Method

    SELA​TM generates topics for segment clusters, represents documents as topic distributions, and uses agentic feedback loops to split or merge topics.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Without refinement, SELATM achieves the best performance in 10 of 15 public evaluation cases, with 9 differences statistically significant, and significantly outperforms baselines in 5 of 6 business evaluation cases.

    Source: §4.1 · p. 5, §4.2 · p. 7

  • Takeaways & Limitations

    Segment-level modeling provides document topic contribution scores while reducing LLM calls relative to document-level assignment.

    Source: §1 · p. 2

    Refinement does not significantly improve CI results, and performance degrades after five iterations in both CCC and CI.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Topic modeling is an effective technique for discovering hidden themes within documents and is widely used in text mining and data analysis across a variety of industry sectors. Recently, large language model (LLM)-based topic models have been emerged that prompt LLMs to generate topics then assign the topics to documents, producing more natural and human-readable topics than conventional topic modeling algorithms. However, the nature of topic assignment process causes certain drawbacks, such as the incapability to produce topic distributions over a document, too broad or narrow topics, and high resource consumption, which increases with the number and length of of documents being assigned topics. These issues are particularly critical for industrial applications, which require high-quality, in-depth analysis and the processing of large volumes of documents. In this context, this paper introduces a framework called SeLATM, which addresses these concerns by employing segment-level topic generation and topic refinement through agentic feedback loops. Experimental results on various datasets demonstrate that SeLATM significantly reduces the LLM resources compared to methods based on topic assignment process, while maintaining superior performance.

1. Introduction

LLM-based topic models produce readable topics but typically cannot quantify how multiple topics contribute to a document, and document-level assignment is resource-intensive. SELATM replaces that assignment step with segment-level topic generation and agentic refinement.

1. Introduction

  • The introduction frames topic modeling as a practical method for discovering corpus themes, with uses in research and industry.LLM approaches generate free-text topics that are more comprehensive and human-understandable than conventional word-distribution representations.Source: §1 · p. 1
  • LLM-based methods typically assign one topic per document and cannot precisely quantify multiple topics’ contributions, limiting document-topic interpretability for exploration.Their assignment cost scales as O(N × L), with N documents and L document length.Source: §1 · p. 1
  • Prior LLM topic refinement either omits refinement or focuses on merging, risking broad, incoherent topics when distinct topics overlap.For example, merging two topics with 60% overlap can combine distinct subtopics into one mega-topic.Source: §1 · pp. 1–2
  • SELATM removes document-level topic assignment, generates topics for segment clusters, and represents each document as a distribution over topics.Evaluator, operator, and planner agents use feedback to assess topic quality, split or merge topics, and decide whether to continue refinement.Source: §1 · p. 2

2. SELATM Framework

SELATM clusters text segments to generate topic labels and document-level topic distributions, then iteratively refines topics through evaluator, operator, and planner agents. It also groups related topics into parent topics using multi-view representations.

2.1. Segment-level Topic Modeling

  • Sentence- and paragraph-level segments are embedded and agglomeratively clustered; the user-set K controls topic granularity, and moderate mis-specification is mitigated by downstream refinement.Syntactic segmentation avoids LLM use, while short segments are intended to reduce long-context and embedding length-collapse issues.Source: §2.1 · p. 2
  • For each cluster, SELATM generates topic names and descriptions, sampling representative segments with MMR for large clusters; document topic shares are computed from segment assignments.LLM calls scale as O(K × nS × l), rather than O(N × L), and the resulting distributions show topic contributions and associated segments.Source: §2.1 · p. 3

2.2. Agentic Topic Refinement

  • Coherence evaluation flags topics for splitting, diversity evaluation identifies groups for merging, and operation agents perform the corresponding changes to topics.A split is skipped when the larger resulting cluster contains more than 80% of the segments.Source: §2.2 · p. 3
  • The planner compares reports before and after refinement, keeps improved results, and continues until the tolerance or iteration limit is reached.The loop uses topic, cluster, and sampled-cluster inputs together with maximum iteration and tolerance settings.Source: §2.2 · p. 4

2.3. Multi-view Topic Parenting

  • Multi-view clustering groups similar topics using segment-centroid, topic-name, and topic-description embeddings, then generates parent-topic names and descriptions.Parent topics provide higher-level topics and their relationships with base topics.Source: §2.3 · p. 4

3. Experiment Design

The experiments use public and in-house business datasets, with different evaluation procedures for labeled and unlabeled data. Baselines span three topic-modeling families, and model settings include dataset-specific topic counts and segment-sampling searches.

3.1. Datasets

  • Experiments cover labeled public datasets Banking77, Bills, and Wiki, plus three unlabeled in-house banking datasets: CCC, BCR, and CI.The in-house datasets contain chatbot messages, chatbot reviews, and summaries of customer issues, respectively.Source: §3.1 · p. 4

3.2. Evaluation Metrics

  • Public datasets use P1, ARI, and NMI, while topic accuracy and completeness are evaluated with LLM-as-a-Judge and five-prediction self-consistency.For SELATM, topic accuracy weights individual topic scores by their document contribution ratios; completeness concatenates generated topics.Source: §3.2 · p. 5

3.3. Baseline Methods

  • Baselines represent embedded topic models, LLM-driven topic generation and assignment, and LLM-assisted neural topic models.The listed methods include BERTopic, C-Top2Vec, TopicGPT, LLoom, TIDE, and LLM-ITL.Source: §3.3 · p. 5

3.4. Models and Hyperparameters

  • Experiments use gpt-4o-2024-08-06 for language generation and text-embedding-3-small-1 for embeddings, with gpt-4.1-mini-2025-04-14 for LLM-as-a-Judge.K is dataset-specific, while the segment-sampling parameter nS is selected through hyperparameter search.Source: §3.4 · p. 5

4. Experimental Results

Across public and business datasets, SELATM generally outperforms baselines, with refinement improving LLM-as-a-Judge metrics in many settings. It also reduces token use in low-iteration settings and produces strong parent-topic results.

4.1. Public Dataset Results

  • SELATM without refinement outperforms baselines in 10 of 15 public-dataset evaluation cases, with 9 differences statistically significant.It also mostly ranks second when a baseline leads; Table 3 reports five-run averages and marks significant gaps.Source: §4.1 · pp. 5–6, Table 3 · p. 6
  • Public-dataset LLM-as-a-Judge metrics improve significantly with refinement, with the best performance mostly occurring at 3 iterations.T_A and T_C gains at three iterations are significant across all three datasets; clustering-based metrics show no significant differences on Bills and Wiki, while Banking77 metrics trend slightly downward with more iterations.Source: §4.1 · p. 6

4.2. Business Dataset Results

  • On business datasets, SELATM without refinement significantly outperforms baselines in 5 of 6 evaluation cases; refinement performs best with 3 iterations for CCC and 5 for BCR.Refinement gains are significant versus no refinement except for T_C on CCC; the CI dataset shows no significant improvement, and performance degrades after five iterations on CCC and CI.Source: Table 4 · p. 6, §4.2 · p. 7

4.3. LLM-Token Consumption Comparison

  • Without refinement, SELATM uses 24.4% of TIDE’s and 8.19% of TopicGPT’s average token consumption.At one refinement iteration it uses 55% of TIDE’s consumption; after three iterations it exceeds TIDE, while remaining below TopicGPT after 3–5 iterations.Source: §4.3 · p. 7, Figure 1 · p. 7

4.4. Parent Topic Modeling Performance

  • Parent topics achieve 92% of base-topic performance on average, and SELATM significantly outperforms TIDE except for the nonsignificant T_A difference on CCC.On Bills and Wiki, SELATM’s T_A and T_C scores are 2.2 and 2.6 times TIDE’s, respectively; Figure 2 presents parent-topic T_A and T_C results.Source: §4.4 · pp. 7–8, Figure 2 · p. 8

5. Related Works

Conventional bag-of-words topic models overlook word semantics, while document-level LLM topic models offer human-readable topics but incur computational overhead. LLM-ITL reduces that overhead by using LLMs primarily for refinement.

5. Related Works

  • Bag-of-words methods such as LDA overlook semantic meanings and relationships between words, motivating embedded topic models that use dense representations.The related-work discussion describes embedded approaches as an attempt to address this limitation.Source: §5 · p. 8
  • LLM-based methods generate interpretable, human-readable topics, but document-level analysis carries significant computational overhead.These methods ask LLMs to identify topics in a corpus and assign them to individual documents.Source: §5 · p. 8
  • LLM-ITL uses neural topic modeling for document analysis and LLMs mainly for refinement, reducing computational overhead.Source: §5 · p. 8

6. Conclusion

SELATM combines segment-level topic modeling with an agentic feedback refinement loop to address limitations of recent LLM-based topic models. Experiments on public and business datasets indicate improved topic quality, topic-contribution interpretability, and reduced LLM computation overhead.

6. Conclusion

  • SELATM addresses three limitations of recent LLM-based topic models: inability to quantify multiple topic contributions, high computational demands, and inaccurate topic naming from insufficient refinement.Source: §6 · p. 8
  • Segment-level topic modeling represents documents as distributions over multiple topics, quantifying each topic's contribution and linking segments to their topics.Source: §6 · p. 8
  • Removing a separate topic-assignment process substantially reduces LLM computational expense, while segment-level modeling and feedback refinement support more precise topics.Source: §6 · p. 8
  • Experiments on public and business datasets show SELATM enhancing interpretability, reducing LLM computation overhead, and producing better-quality topics.Source: §6 · p. 8

A.1. Clustering-based Evaluation Metrics

This appendix defines clustering-based metrics for comparing predicted and ground-truth clusters, including purity, pairwise agreement, adjusted agreement, and normalized mutual information.

A.1. Clustering-based Evaluation Metrics

  • The setup compares two clusterings, U and V, over N instances, with K clusters in U and C clusters in V.u_i and v_i denote the i-th clusters in U and V, respectively.Source: §A.1 · p. 12
  • Purity measures how many ground-truth labels occur within predicted clusters; harmonic mean of purity addresses purity's asymmetry.Purity is described as analogous to precision, and the harmonic mean combines purity in both directions.Source: §A.1 · p. 12
  • Random Index (RI) evaluates agreement between two clustering results using pairs assigned to the same or different clusters in both.Source: §A.1 · p. 12
  • Adjusted Rand Index (ARI) adjusts RI for expected random-cluster agreement; its range is -1 to 1, with 1 indicating perfect agreement.Zero corresponds to random assignments, while negative values indicate less agreement than expected by chance.Source: §A.1 · p. 12
  • Normalized Mutual Information (NMI) measures similarity through mutual information normalized to 0–1, reducing sensitivity to the number of clusters.Mutual information quantifies information shared between two random variables.Source: §A.1 · p. 12

A.2. Baseline Models and Implementation Details

The appendix describes implementation choices for six baseline models and the reporting conventions for the SELATM nS search. The baselines span embedded, LLM-based, and neural topic-modeling approaches.

A.2. Baseline Models and Implementation Details

  • BERTopic clusters pretrained-language-model document embeddings and generates cluster representations with class-based TF-IDF; experiments use all-mpnet-base-v2 and default library settings.Source: §A.2 · p. 12
  • C-Top2Vec segments documents with a sliding window, clusters segment embeddings, derives document-topic distributions from segment assignments, and labels topics using nearby words and phrases.The implementation uses all-mpnet-base-v2 and the authors' hyperparameters.Source: §A.2 · p. 12
  • TopicGPT generates corpus topics and assigns them to documents; experiments use its full pipeline with refinement and low-level topics for comparison with more granular methods.Source: §A.2 · pp. 12–13
  • Table 5 reports five-repetition averages for the nS search, highlighting each dataset's best SELATM result and underlining a baseline that exceeds SELATM's lowest result.Banking77 and BCR omit nS = 80 because their largest clusters contain fewer than 80 instances.Source: Table 5 · p. 13
  • LLoom distills text through summarization, clusters the summaries, then synthesizes high-level concepts; the first and last steps are LLM-driven.Experiments use the full pipeline with default parameters.Source: §A.2 · p. 13
  • LLM-ITL learns topic and document distributions with neural topic modeling, then uses an LLM to name topics and refine topic words.It aligns original and refined topic-word distributions using Optimal Transport; NVDM is the base model in these experiments.Source: §A.2 · p. 13
  • TIDE clusters before generating topics, reducing LLM calls relative to TopicGPT's document-level topic generation, and lets users choose the number of topics.The authors implemented TIDE themselves because its code was not publicly available.Source: §A.2 · p. 13

A.3. Hyperparameter Search for Optimal nS

The hyperparameter search varied nS from 30 to 80 and assessed topic accuracy and completeness. Performance differences were marginal on most datasets, and the authors recommend nS around 40–50 when a full search is impractical.

A.3. Hyperparameter Search for Optimal nS

  • The search varied nS from 30 to 80 in increments of 10 and measured topic accuracy and topic completeness.The authors note that clustering-based metrics were determined during segmentation and clustering and did not depend on the sampled text segments.Source: §A.3 · p. 13
  • In 9 of 12 evaluation cases, SELATM’s lowest nS-specific performance exceeded the best baseline.The effect of nS was less pronounced than expected on most datasets, except CI.Source: §A.3 · p. 13
  • When a thorough search is impractical, the authors recommend setting nS around 40–50.The search selected 70 for Banking77 and CI, 40 for Bills and Wiki, 50 for CCC, and 30 for BCR.Source: §A.3 · pp. 13–14

A.4. Ablation Study

The ablations support the use of both agglomerative clustering and the split operation. Omitting splits lowered performance on Bills and Wiki, while agglomerative clustering generally outperformed K-Means and showed lower variability.

A.4. Ablation Study

  • The clustering comparison used public datasets, clustering-based metrics, and no feedback refinement loop.The ablation compared K-Means with agglomerative clustering.Source: §A.4 · p. 14
  • Agglomerative clustering was more stable and scored better than K-Means except for Bills NMI, where K-Means was marginally higher.Agglomerative clustering was significantly superior across all metrics except that Bills NMI comparison, and had substantially lower standard deviations.Source: §A.4 · p. 14
  • The split ablation removed Ecoh and Asplit from the refinement loop, retaining only Ediv and Amerge.It used public datasets and LLM-as-a-Judge metrics to evaluate the refinement loop.Source: §A.4 · p. 14
  • Without the split operation, Bills and Wiki performance was significantly lower across iterations, while Banking77 showed no significant difference.Performance continued to decline with additional iterations without splits; splits ceased after 2–3 iterations, consistent with optimal performance typically occurring within 1–3 iterations.Source: §A.4 · p. 14

B. Prompt Designs

The prompt designs guide topic generation, coherence and diversity evaluation, refinement planning, and LLM-as-a-Judge assessment of topic accuracy and completeness, including structured responses on how well assigned topics represent a text’s themes.

B.1. Topic Generation Prompt

  • Topic generation asks an agent to name and describe a cluster by covering as many input segments as possible, with a default of two topic words and optional user-defined topic definitions.The prompt can also provide a previous topic and feedback for improvement.Source: §B.1 · pp. 14–15

B.2. Topic Coherence Evaluation Prompt

  • The coherence evaluator scores how well a topic name and description match assigned segments from 1 to 10 in 0.5 increments, and flags low-coherence topics for separation with feedback.For sufficient coherence, it sets need separation to False and leaves feedback null.Source: §B.2 · p. 15

B.3. Topic Diversity Evaluation Prompt

  • The diversity evaluator scores how distinct topics are from one another on a 1-to-10 scale with 0.5 increments, and requests merging when topics are too similar.For each similar group, the prompt requests the two most similar topic IDs and feedback on why they should be merged.Source: §B.3 · p. 16

B.4. Planning Agent Prompt

  • The planning agent summarizes coherence and diversity feedback before and after refinement, reporting strengths, weaknesses, and an overall score from 1 to 10.Its report prompt instructs it not to add suggestions beyond those in the evaluators’ feedback.Source: §B.4 · p. 16
  • The improvement check compares pre- and post-refinement reports using overall scores, strengths and weaknesses, counts of topics needing refinement, and topic-specific feedback.It marks improvement True when criteria are met and otherwise False, with a brief explanation.Source: §B.4 · p. 17

B.5. LLM-as-a-Judge Prompt

  • The topic-accuracy judge assesses whether an assigned topic is represented in the text, ignoring granularity, and rates it Incorrect, Partially Correct, Mostly Correct, or Completely Correct.The prompt defines these levels by how much of the topic’s details are represented in the text.Source: §B.5 · p. 17
  • The topic-completeness judge checks whether assigned topics omit themes present in the text and rates coverage as Not Covered, Minorly Covered, Mostly Covered, or Complete.The prompt defines Not Covered as failing to capture any main themes, with significant topics missing.Source: §B.5 · pp. 17–18
  • The rubric describes a low-coverage case in which only a small portion of relevant themes are represented and important topics are missing.It characterizes coverage as insufficient to represent the text’s content.Source: §B.5 · p. 18
  • “Mostly Covered” means the assigned topics capture most main themes, but still omit one or more significant topics.Coverage is substantial but incomplete.Source: §B.5 · p. 18
  • “Complete” means the assigned topics reflect all major themes, with no significant topics missing.The model’s output is described as a comprehensive representation of the text.Source: §B.5 · p. 18
  • The judge prompt supplies the text and its assigned topic through the fields “Text: {{text}}” and “Assigned Topic: {{topic}}”.Source: §B.5 · p. 18
  • Responses for both evaluation dimensions are generated using structured LLM generation.The structured generation follows a Pydantic schema.Source: §B.5 · p. 18
  • The provided schema excerpt imports Pydantic’s BaseModel and declares an AutoEvalOutput class.Source: §B.5 · p. 18

C.1. Public Dataset Examples

The appendix provides generated-topic examples and topic descriptions for public datasets and business datasets.

C.1. Public Dataset Examples

  • Table 8 presents examples of topics generated by SELATM for public datasets.Source: Table 8 · p. 19
  • Table 9 gives descriptions of the topics shown in Table 8.Source: Table 9 · p. 20

C.2. Business Dataset Examples

  • Table 10 presents business-dataset examples of topics generated by SELATM.Source: Table 10 · p. 21
  • Table 11 gives descriptions of the topics shown in Table 10.Source: Table 11 · p. 21

Related papers

Loading 2609.31460v1…