PaperlayerView exact revision on arXiv

Source-linked AI summary

TopicGPT: A Prompt-based Topic Modeling Framework

Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, Mohit Iyyer

arXiv:2311.01449v2cs.CL

TL;DR

Traditional bag-of-words topics can contain incoherent words that are difficult to interpret, and interactive guidance remains constrained by that format. TopicGPT is a prompt-based topic-modeling framework that generates descriptive topics. It aligns more closely with human-annotated topics than baseline models and is robust across prompt and data settings.

  • Problem

    Traditional bag-of-words topics can contain incoherent words that are difficult to interpret, and interactive guidance remains constrained by that format.

    Source: §1 · p. 1

  • Method

    TopicGPT is a prompt-based topic-modeling framework that produces concise labels paired with broad one-sentence descriptions.

    Source: §3.1 · p. 3, §7 · p. 8

  • Results

    10 points higher absolute purity than LDA on Wikipedia; TopicGPT also aligns significantly more closely with human-annotated topics on Wikipedia and Congressional bills and is robust across prompt and data settings.

    Source: §1 · p. 1, §7 · p. 8

  • Takeaways & Limitations

    Natural-language labels, descriptions, and quoted evidence accompany document-topic associations, making topics and assignments more interpretable.

    Source: §1 · p. 2

    Document truncation can discard valuable context and risk misrepresenting full documents.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Topic modeling is a well-established technique for exploring text corpora. Conventional topic models (e.g., LDA) represent topics as bags of words that often require "reading the tea leaves" to interpret; additionally, they offer users minimal control over the formatting and specificity of resulting topics. To tackle these issues, we introduce TopicGPT, a prompt-based framework that uses large language models (LLMs) to uncover latent topics in a text collection. TopicGPT produces topics that align better with human categorizations compared to competing methods: it achieves a harmonic mean purity of 0.74 against human-annotated Wikipedia topics compared to 0.64 for the strongest baseline. Its topics are also interpretable, dispensing with ambiguous bags of words in favor of topics with natural language labels and associated free-form descriptions. Moreover, the framework is highly adaptable, allowing users to specify constraints and modify topics without the need for model retraining. By streamlining access to high-quality and interpretable topics, TopicGPT represents a compelling, human-centered approach to topic modeling.

1 Introduction

TopicGPT uses prompted LLMs to generate and assign descriptive topics, addressing the interpretability and adaptability constraints of bag-of-words topic models. Its topics align more closely with human labels than competing methods, and users can tailor the topic list.

1 Introduction

  • TopicGPT prompts LLMs to generate topics from sampled documents, refines the list, and assigns documents to topics with supporting quotes.The quotes make assignments more verifiable.Source: §1 · p. 1
  • 30.3% of TopicGPT’s Wikipedia topics are misaligned with human-labeled topics, compared with 62.4% for LDA.TopicGPT also improves absolute purity by 10 points over LDA on Wikipedia and aligns more closely with human-annotated topics than LDA, SeededLDA, and BERTopic across two datasets.Source: §1 · p. 1
  • TopicGPT generates topics as natural-language labels with descriptions and supplies document-topic associations with supporting quotations.The labels and descriptions avoid the ambiguous word lists used by conventional topic models.Source: §1 · pp. 1–2
  • Users can steer topic scope and formatting with example topics, then manually edit or remove topics to curate the list.The example topics need not be data-specific or represent every topic in the corpus.Source: §1 · p. 2
  • In the reported experiments, open-source Mistral-7B-Instruct assigns topics competently but does not competently follow topic-generation instructions.Most experiments use GPT-4 for generation and GPT-3.5-turbo for assignment.Source: §1 · p. 2

2 Related Work

The related work situates TopicGPT among interpretable topic models and prompted LLM approaches to content analysis. It distinguishes TopicGPT from GoalEx by its topic-set organization and evaluation scope.

Related work

  • Prior topic models use topic-word and document-topic distributions, while natural-language topic representations aim to improve interpretability.The section also discusses seeded or anchored models, hierarchical models, and models that impose topic constraints.Source: §2 · pp. 2–3
  • Prompted LLMs have been applied to clustering, summarization, and deductive coding, while other work uses LLMs to label or evaluate topics from existing models.Embedding-based methods have also been used for topic modeling.Source: §2 · p. 3
  • Unlike GoalEx, TopicGPT organizes topics semantically and evaluates both topic alignment and stability, beyond cluster recovery alone.GoalEx focuses on clustering and corpus partitioning; TopicGPT also assigns all prompt topics to each document at once.Source: §2 · p. 3

3 Methodology

TopicGPT separates topic generation from document assignment. The generation stage builds and refines a topic list, and the assignment stage associates documents with topics and supporting quotes.

3.1 Stage 1: Topic Generation

  • Topic generation iteratively prompts an LLM to assign a document to an existing topic or add a new concise label with a broad one-sentence description.A few human-written example topics demonstrate the desired format; they need not be dataset-specific or representative of the corpus.Source: §3.1 · p. 3
  • The topic list can be refined by merging near-duplicates identified with embedding similarity and removing topics below a frequency threshold.TopicGPT compares candidate pairs with cosine similarity ≥0.5 before prompting the LLM about merging them; threshold choice may affect whether important topics are removed.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Stage 2: Topic Assignment

  • For assignment, the LLM receives the topic list, examples, and a document, then returns one or more topics with a document-specific description and supporting quote.A parser detects invalid or hallucinated assignments, which are sent back to the LLM for reassignment.Source: §3.2 · pp. 3–4

4 Experiments

The experiments assess alignment with human-coded topics and robustness across two English-language datasets. They describe sampling, implementation, evaluation metrics, and prompt- and data-variation tests.

4 Experiments

  • The experiments assess whether TopicGPT aligns with human-coded topics and remains robust across settings.The section describes datasets, baselines, model configurations, and evaluation metrics.Source: §4 · p. 4

4.1 Datasets

  • Evaluation uses Wiki articles and Congressional Bills, both of which have human-annotated topic labels.Wiki includes 14,290 articles and 15 high-level, 45 mid-level, and 279 low-level labels; Bills contains 32,661 summaries with 21 high-level and 114 low-level labels.Source: §4.1 · p. 4

4.2 Baselines

  • The baselines are LDA, BERTopic, and SeededLDA, with the number of topics controlled to match TopicGPT’s generated topic count.The section specifies MALLET for LDA, default BERTopic hyperparameters, and default SeededLDA hyperparameters.Source: §4.2 · p. 4

4.3 Sampling documents for TopicGPT

  • Topic generation uses a uniform random sample of 1,000 Bills documents and 1,100 Wiki documents rather than each full corpus.The authors report that refined new-topic counts plateau after around 600 generated documents in both datasets.Source: §4.3 · p. 4

4.4 TopicGPT implementation details

  • Implementation details include truncating longer documents to fit the LLM context window and sampling separate documents for evaluation.For Wiki and Bills, the assigner is set to one topic per document; evaluation samples 8,024 Wiki and 15,242 Bills documents not used in topic generation.Source: §4.4 · pp. 4–5

4.5 Evaluation Setup

  • The evaluation setup assesses topic alignment and stability, following prior work.The alignment comparison uses predicted assignments against ground-truth labels because TopicGPT is not a probabilistic model like LDA.Source: §4.5 · p. 5, §4.5.1 · p. 5

4.5.1 Topical alignment

  • Topical alignment is measured with purity, adjusted Rand index, and normalized mutual information.The metrics offer complementary views of cluster agreement; purity emphasizes cluster purity, while adjusted Rand index corrects for chance and is sensitive to the number of predicted clusters.Source: §4.5.1 · p. 5

4.5.2 Stability

  • Stability tests examine changes to prompts and generation data, including out-of-domain prompts, additional examples, shuffled document order, and a different sample.For the out-of-domain test, Wiki prompts are applied to Bills; the other tests vary example topics or the generation sample.Source: §4.5.2 · p. 5

5 Results

TopicGPT outperforms the baselines on alignment with human-annotated topics, and its topics are more semantically aligned after refinement. Results also examine assignment stability, multi-label assignments, and open-source models.

5.1 TopicGPT is strongly aligned to ground truth labels

  • TopicGPT’s post-refinement harmonic purity was 0.74 on Wiki and 0.57 on Bills, versus 0.64 and 0.52 for LDA.The comparison is reported alongside Table 1, which identifies TopicGPT as best across settings and metrics.Source: §5.1 · p. 6, Table 1 · p. 7
  • With multi-label assignment, three of five Wiki examples received both the ground-truth topic and the originally assigned topic.The authors recommend multi-label prompts to retrieve as many relevant topics as possible.Source: §5.1 · p. 6

5.2 TopicGPT is stable

  • TopicGPT assignments stayed within a 0.05 range across modified Bills settings, with greater stability than LDA on P1 and ARI and comparable NMI.The comparison measures alignment between modified and default settings; identical TopicGPT runs sometimes differed.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 TopicGPT topics are semantically close to ground truth

  • Across annotations, refined TopicGPT topics had a 30.3% misalignment rate, below LDA’s 62.4%; refinement consistently reduced out-of-scope and repeated topics, while increasing missing topics by one in Bills.The missing Bills topic appeared in only 23 of 32,661 documents; practitioners should try different refinement thresholds to avoid filtering out topics important to their research question.Source: §5.3 · p. 7

5.4 Implementing TopicGPT with open-source LLMs

  • Mistral assignment had an absolute purity decrease of about ≈6 points versus GPT-3.5-turbo, while Mistral and GPT-3.5-turbo struggled to generate topics.The authors recommend GPT-4 or models with similar capabilities for topic generation.Source: §5.4 · pp. 7–8

6 Future Work

TopicGPT is designed for inductive content analysis with users familiar with the dataset, while exploratory use by unfamiliar users remains a possible application. The authors identify zero-shot prompting and comparative evaluation of a hierarchical extension as future work.

6 Future Work

  • TopicGPT is designed for inductive content analysis and assumes users have some prior familiarity with the dataset.The authors also note it can be applied to exploration when users are unfamiliar with the data.Source: §6 · p. 8
  • Zero-shot prompting without example topics or documents is a possible direction for exploratory use by users unfamiliar with a dataset.The authors identify this as future work.Source: §6 · p. 8
  • A rigorous comparison of TopicGPT’s hierarchical extension against established hierarchical topic models remains a valuable next step.The extension is presented in Appendix A.Source: §6 · p. 8

7 Conclusion

TopicGPT is a prompt-based framework designed to address interpretability and adaptability limitations in traditional topic models. The authors report stronger ground-truth alignment and robustness across prompts and data subsets, and release the pipeline for researchers and practitioners.

7 Conclusion

  • TopicGPT is a prompt-based framework specifically designed for topic modeling.The authors release its pipeline for researchers and practitioners to try.Source: §7 · p. 8
  • TopicGPT addresses traditional topic models’ interpretability and adaptability limitations by generating high-quality, descriptive topics.Source: §7 · p. 8
  • TopicGPT outperforms baseline topic models on alignment with ground-truth labels and is robust across different prompts and data subsets.Source: §7 · p. 8

Limitations

The authors identify limited transparency into closed-source models, per-run costs, document truncation, and unevaluated non-English datasets as limitations. These boundaries constrain transparency, resource use, document context, and the evaluated language scope.

Limitations

  • TopicGPT’s best-performing setup uses closed-source GPT-4 for generation and GPT-3.5-turbo for assignment, whose training data and architecture are not fully transparent.The authors frame this reliance as reflecting the current imbalance in instruction-following ability, not a permanent limitation.Source: Limitations · p. 9
  • Closed-source models incur costs for each run, though users can skip refinement and assignment or reduce the generation subset to save time and money.Source: Limitations · p. 9
  • Documents must be truncated to fit the context limit, risking loss of valuable context and misrepresentation of full documents.The authors suggest chunking, representative-chunk sampling, summaries, or long-context models as future approaches.Source: Limitations · p. 9
  • TopicGPT has not been evaluated on non-English datasets, and the authors note that OpenAI models’ instruction-following is notably degraded in non-English languages.Source: Limitations · p. 9

Ethical Considerations

The authors state that the framework’s risks are no greater than those inherent in its supporting LLMs. They also describe a hierarchical extension that generates document-grounded subtopics, enabling exploration at multiple levels of specificity.

Ethical Considerations

  • The authors state that TopicGPT’s risks are no greater than those inherent in the LLMs that support it.The human evaluation received IRB approval; US-based annotators gave informed consent and participated voluntarily without compensation.Source: Ethical Considerations · p. 9

A Hierarchical Implementation

  • TopicGPT can extend refined topics into hierarchies by prompting an LLM to generate subtopics grounded in documents associated with each parent topic.When documents do not fit in one prompt, earlier subtopics are included in later prompts.Source: A Hierarchical Implementation · p. 13
  • Generated subtopics capture narrower, nuanced themes within parent topics and are described as informative and well-grounded.The hierarchy lets users explore topics at different specificity levels; Figure 2 shows example Wiki top-level topics.Source: A Hierarchical Implementation · p. 13, Figure 2 · p. 13

B Comparison of TopicGPT topics with human-curated qualitative categories in a different domain

A case study compares TopicGPT’s categories with expert-curated categories for more than 16,000 Reddit responses from formerly suicidal people. TopicGPT categories include direct matches, cross-cutting themes, higher-level themes, and additional categories that capture background content or non-answers.

B Comparison of TopicGPT topics with human-curated qualitative categories in a different domain

  • TopicGPT categories were compared with categories experts curated from more than 16,000 responses using a 30-topic LDA model; expert consensus yielded 24 categories.The experts further grouped these into four meta-categories; Table 6 presents TopicGPT categories for the same responses.Source: B Comparison of TopicGPT topics with… · p. 14, Table 5 · p. 15, Table 6 · p. 16
  • The qualitative comparison identifies direct correspondences between several TopicGPT categories and expert categories, alongside arguable matches.Examples include T1 with E8, T6 with E21, and T11 with E4.Source: B Comparison of TopicGPT topics with… · p. 14
  • Some TopicGPT categories span variations on the experts’ broader theme of concern about others left behind, although the two approaches partition the space differently.The paper identifies T4, T12, and T13 as related to E1–E5.Source: B Comparison of TopicGPT topics with… · p. 14
  • T5 is described as a higher-level theme related to several expert categories, while other TopicGPT categories capture background content or non-answers rather than direct responses.The authors note that qualitative content analysis is shaped by research questions as well as by the data.Source: B Comparison of TopicGPT topics with… · p. 14
  • In a separate evaluation, experts found TopicGPT had fewer missing topics than SeededLDA, similar proportions of out-of-scope and repeated topics, and better usability and interpretability.The comparison used two subject matter experts and avoided LDA as the comparator because LDA helped produce the expert categories.Source: B Comparison of TopicGPT topics with… · p. 14

C Probabilistic Justification for Data Sampling

Across these subsections, the paper gives a sampling procedure for topic generation, guidance on shaping example topics, and a comparison showing TopicGPT’s alignment advantage over LDA across tested topic counts.

C Probabilistic Justification for Data Sampling

  • After an initial topic drought, initially generated topics continue increasing as more documents are processed, while refined topics plateau.The passage reports the pattern for both Bills and Wiki; Figure 3 also shows the expected topic count plateauing.Source: C Probabilistic Justification for Data… · p. 17, Figure 3 · p. 17
  • For Wiki, setting nd to 140 yields a sample size ns of 1,100 with p* approximately 0.005.The procedure sets an upper bound on topics from the minimum desired prevalence and simulates sample sizes to approach a user-defined acceptable probability of missing the least-prevalent topic.Source: C Probabilistic Justification for Data… · p. 17

D Example Topics

  • Example topics can guide the format and specificity of generated topics, and should match the expected granularity at each hierarchy level.The examples illustrate both label length and level-specific granularity, including concise labels and detailed subtopics.Source: D Example Topics · p. 18

E Varying number of topics k for LDA baseline

F Human annotations on topical alignment between TopicGPT and ground truth of Bills dataset

This appendix reports human annotation procedures, defines the topical-alignment metrics, and describes dataset and prompt details alongside results for additional seeded topic models.

F Human annotations on topical alignment between TopicGPT and ground truth of Bills dataset

G Topical alignment metrics

  • Purity scores dominant-class assignments, but can favor singleton clusters; inverse purity addresses grouping, and their harmonic mean balances both objectives.Purity assigns each predicted cluster to its most frequent ground-truth class and counts correctly assigned documents over all clustered items.Source: G Topical alignment metrics · p. 22
  • The Rand Index measures pairwise clustering agreement, while Adjusted Rand Index corrects that agreement for chance using its expected value.ARI scores near 0 indicate random assignments, while scores near 1 indicate strongly consistent assignments.Source: G Topical alignment metrics · p. 22
  • Normalized Mutual Information scales shared information between clusterings to 0–1, making it less sensitive to differing cluster counts.The metric uses mutual information and the entropies of both clusterings.Source: G Topical alignment metrics · p. 22

H Experiment datasets

  • Wiki documents average 3,412 tokens versus 261 for Bills, and running TopicGPT on Wiki costs much more despite its smaller dataset.The passage attributes the higher Wiki cost to its longer documents.Source: Table 11 · p. 23
  • Wiki text-label mappings may have appeared in LLM pretraining, making Wiki a best-case setting; Bills labels instead use numerical codes linked to bills by URLs.Wiki performance is not perfect, and the authors say TopicGPT does not reproduce ground-truth topics unless they are explicitly provided as examples.Source: H Experiment datasets · p. 23

I Additional seeded/anchored topic models

J Prompts

  • The prompt templates cover broad topics, hierarchical subtopics, topic refinement or merging, and document-level topic assignment.Users tailor examples to their dataset, and the assignment prompt can be modified to require one topic.Source: Table 14 · p. 25, Table 15 · p. 26, Table 16 · p. 27
Loading 2311.01449v2…