PaperlayerView exact revision on arXiv

Source-linked AI summary

ClusterLLM: Large Language Models as a Guide for Text Clustering

Yuwei Zhang, Zihan Wang, Jingbo Shang

arXiv:2305.14871v2cs.CL

TL;DR

API-based LLMs cannot be applied directly to text clustering because their embeddings are inaccessible. CLUSTERLLM uses LLM sentence-relation judgments to guide a small embedder, and experiments report improved clustering quality across benchmarks at low cost.

  • Problem

    API-based LLMs have demonstrated language capabilities, but their inaccessible embedding vectors prevent their direct use for text clustering.

    Source: §1 · p. 1

  • Method

    CLUSTERLLM uses LLM triplet judgments to fine-tune a small embedder for user-specified perspectives and pairwise judgments to select cluster granularity.

    Source: §3.1 · p. 3, §6 · p. 9

  • Results

    CLUSTERLLM improves clustering over deep-clustering and self-supervised baselines and outperforms clustering-error methods for granularity, at approximately $0.2 for perspective and $0.4 for granularity with GPT-3.5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework incorporates user-specified perspectives and granularity preferences into clustering using API-based LLM judgments rather than LLM embeddings.

    Source: §1 · p. 2, §6 · p. 9

    The method relies on a pre-trained embedder to identify instances with high clustering-assignment entropy for informative triplet sampling.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We introduce ClusterLLM, a novel text clustering framework that leverages feedback from an instruction-tuned large language model, such as ChatGPT. Compared with traditional unsupervised methods that builds upon "small" embedders, ClusterLLM exhibits two intriguing advantages: (1) it enjoys the emergent capability of LLM even if its embeddings are inaccessible; and (2) it understands the user's preference on clustering through textual instruction and/or a few annotated data. First, we prompt ChatGPT for insights on clustering perspective by constructing hard triplet questions <does A better correspond to B than C>, where A, B and C are similar data points that belong to different clusters according to small embedder. We empirically show that this strategy is both effective for fine-tuning small embedder and cost-efficient to query ChatGPT. Second, we prompt ChatGPT for helps on clustering granularity by carefully designed pairwise questions <do A and B belong to the same category>, and tune the granularity from cluster hierarchies that is the most consistent with the ChatGPT answers. Extensive experiments on 14 datasets show that ClusterLLM consistently improves clustering quality, at an average cost of ~$0.6 per dataset. The code will be available at https://github.com/zhang-yu-wei/ClusterLLM.

1 Introduction

CLUSTERLLM uses API-based LLM judgments to guide a small embedder because LLM embedding vectors are inaccessible. Its triplet and pairwise stages address clustering perspective and granularity, and experiments report improved clustering quality at low query cost.

1 Introduction

  • For perspective, an LLM chooses which of two candidates better matches an anchor, and entropy-based sampling selects informative, nearby triplets.The predicted triplets fine-tune the small embedder toward the user-preferred perspective.Source: §1 · p. 2
  • For granularity, the method samples sentence pairs from a cluster hierarchy and selects the level most consistent with LLM predictions.The hierarchy is built by iteratively merging the closest clusters.Source: §1 · p. 2
  • Experiments cover 14 datasets spanning intent, topic, type, domain, and emotion discovery.Source: §1 · p. 2
  • CLUSTERLLM improves clustering over deep-clustering and self-supervised baselines and outperforms clustering-error methods for determining granularity.The evaluation spans 14 datasets across diverse tasks.Source: §1 · p. 2
  • The framework uses LLM-predicted sentence relations to guide clustering and lets users specify preferences through instructions or a few annotations.Its two stages address perspective—the grouping criterion—and granularity—the scope of clusters.Source: §1 · p. 2
  • With GPT-3.5, the reported costs are approximately $0.2 for perspective and $0.4 for granularity.Source: §1 · p. 2

2 Preliminary

Text clustering maps an unlabeled corpus to cluster indices. CLUSTERLLM can incorporate textual instructions for perspective and few-shot annotations for granularity.

2 Preliminary

  • The clustering task takes an unlabeled corpus D = {x_i} and maps its texts to cluster indices.Source: §2 · p. 2
  • Textual instructions specify the user's clustering perspective, such as selecting examples by entity type.Source: §2 · p. 2
  • Few-shot annotations specify granularity by indicating whether sentence pairs share a category.Source: §2 · p. 2

3 Our CLUSTERLLM

CLUSTERLLM guides a small embedder through two LLM-assisted stages: triplet comparisons refine clustering for a user-specified perspective, and pairwise judgments select cluster granularity.

3 Our CLUSTERLLM

  • CLUSTERLLM uses triplet and pairwise LLM tasks to guide a small embedder on clustering perspective and granularity.The framework treats perspective as the grouping criterion and granularity as the scope of clusters.Source: §3 · p. 3

3.1 Triplet Task for Perspective

  • Stage 1 asks an LLM, under a user-specified instruction, which of two choices better matches an anchor, yielding positive and negative examples.The triplet comprises an anchor and two candidate sentences; the unselected choice is the negative.Source: §3.1 · p. 3

3.1.1 Entropy-based Triplet Sampling

  • Ambiguous anchors are selected by entropy, then paired with choices from their closest clusters; query budget Q is user-defined and independent of dataset size.The closest clusters are those with the largest soft assignment values.Source: §3.1.1 · pp. 3–4

3.1.2 Finetuning Embedder

  • The selected positive and hard negative, plus in-batch negatives, train the base embedder; its refined embeddings can support iterative resampling and clustering.The paper describes applying clustering algorithms to the resulting embeddings.Source: §3.1.2 · p. 4

3.2 Pairwise Task for Granularity

  • Stage 2 asks whether data pairs belong to the same cluster, using user-annotated demonstrations to reflect the desired granularity.Different granularities can be appropriate for the same dataset, such as domains or topics.Source: §3.2 · p. 4

3.2.1 Determine Granularity with Pairwise Hierarchical Sampling

  • Pairs sampled from merged clusters span user-bounded kmin–kmax granularity; F-beta consistency selects the level, and large datasets use mini-batch K-means.The sampled pairs cover the range between the minimum and maximum cluster counts.Source: §3.2.1 · p. 4

4 Experiments

The experiments evaluate CLUSTERLLM across diverse text-clustering tasks and compare its perspective-guided clustering and granularity selection with baseline methods. Results cover clustering quality, triplet sampling and prediction, and granularity decisions.

4.1 Datasets

  • The 14 datasets cover intent, domain, type, topic, and emotion discovery, with small- and large-scale versions that retain the same cluster counts.The small- and large-scale versions differ in data volume.Source: §4.1 · p. 5

4.2 Experiment Details

  • Evaluation runs CLUSTERLLM once per dataset, then uses five seeds for K-means and reports Hungarian-aligned clustering accuracy and normalized mutual information.The K-means evaluation uses the ground-truth number of clusters.Source: §4.2 · p. 5

4.4 Main Results

  • CLUSTERLLM-I beats SCCL-I on 11 of 14 datasets and outperforms self-supervise-I on most datasets, consistently across clustering accuracy and NMI.The comparison uses known granularity; Table 2 reports the 14-dataset averages in its last two columns.Source: Table 2 · p. 6, §4.4 · p. 7

4.3 Compared Methods

  • The comparisons include Instructor and E5 embedders, self-supervised fine-tuning on the same triplets, and SCCL deep clustering.SCCL uses the full dataset for training, while CLUSTERLLM uses fewer training data.Source: §4.3 · p. 6

4.5 Analysis on Triplet Prediction Accuracy

  • GPT-3.5 and GPT-4 improve triplet prediction over Instructor on high-entropy examples, while random sampling yields fewer ground-truth triplets and a smaller or negative accuracy gap.The analysis compares high-entropy and randomly sampled triplets.Source: §4.5 · p. 7

4.6 Ablation Study

  • GPT-4 only marginally improves over GPT-3.5 at much higher cost, human labels yield the highest performance, and random triplet sampling can decrease performance.Accuracy generally increases with higher-entropy selection, except on GoEmo.Source: §4.6 · pp. 7–8, Figure 3 · p. 8

4.7 Determining Cluster Granularity

  • On eight datasets with cluster counts searched from 2 to 200, the proposed methods rank higher and distinguish domains from intents; GPT-4 improves on GPT-3.5, while larger λ generally helps but not always.For MTOP, GPT-3.5 with λ = 3 predicts 92/18 clusters for intent/domain, while BIC predicts 69/64.Source: §4.7 · p. 8

5 Related Works

Related work spans deep clustering, relational supervision, constrained clustering, and pretrained text embeddings. ClusterLLM is positioned among approaches that use relations or LLMs to guide text clustering.

Deep clustering and LLMs

  • Deep clustering research is categorized as multi-stage, iterative, generative, or simultaneous; concurrent work assigns instances to LLM-proposed explanations, while IDAS encodes sentences with LLM summaries.The cited taxonomy organizes methods by how representation learning and clustering modules interact.Source: §5 · pp. 8–9

Clustering with Relations

  • Relational supervision and constrained clustering use pairwise information, but pseudo-labels may be noisy and label-derived must-link or cannot-link constraints limit applicability in this setting.ClusterLLM instead studies relations inferred by API-based LLMs.Source: §5 · p. 9

Pre-trained Embedding Model

  • E5 and Instructor are pretrained embedding models with strong benchmark performance, and CLUSTERLLM aims to improve them using LLMs.E5 uses web-scraped data pairs with a contrastive objective; Instructor is pretrained on 330 tasks with instructions.Source: §5 · p. 9

6 Conclusion

CLUSTERLLM uses API-based LLM feedback in triplet and pairwise tasks to guide small embedders on clustering perspective and granularity. Experiments report improved clustering quality and reasonable granularity at negligible cost, while the framework still relies on an embedding model.

6 Conclusion

  • Triplet predictions fine-tune small embedders toward a user’s clustering perspective, while pairwise predictions and a consistency measure determine cluster granularity.The pairwise task uses few-shot demonstrations to judge whether sentences belong to the same category.Source: §6 · p. 9
  • Experiments report improved clustering quality and reasonable cluster granularity at negligible cost.Source: §6 · p. 9
  • CLUSTERLLM still relies on an embedding model, which the authors describe as inefficient and inapplicable to black-box embedding models.They encourage future work on model-free training, such as constrained clustering.Source: §6 · p. 9

Limitations

The authors identify reliance on pretrained embedders, fine-tuning cost, and suboptimal domain-discovery performance as limitations. The domain-discovery issue is reported for Massive(D) and CLINC(D).

Limitations

  • Finding informative data requires a pretrained embedder to identify high-entropy clustering assignments.The authors suggest combining self-supervised and LLM-predicted triplets to address this reliance.Source: Limitations · p. 10
  • Fine-tuning adds computational cost; constrained clustering was not adopted because unstable training would be heavily affected by LLM prediction errors.The authors hope to reduce fine-tuning cost in future work.Source: Limitations · p. 10
  • On domain-discovery datasets Massive(D) and CLINC(D), performance is usually suboptimal compared with the original Instructor embedding.Source: Limitations · p. 10

Ethics Statement

The appendices describe API data-privacy concerns, a scaling approach for hierarchical clustering, granularity-selection analyses, and embedder and fine-tuning settings.

Ethics Statement

  • Uploading privacy-sensitive data to OpenAI APIs may be risky for some applications and may require removing sensitive information.Source: Ethics Statement · p. 10

A Details of Scaling up Hierarchical Clustering

  • To scale hierarchical clustering, the method first runs mini-batch K-means with kmax, then applies Ward agglomerative clustering and a nearest-neighbor-chain algorithm to current assignments.This is presented as a way to address hierarchical clustering’s O(N^3) time complexity when only a specific granularity range is needed.Source: A Details of Scaling up Hierarchical… · p. 14

B More Details about Determining Cluster Granularity

  • Granularity baselines use the same fine-tuned embeddings and hierarchy, while the proposed method’s F-beta weight is 0.92, selected empirically on Bank77.Figure 4 plots consistency against cluster count with GPT-4 and λ = 1; sampling favors finer-granularity positives and coarser-granularity negatives.Source: B More Details about Determining… · p. 14, Figure 4 · p. 14

C Analysis for Determining Granularity

D Details of Embedders and Fine-tuning

  • Experiments use large Instructor and E5 embedders; fine-tuning follows Su et al. (2022) hyperparameters with a 2e−6 learning rate and specified training limits.SCCL uses maximum token length 128, with learning rates 2e−6 for Instructor and 2e−7 for E5; length 512 did not improve performance.Source: D Details of Embedders and Fine-tuning · pp. 14–15

E Description of Datasets

The paper adapts datasets originally used for classification and other tasks into clustering benchmarks, with task-specific filtering and splits for large- and small-scale evaluation.

E Description of Datasets

  • Bank77 covers fine-grained intents in banking, while CLINC includes multiple domains and supports an in-domain intent dataset and a domain-labeled variant.Source: E Description of Datasets · p. 15
  • For several datasets, rare intents or topics are removed and English-only data is retained; train and test sets serve as large- and small-scale versions.FewRel and FewEvent are randomly split before scale assignment, while FewNerd uses its original splits; StackEx, Reddit, and ArxivS2S splits are combined before filtering and random scale assignment.Source: E Description of Datasets · p. 15
  • Figure 5 reports relative clustering accuracy over four iterations, and Table 9 presents dataset class-distribution entropy.Source: Figure 5 · p. 15, E Description of Datasets · p. 15

F Results of More Iterations

Across four iterations, performance generally rises and then saturates at the fourth iteration, except on GoEmo; GPT-3.5 predictions are almost always beneficial.

F Results of More Iterations

  • Performance generally increases and saturates by iteration four, except on GoEmo.The authors compare iterative fine-tuning with GPT-3.5 predictions against self-supervised training initialized from the GPT-3.5-fine-tuned checkpoint.Source: F Results of More Iterations · p. 15
  • Using GPT-3.5 predictions is almost always beneficial during iterative fine-tuning.Triplets are sampled from the previously fine-tuned embedding space, and the previous checkpoint initializes each iteration.Source: F Results of More Iterations · p. 15

G More Related Works

The related-work discussion contrasts CLUSTERLLM’s minimal-supervision clustering setting with Generalized Category Discovery and prior LLM-feedback work. The accompanying appendix material covers prompts and additional clustering analyses.

More Related Works

  • Generalized Category Discovery assumes annotated known classes and sufficient annotated and unlabeled training data, whereas CLUSTERLLM studies minimal supervision.Prior GCD work uses pseudo-labeling through optimal transport, similarity learning, or prototype-based learning; new intent and relation type discovery address related settings.Source: G More Related Works · p. 15
  • The appendix includes prompt designs for Instructor and triplet tasks, granularity prompts, and an example covering both triplet and pairwise tasks.It also presents inferred granularity and class-distribution entropy for large-scale datasets.Source: Table 7 · p. 16, Table 8 · p. 16, Table 10 · p. 17
  • Prior work uses instruction-tuned LLMs to reproduce or improve human labels and to fine-tune models with LLM feedback, while this paper focuses on clustering..Source: G More Related Works · p. 17

H Sub-optimal Performance on Domain Discovery

CLUSTERLLM offers limited or reduced performance on several domain-discovery datasets, and the authors consider dataset leakage unlikely to be a primary concern. They hypothesize that tighter, more compact embeddings may suit fine-grained rather than coarse-grained clusters.

H Sub-optimal Performance on Domain Discovery

  • On MTOP(D), Massive(D), and CLINC(D), CLUSTERLLM barely improves or can decrease performance; GT&GPT-3.5 also trails self-supervision or random on CLINC(D) and MTOP(D).CLUSTERLLM-I-iter further decreases performance on Massive(D).Source: H Sub-optimal Performance on Domain… · p. 17
  • The authors hypothesize that fine-tuning makes embeddings more compact, favoring fine-grained clusters over coarse-grained ones.They report tighter, more separated embedding-space clusters, especially on CLINC(D).Source: H Sub-optimal Performance on Domain… · p. 17

I Dataset Leakage

  • Although web-scraped evaluation data may have appeared in LLM training, the authors say leakage is not a primary concern because several datasets’ labels were inaccessible during training.They identify Bank77 as synthesized, and FewRel and GoEmo labels as not accessible during training.Source: I Dataset Leakage · p. 17
Loading 2305.14871v2…