PaperlayerView exact revision on arXiv

Source-linked AI summary

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu, Yadan Luo, Wenjie Zhu

arXiv:2609.36648v1cs.CV

TL;DR

Image clustering lacks a standardized benchmark for assessing progress amid inconsistent experimental settings, dataset selection, and evaluation dimensions. VLM4Cluster benchmarks representative methods across diverse datasets, and its results show that language-assisted clustering advances the performance frontier in many dataset–metric comparisons, though gains are less pronounced at scale.

  • Problem

    Without a standardized benchmark, inconsistent settings, inadequate dataset selection, and limited evaluation dimensions make systematic assessment of image-clustering progress difficult.

    Source: §1 · p. 2

  • Method

    VLM4Cluster provides a reproducible benchmark integrating 17 representative clustering methods and evaluating them on 20 datasets spanning five settings.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    LaIC advances the performance frontier in 33 of 45 dataset–metric comparisons, but its advantage is less pronounced on large-scale datasets and it does not systematically reduce relative robustness declines.

    Source: §3.2 · pp. 5–6, §3.3 · p. 8

  • Takeaways & Limitations

    VLM4Cluster is intended to serve as a foundation for continued development in image clustering in the era of pre-trained VLMs.

    Source: §4 · p. 9

    Many augmentation-dependent deep clustering methods are not directly applicable in their original form to a benchmark operating on features extracted from pre-trained VLMs.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision-language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.

1 INTRODUCTION

Visual-only clustering can miss semantic differences between visually similar images, while the rise of language-assisted methods has not been matched by a standardized benchmark. VLM4Cluster addresses this gap with reproducible comparisons across methods, datasets, and evaluation dimensions.

1 INTRODUCTION

  • Visual-only self-supervision is limited when images look similar but differ semantically, motivating LaIC frameworks that use textual semantics to guide clustering.VLMs map visual and textual inputs into a shared embedding space, enriching image representations with cross-modal correspondences.Source: §1 · p. 2
  • The lack of a standardized benchmark, inconsistent experimental settings, inadequate dataset selection, and effectiveness-focused evaluation leave image-clustering progress difficult to assess systematically.Prior comparisons may use different dataset splits, backbones, and image resolutions; commonly used datasets may also be outdated or insufficiently challenging.Source: §1 · p. 2
  • VLM4Cluster provides a unified codebase for 17 representative methods and evaluates them on 20 datasets spanning five dataset categories.The methods cover classical, deep, and language-assisted clustering; the dataset categories are classical, challenging, fine-grained, large-scale, and out-of-distribution.Source: §1 · p. 2
  • The benchmark extends evaluation beyond effectiveness to adversarial robustness, distribution-shift generalization, and computational efficiency, and includes an open-source platform and leaderboard.The platform and leaderboard support comparison across methods, datasets, and VLMs.Source: §1 · p. 2

2 BENCHMARK DESIGN

VLM4Cluster organizes a unified image-clustering benchmark around representative algorithms, diverse datasets and VLM configurations, and four evaluation dimensions. Its design addresses gaps in existing image-clustering benchmarks, including limited method coverage, dataset difficulty, evaluation metrics, and reproducibility.

2 BENCHMARK DESIGN

  • The benchmark integrates 17 clustering algorithms, 20 datasets, VLM configurations, and evaluation across effectiveness, robustness, generalization, and efficiency.Its design is organized around algorithms, datasets, VLMs, evaluation strategy, and comparison with related benchmarks.Source: §2 · p. 2, Table 1 · p. 3

2.1 ALGORITHMS

  • The 17 selected image-clustering methods span three paradigms: four classical methods (k-means, Spectral Clustering, EnSC, SSC-OMP), five deep methods (IDC, SCAN, CPP, TEMI, PRO-DSC), and eight language-assisted methods.Many deep methods that rely heavily on input-space augmentations are not directly applicable in their original form to a benchmark using pre-trained VLM features rather than raw images.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 DATASETS

  • The 20 datasets cover classical, challenging, fine-grained, large-scale, and out-of-distribution clustering scenarios.The suite includes five classical, three challenging, five fine-grained, two large-scale, and five out-of-distribution datasets.Source: §2.2 · p. 3

2.3 VISION LANGUAGE MODELS

  • The benchmark uses OpenCLIP VLMs with ViT-B/32, ViT-B/16, or ViT-L/14 visual encoders, trained on LAION-400M or LAION-2B with CLIP or SigLIP objectives.The configurations pair the visual encoders with correspondingly scaled text encoders.Source: §2.3 · p. 4

2.4 EVALUATION DIMENSIONS

  • Evaluation covers effectiveness, adversarial robustness, generalization under distribution shifts, and computational efficiency.The benchmark associates each dimension with an evaluation protocol designed for image clustering.Source: §2.4 · p. 4

2.5 DISCUSSION ON EXISTING BENCHMARKS

  • Existing image-clustering benchmarks are limited by relatively simple datasets, few evaluation metrics, incomplete paradigm coverage, and a lack of reproducible benchmarking toolboxes for the VLM era.The discussion identifies ClustPy and CLUBench as exceptions to the relatively limited attention given to image-clustering benchmarks.Source: §2.5 · p. 4

3 EXPERIMENTS

The experiments compare language-assisted image clustering (LaIC) with vision-only methods across effectiveness, adversarial robustness, distribution-shift generalization, and computational efficiency. LaIC advances the performance frontier on many benchmarks and generally generalizes better under shifts, but its gains are less consistent at scale and on fine-grained data, and it does not systematically reduce adversarial sensitivity.

3.2 EFFECTIVENESS ANALYSIS (RQ1)

  • LaIC advances the performance frontier in 33 of 45 dataset–metric comparisons, outperforming vision-only methods on 11/15 datasets for NMI, 12/15 for ACC, and 10/15 for ARI.It establishes a new best across all three metrics on 9/15 datasets.Source: §3.2 · p. 5, Table 3 · p. 6, Table 4 · p. 6
  • At large scale and on fine-grained datasets, LaIC gains are less consistent, with LaIC trailing vision-only results on all three metrics for Cars and Food.On ImageNet-1K and Places-365, the reported differences are small; Flowers shows an 8.0-point ACC gain.Source: §3.2 · pp. 6–7

3.3 ROBUSTNESS ANALYSIS (RQ2)

  • LaIC methods generally do not show smaller relative performance drops under adversarial attack, despite often matching or exceeding vision-only post-attack performance.TAC, TAC++, and NTK-SC match or surpass the best vision-only results on all three post-attack metrics.Source: Figure 1 · p. 7, Figure 2 · p. 7, §3.3 · p. 8

3.4 GENERALIZATION ANALYSIS (RQ3)

  • LaIC generally achieves stronger averaged out-of-distribution performance and effective robustness, but no single method generalizes uniformly across shifts.TAC leads average OOD NMI at 69.0%; NTK-SC leads average ACC and ARI at 38.5% and 25.5%.Source: §3.4 · p. 8, Table 6 · p. 9

3.5 EFFICIENCY ANALYSIS (RQ4)

  • On ImageNet-1K, LaIC methods achieve a more favorable trade-off between total pipeline time and clustering effectiveness.Figure 3 compares methods by time consumption and NMI.Source: §3.5 · p. 9, Figure 3 · p. 9

4 CONCLUSION

VLM4Cluster provides a unified benchmark for evaluating image clustering methods across paradigms, datasets, and evaluation dimensions. The supplied background and method passages define clustering and outline representative classical, deep, and language-assisted approaches.

4 CONCLUSION

  • VLM4Cluster evaluates 17 representative methods across three paradigms and four evaluation dimensions on 20 diverse datasets.The benchmark is presented as a unified framework for systematic evaluation.Source: §4 · p. 9

A PRELIMINARY

  • Image clustering groups unlabeled images into K semantically distinct clusters, while language-assisted clustering does so under guidance from wild textual data.The ideal is for each cluster to align with a class.Source: A PRELIMINARY · p. 15
  • Wild textual data may mix positive and negative semantics, motivating its modeling with a Huber contamination model.The empirical wild dataset is described as an i.i.d. sample from that distribution.Source: A PRELIMINARY · p. 15

B.1 CLASSICAL IMAGE CLUSTERING

  • Classical methods include k-means, spectral clustering, and sparse self-expression methods whose coefficients can define affinities for spectral clustering.The spectral-clustering passage defines a normalized Laplacian and its top-K minimum-eigenvector embedding.Source: §B.1 · p. 16

B.2 DEEP IMAGE CLUSTERING

  • Deep methods described include IDC's user-feedback-based clustering head, SCAN's clustering head, and projection-based methods such as CPP, TEMI, and PRO-DSC.TEMI uses student–teacher self-distillation; CPP and PRO-DSC learn projected embeddings and cluster-membership structures.Source: §B.2 · pp. 16–17

B.3 LANGUAGE-ASSISTED IMAGE CLUSTERING

  • LaIC methods are categorized as post-hoc approaches using off-the-shelf clustering or training-based approaches learning parametric clustering heads.Source: §B.3 · p. 18

B.3.1 POST-HOC METHODS

Post-hoc LaIC methods form clustering assignments from image features combined with text-derived counterparts or affinities. Training-based approaches instead learn clustering heads, with the supplied passages describing several ways to refine text features or use image-text counterparts.

B.3.1 POST-HOC METHODS

  • TAC and GradNorm apply k-means to concentrated image features paired with text counterparts; GradNorm filters text features using pseudo-class assignments and gradient magnitudes.TAC derives text counterparts by filtering ambiguous text features and aggregating selected features by similarity.Source: §B.3.1 · pp. 18–19
  • NTK-SC replaces RBF affinities with Neural Tangent Kernel affinities and applies spectral clustering to the resulting affinity matrix.Source: §B.3.1 · p. 19

B.3.2 TRAINING-BASED METHODS

  • Training-based examples include SIC, SAC, MAGIC, TAC++, and SEIC, which learn clustering heads using image features and, in several methods, textual counterparts.SIC constructs a refined text-feature set through prototype-based filtering; SAC and MAGIC use TAC-style counterparts here for fair comparison.Source: §B.3.2 · pp. 20–22

C.1 CLASSICAL DATASETS

The dataset collection spans classical, challenging, fine-grained, large-scale, and out-of-distribution image clustering settings.

C.1 CLASSICAL DATASETS

  • CIFAR-10 is a classical visual-recognition dataset with 60,000 images across 10 classes, split into 50,000 training and 10,000 test images, with 6,000 images per class.CIFAR-20 is a coarse-label re-labeling of CIFAR-100 that collapses its 100 categories into 20 superclasses, not a separate dataset.Source: §C.1 · p. 22

C.2 CHALLENGING DATASETS

  • The study uses DTD, a texture-attribute dataset with 5,640 images across 47 classes; UCF-101 has 13,320 clips across 101 action classes; CIFAR-100 has 60,000 images across 100 classes.For UCF-101, the middle frame of each video clip is used as the input image.Source: §C.2 · pp. 22–23

C.3 FINE-GRAINED DATASETS

  • Aircraft, Food, Pets, Cars, and Flowers comprise the fine-grained group, with datasets ranging from 37 to 196 categories.Aircraft distinguishes visually similar types, while Cars distinguishes car makes, models, and years.Source: §C.3 · p. 23

C.4 LARGE-SCALE DATASETS

  • Places365 and ImageNet-1K form the large-scale group, with approximately 1.8 million and 1.28 million training images, respectively.Places365 spans 365 scene categories, while ImageNet-1K spans 1,000 object categories.Source: §C.4 · p. 23

C.5 OUT-OF-DISTRIBUTION DATASETS

  • ImageNet-C, ImageNet-V2, ImageNet-S, ImageNet-A, and ImageNet-R are used to evaluate generalization under distribution shifts.These datasets include Gaussian-noise images, independently collected reproductions, sketches, naturally occurring hard examples, and artistic-domain images.Source: §C.5 · p. 23

D METRIC DETAILS

The section defines external and internal clustering metrics, including measures of label agreement, cluster structure, and out-of-distribution generalization.

D METRIC DETAILS

  • ACC measures agreement between predicted cluster assignments and ground-truth labels.The accompanying definition describes optimal label alignment with the Hungarian algorithm.Source: D METRIC DETAILS · pp. 23–24
  • NMI measures information shared between clustering labels and ground-truth labels, ranging from 0 for complete irrelevance to 1 for perfect consistency.It is also used when cluster counts differ from ground-truth category counts.Source: D METRIC DETAILS · p. 24
  • ARI measures agreement across sample pairs, correcting the Rand Index for chance so random clustering is near 0 and perfect clustering is 1.Its computation uses a contingency table mapping true classes to predicted clusters.Source: D METRIC DETAILS · p. 24
  • Effective Robustness measures OOD clustering accuracy relative to the level predicted from in-distribution performance by a baseline trend.The baseline predicts OOD performance from a method’s single ID performance.Source: D METRIC DETAILS · pp. 24–25
  • SC measures how well samples fit their assigned clusters relative to the nearest alternative, combining cohesion and separation on a scale from −1 to 1.Values near 1 indicate good matching and separation; values near 0 indicate a boundary, and negative values suggest possible misassignment.Source: D METRIC DETAILS · p. 25
  • DBI evaluates internal cluster quality through worst-case similarity ratios, while CHI compares between-cluster dispersion with within-cluster dispersion.Lower DBI and higher CHI correspond to tighter, better-separated clusters.Source: D METRIC DETAILS · p. 25

E EXPERIMENTAL SETUP

The experimental setup standardizes implementations, splits, runs, and image preprocessing, and provides hyperparameter-search spaces and supplemental result tables.

E EXPERIMENTAL SETUP

  • Classical methods are trained and evaluated on test splits, while deep and LaIC methods are fit on training splits and evaluated on test splits.Results average 10 independent runs; images are resized to 224×224 and seven WordNet prompt templates are used for LaIC.Source: E EXPERIMENTAL SETUP · p. 26
  • Hyperparameters are selected by grid search over the spaces detailed in Table 8.The authors state that this exhaustive tuning maximizes each method’s performance for comparison.Source: E EXPERIMENTAL SETUP · p. 26, Table 8 · p. 26
  • Table 9 presents classical-dataset clustering results for CLIP ViT-B/16 pretrained on LAION-400M, highlighting the best result in each metric column.Experiments primarily used a Linux server with an AMD EPYC 9254 CPU, 188GB RAM, and an NVIDIA RTX 2080 Ti GPU.Source: E EXPERIMENTAL SETUP · p. 25, Table 9 · p. 27

F ADDITIONAL RESULTS ON EFFECTIVE ANALYSIS

Supplemental clustering results are reported across four pretrained vision-language model configurations.

F ADDITIONAL RESULTS ON EFFECTIVE ANALYSIS

G DETAILED RESULTS ON ROBUSTNESS ANALYSIS

Supplementary tables report post-attack clustering results across classical, challenging, large-scale, and fine-grained datasets.

G DETAILED RESULTS ON ROBUSTNESS ANALYSIS

  • Post-attack clustering results are reported for classical, challenging, large-scale, and fine-grained datasets.These tables supplement the results in Table 25.Source: G DETAILED RESULTS ON ROBUSTNESS… · p. 27
  • Table 10 reports clustering results on challenging datasets using CLIP ViT-B/16 pre-trained on LAION-400M.The caption identifies the best result in each metric column but supplies no values here.Source: Table 10 · p. 28
  • Table 11 reports clustering results on large-scale datasets using CLIP ViT-B/16 pre-trained on LAION-400M.The caption identifies the best result in each metric column but supplies no values here.Source: Table 11 · p. 28

H DISCRIMINABILITY ANALYSIS

The benchmark assesses cluster discriminability without class labels, then identifies limitations and research directions for fine-grained clustering, robustness, distribution shifts, and efficiency.

H DISCRIMINABILITY ANALYSIS

  • Discriminability analysis uses SIL, DBI, and CHI to assess cluster separation without class labels or external ground truth.Results are reported for classical, challenging, large-scale, and fine-grained datasets.Source: H DISCRIMINABILITY ANALYSIS · p. 28

I.1 FUTURE DIRECTION

  • Language-assisted clustering gains are less consistent on fine-grained datasets, where textual semantics may miss subtle distinctions between visually similar categories.The authors propose combining detailed visual cues with more discriminative linguistic knowledge, including dynamically generated descriptions.Source: §I.1 · p. 29
  • Language assistance generally improves post-attack clustering performance but does not systematically reduce sensitivity to adversarial perturbations.How perturbations affect visual representations and their alignment with textual semantics remains open.Source: §I.1 · p. 29
  • LaIC methods generally generalize more strongly across distribution shifts, but no single method performs consistently well across all evaluated shifts.The authors call for retaining transferable semantic information while reducing reliance on distribution-specific cues; evaluation of long-tailed and specialized domains would clarify applicability.Source: §I.1 · pp. 29–30
  • LaIC offers a favorable effectiveness–efficiency trade-off, yet its performance advantages become less consistent at scale.Preserving language assistance as dataset size and category diversity increase remains an open question.Source: §I.1 · p. 30

I.2 LIMITATION AND OUTLOOK

The authors identify limits in dataset coverage and fixed lexical resources, outline planned extensions, and report that the study used only public data and models.

I.2 LIMITATION AND OUTLOOK

  • The evaluation primarily covers class-balanced natural image datasets, and fixed lexical databases such as WordNet may limit semantic coverage and adaptability for specialized or emerging concepts.The authors plan to broaden dataset coverage and incorporate more flexible, dynamically generated linguistic knowledge, such as MLLM-provided knowledge.Source: §I.2 · pp. 30–31
  • Planned dataset extensions include long-tailed and medical images, alongside more flexible linguistic knowledge such as that provided by MLLMs.Source: §I.2 · p. 31

K ETHICS STATEMENT

  • The study relies solely on publicly available datasets and models and uses no private or personally identifiable information.Source: K ETHICS STATEMENT · p. 31

Related papers

Loading 2609.36648v1…