Image Clustering with Vision-Language Model Embeddings

• The recovered category structure depends on what a representation emphasizes, since images can share backgrounds, shapes, or colors while depicting different objects.A single collection can also support several meaningful partitions.Source: §1 · p. 2
• Pretrained vision–language models provide reusable representations through visual features or generated descriptions, while prior work also explores prompts and embedding-specific training.These routes yield vectors with different information available to downstream clustering.Source: §1 · p. 2
• The study compares nine representation pipelines across seven image datasets and four clustering configurations, treating each specified extraction–clustering pipeline as the unit of comparison.The comparison includes direct visual features, vision-only references, and a caption-to-text route.Source: §1 · p. 3
• The study addresses comparisons across representations, clustering algorithms, caption-mediated versus direct visual features, and per-image L2 normalization.Its baseline grid contains 252 combinations; the normalization analysis contains 63 paired comparisons at unchanged dimensionality.Source: §1 · p. 3

2.1. Visual and vision–language representations

• Vision–language and vision-only models provide distinct visual representations, but their checkpoint, training, and extraction differences prevent attributing performance differences solely to language supervision.The evaluated BLIP-2 vector averages visual hidden states, not language-model states or generated captions.Source: §2.1 · pp. 3–4

2.2. VLMs trained as embedding models

• VLM2Vec and mmE5 learn embeddings for embedding tasks, a different approach from extracting features from pretrained generative models.These dedicated embedding methods involve additional training outside this study’s scope.Source: §2.2 · pp. 4–5

2.3. Representation learning and adaptation for image clustering

• Image-clustering approaches can learn assignments or modify features, whereas this study applies conventional clustering to fixed representations and separately tests L2 normalization [17].The compared alternatives include learned clustering heads, encoder adaptation, and feature augmentation.Source: §2.3 · p. 5

2.4. Language-guided image clustering

• Language-guided methods use descriptions, prompts, or cross-modal learning for clustering, while this study compares caption embeddings with direct visual features using conventional algorithms [8, 11, 12].Its evaluation measures agreement with supplied labels, not user-defined alternative groupings; Table 1 organizes selected approaches by representation or grouping strategy, additional processing, and relationship to this study.Source: §2.4 · pp. 5–6, Table 1 · p. 7

3. Methods

• Each of nine representations is evaluated independently with each clustering method, without combining model embeddings, across a workflow of 252 baseline combinations.The study also includes a separate L2-normalization sensitivity analysis.Source: Figure 1 · p. 8, §3 · p. 8

3.1. Datasets

• The seven test benchmarks span handwritten digits, characters, clothing, and natural images, with complete test partitions used for feature extraction and clustering.Training partitions are not used in these experiments.Source: §3.1 · p. 9

3.2. Image preprocessing and feature extraction

• Images use each model’s associated preprocessing, and locally executed models generally form embeddings by averaging selected visual-token representations.The included procedures differ by architecture: for example, CLIP averages patch states before projection, while Qwen3-VL averages all returned image-feature tokens.Source: §3.2 · pp. 9–10

3.3. Caption-mediated representation

• Caption + text generates one short description per image and embeds it as a 1,536-dimensional text vector, with dataset-specific prompts supplying semantic guidance.Caption generation is included in extraction timing, and recognition accuracy is not separately evaluated.Source: §3.3 · pp. 10–11
• The comparison evaluates complete pipelines because caption prompts provide task-specific guidance that direct visual pipelines do not receive.Generation uses API defaults without explicit temperature or seed control, so deterministic captions are not assumed.Source: §3.3 · p. 11

3.4. Clustering algorithms and configurations

• Four configurations comprise K-means with random or K-means++ initialization, fuzzy c-means, and AHC with Ward linkage; each uses Euclidean distance and oracle K.The K-means variants differ in both initialization and restart budget, and each stochastic configuration is evaluated once without estimating run-to-run variability.Source: §3.4 · pp. 11–12
• Ward linkage merges the pair producing the smallest increase in within-cluster sum of squares, then cuts the hierarchy at the prespecified cluster count.Fuzzy memberships are converted to a hard partition by assigning each image to its highest-membership cluster for evaluation.Source: §3.4 · p. 12

3.5. Evaluation and aggregation

• ARI is the primary external metric, while V-measure, greedy F1, and silhouette provide complementary assessments; silhouette does not rank representations.Silhouette assesses cohesion and separation in each representation’s embedding space.Source: §3.5 · pp. 12–13
• Across datasets, the analysis reports representation ranks and ARI summaries, paired differences, per-configuration averages, and retrospectively selected best-observed combinations.Dataset averages weight the seven datasets equally; averages across configurations weight each of the four configurations equally.Source: §3.5 · p. 13

3.6. Sensitivity to L2 normalization

• L2 normalization gives each vector unit Euclidean norm while preserving its direction and dimensionality; the clustering settings remain identical across conditions.The analysis adds 126 fits: 63 raw-vector controls and 63 normalized-vector conditions.Source: §3.6 · p. 14
• For each dataset–representation pair, the sensitivity analysis compares ARI from normalized embeddings with its raw-vector control under identical conditions.Mean differences weight each of the seven datasets equally.Source: §3.6 · p. 14

3.7. Visualization and recorded extraction time

• PCA retaining 99% of variance preceded two-dimensional t-SNE for visualization; the projections were not used for clustering or evaluation.The t-SNE settings included perplexity 30, learning rate 150, angle 0.2, and random seed 42.Source: §3.7 · p. 14
• Extraction times began after model loading and included image processing and feature extraction; caption-pipeline times also included generation, embedding requests, and network latency.Reported times are elapsed extraction time per image under the respective local and remote execution conditions.Source: §3.7 · p. 14

3.8. Consistency and reproducibility checks

• Checks covered all 781,551 baseline embeddings and confirmed valid vectors, matching sample counts and dimensions, consistent labels, and finite metrics across 252 combinations.Source: §3.8 · p. 15

3.9. Computational platform

• The local experiments used an NVIDIA DGX Spark GB10 system, while OpenAI caption and text-embedding models were accessed through a remote API.The software environment and library versions were kept the same across baseline and sensitivity experiments.Source: §3.9 · p. 15

4.1. Representation performance with fixed clustering

• With AHC-Ward fixed, Qwen3-VL leads on five datasets and BLIP-2 on both CIFAR datasets, with ARIs of 0.925 on MNIST and 0.925 on CIFAR-10, respectively.Qwen3-VL has the best mean rank (1.286); V-measure identifies the same winner on six datasets, with Fashion-MNIST favoring BLIP-2 (0.720) over Qwen3-VL (ARI 0.573).Source: Table 5 · p. 16, §4.1 · pp. 16–17

4.2. Effect of L2 normalization

• L2 normalization increases ARI in 35 of 63 dataset–representation pairs, decreases it in 26, and leaves two unchanged, while preserving the highest-ARI representation in every dataset.On SVHN, Qwen3-VL’s highest ARI changes from 0.097 on raw vectors to 0.098 after normalization.Source: §4.2 · pp. 18–19, Table 8 · p. 19

4.3. Effect of clustering configuration

• AHC-Ward has the highest mean ARI for eight of nine representations; random-initialized K-means is the exception for Qwen3-VL, increasing its mean ARI from 0.571 to 0.588.Random-initialized K-means uses ten initializations, whereas K-means++ uses one, so these differences describe the tested configurations rather than initialization strategies alone.Source: §4.3 · pp. 19–20, Table 9 · p. 20
• Among 36 evaluated representation–configuration combinations per dataset, the highest observed ARI is 0.970 for BLIP-2 with random-initialized K-means on CIFAR-10.These maxima were selected retrospectively using reference labels; the best configuration varies by dataset.Source: §4.3 · p. 21, Table 10 · p. 21

4.4. Direct visual versus caption-mediated representations

• The caption-mediated pipeline has AHC-Ward ARIs of 0.793 on MNIST and 0.694 on USPS, but does not consistently exceed direct visual features across datasets.The study does not determine the relative contributions of caption recognition errors, wording variation, or shared attributes to weaker results in other domains.Source: §4.4 · pp. 21–22

4.5. Embedding dimensionality, extraction time, and visualization

• Qwen3-VL and LLaVA both use 4,096-dimensional embeddings but have mean ARIs of 0.447 and 0.145; caption-mediated extraction takes approximately 1,314 milliseconds per image, versus 10.40 for Qwen3-VL and 66.07 for BLIP-2.The timing comparison reflects different local and API execution conditions; monetary API expenditure and pretraining costs are excluded.Source: §4.5 · pp. 22–23, Table 11 · p. 22
• The selected PCA–t-SNE projections illustrate Qwen3-VL on MNIST versus SVHN and BLIP-2 versus caption-mediated embeddings on CIFAR-10, but do not provide a basis for ranking representations.Their independently projected and rescaled coordinates can alter apparent separation; colors show reference classes, not predicted clusters.Source: §4.5 · p. 23, Figure 2 · p. 24

4.6. Synthesis of findings

• The findings favor evaluating complete representation–clustering pipelines in the intended domain rather than relying on model characteristics or one benchmark.AHC-Ward generally has the highest dataset-averaged ARI, fuzzy c-means performs poorly on original embeddings, and the caption-mediated route incurs substantially greater extraction time.Source: §4.6 · pp. 23–25

5. Limitations

• The comparisons cover specified extraction paths and clustering settings; other layers, pooling rules, implementations, or dimensionality-reduction choices may yield different outcomes.The L2 sensitivity analysis fixes AHC-Ward and tests vector-length normalization, not other algorithms or feature-wise scaling.Source: §5 · p. 25
• Each baseline configuration was evaluated once, so stochastic-initialization sensitivity was not estimated and clustering methods were not compared under equally tuned conditions.The highest-ARI combinations were selected retrospectively using test labels, and the seven benchmark ranks are descriptive rather than independent repetitions.Source: §5 · p. 25
• Known class counts and relatively small benchmark images leave unknown-K discovery, open-world categories, and larger real-world collections outside the demonstrated scope.Possible pretraining overlap and covarying dataset domain and class count further constrain generalization and attribution.Source: §5 · p. 26
• Because caption recognition accuracy was not assessed separately, downstream errors cannot be assigned specifically to recognition, verbalization, or text encoding.The caption pipeline also receives dataset-specific semantic guidance absent from direct visual extraction.Source: §5 · p. 26

6. Conclusions

• The study compared nine pretrained representation pipelines across seven image datasets and four clustering configurations, for 252 combinations.The pipelines included direct visual features and generated descriptions encoded as text.Source: §6 · p. 26
• With AHC-Ward fixed, Qwen3-VL led in ARI on five datasets, while BLIP-2 led on CIFAR-10 and CIFAR-100; no representation led every evaluated setting.Qwen3-VL also had the best mean rank in all seven datasets.Source: §6 · p. 26
• The cited finding distinguishes digit images by context: clustering agreement was high for isolated handwritten digits, but was not comparable for digits appearing in street-view imagery.Source: §6 · p. 27
• Across the seven datasets, L2 normalization preserved the highest-ARI representation, while ARI increased in 35 pairs, decreased in 26, and was unchanged in two.LLaVA and Qwen3-VL improved on all seven datasets.Source: §6 · p. 27
• AHC-Ward achieved the highest dataset-averaged ARI for eight of nine representations, while random-initialized K-means led for Qwen3-VL.Fuzzy c-means performed substantially worse overall; DINOv3 had its highest mean ARI with that method.Source: §6 · p. 27
• The evaluated caption-mediated pipeline agreed more on isolated handwritten digits than on clothing, mixed characters, or natural objects, but did not consistently exceed direct visual representations.These findings apply to the evaluated captioning model, text encoder, and prompts; recognition errors and lost visual distinctions are proposed explanations requiring separate assessment.Source: §6 · p. 27
• The observed rankings and dataset wins provide a descriptive basis for selecting candidate representation–clustering pipelines within the evaluated settings.The authors identify initialization, parameter sensitivity, extraction, pooling, normalization, dimensionality reduction, and captioning choices for future study.Source: §6 · pp. 27–28

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

• ChatGPT supported language editing, text flow, limited technical clarifications, and suggestions on organization and draft wording.The reported technical clarifications concerned LaTeX and Python code.Source: Declaration of generative AI and… · p. 28
• The authors state that scientific ideas, model choices, experimental rationale, data analysis, interpretation, and conclusions were developed by them.Source: Declaration of generative AI and… · p. 28
• The authors report reviewing, editing, and where appropriate substantially modifying AI-assisted content, and take full responsibility for the final manuscript.Source: Declaration of generative AI and… · p. 28

Appendix A. Complementary V-measure results for AHC-Ward clustering

• The appendix reports V-measure for AHC-Ward across nine representations and seven datasets in Table A.12 and Table A.13.These results correspond to the ARI results in Tables 5 and 7, respectively.Source: Appendix A · p. 35
• Table A.12 gives V-measure with AHC-Ward on the original embeddings, with bold values marking the best representation for each dataset.Source: Table A.12 · p. 35
• Table A.13 gives V-measure after per-image L2 normalization while retaining original embedding dimensions; bold marks each dataset’s highest value before rounding.Source: Table A.13 · p. 35

• Tables B.14–B.17 cover all clustering configurations on MNIST, Fashion-MNIST, EMNIST, and USPS, reporting ARI, V-measure, weighted F1 (greedy), and silhouette.Source: Appendix B · p. 36, Table B.14 · p. 37, Table B.15 · p. 38
• Tables B.18–B.20 report the same four metrics across all clustering configurations for SVHN, CIFAR-10, and CIFAR-100.Source: Appendix B · p. 36, Table B.18 · p. 41, Table B.19 · p. 42
• The detailed tables complement the aggregate results presented in Section 4.3.Source: Appendix B · p. 36
• Table B.14 marks pre-rounding column maxima for ARI, V-measure, and greedy-mapping F1; silhouette is not compared across feature spaces.Source: Table B.14 · p. 37