PaperlayerView exact revision on arXiv

Source-linked AI summary

Supporting Clustering with Contrastive Learning

Dejiao Zhang, Feng Nan, Xiaokai Wei, Shangwen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew Arnold, Bing Xiang

arXiv:2103.12953v2cs.LGcs.CL

TL;DR

Distance-based clustering can struggle when semantic categories overlap in representation space. SCCL jointly combines top-down clustering with bottom-up instance-wise contrastive learning, and it advances results on most short-text clustering benchmarks while improving cluster distances under ground-truth evaluation.

  • Problem

    Overlapping semantic categories make distance-based clustering unable to achieve good separation in the representation space.

  • Method

    SCCL jointly optimizes a top-down clustering loss with a bottom-up instance-wise contrastive loss.

  • Results

    SCCL substantially outperforms or performs highly comparably to state-of-the-art methods on eight benchmark short text clustering datasets.

  • Takeaways & Limitations

    Combining instance discrimination with clustering yields high-quality clusters with better intra-cluster and inter-cluster distances under ground-truth evaluation.

  • Takeaways & Limitations

    Effective textual transformations remain challenging because natural language is discrete; data mixing is left for future work.

Abstract

from arXiv · show

Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning process, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) -- a novel framework to leverage contrastive learning to promote better separation. We assess the performance of SCCL on short text clustering and show that SCCL significantly advances the state-of-the-art results on most benchmark datasets with 3%-11% improvement on Accuracy and 4%-15% improvement on Normalized Mutual Information. Furthermore, our quantitative analysis demonstrates the effectiveness of SCCL in leveraging the strengths of both bottom-up instance discrimination and top-down clustering to achieve better intra-cluster and inter-cluster distances when evaluated with the ground truth cluster labels.

1 Introduction

SCCL addresses category overlap in representation space by jointly combining top-down clustering with bottom-up instance-wise contrastive learning for unsupervised short text clustering.

  • 1 Introduction: Distance-based clustering struggles when high-dimensional data overlap across semantic categories in the learned representation space.This overlap can produce clusters with poor purity.
  • 1 Introduction: Instance-wise contrastive learning separates different instances, while clustering explicitly tightens instances belonging to the same semantic cluster.The combination targets both inter-cluster separation and intra-cluster compactness.
  • 1 Introduction: SCCL jointly optimizes a top-down clustering loss and a bottom-up instance-wise contrastive loss.The framework uses contrastive learning to separate overlapped categories while clustering brings semantically related instances together.
  • 1 Introduction: SCCL is evaluated on short text clustering, where noise, sparsity, and weak signals make clustering difficult.The framework targets applications including topic discovery, recommendation, and visualization.
  • 1 Introduction: SCCL explores text augmentation strategies and finds that composing augmentations is not always beneficial in the text domain.This differs from the reported behavior in the image domain.

2 Related Work

Prior work uses self-supervised and neural representation learning to improve clustering, but instance-wise contrastive learning can ignore semantic similarity and short-text methods often require multistage training. SCCL combines clustering and contrastive objectives in an end-to-end framework and reports improved benchmark performance.

  • 2 Related Work: Instance-wise contrastive learning pulls together augmented views of each instance and pushes apart representations from different instances.It produces well-separated instances while preserving local invariance.
  • 2 Related Work: Instance-wise contrastive learning may separate semantically similar instances because it treats different original instances as distinct regardless of semantic similarity.Its implicit grouping of similar instances is therefore less stable and more data-dependent.
  • 2 Related Work: Short text clustering has weak signals, making BoW and TF-IDF representations sparse and less expressive.Earlier approaches use neural networks and word embeddings to enrich representations.
  • 2 Related Work: Existing short-text approaches often divide learning into multiple independently optimized stages, while SCCL optimizes a pretrained transformer end to end.SCCL uses contextualized representations that had been largely unexplored for short text clustering.

3 Model

SCCL jointly uses instance-wise contrastive learning and unsupervised clustering through separate heads over a shared representation, combining instance separation with semantic grouping.

  • Model architecture: The model maps inputs with ψ(·) and applies contrastive and clustering losses through separate heads g(·) and f(·).The neural network produces a representation before the two task-specific heads.
  • Instance-wise Contrastive Learning: Each original minibatch instance receives two augmentations, forming an augmented batch of size 2M for Instance-CL.The augmented pairs define positive examples, while the remaining 2M−2 examples serve as negatives.
  • Instance-wise Contrastive Learning: The Instance-CL objective separates augmented views of different instances while preserving similarity between views of the same instance.Similarity uses normalized-output dot products, with temperature τ set to 0.5.
  • Clustering: The clustering component represents semantic categories through cluster centroids and Student’s t-distribution assignment probabilities.A linear clustering head approximates the centroids, which are iteratively refined using an auxiliary target distribution.
  • Clustering: The auxiliary distribution sharpens high-confidence assignments and compensates for imbalanced cluster frequencies before KL-divergence optimization.The target raises assignment probabilities to the second power and normalizes them by estimated cluster frequency.
  • Overall objective: SCCL simplifies training to end-to-end optimization with the target distribution updated at every iteration, while balancing contrastive and clustering losses with η = 10.The clustering loss is optimized over original data, with augmented-data consistency discussed as an alternative.

4 Numerical Results

SCCL is evaluated on eight short-text datasets against established baselines and ablated variants, with results also examining representation distances and augmentation choices. It outperforms most baselines and benefits from jointly optimizing clustering and instance-contrastive objectives, while augmentation composition depends on dataset characteristics.

  • Comparison with State-of-the-art: SCCL outperforms all baselines by a large margin on most datasets, although it trails Hadifar et al. (2019) on Biomedical.Biomedical is less related to the general domains used to pretrain the transformers, whereas Hadifar et al. (2019) use in-domain biomedical embeddings and layer-wise autoencoder pretraining.
  • Ablation Study: SCCL consistently outperforms Instance-CL, Clustering, and SCCL-Seq, validating joint rather than sequential optimization.SCCL-Seq first trains with Instance-CL and then optimizes the clustering objective.
  • Cluster-level evaluation: SCCL produces better-separated and less-dispersed clusters than the component methods in ground-truth cluster evaluations.The analysis tracks intra-cluster distance to the centroid and inter-cluster distance to the closest neighboring cluster.
  • Exploration of Data Augmentations: Contextual Augmenter substantially outperforms WordNet augmenter and paraphrase via back translation among the tested augmentation techniques.The comparison summarizes the best results for each technique.
  • Exploration of Data Augmentations: Composition of augmentations improves SCCL on GoogleNews-TS but reduces performance on StackOverflow.GoogleNews-TS averages 28 words per instance, whereas StackOverflow averages 8; augmented StackOverflow pairs also diverge substantially after the second augmentation round.

5 Conclusion

SCCL combines instance-wise contrastive learning with unsupervised clustering and is evaluated across eight short-text clustering benchmarks. The authors report strong clustering quality, while identifying text augmentation design as an important open challenge.

  • Across eight benchmark short text clustering datasets, SCCL substantially outperforms or performs highly comparably to state-of-the-art methods.
  • The framework combines bottom-up instance discrimination and top-down clustering to generate clusters with better intra-cluster and inter-cluster distances.
  • Effective textual transformations remain challenging because natural language is discrete, motivating future work on data mixing strategies.

A.1 Experiment Setup

The experiments use Adam with a batch size of 400 and a DistilBERT Sentence Transformers backbone. Optimization uses different learning rates for the backbone and the two task-specific heads.

  • The experiments use the Adam optimizer with batch size 400.
  • The backbone is distilbert-basenli-stsb-mean-tokens from the Sentence Transformers library, with maximum input length 32.
  • The learning rate is 5e-6 for the backbone and 5e-4 for both the Clustering head and Instance-CL head.

A.2 Data Augmentation

The study compares augmentation strengths and paraphrase quality for short-text clustering. Results vary by text length and augmentation type, while composed augmentations can damage short-text semantics.

  • WordNet and Contextual Augmenters are tested at 10%, 20%, and 30% word-substitution ratios, generating an augmentation pair for each text instance.
  • Back-translation selects three augmentation pairs from ten candidate paraphrases using the highest, medium, and lowest BLEU scores.
  • Longer text instances in AgNews, SearchSnippets, GoogleNewsTS, and GoogleNewsS benefit from stronger WordNet and Contextual augmentation.
  • Back-translation performs better with augmented pairs having the two lowest BLEU scores among ten candidates.
  • Composing augmentations is not always beneficial because changing one word can invert a short sentence's meaning, especially after a second augmentation round.

A.3 Alternative Clustering Loss for SCCL

The paper discusses two alternatives for applying the clustering loss to augmented instances rather than only the original dataset. These alternatives are left for future exploration because they may improve joint instance-level and cluster-assignment contrastive learning.

  • A.3 Alternative Clustering Loss for SCCL: In current SCCL, the clustering loss is optimized only on the original dataset.
  • A.3 Alternative Clustering Loss for SCCL: Alternative 1 pushes each augmented instance's cluster assignment toward the target distribution obtained from its paired augmentation.
  • A.3 Alternative Clustering Loss for SCCL: Alternative 2 uses the original instance as an anchor and pushes both augmented instances' cluster assignments toward it.
  • A.3 Alternative Clustering Loss for SCCL: The proposed alternatives are outside the paper's scope but may improve joint optimization of instance-level and cluster-assignment-level contrastive losses.

A.4 Supplement materials for ablation study

Figure 6 provides the full version of Figure 3 in Section 4.

  • Figure 6 is the full version of Figure 3.
  • The supplementary material expands Figure 3 in Figure 6.
  • Figure 6 presents the complete version of the earlier Figure 3.

A.5 Comparison with Rakib et al. (2020)

Rakib et al. (2020) achieve better Accuracy on Tweet and GoogleNews-T, while SCCL is presented as more scalable to large-scale data because its optimization scales linearly with the number of examples.

  • Rakib et al. (2020) achieve better Accuracy on Tweet and GoogleNews-T.
  • HAC-SD can require O(N2 log N) time and O(N2/K) storage before clustering.
  • The agglomerative clustering stage can require O(N2 log(N/K)) time, making HAC-SD less feasible for large-scale data.
  • SCCL uses standard stochastic optimization whose time complexity scales linearly with N and typically converges in 20−100 epochs.
Loading 2103.12953v2…