PaperlayerView exact revision on arXiv

Source-linked AI summary

Twin Contrastive Learning for Online Clustering

Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, Xi Peng

arXiv:2210.11680v1cs.LG

TL;DR

Online clustering requires methods that can learn representations and assign clusters without repeatedly processing the whole dataset. TCL addresses this by jointly applying instance- and cluster-level contrastive learning, then using confident pseudo-labels to reduce false negatives and correct assignments; experiments demonstrate its effectiveness across image and text benchmarks.

  • Problem

    Existing deep clustering methods often require the entire dataset for offline clustering, limiting direct support for online or streaming data.

  • Method

    TCL projects data into a feature space whose rows and columns represent instances and clusters, then performs twin contrastive learning with mixed augmentations and confidence-based pseudo-label boosting.

  • Results

    Extensive experiments demonstrate TCL’s effectiveness on five image benchmarks and two text datasets.

  • Takeaways & Limitations

    TCL simultaneously learns representations and performs clustering while independently predicting cluster assignments for instances, supporting online clustering.

  • Takeaways & Limitations

    The paper identifies augmentation effects and better pseudo-label use for false-negative detection as directions requiring further investigation.

Abstract

from arXiv · show

This paper proposes to perform online clustering by conducting twin contrastive learning (TCL) at the instance and cluster level. Specifically, we find that when the data is projected into a feature space with a dimensionality of the target cluster number, the rows and columns of its feature matrix correspond to the instance and cluster representation, respectively. Based on the observation, for a given dataset, the proposed TCL first constructs positive and negative pairs through data augmentations. Thereafter, in the row and column space of the feature matrix, instance- and cluster-level contrastive learning are respectively conducted by pulling together positive pairs while pushing apart the negatives. To alleviate the influence of intrinsic false-negative pairs and rectify cluster assignments, we adopt a confidence-based criterion to select pseudo-labels for boosting both the instance- and cluster-level contrastive learning. As a result, the clustering performance is further improved. Besides the elegant idea of twin contrastive learning, another advantage of TCL is that it could independently predict the cluster assignment for each instance, thus effortlessly fitting online scenarios. Extensive experiments on six widely-used image and text benchmarks demonstrate the effectiveness of TCL. The code will be released on GitHub.

1 Introduction

TCL addresses the need for online deep clustering by jointly learning instance and cluster representations through twin contrastive learning. It combines mixed augmentations and confidence-based pseudo-labeling to reduce false negatives, rectify assignments, and support streaming data.

  • Motivation: Traditional deep clustering methods often require the entire dataset for offline clustering, limiting their suitability for online settings.TCL is designed for batch-wise optimization and can assign clusters to new streaming data without accessing the whole dataset.
  • Twin contrastive learning: TCL interprets feature-matrix rows as instance representations and columns as cluster representations, enabling contrastive learning at both levels.The two contrastive objectives jointly learn representations while performing clustering.
  • Twin contrastive learning: Mixed weak and strong transformations construct contrastive pairs in a strategy reported effective for both image and text data.This augmentation strategy is presented as fitting the TCL framework rather than directly applying strong augmentation alone.
  • Confidence-based boosting: Confidence-based pseudo-labels reduce intrinsic false negatives and rectify cluster assignments during fine-tuning of both contrastive objectives.The criterion selects confident predictions to improve the instance- and cluster-level learning process.
  • Online clustering: TCL independently predicts each instance’s cluster assignment, allowing batch-wise optimization and timely assignment of newly arriving streaming data.This property supports online clustering without repeatedly accessing the complete dataset.

2 Related Work

Related work generally treats contrastive learning, representation learning, and clustering as separate or staged processes. TCL instead unifies row- and column-wise contrastive learning, mixed augmentation, and confidence-based boosting in a one-stage framework.

  • 2.1 Contrastive Learning: Contrastive learning constructs positive and negative pairs and optimizes their projected similarities, while unsupervised settings lack labels for directly guiding pair construction.This motivates alternative strategies for forming informative pairs without supervision.
  • 2.1 Contrastive Learning: Unlike methods that mainly perform instance-level row-wise learning, TCL concurrently conducts row- and column-wise contrastive learning for instances and clusters.The distinction follows from the feature-matrix interpretation of rows and columns.
  • 2.1 Contrastive Learning: TCL uses mixed weak and strong transformations to facilitate contrastive learning, addressing uncertainty about how to use strong augmentations directly.The proposed strategy is described as suitable for different data types, including images and text.
  • 2.2 Deep Clustering: Unlike multi-stage deep clustering methods, TCL unifies representation learning and clustering in a one-stage twin contrastive framework.The framework is intended to learn representations favorable for clustering while performing both tasks together.
  • 2.3 Connection between contrastive learning and deep clustering: Confidence-based boosting selects confident predictions as pseudo-labels to remove within-class false negatives and rectify cluster assignments.The method uses pseudo-labels for self-supervised instance-level contrastive loss and cross-entropy-based cluster-level correction.
  • 2.3 Connection between contrastive learning and deep clustering: Experiments report TCL effectiveness across image and text clustering, with a comparable performance gain over the previous conference paper.The cited extension evaluates the framework, mixed augmentation, and confidence-based boosting strategy.

3 Method

TCL jointly learns instance and cluster representations through twin contrastive learning, then uses confidence-based pseudo-labels to reduce false negatives and rectify cluster assignments. Its decoupled pipeline combines contrastive pair construction, two contrastive heads, and confidence-based boosting for online clustering.

  • Online Clustering: After convergence, the cluster-level head independently predicts each instance’s cluster assignment, supporting online clustering.Decoupling the instance- and cluster-level learning into two subspaces experimentally improves clustering performance.
  • Contrastive Pair Construction: TCL constructs two correlated samples per instance through weak and strong data augmentations, then processes them with a shared backbone.Image data uses SimCLR and RandAugment transformations, while text data uses synonym replacement and sentence operations.
  • Twin Contrastive Learning: The training objective jointly optimizes instance-level and cluster-level contrastive losses through separate contrastive heads.Instance-level learning compares augmented samples, while cluster-level learning compares column representations of the assignment matrix.
  • Instance-level Contrastive Learning: Instance-level contrastive learning treats same-instance augmentations as positives and other augmented samples as negatives using an InfoNCE objective.A nonlinear projection head maps backbone features into the subspace where pairwise cosine similarities are optimized.
  • Cluster-level Contrastive Learning: Cluster-level representations arise from feature-matrix columns when projected features have cluster-number dimensionality, with corresponding augmentations forming positive cluster pairs.The columns represent cluster assignment distributions over the mini-batch.

4 Experiments

The experiments evaluate TCL on image and text datasets using performance analyses, qualitative analyses, and ablation studies.

  • Evaluation Setup: TCL is evaluated on five image datasets and two text datasets.The evaluation covers both image and text modalities.
  • Evaluation Setup: The study includes qualitative analyses to provide an intuitive examination of the method.
  • Evaluation Setup: Ablation studies are conducted to help assess the method’s components and behavior.

4.1 Datasets

The evaluation uses five image datasets and two text datasets, with CIFAR-100 evaluated using its 20 super-classes rather than its fine-grained classes. The datasets include subsets from Kaggle’s challenge data and PubMed data distributed by BioASQ.

  • Image Datasets: The image datasets are CIFAR-10, CIFAR-100, STL-10, ImageNet-10, and ImageNet-Dogs.
  • Image Datasets: CIFAR-100 uses its 20 super-classes as ground truth instead of its 100 fine-grained classes.
  • Text Datasets: The text datasets are StackOverflow and Biomedical, drawn respectively from Kaggle challenge data and PubMed data distributed by BioASQ.

4.2 Experimental Settings

TCL uses modality-specific backbones and mixed weak/strong augmentations, with separate contrastive-head dimensions and shared optimization settings.

  • Image inputs use ResNet34, while text inputs use distilbert-base-nli-stsb-mean-tokens from Sentence Transformers.
  • Weak image augmentation follows SimCLR, omitting GaussianBlur for CIFAR-10/100 because up-scaling already blurs small images.
  • Strong image augmentation applies four randomly selected RandAugment transformations followed by Cutout with size 75 × 75.
  • Text augmentation randomly substitutes 20% of words using suitable RoBERTa-derived alternatives for weak augmentation.
  • Strong text augmentation uses EDA operations—SynonymReplacement, RandomInsertion, RandomSwap, and RandomDeletion—with probability 0.2 each.
  • ICH has dimension 128, CCH matches the target cluster number, and temperatures are τI = 0.5 and τC = 1.0.

4.3 Compared Methods

The study compares TCL with broad image and text clustering baselines, standardizing evaluation backbones and clarifying differences in pretraining and supervision.

  • Image experiments compare TCL with 21 representative methods spanning conventional, representation-based, and deep clustering approaches.
  • Recent image methods use a common ResNet34 backbone, while SPICE is compared using self-trained ResNet34 results for fairness.
  • SPICE additionally uses ImageNet pretraining for ImageNet-10/Dogs, whereas other methods, including TCL, train from scratch.
  • Text experiments compare TCL with 11 benchmarks, including traditional feature, embedding, hierarchical, and contrastive clustering methods.

4.4 Evaluation Metrics

Clustering quality is evaluated with three standard metrics, where higher scores indicate better performance.

  • The evaluation uses Normalized Mutual Information (NMI), Clustering Accuracy (ACC), and Adjusted Rand Index (ARI).

4.5 Results

TCL is evaluated quantitatively and qualitatively across image and text benchmarks, with results indicating strong clustering performance and progressively improved representations during training and boosting.

  • Quantitative comparisons with state-of-the-art baselines and visualizations across training evaluate TCL on image and text benchmarks.
  • Image comparisons: TCL shows promising clustering performance across five object-image benchmarks.
  • Image comparisons: TCL outperforms SPICE on ImageNet-Dogs without ImageNet pretraining.
  • Text comparisons: Table 4 reports TCL clustering results on two text datasets using NMI and ACC comparisons.
  • Text comparisons: TCL dominates SCCL, a contrastive method that clusters representations learned through instance-level contrastive learning.
  • Evolution analysis: During training, instance features become more compact and separated, while confidence-based boosting improves cluster assignments.

4.6 Ablation Study

The ablations examine boosting, augmentation, decoupling, contrastive-head design, and clustering-resolution choices. Results generally support confidence-based fine-tuning, mixed augmentation, separate contrastive subspaces, and online-capable cluster assignments, while revealing sensitivity to batch composition and manually specified cluster numbers.

  • Effectiveness of the boosting strategy: Both instance- and cluster-level boosting losses improve clustering performance, while the self-labeling loss directly affects cluster assignments and is essential for performance boosting.
  • Effectiveness of the boosting strategy: Confident predictions become more likely to be correct after training, and their number increases during training and especially at the boosting stage.Pseudo-label selection uses ranking rather than treating every prediction above an absolute confidence threshold as a pseudo-label.
  • Hyper-parameter analysis: Boosting remains stable across reasonable α and γ ranges but degrades when either the confidence threshold or selection-ratio criterion is abandoned.
  • Different combinations of data augmentation: Mixed weak and strong augmentation produces the best clustering performance for both image and text data, but its effectiveness is closely tied to the TCL framework.Compared with SimCLR+k-means, TCL benefits more from mixed augmentation on CIFAR-10.
  • Effectiveness of the decoupling strategy: Decoupling instance- and cluster-level contrastive learning into separate subspaces improves performance because a shared subspace can limit instance information and produce interacting objectives.The two losses still act on the same representation and jointly optimize the network.
  • Over-clustering experiments: Over-clustering can improve performance on CIFAR-100 but hurt coarse-grained CIFAR-10 structure; very fine partitions can also suffer when mini-batches contain too few or no samples per cluster.The target cluster number must be manually set, although cluster-number search or community-detection methods can estimate it.

5 Conclusion

TCL performs online deep clustering by jointly learning instance and cluster representations through twin contrastive learning, with confidence-based pseudo-label boosting. Experiments support its effectiveness across image and text datasets, while the authors identify augmentation analysis and better pseudo-label use as future directions.

  • TCL jointly conducts instance- and cluster-level contrastive learning and uses confidence-selected pseudo-labels to reduce false negatives and rectify cluster assignments.
  • Extensive experiments demonstrate TCL’s effectiveness on five image benchmarks and two text datasets.
  • Future work will examine augmentation effects more deeply and improve pseudo-label use for identifying false-negative pairs.
Loading 2210.11680v1…