Source-linked AI summary
Online Deep Clustering for Unsupervised Representation Learning
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew Soon Ong, Chen Change Loy
TL;DR
Alternating clustering and network updates can destabilize unsupervised representation learning. ODC integrates batch-wise label reassignment and memory-based centroid evolution into network iterations, and experiments report improved benchmark performance and fine-tuning utility.
Problem
Deep Clustering’s alternating offline clustering and network-update schedule permutes pseudo-labels and destabilizes representation learning.
Method
ODC decomposes global clustering into online label reassignment and memory updates integrated with network iterations, using samples and centroids memories.
Results
ODC shows promising performance across benchmarks, including 57.6% center-crop accuracy on ResNet-50 conv5 and 78.2% mAP on VOC2007.
Takeaways & Limitations
ODC provides stable unsupervised representation learning and can substantially improve previous self-supervised methods through unsupervised fine-tuning.
Takeaways & Limitations
ODC remains 9.8% below a supervised model pretrained with ImageNet labels, leaving room for further exploration.
Abstract
from arXiv · showhide
Joint clustering and feature learning methods have shown remarkable performance in unsupervised representation learning. However, the training schedule alternating between feature clustering and network parameters update leads to unstable learning of visual representations. To overcome this challenge, we propose Online Deep Clustering (ODC) that performs clustering and network update simultaneously rather than alternatingly. Our key insight is that the cluster centroids should evolve steadily in keeping the classifier stably updated. Specifically, we design and maintain two dynamic memory modules, i.e., samples memory to store samples labels and features, and centroids memory for centroids evolution. We break down the abrupt global clustering into steady memory update and batch-wise label re-assignment. The process is integrated into network update iterations. In this way, labels and the network evolve shoulder-to-shoulder rather than alternatingly. Extensive experiments demonstrate that ODC stabilizes the training process and boosts the performance effectively. Code: https://github.com/open-mmlab/OpenSelfSup.
1. Introduction
Clustering-based representation learning jointly learns features and clusters without manual annotations, but Deep Clustering’s alternating schedule destabilizes training. ODC integrates clustering into network updates so labels and representations evolve continuously, improving stability and performance.
- Clustering-based methods learn inter-image similarity while jointly optimizing clustering and feature learning without manual annotations.
- Deep Clustering alternates whole-dataset feature clustering with CNN updates, causing label permutations, classifier reinitialization, and representation instability.
- ODC integrates mini-batch label updates into network iterations, maintaining samples and centroids memories for continuous label and centroid evolution.
- ODC trains uninterrupted and unsupervised, with labels and network parameters evolving shoulder-to-shoulder rather than alternatingly.
- ODC provides unsupervised representation learning and fine-tuning, with promising performance across benchmarks.
2. Related Work
Prior unsupervised representation learning includes generative and self-supervised approaches, while ODC’s clustering process is illustrated as an online four-step update cycle. The cycle links feature extraction, label use, memory updates, and centroid recomputation.
- Generative unsupervised methods learn latent representations by reconstructing input images, including auto-encoders, variational auto-encoders, and generative adversarial networks.
- Self-supervised methods derive proxy labels from raw data through pretext tasks such as jigsaw solving, colorization, inpainting, rotation prediction, and contrastive learning.
- ODC iteration: ODC iterations perform forward mapping, back-propagation using stored labels, samples-memory updates with new labels, and centroids-memory recomputation.
- Clustering-based methods complement self-supervised learning by focusing on inter-image information and can serve as unsupervised fine-tuning schemes.
3. Methodology
ODC replaces alternating global clustering with uninterrupted, iteration-level updates supported by dynamic samples and centroids memories. Its methodology also reweights losses and handles small clusters to maintain a stable clustering distribution.
- ODC versus DC: DC globally clusters extracted features offline, repeatedly permuting pseudo-labels and forcing rapid network adaptation in subsequent epochs.
- Framework Overview: Unlike DC, ODC avoids extra feature extraction and evolves labels alongside network parameters through samples and centroids memories.The samples memory stores dataset features and pseudo-labels; the centroids memory stores class-centroid features.
- An ODC Iteration: ODC initializes memories with global clustering, then iterates forward propagation, pseudo-label-based back-propagation, memory updates, and centroid updates.
- An ODC Iteration: ODC updates stored features with momentum and reassigns involved samples to their nearest centroids, while recomputing affected centroids periodically.
- Handling Clustering Distribution: Loss re-weighting uses wc ∝ 1 √Nc so smaller clusters contribute more during back-propagation and huge-cluster formation is discouraged.
- Handling Clustering Distribution: ODC addresses extremely small clusters by splitting the largest cluster with K-Means until clusters meet the minimum-size condition.
- Unsupervised Fine-tuning: ODC can complement self-supervised approaches because clustering emphasizes inter-image information and is not tied to a specifically designed pretext objective.
4. Experiments
ODC consistently improves unsupervised representations across ImageNet, Places205, VOC2007, and low-shot VOC2007 evaluations. Further analyses show strong fine-tuning gains, stable hyper-parameter behavior, declining label changes, and discovery of semantic clusters beyond ImageNet annotations.
- ImageNet classification: ODC improves AlexNet representations over DC across all evaluated layers, with a 6.7% largest gain at conv1.On ImageNet, ODC reaches 41.4% on conv4, while ranking second to Rot-Decoupling at the best-performing layer.
- ImageNet classification: ODC achieves 57.6% center-crop accuracy on ResNet-50 conv5, 5.4% above the best layer of re-implemented DC.ODC outperforms LA from conv1 through conv4, although conv5 is slightly below LA.
- Places205 classification: On Places205, ODC surpasses previous methods in every ResNet-50 layer and reaches 49.3% center-crop accuracy on conv5.The largest margin over the runner-up is 3.1% at conv2, and ODC exceeds re-implemented DC by 3.2% at the respective best layer.
- VOC2007 SVM classification: ODC reaches 78.2% mAP on VOC2007 SVM classification, 9.1% higher than DC but 9.8% below supervised ImageNet pre-training.The remaining supervised gap leaves room for further exploration.
- Low-shot VOC2007 classification: ODC reaches 57.1% mAP in low-shot VOC2007 SVM classification, 10.2% above DC, with the gap increasing as more positive examples are allowed.The gap to supervised ImageNet pre-training narrows as the training-shot count increases.
- Further analysis: ODC improves self-supervised fine-tuning and remains relatively insensitive to centroid-update frequency and minimal cluster size within reasonable ranges.Reported VOC2007 gains are 16.7% for Colorization, 9.9% for Jigsaw, 7.1% for Rotation, and 7.9% for DC; changed-label ratios also gradually decline during training.
5. Conclusion
ODC provides a joint clustering and feature-learning paradigm for effective and stable unsupervised training. It also performs well standalone and improves previous self-supervised learning methods through fine-tuning.
- ODC decomposes feature clustering and integrates it into network-update iterations to stabilize unsupervised training.
- ODC performs compellingly as an unsupervised representation learning scheme on its own.
- ODC can fine-tune and substantially improve previous self-supervised learning methods.