PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Clustering for Unsupervised Learning of Visual Features

Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze

arXiv:1807.05520v2cs.CV

TL;DR

Visual feature learning needs approaches that extend beyond limited supervised annotation and adapt clustering to end-to-end convnet training at large scale. DeepCluster alternates k-means clustering with pseudo-label prediction to train convnets, achieving significantly better performance than prior unsupervised methods across standard transfer tasks. Its ImageNet advantage is examined against the less class-balanced YFCC100M distribution.

  • Problem

    Clustering had seen little adaptation to end-to-end visual feature training on large-scale datasets, despite broad use in computer vision and usefulness where annotations are scarce.

  • Method

    DeepCluster alternates k-means clustering of convnet features with updating network weights to predict the resulting cluster assignments as pseudo-labels.

  • Results

    DeepCluster achieves significantly better performance than previous unsupervised methods across standard ImageNet classification and transfer tasks.

  • Takeaways & Limitations

    The approach is a candidate for learning domain-specific deep representations where annotations are scarce because it makes little assumption about inputs and requires little domain knowledge.

  • Takeaways & Limitations

    ImageNet’s balanced, object-oriented distribution may favor DeepCluster because its clustering objective favors balanced clusters; YFCC100M provides a less favorable unbalanced distribution.

Abstract

from arXiv · show

Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

1 Introduction

The paper targets the limited adaptation of clustering to large-scale, end-to-end visual feature learning. DeepCluster alternates feature clustering with convnet training and reports strong performance across classification and transfer evaluations.

  • Motivation: ImageNet’s near-saturated supervised performance motivates exploring unsupervised visual feature learning on larger or different data distributions.The introduction connects limited remaining ImageNet error with interest in methods that can use domains where annotations are scarce.
  • Approach: DeepCluster alternates clustering image descriptors with updating convnet weights to predict cluster assignments as pseudo-labels.The method focuses on k-means, while noting that other clustering algorithms can also be used.
  • Approach: The framework requires little domain knowledge and no input-specific signal compared with self-supervised methods.
  • Results: The paper reports significantly higher performance than previously published unsupervised methods on ImageNet classification and transfer tasks.
  • Evaluation: The experiments examine robustness to architecture and training distribution, including VGG networks, YFCC100M images, and image-retrieval evaluation.The authors also broaden evaluation beyond class-level information to instance-level retrieval.

2 Related Work

Prior unsupervised visual learning work includes clustering, self-supervised pretext tasks, and generative models. DeepCluster differs by adapting clustering to end-to-end convnet training at modern scale.

  • Unsupervised feature learning: Earlier clustering methods used handcrafted descriptors or adapted density-estimation and dimensionality-reduction techniques to deep models.
  • Unsupervised feature learning: Coates and Ng used k-means for layerwise convnet pre-training, whereas DeepCluster learns features end to end.
  • Unsupervised feature learning: Other clustering losses jointly learned convnet features and image clusters but had not been tested at a scale supporting thorough modern-architecture studies.
  • Self-supervised learning: Self-supervised methods replace human labels with pseudo-labels computed from raw inputs through spatial, temporal, or other pretext signals.
  • Generative models: Generative approaches learn mappings from random noise or reconstruction objectives, while GAN discriminators can provide visual features.The cited related work describes autoencoders, GANs, and reconstruction-based models.

3 Method

DeepCluster learns visual features by alternating k-means clustering with discriminative prediction of the resulting assignments. The method includes scalable safeguards against degenerate clusters and uses standard convnet training procedures.

  • Preliminaries: The discriminative training objective uses a multinomial logistic loss optimized with mini-batch stochastic gradient descent and backpropagation.
  • Unsupervised learning by clustering: k-means groups convnet feature vectors into k clusters by jointly learning a d × k centroid matrix and image assignments.
  • Unsupervised learning by clustering: DeepCluster alternates clustering convnet features to produce pseudo-labels with updating convnet parameters to predict those assignments.The alternating procedure uses k-means assignments as supervision and does not use the learned centroid matrix afterward.
  • Avoiding degenerate solutions: Empty-cluster reassignment and cluster-size constraints address trivial solutions that can collapse predictions or concentrate images in a few clusters.The method discussion identifies empty clusters and highly unbalanced assignments as sources of degeneration.
  • Implementation details: The implementation trains AlexNet or VGG-16 with batch normalization, augmentation, dropout, weight regularization, momentum, and 256-image mini-batches.Features are PCA-reduced, whitened, and ℓ2-normalized before k-means; hyperparameters are selected using Pascal VOC classification without fine-tuning.

4 Experiments

The experiments examine DeepCluster’s training dynamics, learned filters, and transfer performance across ImageNet, Places, and Pascal VOC. The analyses show progressively more class-related information, semantically coherent higher-layer filters, and strong transfer results across tasks.

  • Preliminary study: NMI between cluster assignments and ImageNet labels increases during training, indicating progressively greater capture of object-class information.The measure is used only for analysis, not model selection.
  • Preliminary study: NMI between consecutive epoch assignments increases while remaining below 0.8, indicating stabilization with a significant fraction of changing assignments.The measure compares clusters at epochs t−1 and t.
  • Preliminary study: k = 10,000 gives the best reported Pascal VOC 2007 classification mAP after 300 epochs among the tested cluster counts.Comparisons at the same epoch count may not be directly comparable across k.
  • Visualizations: Sobel preprocessing yields first-layer edge-detector filters, whereas raw RGB training produces filters capturing mostly color information.The raw-input and Sobel-filtered conditions are compared in Figure 3.
  • Visualizations: Deeper layers capture larger textural structures, while some conv5 filters respond coherently to object-related structures or stylistic effects.Figure 5 examples include filters sensitive to background blur and depth of field.
  • Linear classification on activations: DeepCluster outperforms the state of the art from conv3 to conv5 on ImageNet by 3−5%, with the largest improvement at conv4.At conv5, the AlexNet likely stores most class-level information; an MLP on the last layer improves over the state of the art by 8%.
  • Linear classification on activations: On Places, DeepCluster conv3-4 features are comparable to ImageNet-supervised features, while higher-layer performance decreases.The authors suggest labels are less important when the target task is sufficiently far from ImageNet’s domain.
  • Pascal VOC transfer: On Pascal VOC, DeepCluster outperforms previous unsupervised methods across classification, detection, and segmentation settings.Fine-tuning improves semantic segmentation over the state of the art by 7.5%, while the frozen fc6-8 setting reaches up to a 9% classification gap.

5 Discussion

The discussion tests DeepCluster beyond the standard ImageNet-AlexNet, class-level evaluation pipeline by varying the training distribution, architecture, and task. These experiments show robustness to uncured Flickr images, gains from deeper architectures, and the importance of instance-level retrieval evaluation.

  • Evaluation protocol: The evaluation changes the training set, architecture, and task to measure biases introduced by the standard AlexNet-on-ImageNet, class-level pipeline.The alternative task is instance-level recognition through image retrieval.
  • ImageNet versus YFCC100M: Up to +4.3% on classification and +4.5% on semantic segmentation, DeepCluster exceeds the state of the art after pre-training on 1M uncured YFCC100M images.Performance drops by a few percent relative to ImageNet because YFCC100M is not object oriented, yet DeepCluster remains competitive or better on most tasks.
  • AlexNet versus VGG: DeepCluster with VGG-16 reaches only 1.4% below the supervised topline on Pascal VOC 2007 object detection.The unsupervised-supervised gap remains about 1.4% for both AlexNet and VGG-16.
  • AlexNet versus VGG: A deeper architecture significantly improves target-task performance regardless of the unsupervised approach.The gap with a random baseline grows for larger architectures, supporting the relevance of unsupervised pre-training when supervised data are limited.
  • Evaluation on instance retrieval: Image retrieval evaluates instance-level discrimination, which class-level benchmarks do not measure; random convnets perform particularly poorly on this task.The protocol uses Oxford Buildings and Paris, and pre-training is described as essential for retrieval.

6 Conclusion

The conclusion presents DeepCluster as a scalable unsupervised method that alternates k-means feature clustering with pseudo-label-based convnet updates. On large datasets, it reports better performance than prior unsupervised methods across standard transfer tasks and requires few assumptions about inputs.

  • 6 Conclusion: DeepCluster is a scalable approach for unsupervised convnet learning that alternates k-means clustering of features with network-weight updates.The cluster assignments serve as pseudo-labels in a discriminative loss.
  • 6 Conclusion: DeepCluster achieves significantly better performance than previous state-of-the-art methods on every standard transfer task when trained on ImageNet or YFCC100M.The conclusion describes the method as making little assumption about inputs and requiring little domain-specific knowledge.
  • 6 Conclusion: Its limited assumptions and domain-specific requirements make DeepCluster a candidate for learning representations in domains where annotations are scarce.This consequence is stated for domain-specific representation learning under limited annotation availability.

1 Additional results

Additional results compare DeepCluster with prior unsupervised methods, monitor feature quality during training, and report an 8% accuracy advantage over the previous best method. The study uses downstream validation performance to select hyperparameters and track feature improvement.

  • 1.1 Classification on ImageNet: The comparison includes DeepCluster, other unsupervised AlexNet networks, random networks, and supervised networks.The convolutional layers are frozen while fully connected layers are retrained using ImageNet labels.
  • 1.1 Classification on ImageNet: 8% better accuracy than the previous best-performing method, DeepCluster halves the gap with supervised networks.This result concerns the reported ImageNet classification comparison against state-of-the-art unsupervised methods.
  • Training analysis: Downstream Pascal VOC classification accuracy and clustering NMI are monitored throughout training to select hyperparameters and identify when features stop improving.The validation task uses no fine-tuning.

2 Further discussion

DeepCluster is examined with alternative clustering, transfer tasks, visualization, and robustness experiments. PIC avoids preselecting the cluster count, but k-means generally performs better across the reported comparisons.

  • 2.1 Alternative clustering algorithm: PIC constructs a nearest-neighbor graph from image descriptors and forms clusters through graph-based connected components.Images are connected to their five nearest neighbors in descriptor space; clusters are connected components of a directed subgraph.
  • 2.1 Alternative clustering algorithm: PIC does not require the number of clusters to be set beforehand, although σ influences the resulting cluster count.The experiments use σ = 0.2.
  • 2.1 Alternative clustering algorithm: Nearest-neighbor retrieval with PIC DeepCluster is compared against a randomly initialized network on Flickr query images.The random network performs well for some queries with simple low-level structure.
  • 2.2 PIC results: PIC and k-means DeepCluster models perform in comparable ranges on the reported Pascal VOC transfer tasks.The comparison covers classification, detection, and segmentation tasks.
  • 2.3 Experimental variants: Raw RGB input causes DeepCluster performance to degrade significantly.This result is reported among experiments varying the training set, clustering method, and input preprocessing.
  • 2.2 PIC results: k-means DeepCluster yields better results than PIC on both ImageNet and YFCC100M in the reported comparisons.The comparison varies both the clustering method and the pre-training dataset.

3 Additional visualisation

The visualizations indicate that unsupervised DeepCluster features can organize images and activate on recognizable semantic, stylistic, and textural patterns. Cluster purity varies across ImageNet concepts, with the strongest correlations for animals and plants.

  • 3 Additional visualisation: DeepCluster filters in VGG-16 show semantically or stylistically coherent top-activated images and synthetic visualizations.The visualized filters were selected from the last convolutional layer.
  • 3.1 VGG-16: The correlation between learned clusters and ImageNet categories varies substantially, with the highest correlations for animals and plants.The analysis uses cluster purity across different synsets in the ImageNet ontology.
  • 3.2 AlexNet: On YFCC100M, some unsupervised filters respond to particular object classes, while others respond to stylistic effects or textures.The analysis uses the top nine activated images for the first 100 target filters.

4 Erratum [18/03/2019]

The erratum revisits linear-classification evaluation on ImageNet and Places by adding single-crop results to the originally reported ten-crop results. It states that the comparison issue does not alter the conclusion that DeepCluster outperforms prior methods from conv3 to conv5.

  • 4 Erratum [18/03/2019]: The original linear-classifier results averaged accuracy over 10 crops, whereas other methods reported central-crop accuracy.The authors identify this mismatch as an unfair comparison and provide both single- and ten-crop results in Table 5.
  • 4 Erratum [18/03/2019]: The crop-evaluation discrepancy does not change the conclusion that DeepCluster outperforms the state of the art from conv3 to conv5 layers.The revised table reports results for both single and 10 crops.
  • 4 Erratum [18/03/2019]: Table 5 reports linear classification on ImageNet and Places using activations from AlexNet convolutional layers.The table covers representations from the convolutional layers used as features.
Loading 1807.05520v2…