Source-linked AI summary
Unsupervised Pre-Training of Image Features on Non-Curated Data
Mathilde Caron, Piotr Bojanowski, Julien Mairal, Armand Joulin
TL;DR
The paper addresses the challenge of learning transferable features from large, non-curated datasets without annotations, where prior raw-data approaches reported degraded feature quality. It combines self-supervision with clustering, and reports strong unsupervised benchmark performance plus an ImageNet pre-training gain.
Problem
Unsupervised feature learning has focused largely on curated datasets, while prior work using non-curated raw data reported degraded feature quality.
Source: §1 · p. 1Method
DeeperCluster combines self-supervision and clustering, generating cluster targets from dataset features under self-supervision constraints.
Source: §1 · p. 2Results
Takeaways & Limitations
The results support the potential of unsupervised pre-training on large non-curated datasets when annotations are scarce or curation is not trivial.
Source: §6 · p. 8The alternating clustering optimization is prone to trivial solutions, so the interaction between its objectives must be controlled.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Pre-training general-purpose visual features with convolutional neural networks without relying on annotations is a challenging and important task. Most recent efforts in unsupervised feature learning have focused on either small or highly curated datasets like ImageNet, whereas using uncurated raw datasets was found to decrease the feature quality when evaluated on a transfer task. Our goal is to bridge the performance gap between unsupervised methods trained on curated data, which are costly to obtain, and massive raw datasets that are easily available. To that effect, we propose a new unsupervised approach which leverages self-supervision and clustering to capture complementary statistics from large-scale data. We validate our approach on 96 million images from YFCC100M, achieving state-of-the-art results among unsupervised methods on standard benchmarks, which confirms the potential of unsupervised learning when only uncurated data are available. We also show that pre-training a supervised VGG-16 with our method achieves 74.9% top-1 classification accuracy on the validation set of ImageNet, which is an improvement of +0.8% over the same network trained from scratch. Our code is available at https://github.com/facebookresearch/DeeperCluster.
1. Introduction
The paper seeks transferable visual features from large, uncurated datasets without annotations, addressing the cost of curated labels and the degradation reported when learning from raw data. It combines clustering and self-supervision to scale feature learning while handling changing data distributions.
1. Introduction
- Curated, fully annotated datasets require substantial data cleansing and manual labeling, while metadata may be unavailable or incomplete.Prior work also reported degraded feature quality when training on non-curated raw data.Source: §1 · p. 1
- As the number of training images increases, Pascal VOC feature quality improves and eventually surpasses unsupervised features trained on curated datasets.Figure 1 reports validation mAP on Pascal VOC classification in the FC68 setting.Source: Figure 1 · p. 1, §1 · p. 2
- Large raw datasets demand increasing model complexity and stability to distribution changes, while clustering-based learning can be unstable and distribution-sensitive.The method avoids assuming that the distribution of latent classes is known.Source: §1 · p. 2
- 74.9% top-1 accuracy on ImageNet validation, +0.8% over training from scratch, follows supervised VGG-16 pre-training with DeeperCluster.The authors also report that feature quality improves with more training images, eventually surpassing unsupervised features trained on curated datasets.Source: §1 · p. 2
- DeeperCluster generates dataset-wide cluster targets under self-supervision constraints and uses a hierarchical formulation suited to distributed training.The approach combines large-rotation classification with clustering to discover latent categories in the long tail.Source: §1 · p. 2
2. Related Work
Prior unsupervised feature-learning work includes self-supervised pretext tasks, deep clustering, and learning from non-curated data using metadata. This paper builds on rotation prediction and clustering while differing from methods that use curated data or extrinsic supervision.
Self-supervision
Deep clustering
- Caron et al. [6] use k-means to cluster visual representations, mainly training on unlabeled ImageNet; Noroozi et al. [34] use clustering as post-processing for distillation.The latter does not combine clustering and self-supervision to improve feature quality.Source: §2 · pp. 2–3
3. Preliminaries
The section defines visual features as penultimate-layer convnet representations and introduces fixed self-supervised targets alongside changing cluster assignments.
3. Preliminaries
- A feature or representation is the convnet’s penultimate-layer vector, intended to be useful on downstream tasks.Source: §3 · p. 3
3.1. Self-supervision
- RotNet, selected for strong standard-benchmark performance, trains on four rotation-angle classes as targets extracted from images.Source: §3.1 · p. 3
3.2. Deep clustering
- Deep clustering alternates feature and classifier optimization with cluster-label reassignment, using k-means targets refreshed every T epochs.The authors note the scheme can reach trivial solutions and cite empty-cluster reassignment and uniform cluster sampling as workarounds.Source: §3.2 · p. 3
4. Method
DeeperCluster combines rotation self-supervision and clustering in a joint target space, then approximates that objective with a hierarchy designed to scale to large datasets and many targets. Its data partitions also support distributed training.
4.1. Combining self-supervision and clustering
- The joint Y × Z target space models interactions between rotation and cluster assignments, but its O(|Y||Z|) cost motivates a scalable hierarchical approximation.Source: §4.1 · p. 4
4.2. Scaling up to large number of targets
- A two-level target hierarchy predicts a super-class first, then a sub-class within it, using jointly learned super- and sub-class classifiers.Source: §4.2 · p. 4
- DeeperCluster alternates clustering non-rotated features with training to predict rotation and rotation-linked clusters; m=4 and k=80k yield 320k clusters.The method pairs m k-means data partitions with rotation classes to form super-classes, then divides each into k sub-classes.Source: §4.2 · pp. 4–5, Figure 2 · p. 4
- Data-based super-classes support distributed training: 4m GPU groups share the backbone and super-class head, while each group trains its own sub-class head.Assignments are recomputed every T epochs by two distributed k-means passes; empirically, k-means converges in 10 iterations.Source: §4.2 · pp. 4–5
4.3. Implementation details
- Table 2 compares methods using curated and non-curated data on PASCAL VOC 2007 classification and detection, reporting test results averaged over five runs.Source: Table 2 · p. 5
- Training uses VGG-16 with batch normalization on 96M downloaded YFCC100M images, reassigning clusters every three epochs after whitening and L2 normalization.Source: §4.3 · p. 5
5. Experiments
The experiments assess DeeperCluster features across transfer tasks, ImageNet pre-training, and analyses of method comparisons, scaling, metadata alignment, and cluster examples. They report strong benchmark performance and improvements from both pre-training and increased data scale.
5.1. Evaluating unsupervised features
- DeeperCluster achieves state-of-the-art VGG-16 unsupervised performance across curated- and non-curated-data methods; fine-tuning reduces the supervised gap to under 5% on both tasks.With frozen convolutions, the gap remains 6% for detection and 10% for classification.Source: §5.1 · p. 6
- DeeperCluster matches supervised features on Places205 at every tested layer and on ImageNet through the fourth convolutional block, after which the ImageNet gap reaches about 20%.The supervised comparison is trained on ImageNet itself, whereas DeeperCluster is trained on YFCC100M.Source: §5.1 · p. 6, Figure 3 · p. 6
5.2. Pre-training for ImageNet
- 74.9% top-1 accuracy on ImageNet follows supervised VGG-16 initialization with DeeperCluster pre-training, a +0.8% improvement over standard initialization.The pre-training uses YFCC100M images.Source: §5.2 · p. 6, Table 3 · p. 7
5.3. Model analysis
- DeeperCluster outperforms RotNet and DeepCluster across the reported linear-classifier comparisons, except that ImageNet-trained DeepCluster performs best on ImageNet classification.DeepCluster could not scale to all of YFCC100M and was trained on a random 1.3M-image subset.Source: §5.3 · p. 7, Table 4 · p. 7
- Feature quality improves with both image count and cluster count, but increasing images has the larger impact; 4M–20M YFCC100M images are needed to meet the method's ImageNet performance.The improvement corresponds to a reduction of more than 10% in relative error versus the supervised model.Source: §5.3 · p. 7
- Clustering NMI improves over training for every measured metadata type, and the authors report that it captures enough input information to predict metadata at least as well as supervised features.The ImageNet-label comparison uses predictions with confidence above 75%, omitting a large amount of data.Source: §5.3 · p. 7, Figure 4 · p. 8
- GPS-pure but user-mixed clusters contain images taken by different users at the same place, including tourist landmarks.Other illustrated clusters align with hashtags or scanner device metadata.Source: §5.3 · p. 8, Figure 5 · p. 8
6. Conclusion
DeeperCluster is designed for large non-curated datasets and distributed training, and is evaluated on 96M images. The conclusion reports that it surpasses unsupervised methods trained on curated data and improves supervised ImageNet training when used for pre-training.
6. Conclusion
- DeeperCluster handles large non-curated datasets through a formulation suited to distributed training, and its 96M-image results surpass unsupervised methods trained on curated datasets.The authors frame this as evidence for unsupervised learning where annotations are scarce or curation is not trivial.Source: §6 · p. 8
1. Evaluating unsupervised features
The passage directs readers to Table 5 for the numerical values from Figure 2.
2. Evaluating unsupervised features
- The numerical values from Figure 2 are provided in Table 5.Source: §1 · p. 12
2. YFCC100M and Imagenet label distribution
YFCC100M has a highly unbalanced, long-tail hashtag distribution unlike ImageNet's more balanced label distribution. The passage illustrates the difference with tags that have far fewer associated YFCC100M images than ImageNet labels.
3. YFCC100M and Imagenet label distribution
- YFCC100M's hashtag distribution is long-tailed, unlike ImageNet's well-behaved label distribution; guenon and baseball have 226 and 256,758 YFCC100M images, respectively, versus 1,300 ImageNet images each.Source: §2 · p. 12
3. Pre-training for ImageNet
The section compares supervised ImageNet fine-tuning from scratch with initialization from DeeperCluster or RotNet, using a changed training schedule. At convergence, the DeeperCluster gap over the scratch baseline is smaller than before convergence.
3. Pre-training for ImageNet
- The comparison fine-tunes supervised ImageNet models initialized from scratch, DeeperCluster, or RotNet for 90 epochs with a learning-rate schedule differing from the main-paper setup.The schedule uses learning-rate reductions at epochs 30 and 60; the authors say this setting is unfair to the scratch baseline because the pretrained models converge earlier.Source: §3 · p. 12
- At convergence, DeeperCluster’s gap over the scratch baseline shrinks from 1.0 to 0.8, while RotNet’s gap remains 0.4.The earlier 1.0 gap is reported before convergence.Source: §3 · p. 12
- Evaluating at convergence rather than before convergence narrows the DeeperCluster gap, while leaving the RotNet gap unchanged.The comparison uses ImageNet validation accuracy.Source: §3 · p. 12
4. Model analysis
The analysis compares unsupervised features on instance retrieval and examines how color-focused clustering relates to input preprocessing. It reports that Sobel filtering improves downstream feature performance, while retrieval outcomes vary across benchmarks.
4.1. Instance retrieval
- Instance retrieval uses R-MAC features and is motivated as a task where class-level invariance may be unhelpful; the evaluation requires no additional training.The experimental setup follows prior work cited by the authors.Source: §4.1 · p. 12
- RotNet features perform significantly worse than DeepCluster on Oxford5K and Paris6K; the combined method loses substantially on Oxford5K but little on Paris6K.The authors attribute this pattern to the landmarks’ shared orientation and emphasize evaluating on diverse benchmarks.Source: §4.1 · p. 12
- Table 7 reports instance-retrieval mAP on Oxford and Paris using R-MAC, with DeepCluster trained on a random 1.3M-image subset because it does not scale to full YFCC100M.The evaluation uses a 1024-pixel resolution and three grid levels.Source: Table 7 · p. 13
4.2. Influence of data pre-processing
- Table 8 reports validation mAP for Pascal VOC classification in the FC68 setting, comparing Sobel-filtered inputs with raw RGB inputs.The prose reports that Sobel filtering also slightly improves RotNet features.Source: Table 8 · p. 13, §4.2 · p. 13
- Clustering on the method’s features focuses more on color than RotNet clustering, and Sobel filtering improves downstream feature performance by replacing RGB inputs with image gradients.Figure 7 sorts 3,000 clusters by standard deviation from mean color; Figure 8 shows low-variation clusters, and Figure 9 visualizes Sobel’s two derivative channels.Source: Figure 7 · p. 12, §4.2 · p. 13, Figure 8 · p. 14
5. Hyperparameters
The section specifies input preprocessing, hyperparameter selection, and training settings for the compared methods. It also lists photographer usernames and their ordering for images shown in a cluster visualization.
5. Hyperparameters
- Images are resized to 3×224×224, and hyperparameters are selected using Pascal VOC 2007 classification without fine-tuning; training stops when that task’s performance saturates.All methods use SGD with momentum 0.9, and the experiments use PyTorch 1.0.Source: §5 · p. 14
- RotNet on both YFCC100M and ImageNet uses batch size 512, learning rate 0.05, weight decay 0.00001, and dropout 0.3.Source: §5 · p. 14
- DeepCluster on 1.3M YFCC100M images uses batch size 256 and 10,000 clusters, formed every two training epochs from whitened, normalized, PCA-reduced features.The features are reduced to 256 dimensions, and preprocessing uses a Sobel filter.Source: §5 · p. 14
- DeeperCluster on YFCC100M uses batch size 3072 and hierarchical k-means with 320,000 clusters, while its ImageNet setup uses batch size 748 and 10,000 clusters.Both configurations use Sobel preprocessing and whitened, normalized 4096-dimensional features; YFCC100M clustering runs every three epochs and ImageNet clustering every five.Source: §5 · p. 14
6. Usernames of cluster visualization images
- Photographer usernames for the Figure 5 images are listed by cluster, ordered left to right and then top to bottom.Source: §6 · p. 14