PaperlayerView exact revision on arXiv

Source-linked AI summary

ClusterFit: Improving Generalization of Visual Representations

Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, Dhruv Mahajan

arXiv:1912.03330v1cs.CVcs.LG

TL;DR

Weakly and self-supervised pretraining can produce representations that overfit proxy-task artifacts, limiting transfer. ClusterFit clusters features from a pretrained network and retrains from scratch on cluster assignments; the paper reports consistent improvements across 11 image and video benchmarks.

  • Problem

    Weak and self-supervised proxy objectives may lack strong discriminative signals, leaving representations overfit to objective-specific artifacts and less suitable for transfer tasks.

    Source: §1 · p. 1

  • Method

    ClusterFit uses k-means to cluster features from a pretrained network, then trains a new network from scratch on the cluster assignments as pseudo-labels.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    ClusterFit consistently improves performance across 11 public image and video benchmarks, spanning pretraining methods, modalities, architectures, and datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that lossy clustering captures essential visual invariances and makes retrained features less sensitive to the original pretraining objective and more transferable.

    Source: §6 · p. 8

    In its current unsupervised form, iterative ClusterFit provides little improvement; domain knowledge from downstream tasks may guide clustering and possibly improve transfer.

    Source: §6 · p. 8

Abstract

from arXiv · show

Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks. However, due to the lack of strong discriminative signals, these learned representations may overfit to the pre-training objective (e.g., hashtag prediction) and not generalize well to downstream tasks. In this work, we present a simple strategy - ClusterFit (CF) to improve the robustness of the visual representations learned during pre-training. Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels. We empirically show that clustering helps reduce the pre-training task-specific information from the extracted features thereby minimizing overfitting to the same. Our approach is extensible to different pre-training frameworks -- weak- and self-supervised, modalities -- images and videos, and pre-training tasks -- object and action classification. Through extensive transfer learning experiments on 11 different target datasets of varied vocabularies and granularities, we show that ClusterFit significantly improves the representation quality compared to the state-of-the-art large-scale (millions / billions) weakly-supervised image and video models and self-supervised image models.

1. Introduction

Weakly and self-supervised proxy objectives can leave representations overfit to task-specific artifacts, limiting transfer. ClusterFit smooths pretrained features through clustering and retraining, and improves results across 11 image and video benchmarks.

1. Introduction

  • Weakly and self-supervised methods use proxy objectives, but weak discriminative signals can make learned features overfit to objective- and dataset-specific artifacts.Reported factors include label noise, polysemy, linguistic ambiguity, and tags with little visual content; later-layer self-supervised features can also align more closely with the proxy objective.Source: §1 · p. 1
  • ClusterFit clusters features from a pretrained network with k-means, then trains a new network from scratch using cluster memberships as pseudo-labels.The paper presents post-hoc clustering as a way to smooth the feature space and retain essential visual invariances while reducing proxy-objective artifacts.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • ClusterFit applies post-hoc smoothing across weak- and self-supervised frameworks and scales to millions or billions of images and videos.The paper distinguishes its focus on post-hoc smoothing from prior work that uses clustering as the training objective itself.Source: §1 · p. 2
  • Across 11 public image and video benchmarks, ClusterFit consistently improves performance across pretraining methods, modalities, architectures, and datasets.Table 1 summarizes results as absolute top-1 accuracy gains, except for VOC07, which uses mAP; the caption reports improvements with the same architecture and no additional data or supervision.Source: Table 1 · p. 1, §1 · p. 2

2. Related Work

Prior work studies weakly supervised proxy tasks, self-supervised pretext tasks, and model distillation. ClusterFit builds on these areas while targeting pretraining-task bias through post-hoc clustering.

2. Related Work

  • Weakly supervised image pretraining uses metadata-based proxy tasks, and ClusterFit uses features from those models to mitigate noise while retaining data for complementary visual information.Unlike target-specific noise removal that ranks images and discards most of the data, ClusterFit is not tied to a particular target task and leverages all the data.Source: §2 · p. 2
  • Self-supervised methods use visual pretext tasks, while ClusterFit reports larger improvements for self-supervised models because pretext and semantic transfer tasks differ substantially.The paper evaluates Jigsaw and RotNet image-based methods.Source: §2 · p. 2
  • Compared with clustering-based self-supervised methods, ClusterFit uses post-hoc clustering without alternate optimization; unlike, it targets pretraining-task bias rather than distilling to a common architecture.The paper says this makes ClusterFit more stable, computationally efficient, and applicable across fully supervised, noisy, weakly supervised, and self-supervised models.Source: §2 · p. 2
  • Standard distillation transfers teacher knowledge through predictions, but the paper reports that retaining more original-task knowledge performs poorly under weak supervision and cites similar findings for self-supervision.The paper contrasts this with ClusterFit’s lossy-clustering approach.Source: §2 · p. 3

3. Approach

ClusterFit extracts features from another dataset with a pretrained network, clusters them, and trains a new network on the resulting pseudo-labels. In a synthetic-noise control, its transfer advantage grows with noise and is strongest on the finer-grained targets.

3.1. ClusterFit Framework

  • The framework extracts penultimate-layer features from Dcf, clusters them into K groups with k-means, and trains a different network from scratch on the assignments with cross-entropy.The paper describes this retraining as unsupervised and evaluates transfer with linear classifiers on fixed features.Source: §3.1 · p. 3

3.2. Control Experiment using Synthetic Noise

  • The control experiment fixes ImageNet-1K as both pretraining and clustering data and uses ResNet-50 for both networks, varying uniform random label noise and K.It evaluates fixed res5 features on ImageNet-1K, ImageNet-9K, and iNaturalist; the authors caution that synthetic noise oversimplifies real-world noise.Source: §3.2 · p. 3
  • At p = 0, the clean pretrained model significantly outperforms ClusterFit on ImageNet-1K, while ClusterFit can match or exceed it on ImageNet-9K and iNaturalist at higher K.The reported comparison distinguishes an aligned pretraining and transfer task from the finer-grained targets.Source: §3.2 · pp. 3–4
  • As label noise increases, ClusterFit’s performance gap over the noisy pretrained model widens; it consistently wins on ImageNet-9K and iNaturalist and wins on ImageNet-1K when p ≥ 50.Model distillation gives some gains over the pretrained model but is consistently outperformed by ClusterFit in noisy-label settings.Source: §3.2 · p. 4
  • The authors suggest that clustering captures useful visual invariances and creates more noise-resilient pseudo-labels; larger K generally improves transfer, especially on challenging, fine-grained datasets.They say the gains are larger on ImageNet-9K and iNaturalist.Source: §3.2 · p. 4

4. Experiments

The experiments compare ClusterFit with the original pre-training across weakly supervised images and videos, and self-supervised images. Across these settings, ClusterFit improves transfer performance, with task-specific exceptions and patterns described below.

4. Experiments

  • Npre and Ncf use the same dataset, architecture, and training duration in the common ClusterFit setting, and transfer is evaluated with linear classifiers on fixed features.The experiments cover weakly supervised videos, weakly supervised images, and self-supervised images.Source: §4 · p. 4, Table 3 · p. 5

4.1.1 Weakly-supervised image pre-training

  • Ncf gains 4.6 points on ImageNet-9K and 5.8 on iNaturalist over Npre, but loses 1.5 points on ImageNet-1K.The authors attribute the ImageNet-1K decrease to the hand-crafted label alignment between IG-ImageNet-1B and ImageNet-1K.Source: Table 4 · p. 5, §4.1.1 · p. 5

4.1.2 Weakly-supervised video pre-training

  • 3.2–4.3%: ClusterFit improves transfer over Npre across all weakly supervised video datasets.The best cluster count varies by target dataset and is typically an order of magnitude larger than the original 438-verb label space.Source: Table 5 · p. 5, §4.1.2 · p. 6

4.2. Self-Supervised pre-training for Images

  • ClusterFit consistently improves Jigsaw and RotNet across pre-training and target tasks, with 3–5-point gains on Places205 and 5–8-point gains on iNaturalist.The layerwise analysis also finds improved transfer at every layer except conv1, with a smaller res4–res5 performance gap after ClusterFit.Source: Table 6 · p. 6, §4.2 · p. 6

5. Analyzing ClusterFit

The analysis examines how model capacity, clustering strategy, and the pre-training label space relate to ClusterFit transfer performance. Results favor richer pre-training features, unsupervised clustering, and greater label-space diversity.

5. Analyzing ClusterFit

  • After ClusterFit, transfer improves at every examined layer except conv1, and the res4–res5 performance gap is reduced.Before ClusterFit, res5 features transfer worse than res4 features.Source: Figure 4 · p. 7

5.1. Relative model capacity of Npre and Ncf

  • With Ncf fixed as R(2+1)D-18 on Kinetics, using a higher-capacity Npre, R(2+1)D-34, rather than matching Npre consistently improves transfer performance by 2%–3% across different K values.Source: Figure 5 · p. 7, §5.1 · p. 7

5.2. Unsupervised vs. Per-Label Clustering

  • Unsupervised clustering outperforms per-label clustering on Kinetics and Sports1M across all tested K values.The authors suggest feature-based clustering captures visual coherence and addresses factors around label noise, including wrong or missing labels and lexical ambiguity.Source: Figure 6 · p. 7, §5.2 · p. 8

5.3. Properties of Dpre

  • Using the top 10 pretraining verbs (m=10) drops accuracy around 9% versus m=438; with m=100, Ncf is within 2% of all-438 accuracy and outperforms the weakly supervised counterpart using 62M videos and all 438 verbs.Transfer performance increases log-linearly with the number of pretraining labels.Source: §5.3 · p. 8

6. Discussion

The discussion frames ClusterFit as a way to improve generalization by relearning from clustered features. It also notes a limitation of unsupervised iteration and identifies the framework's flexibility.

6. Discussion

  • ClusterFit treats clustering as lossy compression of the original feature space, retaining essential visual invariances for a less pre-training-sensitive, more transferable representation.The new network is trained to predict the cluster assignments.Source: §6 · p. 8
  • In its current unsupervised form, iterating ClusterFit provides little improvement; downstream domain knowledge may help guide clustering and transfer.Source: §6 · p. 8
  • ClusterFit is described as scalable, with no restrictions on model architectures, data modalities, or forms of supervision.The discussion proposes combining different types of pre-trained models for multi-task cluster assignment as a future direction.Source: §6 · p. 8

A. Details on baseline methods

The section describes a distillation loss combining teacher soft-target matching with ground-truth supervision, and a prototype baseline that reassigns examples to label-derived cluster centers.

A. Details on baseline methods

  • The distillation loss combines teacher soft-target matching and ground-truth cross-entropy with weights α and 1−α.The reported grid-search settings are T = 20 and α = 0.75.Source: A. Details on baseline methods · p. 11
  • The prototype baseline initializes centers with label-wise mean features, reassigns examples to their nearest centers, then trains on the reassigned cluster ids.It requires labels and yields as many clusters as the input label space has classes.Source: A. Details on baseline methods · p. 11

B. Weakly supervised Images

The appendix gives dataset, pre-training, clustering, and transfer-learning details for weakly supervised images and videos.

B. Weakly supervised Images

  • IG-ImageNet contains 1,500 or more hashtags because multiple hashtags can map to the same ImageNet synset.Source: B. Weakly supervised Images · p. 11
  • Image pre-training follows on 128 GPUs across 16 machines, processes 2B images, and uses an initial learning rate of 1.6.Each GPU processes 32 images; the learning rate is halved at 13 equally spaced steps.Source: B. Weakly supervised Images · p. 11
  • Image ClusterFit runs 30 k-means iterations on a random 200M-image sample, then five more iterations using all 1B images.Source: B. Weakly supervised Images · p. 11

C. Weakly supervised Videos

  • IG-Verb-62M retains labels with at least 50 matching videos and contains 438 verbs and 62M videos; IG-Verb-19M is a tail-preserving subset.When a video has multiple hashtags, one verb is randomly selected as its label.Source: C. Weakly supervised Videos · p. 11
  • IG-Kinetics-19M has 359 Kinetics labels, while IG-Noun-19M has 1,438 ImageNet synset labels; both support studies of label type.Source: C. Weakly supervised Videos · p. 11
  • Video pre-training for Npre and Ncf uses 128 GPUs across 16 machines and processes 490M videos across all epochs.Each GPU processes six videos, with batch normalization on convolutional layers; transfer learning follows [20] with a learning-rate sweep.Source: C. Weakly supervised Videos · p. 11

D. Self-supervised Images

The section details self-supervised image ClusterFit, its transfer evaluation, and a limitation of distillation baselines for transformation-dependent predictions.

D. Self-supervised Images

  • ClusterFit clusters l2-normalized, 2,048-dimensional res5 features from self-supervised Npre models on ImageNet-1K to generate pseudo-labels.Source: D. Self-supervised Images · p. 12
  • Clustering res4 features produces performance similar to clustering res5 features for training Ncf.Source: D. Self-supervised Images · p. 12
  • Transfer evaluation trains linear classifiers on fixed features, with layer-wise average pooling and roughly constant parameter updates across datasets.The section uses linear SVMs for VOC07.Source: D. Self-supervised Images · p. 12
  • ClusterFit improves over Npre even at K = 1000, while Ncf performance increases as the number of clusters grows.Table 7 reports linear evaluation across four datasets, using VOC07 mAP and top-1 accuracy elsewhere.Source: D. Self-supervised Images · p. 12, Table 7 · p. 12
  • Distillation is not a valid baseline for Jigsaw and RotNet because their predictions depend on the input transformation applied.An untransformed image does not provide a meaningful output distribution for distillation in these methods.Source: D. Self-supervised Images · p. 12

E. Bonus: Multi-task Self-supervised Learning

ClusterFit trains a shared network using pseudo-labels from Jigsaw and RotNet, and this multi-task model improves transfer results over the single-task models.

E. Bonus: Multi-task Self-supervised Learning

F. Analysis of ClusterFit

The analysis varies the labels used to pre-train the network while applying ClusterFit on IG-Verb-62M. All three label choices improve transfer performance over the weakly-supervised upper bound, with a slight Kinetics advantage for Kinetics pre-training.

F. Analysis of ClusterFit

Related papers

Loading 1912.03330v1…