PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta

arXiv:1707.02968v2cs.CVcs.AI

TL;DR

The paper examines how scaling beyond ImageNet-sized datasets changes visual deep learning and uses JFT-300M for representation learning. It finds logarithmic performance growth with training-data volume, continued gains from better base models, and new state-of-the-art results across vision tasks.

  • Problem

    The paper asks how increasing visual training data by 10x or 100x affects performance, given that datasets have lagged behind advances in models and computation.

  • Method

    The paper uses JFT-300M, containing 300M images and 375M automatically generated labels, to study large-scale representation learning for vision tasks.

  • Results

    Performance increases logarithmically with training-data volume, representation learning improves every studied vision task, and models achieve new state-of-the-art benchmark results.

  • Takeaways & Limitations

    Better base models and features remain valuable, while ImageNet is insufficient to use all model parameters or representational capacity.

  • Takeaways & Limitations

    The reported quantitative performance may underestimate data's impact because the training regime and hyperparameters were based on experience with 1M images and training was shortened to four epochs.

Abstract

from arXiv · show

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

1. Introduction

The paper examines what happens when visual representation learning scales beyond stagnant dataset sizes, using noisy JFT data to study data volume, model capacity, and downstream performance. It reports logarithmic gains, stronger representations, and state-of-the-art results across vision benchmarks.

  • 1M labeled ImageNet images remained standard even as GPU computation and model capacity increased, motivating larger-scale pre-training.
  • JFT contains more than 300M automatically labeled images, with approximately 20% label-precision error, enabling analysis of enormous noisy data.
  • Better representation learning improves performance on every vision task studied, supporting continued investment in large-scale pre-training.
  • Larger training volumes improve vision-task performance logarithmically during representation learning.
  • Higher model capacity is needed to fully exploit 300M images: COCO detection gains were 1.87% with ResNet-50 versus 3% with ResNet-152.
  • A single JFT-trained model achieved 37.4 AP versus 34.3 AP on COCO detection, alongside new state-of-the-art results on several benchmarks.

2. Related Work

Prior work increased model capacity, used pre-training and web or unsupervised supervision, and studied dataset scale and bias. The paper extends this line by directly examining how data affects visual deep learning and representation learning.

  • ConvNet research increasingly improved accuracy by expanding network width, depth, and representational capacity.
  • Large-scale visual representations support many tasks, but detection and segmentation datasets remain much smaller than ImageNet, encouraging pre-training.
  • Pre-training data remained largely fixed because human-verified datasets larger than ImageNet were scarce, prompting web-supervised and unsupervised alternatives.
  • Earlier studies linked data expansion, category selection, and dataset bias to downstream performance, but the paper focuses specifically on data and visual deep learning.
  • Unlike prior YFCC-100M work reporting plateauing detection performance, this paper studies JFT, which offers broader web diversity and reduced label noise.

3. The JFT-300M Dataset

JFT-300M is a massive, hierarchically organized dataset of automatically labeled web images. Its scale and diversity come with noisy, incomplete annotations and a heavily long-tailed category distribution.

  • JFT-300M contains 300M images and 375M labels across 18,291 categories, averaging 1.26 labels per image.
  • Its categories form a rich hierarchy with maximum depth 12 and up to 2,876 children per parent category.
  • Labels are generated automatically from web signals, webpage connections, and user feedback, with approximately 20% noisy labels and unknown recall.
  • Figure 2 characterizes label confusion and incorrect labels, using quantized distances to visual-feature K-Means centroids on the x-axis.
  • The category distribution is heavily long-tailed, ranging from more than 2M flower images to only 131 train-conductor images.

4. Training and Evaluation Framework

The paper trains and evaluates ResNet-101 representations on JFT-300M, comparing initialization strategies and assessing transfer across vision tasks. Training is computationally distributed, and the full schedule is shortened for practical reasons.

  • Training setup: ResNet-101 is trained on JFT-300M with 18,291 output labels using per-label logistic loss for non-mutually-exclusive annotations.The model uses a standard 101-layer Residual Network architecture with a fully connected classification layer.
  • Training setup: The training pipeline uses asynchronous gradient descent on 50 NVIDIA K80 GPUs with Downpour SGD and distributed parameter servers.The final classification layer is split across 50 parameter-server sub-layers because it contains over 36M parameters.
  • Baseline: The ImageNet baseline uses separately selected hyperparameters because JFT-300M training settings are sub-optimal for ImageNet.The baseline trains for 5M steps with momentum optimization and scheduled learning-rate reductions.
  • Baseline: The baseline ResNet-101 performs 1% better than the authors’ open-sourced ResNet-101 checkpoint under the same evaluation protocol.This establishes the paper’s baseline relative to the referenced released checkpoint.
  • Evaluation: FastEval14k monitors JFT-300M training using mAP@100 on 14,000 densely annotated images spanning 6,000 classes.The metric is mean average precision over the top 100 predictions.
  • Evaluation: ImageNet initialization performs better during the first 15M iterations, then becomes comparable to random initialization on FastEval14k.Figure 3 plots training steps against mAP@100 for the two initialization strategies.
  • Training scope: The complete training schedule is 90M iterations, but experiments use 36M iterations because the full schedule takes approximately two months.The shortened run corresponds to roughly four rather than ten epochs.
  • Downstream evaluation: Visual representations are evaluated both as frozen feature extractors and through fine-tuning on object detection, semantic segmentation, and human pose estimation.The framework also reports JFT-300M improvements over the ImageNet ResNet baseline for the other tasks.

5. Experiments

The experiments evaluate JFT-300M representations across image classification, object detection, semantic segmentation, and human pose estimation, with detailed analyses of data size, training duration, initialization, and model capacity. Across these evaluations, larger-scale pretraining improves downstream performance, often logarithmically, and JFT-300M checkpoints outperform ImageNet baselines on detection and segmentation comparisons.

  • The experiments fine-tune JFT-300M ResNet-101 checkpoints on image classification, object detection, semantic segmentation, and human pose estimation.
  • Object Detection: 3.3% and 4.4% boosts in mAP@.5, and 2.4% and 3.1% boosts in mAP@[.5,.95], are reported for JFT-300M checkpoints over the ImageNet baseline on COCO test-dev.The two JFT-300M settings are trained from scratch and from ImageNet initialization, respectively.
  • Object Detection: 5.1% and 5.0% mAP@.5 improvements are reported for the two JFT-300M checkpoints over the ImageNet baseline on PASCAL VOC 2007 test.
  • Impact of Epochs: Performance improves as JFT-300M training steps increase, while ImageNet checkpoints improve faster with respect to epochs.The comparison uses JFT-300M checkpoints trained for 1.3, 2.6, and 4 epochs, versus ImageNet checkpoints trained for 3, 6, 12, and 150 epochs.
  • Impact of Data Size: Detection performance grows logarithmically as pretraining data expands, with fine-tuning outperforming frozen feature extractors at every data size.
  • Impact of Classes: Models trained with 941 labels perform on par with models trained using the full label set, indicating that the benefit comes from more training images instead of more labels.
  • Semantic Segmentation: JFT-300M initialization outperforms ImageNet by 1.7% points, while additional JFT-300M training from an ImageNet checkpoint provides a 2.9% points boost over vanilla ImageNet initialization.

6. Discussions

The discussion argues that data remains a major source of progress: larger datasets improve representation learning and can yield gains even as models plateau. However, the reported improvements may underestimate data’s impact because training settings were inherited from 1M-image regimes.

  • Modest performance improvements remain possible with exponential increases in data, while models appear to be plateauing.
  • Larger-scale data improves representation learning across every vision task studied.
  • Improved base models or features can produce significant performance gains on downstream tasks.
  • The reported quantitative gains may underestimate data’s impact because training regimes and schedules were based on 1M-image ConvNet training.Training a JFT model for four epochs required two months on 50 K-80 GPUs.

De-duplication Experiments

The experiments address possible train–validation overlap in large datasets by identifying near duplicates and reporting results before and after deduplication. COCO test-dev remains outside the deduplication analysis because its ground-truth annotations are unavailable.

  • 300M-image datasets can contain images overlapping target-task validation sets; ImageNet already has 890 near duplicates among 50K validation images.
  • Near duplicates were identified across ImageNet, COCO, and Pascal VOC validation or test sets using visual embeddings.The reported counts are 5536 ImageNet, 1648 COCO, 201 Pascal VOC 2007, and 84 Pascal VOC 2012 images.
  • Deduplication comparisons cover ImageNet classification, COCO object detection, and Pascal VOC detection and segmentation metrics.The associated tables report top-1/top-5 accuracy, COCO mAP metrics, and Pascal VOC mAP@0.5 and mIOU.
  • COCO test-dev detection and pose-estimation experiments were not deduplicated because their ground-truth annotations are unavailable.

Detailed and Per-category Results: Object Detection

The object-detection appendix reports detailed results on COCO test-dev, including performance by initialization and per-category metrics.

  • The appendix reports detailed AP and AR results for different initializations and per-category AP and AP@.5 results.
  • Per-category object-detection performance is presented alongside aggregate initialization comparisons.
  • Object-detection results are evaluated on the COCO test-dev split using different model initializations.

Per-category Results: Semantic Segmentation

The semantic-segmentation appendix reports per-class results on the PASCAL VOC 2012 validation set, with improvements exceeding seven points for some categories.

  • More than 7 point improvement is observed for categories including boat and horse.
  • Per-class semantic-segmentation results are reported on the PASCAL VOC 2012 validation set.

Detailed Results: Human Pose Estimation

The appendix reports human pose estimation performance on the COCO test-dev split using AP and AR results, while a separate table covers per-class object detection.

  • Human Pose Estimation: Table 13 reports all AP and AR results for the human pose estimation performance presented in Table 7.
  • Human Pose Estimation: The reported human pose estimation results use the COCO test-dev split.
  • Related Object Detection Results: Table 14 separately reports per-class object detection performance on COCO test-dev using different model initializations.
Loading 1707.02968v2…