PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels

Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, Sanghyuk Chun

arXiv:2101.05022v2cs.CV

TL;DR

ImageNet’s single-label training annotations mismatch images and random crops containing multiple objects, producing noisy supervision and making exhaustive human relabeling impractical. The paper uses a strong external classifier to generate localized multi-label maps and trains with crop-specific LabelPooling. ReLabel improves ImageNet classification, transfer learning, multi-label evaluation, and robustness, reaching 78.9% top-1 accuracy for ResNet-50 and 80.2% with CutMix.

  • Problem

    ImageNet contains many effectively multi-label images, while single-label training and random crops create noisy or incorrect supervision; exhaustive human relabeling is impractical.

  • Method

    ReLabel uses a strong classifier trained on external data to generate localized pixel-wise multi-label maps, and LabelPooling pools map scores over each random crop.

  • Results

    ReLabel consistently improves ImageNet, transfer-learning, multi-label, and robustness results; ResNet-50 reaches 78.9% top-1 accuracy and 80.2% with CutMix.

  • Takeaways & Limitations

    Localized multi-label supervision provides consistent gains across classification, downstream tasks, multi-label evaluation, and robustness benchmarks.

Abstract

from arXiv · show

ImageNet has been arguably the most popular image classification benchmark, but it is also the one with a significant level of label noise. Recent studies have shown that many samples contain multiple classes, despite being assumed to be a single-label benchmark. They have thus proposed to turn ImageNet evaluation into a multi-label task, with exhaustive multi-label annotations per image. However, they have not fixed the training set, presumably because of a formidable annotation cost. We argue that the mismatch between single-label annotations and effectively multi-label images is equally, if not more, problematic in the training setup, where random crops are applied. With the single-label annotations, a random crop of an image may contain an entirely different object from the ground truth, introducing noisy or even incorrect supervision during training. We thus re-label the ImageNet training set with multi-labels. We address the annotation cost barrier by letting a strong image classifier, trained on an extra source of data, generate the multi-labels. We utilize the pixel-wise multi-label predictions before the final pooling layer, in order to exploit the additional location-specific supervision signals. Training on the re-labeled samples results in improved model performances across the board. ResNet-50 attains the top-1 classification accuracy of 78.9% on ImageNet with our localized multi-labels, which can be further boosted to 80.2% with the CutMix regularization. We show that the models trained with localized multi-labels also outperforms the baselines on transfer learning to object detection and instance segmentation tasks, and various robustness benchmarks. The re-labeled ImageNet training set, pre-trained weights, and the source code are available at {https://github.com/naver-ai/relabel_imagenet}.

1. Introduction

ImageNet’s single-label annotations conflict with images and random crops containing multiple or absent foreground objects, creating noisy training supervision. ReLabel addresses this by generating localized multi-label maps with a strong external classifier, improving classification and downstream performance.

  • Motivation: ImageNet often contains multiple possible classes, but single-label evaluation penalizes predictions beyond the annotated class.Recent work therefore introduced multi-label validation annotations and metrics.
  • Motivation: Random crops can contain a different object or no foreground object, making single-label supervision noisy during training.Only 23.5% of sampled crops have IoU greater than 50% with ground-truth boxes, while 8% have no overlap.
  • Annotation challenge: Human pixel-wise multi-label and localization annotations are desirable but impractical for 1.28 million ImageNet training images.Five human experts needed more than three months to label 2,000 images.
  • Method: ReLabel uses a strong classifier trained on external data to generate pixel-wise multi-label maps, and LabelPooling pools scores over each random crop.The labels are generated once per dataset, and LabelPooling adds only a small computational cost beyond standard single-label supervision.
  • Results: 78.9% top-1 accuracy is achieved by ResNet-50 with ImageNet ReLabel, a +1.4 pp gain over original-label training.CutMix further raises accuracy to 80.2%, while ReLabel also improves multi-label evaluation, transfer learning, and robustness benchmarks.

2. Related Works

Prior work identifies systematic single-label errors in ImageNet and improves validation labels or removes questionable training samples. ReLabel instead corrects training labels with machine-generated, region-specific multi-label supervision.

  • ImageNet labeling issues: ImageNet contains systematic erroneous single labels because many images include multiple classes, synonymous or hierarchical labels, or inherent ambiguity.These error categories were identified in analyses of ImageNet label reliability.
  • Prior approaches: Prior studies refined validation labels into multi-labels, while other work removed training samples with potentially erroneous labels using strong-classifier predictions.Those approaches addressed validation or sample removal rather than correcting all training labels.
  • ReLabel distinction: ReLabel uses strong classifiers to correct ImageNet training labels rather than remove samples, while assigning labels per image region.The paper reports improved results compared with the removal-based approach.

3. Method

ReLabel uses strong external-data classifiers to generate localized multi-label maps, while LabelPooling adapts supervision to each random crop. The method preserves both multi-labelness and localization, which jointly drive its gains over machine-generated global single labels.

  • Re-labeling ImageNet: ReLabel uses strong classifiers trained on extra data to generate dense, pixel-level multi-label maps for ImageNet training images.The maps use predictions before final pooling and are intended to encode both multiple classes and object locations.
  • Re-labeling ImageNet: Single-label cross-entropy trained on noisy multi-class examples tends to produce multi-label predictions.When examples with multiple correct categories receive competing single labels, the loss is minimized by distributing probability across those categories.
  • Training a Classifier with Dense Multi-labels: LabelPooling pools the precomputed label map over each random crop before applying global average pooling and softmax.It uses RoIAlign to derive crop-specific multi-label supervision rather than retaining the image-level single label.
  • Discussion: Saving all 15×15×1000 label maps for 1.28 million images would require more than 1 TB of storage.The implementation therefore faces substantial storage demands even with relatively low-resolution maps.
  • Discussion: LabelPooling adds only 0.5% training time, whereas knowledge distillation with an EfficientNet-B7 teacher takes more than four times the original training time.ReLabel is computed once, avoiding teacher inference at every training iteration.
  • Discussion: 78.9% performance falls by 0.5 pp without multi-labels, 0.4 pp without localization, and 1.4 pp without both.Global machine-generated single labels provide no gain over the original ImageNet labels, indicating that both properties are important.

4. Experiments

Experiments show that ReLabel improves ImageNet classification across supervision metrics and architectures, transfers to downstream tasks, and strengthens robustness. Its gains remain competitive with knowledge distillation while requiring substantially less training time, and localized labels also help multi-label classification.

  • Multiple architectures: ReLabel improves validation top-1 accuracy across multiple network architectures compared with original-label supervision.The experiments include ResNet, EfficientNet, and ReXNet variants.
  • ImageNet classification: ReLabel consistently achieves the best performance across single-label and multi-label ImageNet evaluation metrics.It reaches 78.9% validation accuracy, a +1.4 pp gain over original labels, while also improving ImageNetV2, ReaL, and Shankar et al. metrics.
  • Additional training strategies: 80.2% ImageNet top-1 accuracy is achieved by combining ReLabel with CutMix for ResNet-50.ReLabel with CutMix reaches 81.6% for ResNet-101, while adding extra ImageNet-21K data raises ResNet-50 to 81.2%.
  • Knowledge distillation: ReLabel reaches comparable accuracy to larger-teacher knowledge distillation with 13.6 GPU days, versus 41 or 78 GPU days for KD with EfficientNet-B5 or B7.KD with B7 reaches 79.0%, compared with 78.8% for ReLabel.
  • Robustness: ReLabel improves robustness to adversarial, natural, and background perturbations, including a +8.7% gain on background robustness.Combining ReLabel with CutMix and extra data further boosts performance across robustness benchmarks.
  • Transfer learning: ReLabel-trained models improve transfer learning for fine-grained classification, object detection, and instance segmentation.On FGVC Aircraft, ReLabel pre-training improves downstream performance by +3.8 pp.
  • Multi-label classification: Machine-generated localized label maps improve COCO multi-label classification by +3.7 pp and +2.4 pp mAP in the reported settings.The experiments compare machine-generated and segmentation-ground-truth label maps using LabelPooling.

5. Conclusion

The paper introduces ReLabel for converting ImageNet’s single-class image labels into localized multi-class labels and LabelPooling for training with them. Across ImageNet, transfer-learning, and multi-label tasks, the approach produces consistent performance gains.

  • Contribution: ReLabel assigns machine-generated multi-class labels to every image region in ImageNet’s 1.28 million training images.The machine annotator is a strong classifier trained on an additional source of visual data.
  • Contribution: LabelPooling trains classifiers with localized multi-class labels rather than the original single-class labels.The paper presents ReLabel and LabelPooling as complementary components of the training approach.
  • Findings: ReLabel yields consistent gains across ImageNet benchmarks, transfer-learning tasks, and multi-label classification tasks.The authors state that localized labels and the corresponding training technique induce significant performance gains.

A. ReLabel Algorithm

The ReLabel algorithm loads an image and its label map, applies a random crop, pools the corresponding label-map region, and uses the resulting multi-label target for model training. The label-map construction removes global average pooling so predictions retain spatial structure.

  • Training loop: Each iteration loads an image and saved label map, then applies random cropping and resizing to the input.The pseudocode assumes minibatch size one for simplicity.
  • LabelPooling: LabelPooling applies RoIAlign to the cropped label-map region and softmaxes the pooled scores into a multi-label target.The pooled target has one value per class and corresponds to the crop coordinates.
  • Model update: The model prediction is compared with the pooled target using cross-entropy loss before updating the model.The update completes each training iteration.
  • Label-map construction: The modified classifier outputs a spatial label map by replacing global average pooling with a 1 × 1 convolution sharing the original fully connected weights.The original classifier outputs a 1 × 1 × C label, whereas the modified classifier outputs an H × W × C map.
  • Examples: Label-map examples visualize heatmaps for the top-1 and top-2 predicted classes at their object locations.The figure presents the input image alongside the two class-specific heatmaps.

B. Re-labeling ImageNet: Detailed Procedure and Examples

The detailed procedure uses a strong classifier without global average pooling to generate spatial class scores for every ImageNet training image. These maps provide localized heatmaps whose top predicted classes occupy distinct object regions.

  • Classifier modification: The original classifier produces global class predictions after feature extraction and global average pooling, while the modified classifier retains spatial predictions.Both classifiers use equivalent final class weights, implemented as a fully connected layer or 1 × 1 convolution.
  • Machine annotator: EfficientNet-L2 generates label maps from uncropped 475 × 475 ImageNet images with spatial size 15 × 15 and 1,000 class channels.The classifier’s feature depth is 5,504.
  • Label-map examples: Top-1 and top-2 class heatmaps are disjointly located at the positions of their respective objects.The examples extract one heatmap per class from the H × W × C label map.

C. Results on ImageNetV2

ReLabel improves ResNet-50 performance across three ImageNetV2 evaluation metrics compared with the vanilla baseline.

  • 80.5 accuracy on ImageNetV2 “Top-Images” represents a +1.5 pp gain over vanilla ResNet-50.
  • 67.3 accuracy on ImageNetV2 “Matched Frequency” represents a +2.1 pp gain over vanilla ResNet-50.
  • 76.0 accuracy on ImageNetV2 “Threshold 0.7” represents a +1.7 pp gain over vanilla ResNet-50.

D. Implementation details

This section presents the implementation details for the paper’s experiments.

  • The paper introduces this section as the location for its implementation details.

D.1. Training Hyper-parameters

Training generally uses SGD, while AdamP is used for experiments with additional performance-improving techniques or extra training data.

  • Most experiments use SGD with learning rate 0.1 and weight decay 0.0001.
  • AdamP uses learning rate 0.002 and weight decay 0.01 when applying CutMix or extra ImageNet-21K training data.
  • EfficientNet variants are trained for 400 epochs with minibatch size 512, AdamP, and multiple regularization and augmentation techniques.
  • Knowledge distillation uses official EfficientNet B1-B7 teachers and applies only the teacher-student cross-entropy distillation loss.
  • COCO multi-label fine-tuning replaces the ImageNet classifier’s 1000-class output with an 80-class randomly initialized layer.

E. ReLabel Examples on ImageNet

ReLabel examples compare full ImageNet training images with random crops and show location-specific multi-label supervision alongside the original labels.

  • ReLabel examples display each full training image beside its random cropped patch, with crop coordinates marked by blue bounding boxes.
  • ReLabel generates location-specific multi-labels that provide more precise supervision than the original ImageNet label.
  • The examples present both the original ImageNet label and ReLabel’s new multi-labels for each training image and crop.
Loading 2101.05022v2…