PaperlayerView exact revision on arXiv

Source-linked AI summary

CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo

arXiv:1905.04899v2cs.CVcs.LG

TL;DR

Regional dropout can improve generalization and localization but wastes informative training pixels by replacing regions with zeros or noise. CutMix replaces those regions with patches from other images and mixes labels proportionally, achieving significant gains across classification, localization, and transfer tasks.

  • Problem

    Regional dropout improves classification and localization but wastes informative pixels by zeroing or noising deleted regions, raising how to use them while retaining its benefits.

  • Method

    CutMix replaces dropped image regions with patches from other training images and mixes labels according to the combined pixels’ proportions.

  • Results

    CutMix consistently achieves significant enhancements across three tasks, including +2.28% top-1 accuracy improvements on ImageNet with ResNet-50.

  • Takeaways & Limitations

    CutMix offers a single augmentation strategy that improves classification, localization, and transfer-task performance without the tradeoffs reported for Mixup and Cutout.

Abstract

from arXiv · show

Regional dropout strategies have been proposed to enhance the performance of convolutional neural network classifiers. They have proved to be effective for guiding the model to attend on less discriminative parts of objects (e.g. leg as opposed to head of a person), thereby letting the network generalize better and have better object localization capabilities. On the other hand, current methods for regional dropout remove informative pixels on training images by overlaying a patch of either black pixels or random noise. Such removal is not desirable because it leads to information loss and inefficiency during training. We therefore propose the CutMix augmentation strategy: patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. By making efficient use of training pixels and retaining the regularization effect of regional dropout, CutMix consistently outperforms the state-of-the-art augmentation strategies on CIFAR and ImageNet classification tasks, as well as on the ImageNet weakly-supervised localization task. Moreover, unlike previous augmentation methods, our CutMix-trained ImageNet classifier, when used as a pretrained model, results in consistent performance gains in Pascal detection and MS-COCO image captioning benchmarks. We also show that CutMix improves the model robustness against input corruptions and its out-of-distribution detection performances. Source code and pretrained models are available at https://github.com/clovaai/CutMix-PyTorch .

1. Introduction

CutMix replaces erased image regions with patches from other training images and mixes labels according to pixel area, preserving informative pixels while retaining regional-dropout regularization. It consistently improves classification, localization, and detection performance across the evaluated tasks, including ImageNet gains and a 14.47% top-1 error.

  • Motivation: Regional dropout improves generalization and localization by preventing CNNs from focusing exclusively on limited activations or small image regions.Prior methods include dropout over hidden activations and regional dropout over input-image regions.
  • Motivation: Zeroing or noise-filling deleted regions reduces informative training pixels, a conceptual limitation for data-hungry CNNs.CutMix is introduced to maximize use of those deleted regions without giving up regional-dropout benefits.
  • CutMix: CutMix replaces removed regions with patches from another image and mixes ground-truth labels proportionally to the combined images’ pixel counts.This avoids uninformative training pixels while retaining regional dropout’s tendency to direct attention toward non-discriminative object parts.
  • Results: CutMix consistently enhances image classification, weakly supervised localization, and transfer learning for object detection, unlike Mixup and Cutout’s declines on localization or detection.The comparison is summarized in Table 1 across the three tasks.
  • Results: 14.47% top-1 error is CutMix’s reported state-of-the-art result, while ResNet-50 and ResNet-101 ImageNet classification accuracy improves by +2.28% and +1.70%, respectively.The reported improvements apply CutMix to the two named architectures.

2. Related Works

The related work positions CutMix among regional dropout, synthetic-data, Mixup, and deep-network training methods. Its distinguishing approach is to combine samples by replacing removed image regions with patches from other training images while mixing labels proportionally.

  • Regional dropout: Regional dropout removes random image regions to improve CNN generalization, while object-localization methods use it to improve localization.CutMix differs by filling removed regions with patches from another training image; DropBlock extends regional dropout to feature space.
  • Synthesizing training data: Synthetic-data methods such as Stylized ImageNet encourage shape-focused representations, whereas CutMix cuts and pastes mini-batch patches with lower computational cost than stylization.Both approaches are associated with improved classification and detection performance, while CutMix generates new samples through patch operations.
  • Mixup: CutMix resembles Mixup because both combine two samples and assign labels through linear interpolation of one-hot labels.The paper contrasts Mixup’s locally ambiguous, unnatural samples with CutMix’s patch-based construction, especially for localization.
  • Tricks for training deep networks: Deep-network training commonly uses weight decay, dropout, and Batch Normalization, alongside feature-noise methods and architectures with extra paths.These methods address the computational and data demands of training deep networks and aim to enhance image classification performance.

3. CutMix

CutMix trains on samples formed by replacing a region of one image with a patch from another and mixing labels by the corresponding area. This preserves pixel efficiency while encouraging recognition from partial object views, accurate localization of multiple objects, and reduced overfitting.

  • Algorithm: CutMix combines two training samples by replacing a masked region of one image with a patch from another and training with the original loss.The mask is binary, and the generated image-label pair is used as a new training sample.
  • Algorithm: The combination ratio λ is sampled from Beta(α, α); with α = 1 in all experiments, λ follows a uniform distribution on (0, 1).The cropped area ratio is tied to 1−λ.
  • Motivation: CutMix uses two objects in one image to promote recognition from partial views while retaining the full-object cue motivation of regional dropout.This combines training efficiency with the goal of attending to less discriminative object parts.
  • Visualization: CutMix localizes both object classes accurately, whereas Cutout wastes pixels and Mixup introduces unnatural artifacts that confuse recognition cues.The comparison uses class activation maps for Saint Bernard and Miniature Poodle with a vanilla ResNet-50.
  • Training stability: CutMix achieves lower validation error than the baseline at the end of training and avoids the overfitting-related increase seen after epoch 150.The experiments use ResNet-50 on ImageNet and PyramidNet-200 on CIFAR-100; diverse training samples produce a steady decrease in validation error.

4. Experiments

The experiments evaluate CutMix across classification, weakly supervised localization, transfer to detection and captioning, robustness, and uncertainty. Results show improvements over competing augmentation and regularization methods across these tasks.

  • Experimental scope: The study evaluates CutMix for model localizability and generalizability across classification, weakly supervised localization, transfer learning, robustness, and uncertainty tasks.Classification and weakly supervised localization are studied first, followed by transfer to object detection and image captioning, then robustness and uncertainty evaluations.
  • Image classification: CutMix achieves 14.47% top-1 classification error on CIFAR-100, compared with the baseline performance 16.45%, and reaches 13.81% when combined with ShakeDrop.It also improves weaker CIFAR-100 architectures and enhances CIFAR-10 classification performance by +0.97%, outperforming Mixup and Cutout.
  • Image classification: CutMix performs best when applied at the input-image level, while most feature-level variants improve accuracy over the 16.45% baseline.The layer index ranges from 0 for image-level application to 4 after layer3; the layer3 case is the exception.
  • Weakly supervised localization: CutMix outperforms Cutout by +2.03% on CUB200-2011 and +0.56% on ImageNet in localization accuracy, while remaining comparable to dedicated WSOL methods.The passage attributes Mixup’s lower localization accuracy to greater ambiguity that encourages attention to more discriminative object parts.
  • Transfer learning: The experiments transfer ImageNet-pretrained ResNet-50 models to Pascal VOC object detection and MS-COCO image captioning, where CutMix outperforms Mixup and Cutout for captioning.Detection is evaluated with SSD and Faster R-CNN using mAP, while captioning uses NIC with metrics including METEOR and CIDER.
  • Robustness and uncertainty: CutMix significantly improves robustness to adversarial attacks and occlusions, nearly matching Cutout under occlusion, and alleviates model over-confidence.Mixup only marginally improves occlusion robustness and aggravates over-confidence, whereas CutMix significantly alleviates it.

5. Conclusion · A. CutMix Algorithm

CutMix is presented as an easy-to-implement augmentation method that improves CNN classification and localization across tasks without computational overhead. The algorithm mixes shuffled image patches and interpolated target labels, making it practical for training.

  • 5. Conclusion: CutMix improves CNN classification and localization performance across various tasks without computational overhead.The conclusion describes CutMix as surprisingly effective while retaining implementation simplicity and no added computation.
  • 5. Conclusion: +2.28% and +1.70% top-1 accuracy improvements are obtained on ImageNet with ResNet-50 and ResNet-101, respectively.These gains are reported for applying CutMix to the two ImageNet classification baselines.
  • 5. Conclusion: +1.98% improvement over the CIFAR classification baseline leads to a state-of-the-art top-1 error of 14.47%.The reported CIFAR result combines the baseline improvement with the resulting top-1 error.
  • A. CutMix Algorithm: CutMix shuffles minibatch inputs and targets, samples lambda and a cropping region, and replaces one input region with a region from another input.The algorithm operates along the first tensor axis before performing the patch replacement.
  • A. CutMix Algorithm: The target label is mixed by interpolation in proportion to the patch-mixing operation.The code-level description states that target labels are mixed alongside the input images.
  • A. CutMix Algorithm: CutMix is easy to implement in few lines of code and is practical for achieving significant impact across a wide range of tasks.The implementation note identifies lines 4 through 15 as the core code span.

B. Weakly-supervised Object Localization

The WSOL procedure largely follows image-classification training but modifies feature-map resolution and evaluates localization using a joint class-correctness and IoU criterion. Experiments also use dataset-specific randomly initialized final layers and a controlled augmentation setup.

  • Training procedure: WSOL starts from ImageNet-pretrained VGG-16 or ResNet-50 models and uses 14×14 feature maps instead of the original 7×7 maps.For VGG, the method uses VGG-GAP.
  • Training procedure: The final fully connected layer is randomly initialized with output dimension 200 for CUB200-2011 and 1000 for ImageNet.The initialization accounts for network modification and target datasets differing from ImageNet.
  • Evaluation metric: Top-1 localization accuracy counts a prediction as correct only when its class label is correct and its estimated bounding box has IoU greater than 0.5 with the ground truth.Otherwise, the localization estimate is considered wrong.

B.1. CUB200-2011

The CUB-200-2011 experiments used over 11 K bird images across 200 categories and trained models for 600 epochs with SGD and scheduled learning-rate decay.

  • B.1. CUB200-2011: CUB-200-2011 contains over 11 K images spanning 200 bird categories, and experiments ran for 600 training epochs.The dataset and training setup are described directly in the passage.
  • B.1. CUB200-2011: Learning rates decayed by a factor of 0.1 every 150 epochs under SGD optimization.The passage also specifies minibatch-size settings, but the provided text truncates before stating the value.

B.2. ImageNet dataset · C. Transfer Learning to Object Detection · D. Transfer Learning to Image Captioning

The paper specifies ImageNet-1K training settings, then describes transfer-learning protocols for Pascal VOC object detection and MS-COCO image captioning using ResNet-50 backbones.

  • B.2. ImageNet dataset: ImageNet-1K contains 13 M training samples and 50 K validation samples for general-object recognition.Training runs for 20 epochs with SGD, minibatch size 256, momentum 0.9, and weight decay 0.0001.
  • B.2. ImageNet dataset: ImageNet training uses learning rates of 0.1 for the last fully-connected layer and 0.01 for other layers.The learning rate decays by a factor of 0.1 every 6 epochs.
  • C. Transfer Learning to Object Detection: Pascal VOC 2007 evaluation uses 5 K test images across 20 object categories.Training combines VOC2007 and VOC2012 trainval, denoted VOC07+12.
  • C. Transfer Learning to Object Detection: For SSD2 finetuning, input images are resized to 300×300 and the original data augmentation, prior boxes, and extra layers are retained.The backbone changes from VGG16 to ResNet-50, requiring modification of the VGG16 conv4 3 pooling location.
  • C. Transfer Learning to Object Detection: CutMix shuffles each minibatch, samples lambda from Unif(0,1), and samples patch-center coordinates uniformly within the image dimensions.The minibatch input is an N×C×W×H tensor and the target is an N×K tensor.
  • C. Transfer Learning to Object Detection: The CutMix patch dimensions are set to Sqrt(1 - lambda), with its coordinates rounded and clipped to the image boundaries.This procedure defines x1, x2, y1, and y2 from uniformly sampled patch-center coordinates.
  • C. Transfer Learning to Object Detection: CutMix replaces the selected image region, recalculates lambda from the exact area ratio, and mixes targets as lambda * target + (1 - lambda) * target s.The mixed input is passed through the model, and loss is computed against the mixed target before the model update.
  • D. Transfer Learning to Image Captioning: MS-COCO captioning uses 120 K trainval images and 40 K test images, changes the NIC4 backbone from GoogLeNet to ResNet-50, and trains for 100 epochs.Batch size is 20, learning rate is 0.001, and beam size is 20 for evaluation.

E. Robustness and Uncertainty

This section describes the experimental setting and evaluation methods for assessing robustness and uncertainty.

  • E. Robustness and Uncertainty: The section outlines the experimental setting and evaluation methods.

E.1. Robustness

The section evaluates ImageNet-trained ResNet-50 models for robustness to adversarial perturbations, occlusion, and in-between samples. It uses FGSM with ϵ = 8/255 and tests top-1 validation accuracy across varying occlusion sizes.

  • Robustness: Robustness experiments compare unregularized, Mixup-, Cutout-, and CutMix-trained ResNet-50 models under adversarial perturbations, occlusion, and in-between samples.All models are trained on ImageNet using the settings from Section 4.1.1.
  • Robustness: FGSM generates adversarial samples from an image, ground-truth label, and noise size, with experiments using ϵ = 8/255.The loss function may be cross entropy.
  • Robustness: Occlusion experiments zero out either the center hole or its exterior and measure top-1 ImageNet validation accuracy for hole sizes from 0 to 224.Center-occluded samples zero the hole interior, while boundary-occluded samples zero the region outside the hole.

E.2. Uncertainty

The section evaluates out-of-distribution detection using CIFAR-100-trained PyramidNet-200 models and finds that CutMix outperforms the baseline, Mixup, and Cutout, especially under noise-based shifts.

  • The comparison uses PyramidNet-200 CIFAR-100 models trained without regularization or with Mixup, Cutout, or CutMix.
  • OOD detection is evaluated by TNR at 95% TPR, AUROC, and detection accuracy across seven out-of-distribution datasets.The datasets include TinyImageNet, LSUN, iSUN, Uniform noise, and Gaussian noise variants.
  • CutMix outperforms the baseline, Mixup, and Cutout in overall OOD detector performance.
  • Mixup and Cutout improve classification performance but substantially degrade baseline OOD detection, particularly for Uniform noise and Gaussian noise.
  • CutMix dramatically improves OOD detection performance for Uniform noise and Gaussian noise.
Loading 1905.04899v2…