Source-linked AI summary
Data Augmentation using Random Image Cropping and Patching for Deep CNNs
Ryo Takahashi, Takashi Matsubara, Kuniaki Uehara
TL;DR
Deep CNNs risk overfitting when training data are limited, motivating augmentation methods that increase image variety. The paper proposes RICAP, which randomly crops and patches four images while mixing their labels, and reports improved results across classification datasets and image-caption retrieval.
Problem
CNNs with many parameters risk overfitting when training samples are insufficient, while collecting more samples is prohibitively costly.
Method
RICAP randomly crops four training images, patches them into one image, and mixes their class labels according to their areas.
Results
RICAP improves classification accuracy across network architectures and datasets, and is applicable to image-caption retrieval beyond classification.
Takeaways & Limitations
RICAP increases training-image variety and prevents overfitting to the most apparent features, supporting its use beyond image classification.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) have achieved remarkable results in image processing tasks. However, their high expression ability risks overfitting. Consequently, data augmentation techniques have been proposed to prevent overfitting while enriching datasets. Recent CNN architectures with more parameters are rendering traditional data augmentation techniques insufficient. In this study, we propose a new data augmentation technique called random image cropping and patching (RICAP) which randomly crops four images and patches them to create a new training image. Moreover, RICAP mixes the class labels of the four images, resulting in an advantage similar to label smoothing. We evaluated RICAP with current state-of-the-art CNNs (e.g., the shake-shake regularization model) by comparison with competitive data augmentation techniques such as cutout and mixup. RICAP achieves a new state-of-the-art test error of $2.19\%$ on CIFAR-10. We also confirmed that deep CNNs with RICAP achieve better results on classification tasks using CIFAR-100 and ImageNet and an image-caption retrieval task using Microsoft COCO.
I. INTRODUCTION
Deep CNNs achieve strong image-processing results but can overfit when training data are limited, motivating increasingly sophisticated data augmentation. RICAP addresses this by spatially combining cropped images and mixing their labels, with evaluations across several tasks and datasets.
- Motivation: Limited training samples can cause high-parameter CNNs to memorize image details that do not generalize, motivating data augmentation.Existing approaches manipulate images through flipping, resizing, cropping, color changes, and pixel dropping.
- Related methods: Cutout and random erasing disrupt image regions, whereas mixup alpha-blends two images and mixes their labels as soft targets.These methods extend augmentation beyond simply enriching datasets with natural-looking images.
- Proposed method: RICAP randomly crops four training images, patches them into an original-sized image, and mixes their labels according to patch areas.Compared with mixup, it spatially mixes partial images without creating features absent from the original dataset except at patch boundaries.
- Evaluation: RICAP was evaluated with deep CNNs on CIFAR-10, CIFAR-100, and ImageNet classification, plus image-caption retrieval and other tasks.The study reports that RICAP outperformed existing augmentation methods and achieved state-of-the-art CIFAR-10 performance.
- Extensions: The paper extends preliminary conference results by adopting beta-distributed boundary positions, visualizing feature learning, conducting ablations, and testing additional tasks.The authors report that RICAP supports broader feature use and deeper foreground-background relationships.
3) Data Mixing Method:
Data-mixing methods construct training examples from multiple images or labels, while RICAP combines four cropped images into one fixed-size sample. Its procedure samples patch boundaries from beta distributions and uses the resulting image areas for label mixing.
- Data Mixing Method: Mixup alpha-blends two images and their labels, encouraging simple linear behavior between training samples.The method also increases image variety and is described as providing smoother estimation and margin maximization.
- Auto-adjustment Method: AutoAugment searches for effective hyperparameters of existing augmentation methods through reinforcement learning rather than augmenting images itself.The paper presents it as an external framework.
- Soft Labels: Soft labels replace hard 0-or-1 targets with intermediate probabilities, reducing unjustly high confidence during CNN training.The supplied passage introduces soft labels through knowledge distillation.
- RICAP: RICAP selects four images, crops them, and patches them into a new image while preserving the original image size.The four crops occupy upper-left, upper-right, lower-left, and lower-right regions.
- RICAP: RICAP samples horizontal and vertical patch boundaries from Beta(β, β), then derives crop sizes and positions that preserve image dimensions.The boundary coordinates are rounded from the sampled proportions and scaled by the original width and height.
- RICAP: RICAP increases image variety and is reported to prevent overfitting in deep CNNs with numerous parameters.This claim is stated as an effect of the simple crop-and-patch procedure.
B. Label Mixing of RICAP for Classification
RICAP mixes four cropped images spatially and assigns class-label probabilities according to their areas. This combines spatial patching with soft-label supervision and occupancy-based learning.
- Label construction: RICAP mixes four one-hot class labels using ratios proportional to the areas of their cropped image patches.The target label is therefore a soft mixture determined by patch occupancy.
- Boundary position: The boundary-position hyperparameter controls whether RICAP emphasizes overall features and soft labels or occupancy estimation.Positions near corners favor the former, while positions near the center favor the latter.
- Comparison with cutout: RICAP differs from cutout because every region of the patched image contributes features, rather than masking and reducing available features.Both alter apparent image features during training, but RICAP adds content from other images.
- Comparison with mixup: RICAP spatially patches partial images, unlike mixup’s pixel-wise alpha-blending, while preserving original pixel-level features except at patch boundaries.RICAP’s construction avoids creating most pixel-level features absent from the original dataset.
- Occupancy estimation: RICAP replaces classification with occupancy estimation by forcing the CNN to classify pixels and use minor, partial, background, and other overlooked information.This role is especially associated with boundary positions near the patched image center.
E. Object Existence in Cropped Areas
The paper contrasts RICAP’s spatial patching and cropping with mixup and evaluates its training behavior and hyperparameter on CIFAR datasets. RICAP is presented as a regularizer that makes training harder to overfit.
- Object existence in cropped areas: RICAP cropping can leave a patch containing only background, yet its area-weighted label makes the CNN associate that background with the subject class.This is the paper’s described extreme case for cropped regions with no object.
- Differences from mixup: RICAP patches preserve original local features, whereas mixup can create absent features that make recognition excessively difficult and disrupt training.The paper states that RICAP’s local features support model training.
- Differences from mixup: RICAP cropping removes object parts that mixup may leave recognizable, reducing reliance on salient features such as a penguin’s head or an aircraft’s empennage.The paper links this design to broader feature use rather than focus on a limited object region.
- Regularization: RICAP works as an appropriate regularizer, while mixup tends to produce tasks that are either too easy or too difficult.This comparison follows the paper’s discussion of spatial versus pixel-wise blending and cropping.
- Hyperparameter analysis: β = 0.3 produced the best test error rates on both CIFAR-10 and CIFAR-100, while excessively large β values performed worse than the baseline.Decreasing β caused performance to converge toward baseline results.
- Training analysis: With RICAP, training losses and error rates did not converge to zero, while test error rates were better than baseline despite similar test losses.The paper interprets this as continued training that is harder to overfit because of image and label mixing.
B. Classification of ImageNet
RICAP was evaluated with WideResNet on ImageNet against baseline and competitive augmentation methods. It clearly outperformed them at 100 epochs and remained superior or competitive at 200 epochs.
- ImageNet results: RICAP clearly outperformed the baseline and competitive methods with 100-epoch training on ImageNet.The comparison used WideResNet 50-2-bottleneck and included cutout and mixup.
- ImageNet results: After 200 epochs, RICAP remained superior or competitive to the other methods on ImageNet.Cutout and mixup required longer training to obtain results better than the baseline.
- Cross-dataset findings: RICAP achieved significant results on both CIFAR and ImageNet, with robustness to the hyperparameter value shown in the lower panels of Fig. 5.The passage reports this as a cross-dataset result.
C. Classification by Other Architectures
RICAP was tested across DenseNet, pyramidal ResNet, and shake-shake architectures on CIFAR-10. It outperformed competitive methods and achieved a 2.19% test error with shake-shake regularization.
- Cross-architecture results: RICAP outperformed competitive methods across DenseNet, pyramidal ResNet, and shake-shake models evaluated on CIFAR-10.The experiments used multiple deep CNN architectures rather than a single backbone.
- Cross-architecture results: 2.19% test error was achieved by the shake-shake regularization model with RICAP on CIFAR-10.The paper identifies this as a new record under the same comparison conditions.
- Cross-architecture results: RICAP’s results indicate applicability to various CNN architectures, with the appropriate hyperparameter not depending on the architecture.The reported experiments used β = 0.3.
- Visualization analysis: CAM visualizations compare baseline and RICAP attention, including limited object regions and foreground-only focus in the baseline.The cited passages describe the input, baseline CAM, and RICAP CAM rows.
- Visualization analysis: RICAP-trained WideResNet correctly attends to extremely cropped labeled objects and potentially becomes robust to occlusion and cutting off.The model also learns to ignore boundary patching caused by RICAP.
B. Case with No Objects in Cropped Areas
RICAP helps CNNs use object and background cues even when cropped regions contain no objects. Its image and label mixing jointly contribute to the method’s performance.
- RICAP-trained WideResNets focused on objects when objects were present and on backgrounds when cropped regions contained no objects.Class Activation Mapping showed broader use of background clues in the latter case.
- RICAP crops and patches four images while mixing their class labels according to patch areas.
- Image mixing alone improved substantially over baseline but was not competitive with label mixing.The image-mixing-only variant assigned the largest-area patch’s class as the target label.
- Label mixing alone performed much worse, indicating harmful effects from extreme soft labeling without image mixing.
- Both image mixing and label mixing jointly play an important role in RICAP.
B. Comparison with Mixup of four Images
RICAP differs from four-image mixup by spatially patching cropped images rather than alpha-blending them. Experiments also show that RICAP improves image-caption retrieval when applied to VSE++.
- RICAP mixes four images spatially, crops partial images, and mixes labels by patch area, whereas mixup alpha-blends two images using an alpha value.
- 4-mixup outperformed baseline but underperformed both original mixup and RICAP.This suggests RICAP’s performance does not come merely from mixing four images.
- In VSE++, RICAP constructs a patched image from four randomly selected images for training the image encoder.
- RICAP forms a caption representation by area-weighting the encoded representations of four paired captions.
- RICAP improved VSE++ performance on Microsoft COCO image-caption retrieval.The evaluation used scores averaged over five folds of 1,000 test images.
B. Evaluation on Person Re-identification
For person re-identification, RICAP is adapted into FICAP, which preserves meaningful absolute positions through fixed cropping. FICAP improved IDE identification performance but conflicted with PCB’s multi-part image division.
- FICAP fixes crop origins to preserve absolute human-part positions while retaining random boundary locations.This adapts RICAP to aligned, center-cropped person images.
- FICAP improved IDE identification performance on Market-1501.The cited evaluation used 19,732 test images from 750 identities.
- FICAP degraded performance when applied to PCB’s horizontally divided subparts.Applying FICAP produced 24 patches per person; reducing this to two patches per subpart also degraded performance.
- The authors attribute the PCB incompatibility to conflict between image patching and existing image division, which can make patches too small to recognize.
C. Evaluation on Object Detection
For object detection, RICAP performs image-only cropping and patching while correcting bounding-box geometry. On MS-COCO, it improved YOLOv3 detection and addressed several occlusion-related errors.
- RICAP cannot mix bounding-box labels for object detection, so it applies only random cropping and patching to input images.
- With RICAP, YOLOv3 resolved examples involving a split zebra detection, missed side-by-side buses, and a horse tail mistaken for a dog.The examples indicate greater robustness to occlusion.
- Bounding-box coordinates, widths, and heights are corrected according to each object’s cropped region.
- RICAP improved YOLOv3 detection performance on MS-COCO.The evaluation reported mAP, precision, and recall for β = 0.3 and β = 1.0.
VIII. CONCLUSION
The paper proposes RICAP, which randomly crops four training images and patches them into a new image. Experiments report improved classification across architectures and datasets, reduced overfitting to salient features, and applicability to image-caption retrieval.
- RICAP randomly selects and crops four training images, then patches them into one new training image.
- RICAP improves classification accuracy across various network architectures and datasets by increasing training-image variety and preventing overfitting.
- Visualization results show that RICAP prevents deep CNNs from overfitting to the most apparent image features.
- RICAP is applicable to image processing tasks beyond classification, as demonstrated by image-caption retrieval results.
APPENDIX
The appendix provides implementation resources for reproducing RICAP and WideResNet classification experiments.
- The appendix provides Python code implementing RICAP in Algorithm 1.
- The implementation uses NumPy and PyTorch modules and follows naming conventions from official PyTorch examples.
- Executable WideResNet classification code is released at the paper’s GitHub repository.