Source-linked AI summary
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D. Cubuk, Quoc V. Le, Barret Zoph
TL;DR
Data-hungry instance segmentation needs more data-efficient ways to handle rare categories. This paper systematically studies randomly placed Copy-Paste augmentation, finding strong gains across COCO and LVIS, including additive benefits with self-training.
Problem
Instance segmentation models are data-hungry, annotation is expensive, and rare object categories have limited labeled data.
Method
The paper systematically studies Copy-Paste by randomly selecting object instances and pasting them onto other images, including in self-training and LVIS settings.
Results
Copy-Paste improves data efficiency on COCO, achieves 57.3 Box AP and 49.1 Mask AP with self-training, and improves LVIS mask AP by 6.1 and 3.7 on rare and common categories.
Takeaways & Limitations
Random placement is effective without modeling surrounding context, and Copy-Paste is easy to integrate without increasing training cost or inference time.
Takeaways & Limitations
Random placement can produce images with unrealistic object co-occurrences or scale relationships.
Abstract
from arXiv · showhide
Building instance segmentation models that are data-efficient and can handle rare object categories is an important challenge in computer vision. Leveraging data augmentations is a promising direction towards addressing this challenge. Here, we perform a systematic study of the Copy-Paste augmentation ([13, 12]) for instance segmentation where we randomly paste objects onto an image. Prior studies on Copy-Paste relied on modeling the surrounding visual context for pasting the objects. However, we find that the simple mechanism of pasting objects randomly is good enough and can provide solid gains on top of strong baselines. Furthermore, we show Copy-Paste is additive with semi-supervised methods that leverage extra data through pseudo labeling (e.g. self-training). On COCO instance segmentation, we achieve 49.1 mask AP and 57.3 box AP, an improvement of +0.6 mask AP and +1.5 box AP over the previous state-of-the-art. We further demonstrate that Copy-Paste can lead to significant improvements on the LVIS benchmark. Our baseline model outperforms the LVIS 2020 Challenge winning entry by +3.6 mask AP on rare categories.
1. Introduction
Instance segmentation is data-hungry and expensive to annotate, motivating object-aware augmentation. The paper finds that randomly pasting objects improves data efficiency and performance across COCO, self-training, and LVIS settings.
- Instance segmentation models are data-hungry, while producing 1000 COCO instance masks required 22 worker hours.
- Copy-Paste creates training data by randomly selecting instances from one jittered image and pasting them onto another.
- Randomly picking objects and paste locations provides significant gains across backbones, scale jittering levels, training schedules, and image sizes.
- 2× data-efficiency improvement over standard scale jittering is achieved on COCO when Copy-Paste is combined with large scale jittering.
- +10 Box AP is observed in the low-data regime using only 10% of COCO training data.
- 57.3 Box AP and 49.1 Mask AP are achieved on COCO with self-training, while LVIS gains reach 6.1 and 3.7 mask AP on rare and common categories.
- Copy-Paste is easy to integrate, can use unlabeled images, and adds no training or inference overhead.
2. Related Work
Related augmentation methods combine images or image regions, but many are not object-aware or designed specifically for instance segmentation. Copy-Paste instead transfers exact object pixels and uses random placement without modeling surrounding context.
- General-purpose augmentations such as random crop, color jittering, and Auto/RandAugment have supported strong results in several vision settings.
- Mixing Image Augmentations: Mixup, CutMix, and Mosaic combine images or cropped regions with corresponding labels to create new training data.
- Mixing Image Augmentations: These mixing methods are not object-aware and were not designed specifically for instance segmentation.
- Copy-Paste Augmentation: Copy-Paste copies exact object pixels between images, unlike CutMix, which pastes rectangular crops or bounding-box regions.
- Copy-Paste Augmentation: Unlike contextual methods, this work places copied instances randomly without modeling surrounding visual context.
- Instance Segmentation: Instance segmentation jointly detects object instances and segments the pixels belonging to each instance, with COCO serving as a widely used benchmark.
- Instance Segmentation: The paper reports COCO results surpassing SpineNet by 2.8 AP and DetectoRS by 0.6 AP.
- Copy-Paste Augmentation: Weakly supervised Copy-Paste uses generated masks and adversarial realism checks, whereas this work uses Copy-Paste as augmentation.
3. Method
The method combines Copy-Paste with scale jittering, optional blending, and self-training to generate augmented instance-segmentation data. Random placement does not model surrounding context, so generated images can contain unrealistic object co-occurrences or scale relationships.
- Copy-Paste procedure: Copy-Paste randomly selects two augmented images, pastes a random subset of objects between them, and updates annotations for occlusion.Fully occluded objects are removed, while masks and bounding boxes for partially occluded objects are updated.
- Scope and caveat: Because surrounding context is not modeled, generated images may contain unrealistic object co-occurrences or mismatched object scales.The paper gives giraffes and soccer players with very different scales as an example.
- Scale jittering: Large Scale Jittering resizes images from 0.1 to 2.0 of their original size, whereas Standard Scale Jittering uses 0.8 to 1.25.Both methods crop images and apply horizontal flips; smaller images are padded with gray pixels.
- Blending pasted objects: Pasted objects can be blended by filtering their binary masks, although composing them without blending has similar performance.The composed image is formed from the pasted and main images using the object mask α.
- Self-training Copy-Paste: Self-training Copy-Paste trains on labeled data, generates pseudo-labels for unlabeled data, and pastes ground-truth instances into labeled and pseudo-labeled images.The resulting data is used to train a new model.
4. Experiments
Across architectures, image sizes, training configurations, dataset sizes, and semi-supervised settings, Copy-Paste consistently improves instance segmentation performance and data efficiency. It also yields strong COCO and LVIS results, including gains on rare categories.
- Robustness to training configurations: Copy-Paste remains helpful with ImageNet or random backbone initialization and across standard training schedules, with gains increasing for longer schedules.The method provides benefits at typical 2× and 3× schedules without requiring longer training to see improvement.
- Robustness to training configurations: Copy-Paste is additive to standard and large-scale jittering, while mixup does not help when combined with large-scale jittering.Copy-Paste produces a higher boost on top of standard scale jittering.
- Robustness to training configurations: Copy-Paste improves models across ResNet and EfficientNet backbones and 640×640, 1024×1024, and 1280×1280 image sizes.Across seven models, it averages +1.3 box AP and +0.8 mask AP over large-scale-jittering baselines.
- Data efficiency: Copy-Paste is most beneficial with limited COCO data, yielding +6.9 box AP with standard scale jittering and +4.8 box AP with large-scale jittering at 10% of the dataset.Using Copy-Paste and large-scale jittering with 75% of COCO achieves similar AP to using large-scale jittering with all COCO data.
- Self-training and COCO results: Combining Copy-Paste with self-training improves performance beyond either method alone and reaches 57.3 box AP and 49.1 mask AP on COCO test-dev.The combined model gains +2.5 box AP and +2.2 mask AP in the reported comparison, without test-time augmentation or model ensembling.
5. Conclusion
The paper finds Copy-Paste effective and robust across COCO and LVIS settings, with practical benefits for training instance segmentation models and incorporating unlabeled data.
- Copy-Paste performs well across multiple experimental settings and improves strong baselines on COCO and LVIS instance segmentation.
- Copy-Paste is simple to integrate, without increasing training cost or inference time.
- Copy-Paste incorporates extra unlabeled images during training and adds gains on top of self-training techniques.
- Cascade improves APf but hurts APr in the reported experiments.
A. Ablation on the Copy-Paste method
The ablations examine implementation choices for Copy-Paste, including object selection, blending, and scale jittering for the main and pasted images.
- The ablations use Mask R-CNN EfficientNetB7-FPN with 640×640 images.
- Subset of pasted objects: Pasting a random subset of objects produces the best improvement among pasting one object, all objects, or a random subset.The added randomness from selecting a subset is reported as helpful.
- Blending: Removing alpha blending produces the same results as using blending, indicating that smoothing pasted-object edges is not important here.
- Scale jittering: Copy-Paste applies independent random scale jittering to both the pasted image and the main image.The study evaluates the importance of scale jittering on each image separately.
B. Copy-Paste provides more gain on harder categories of COCO
Copy-Paste improves AP across nearly all COCO categories, with the largest gains concentrated among categories having lower baseline AP.
- Copy-Paste improves AP for every COCO class except hair drier.
- The classes with the highest improvements mostly have lower baseline AP, indicating greater gains on harder categories.
C. How likely objects are copied to an unmatched scene?
The method copies objects between random images without modeling scene context, so many generated images pair objects with unmatched indoor or outdoor scenes.
- Objects are copied from one random image to another without considering image context.
- 46.8% of generated images copy objects to an unmatched indoor or outdoor scene category.COCO images are assigned indoor or outdoor categories using COCO-panoptic labels.
- The scene-category transfer probabilities are summarized in Table 12.
D. Benchmark results on different object sizes
This section reports additional Copy-Paste benchmarks across object sizes, model backbones, and image resolutions, alongside per-category COCO improvements.
- The section also reports Copy-Paste performance across a variety of model architectures.
- Figure 6 shows per-category relative AP improvement from Copy-Paste across 80 COCO classes.The figure lists baseline and Copy-Paste AP values for each category, averaged over 5 runs and sorted by baseline AP.
- Table 13 benchmarks Box AP and Mask AP across different object sizes.
- The object-size benchmarks compare models trained with different backbones and image sizes.