PaperlayerView exact revision on arXiv

Source-linked AI summary

InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting

Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, Cewu Lu

arXiv:1908.07801v1cs.CV

TL;DR

Instance segmentation needs many labor-intensive mask annotations, while prior augmentation methods do not fully exploit existing masks and can suffer from poor performance or computational overhead. InstaBoost uses random object jittering and appearance-consistency-guided crop-pasting, achieving improvements on COCO and Pascal VOC without changing the model architecture. The method is designed for efficient integration into existing instance-segmentation pipelines.

  • Problem

    Pixel-wise mask annotations are labor-intensive, and existing augmentation methods do not fully exploit available mask annotations while some prior crop-and-paste approaches perform poorly or add computational overhead.

  • Method

    InstaBoost augments training data by separating annotated instances from backgrounds, applying restricted random jittering, and sampling visually compatible paste locations from an appearance-consistency heatmap.

  • Results

    2.2 mAP improvement is reported on COCO instance segmentation with appearance-consistency heatmap guided InstaBoost, while random InstaBoost yields 1.7 mAP improvement with Mask R-CNN.

  • Takeaways & Limitations

    InstaBoost can be embedded in existing instance-segmentation frameworks with little CPU overhead and without modifying the backbone or network structure.

  • Takeaways & Limitations

    Prior context-based crop-and-paste methods can prune semantically consistent placements with different background patterns, while the heatmap assumes nearby ambient appearance matters more.

Abstract

from arXiv · show

Instance segmentation requires a large number of training samples to achieve satisfactory performance and benefits from proper data augmentation. To enlarge the training set and increase the diversity, previous methods have investigated using data annotation from other domain (e.g. bbox, point) in a weakly supervised mechanism. In this paper, we present a simple, efficient and effective method to augment the training set using the existing instance mask annotations. Exploiting the pixel redundancy of the background, we are able to improve the performance of Mask R-CNN for 1.7 mAP on COCO dataset and 3.3 mAP on Pascal VOC dataset by simply introducing random jittering to objects. Furthermore, we propose a location probability map based approach to explore the feasible locations that objects can be placed based on local appearance similarity. With the guidance of such map, we boost the performance of R101-Mask R-CNN on instance segmentation from 35.7 mAP to 37.9 mAP without modifying the backbone or network structure. Our method is simple to implement and does not increase the computational complexity. It can be integrated into the training pipeline of any instance segmentation model without affecting the training and inference efficiency. Our code and models have been released at https://github.com/GothicAi/InstaBoost

1. Introduction

Instance segmentation needs many pixel-wise mask annotations, but these are labor-intensive and existing augmentation methods do not fully exploit available masks. InstaBoost augments instances through restricted random jittering and appearance-consistency-guided placement, improving benchmark performance.

  • Pixel-wise mask annotation is labor-intensive, limiting the number of available instance-segmentation training samples.
  • Existing weakly supervised methods use annotations from other domains, while few methods leverage existing mask annotations to augment training data.
  • Crop-and-paste methods previously used random or context-based placement, but performed poorly for instance segmentation and could incur significant computational overhead.
  • 1.7 mAP improvement is achieved on COCO instance segmentation with Mask R-CNN using random InstaBoost, which pastes objects near their original positions with small scale and rotation jitter.
  • 2.2 mAP improvement on COCO is achieved by sampling paste locations from an appearance-consistency heatmap based on local contour similarity.
  • The authors report 3.9 mAP improvement on Pascal and 2.2 mAP improvement on COCO using appearance-consistency heatmap guided InstaBoost.

2. Related work

Instance segmentation methods have progressed from proposal-based and pixel-association approaches to fully convolutional and two-stage models. Despite strong benchmark performance, these models require extensive data, motivating instance-level augmentation that better exploits existing annotations.

  • Instance mask segmentation: Instance segmentation combines instance detection with semantic segmentation and assigns pixels to individual object instances.
  • Instance mask segmentation: Mask R-CNN extends Faster R-CNN with a mask head and achieves state-of-the-art performance on instance segmentation and pose estimation.
  • Instance mask segmentation: Heavy deep models require extremely large training datasets, which are often unavailable in real-world applications.
  • Instance-level augmentation: Instance-level augmentation seeks to exploit supervised information already present in datasets through more precise image transformations.

3.1. Overview

InstaBoost models object placement as a probability distribution over affine transformations. Random jitter samples near the identity transform, while an appearance-consistency heatmap expands placement to visually compatible background locations.

  • An object placement is represented by an affine transformation with coordinate shifts, scale variation, and rotation.
  • The transformation space is a 4D tuple (tx, ty, s, r), where tx and ty shift coordinates, s changes scale, and r specifies rotation.
  • A probability density f measures how reasonable it is to paste object O on image I under each transformation tuple.
  • The identity transform at the object's original position is assumed to have the highest placement probability.
  • Random object jitter samples transformations near the identity transform, exploiting local pixel continuity and redundancy.
  • An appearance-consistency heatmap extends feasible placement beyond the local neighborhood when backgrounds share similar patterns.

3.2. Random InstaBoost

Random InstaBoost prepares a foreground instance and an inpainted background, then applies a restricted random affine transformation to create an augmented image.

  • Random InstaBoost separates a target instance and background using ground-truth annotations, matting, and inpainting before augmentation.
  • Figure 2 presents the original image, its inpainting result, and the instance obtained through matting.
  • Matting smooths object outlines from boundary-based annotations, while inpainting fills the instance-shaped hole left in the background.
  • The method applies a random transformation sampled from the neighboring space of the identity transform to paste the instance.

3.3. Appearance consistency heatmap guided InstaBoost

Appearance consistency heatmap guided InstaBoost samples object placements using local background appearance similarity, refining feasible locations beyond simple neighboring jitter. It represents placement likelihoods through heatmaps generated from weighted contour descriptors and appearance distances.

  • Appearance distance: The method evaluates candidate placements by comparing local RGB appearance around the original and transformed object centers.Appearance distance uses descriptors built from neighboring contour areas and adopts Euclidean distance in the implementation.
  • Visual examples: The heatmap examples show original images beside maps whose red regions indicate high appearance consistency and blue regions indicate low consistency.Figure 5 presents original image, heatmap, and processed image from left to right.
  • Appearance descriptor: The appearance descriptor combines three fixed-width contour areas at different scales, assigning greater weight to contours nearer the original object.The innermost contour receives the greatest weight, with w1 > w2 > w3.
  • Boundary handling: Locations whose effective consistency area extends outside the background are assigned infinite distance and ignored.This handles pixels for which the relevant consistency region is unavailable.
  • Heatmap generation: The heatmap is generated by scanning appearance distances from the original descriptor, then normalizing and applying a negative-log mapping.The maximum and minimum candidate-center distances determine the normalization range.
  • Coordinate shift: Heatmap values are treated as placement probabilities, enabling Monte Carlo sampling of candidate coordinates while avoiding semantically inconsistent backgrounds.Scale and rotation are sampled independently from uniform distributions near (1, 0).
  • Efficiency: The full heatmap computation is initially inefficient because it requires W × H distances for each point in the effective area, yielding O(W^2H^2) complexity.The paper identifies this complexity as unacceptable for real-world applications before introducing an efficiency measure.

3.4. Training

InstaBoost is applied during training by the dataloader alongside other augmentation strategies, with little CPU overhead and parallel processing supporting efficiency.

  • Training pipeline: The dataloader applies InstaBoost with a given probability during training, together with other data augmentation strategies.The strategy is integrated into the existing training pipeline rather than changing the model structure.
  • Efficiency: The implementation introduces little CPU overhead, while parallel dataloader processing maintains training efficiency.The stated efficiency concern applies to the training phase.

3.5. Discussion

InstaBoost favors background-pattern consistency over explicit semantic consistency when placing pasted objects, producing visually coherent images with fewer blending artifacts and efficient implementation.

  • Figure 6 compares vanilla Mask R-CNN outputs with outputs from Mask R-CNN trained using InstaBoost.The caption characterizes the InstaBoost results as finer instance segmentation.
  • The appearance consistency map places objects where the original image has similar background patterns rather than explicitly modeling semantic consistency.This constraint may prune semantically consistent configurations with different background patterns, but usually preserves visual coherence.
  • Compared with the context model, InstaBoost generates more photorealistic images with fewer blending artifacts.The authors attribute this to introducing less noise during neural-network training.
  • The method reports superior qualitative and quantitative performance while using a more efficient implementation than the context-based approach.The context model requires an extra training step and offline preprocessing, whereas InstaBoost is integrated into the training pipeline.

4. Experiments

Experiments evaluate InstaBoost on Pascal VOC and COCO with Mask R-CNN variants, showing improved detection and segmentation, resistance to overfitting, and efficient comparison with prior augmentation methods.

  • 4.1. Datasets: Experiments evaluate bounding-box detection and instance segmentation on Pascal VOC/VOCSDS and COCO benchmarks.COCO contains 80 categories, more than 200,000 labeled images, many small objects, occlusion, and noisy backgrounds.
  • 4.2. Baselines: The evaluation uses Mask R-CNN with Res-50-FPN and Res-101-FPN backbones, plus Res-101-FPN Cascaded Mask R-CNN baselines.Baselines are retrained using corresponding open implementations.
  • 4.3. Implementation details: The augmentation uses small rotation, with the rotation range set to [−5, 5], and appearance descriptors built from three 5-pixel contour areas.Contour weights are 0.4, 0.35, and 0.25 from inside to outside; map generation uses fixed size (180, 120).
  • 4.4. Main results: InstaBoost elevates state-of-the-art models on COCO detection and instance segmentation, while VOC instance-segmentation improvement is around 4 mAP.The reported COCO and VOC evaluations cover both the augmentation method and multiple model settings.
  • 4.4. Main results: With InstaBoost, Mask R-CNN predicts correct masks in examples where vanilla Mask R-CNN produces incomplete masks or ignores objects.The qualitative comparison is shown in Figure 6.
  • 4.5. Analysis: Compared with a context model, InstaBoost improves both tasks without tedious preparation or affecting training efficiency.The context model requires an extra training step and offline preprocessing.
  • 4.5. Analysis: Randomly pasting instances degrades instance-segmentation performance by 1.3 and 1.1 mAP relative to the original baseline on VOC and COCO.This comparison supports appearance consistency as an essential component of InstaBoost.
  • 4.5. Analysis: InstaBoost resists overfitting: original Mask R-CNN accuracy stops increasing after 24 epochs, whereas augmented models continue improving at longer training durations.The comparison covers both detection and segmentation accuracy.

5. Conclusion

The paper addresses limited instance-segmentation training data with random and appearance-guided InstaBoost augmentation, reporting gains with little CPU overhead and easy pipeline integration.

  • Random InstaBoost samples neighboring identity transformations in a 4D transformation tuple and improves Mask R-CNN by 1.7 mAP on COCO instance segmentation.The transformation-based augmentation is described as simple and effective.
  • Appearance consistency heatmap guidance raises the total COCO instance-segmentation improvement to 2.2 mAP.The heatmap guides placement using appearance consistency.
  • The online implementation integrates into existing instance-segmentation frameworks and offers improvement with little CPU overhead.The conclusion describes the method as easy to embed in existing pipelines.
Loading 1908.07801v1…