PaperlayerView exact revision on arXiv

Source-linked AI summary

Path Aggregation Network for Instance Segmentation

Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, Jiaya Jia

arXiv:1803.01534v4cs.CV

TL;DR

Proposal-based instance segmentation has limited information flow from low-level features and incomplete access to feature levels and proposal views. PANet improves propagation through bottom-up path augmentation, adaptive feature pooling, and complementary mask prediction, achieving first place in COCO 2017 instance segmentation and second place in object detection without large-batch training.

  • Problem

    Proposal-based instance segmentation has long paths from low-level features, heuristic single-level pooling for proposals, and single-view mask prediction that limit access to useful information.

  • Method

    PANet enhances feature propagation with bottom-up path augmentation, adaptive pooling across all feature levels, and a complementary mask-prediction path.

  • Results

    PANet achieves top-ranking results across COCO 2017, Cityscapes, and MVD, including 1st place in instance segmentation and 2nd place in object detection.

  • Takeaways & Limitations

    PANet is presented as a practical, top-performing framework with simple components and subtle computational overhead.

Abstract

from arXiv · show

The way that information propagates in neural networks is of great importance. In this paper, we propose Path Aggregation Network (PANet) aiming at boosting information flow in proposal-based instance segmentation framework. Specifically, we enhance the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation, which shortens the information path between lower layers and topmost feature. We present adaptive feature pooling, which links feature grid and all feature levels to make useful information in each feature level propagate directly to following proposal subnetworks. A complementary branch capturing different views for each proposal is created to further improve mask prediction. These improvements are simple to implement, with subtle extra computational overhead. Our PANet reaches the 1st place in the COCO 2017 Challenge Instance Segmentation task and the 2nd place in Object Detection task without large-batch training. It is also state-of-the-art on MVD and Cityscapes. Code is available at https://github.com/ShuLiu1993/PANet

1. Introduction

PANet improves proposal-based instance segmentation by strengthening information flow through bottom-up path augmentation, adaptive feature pooling, and complementary mask prediction. The framework achieves top-ranking results across COCO, Cityscapes, and MVD with modest implementation changes and without large-batch training.

  • Our Findings: PANet targets information-propagation weaknesses in Mask R-CNN, where low-level features aid large-instance identification but face long paths to topmost features.Each proposal also uses feature grids from only one heuristically assigned feature level.
  • Our Contributions: Bottom-up path augmentation shortens paths from low-level features and propagates accurate localization signals through the entire feature hierarchy.The paper identifies this as unexplored for instance recognition despite prior use of low-level features.
  • Our Contributions: Adaptive feature pooling aggregates features from all feature levels for each proposal, replacing arbitrary single-level assignment with cleaner information paths.The component is designed to recover broken paths between proposals and all feature levels.
  • Our Contributions: Complementary fully connected layers add a second view to the FCN mask branch, and fusing both predictions increases information diversity and mask quality.The added layers are described as tiny and complementary to the FCN originally used by Mask R-CNN.
  • Experimental Results: PANet achieves 1st place in the COCO 2017 Instance Segmentation task and 2nd place in Object Detection without large-batch training.With ResNet-50 and single-scale testing, it also outperforms the COCO 2016 champion in both tasks despite that champion using larger models, multi-scale, and horizontal-flip testing.

2. Related Work

Related work spans proposal-based and segmentation-based instance segmentation, along with multi-level feature fusion and larger-context representations. PANet differs by simplifying feature integration, using all in-network feature levels with single-scale input, and enabling end-to-end training.

  • Instance Segmentation: Instance segmentation methods mainly follow proposal-based or segmentation-based pipelines.Proposal-based methods are closely connected to object detection, whereas segmentation-based methods predict transformations or instance boundaries before decoding masks.
  • Instance Segmentation: Proposal-based methods evolved from externally supplied region proposals toward generating instance masks within networks as proposals or final results.R-CNN-family methods extracted pooled features for classification, while later systems integrated mask generation into the network.
  • Multi-level Features: Prior multi-level approaches fused lower- and higher-layer features through skip connections, lateral connections, or feature-grid concatenation.These approaches support finer segmentation details or stronger prediction, but concatenation-based methods require normalization, concatenation, and dimension reduction.
  • Multi-level Features: PANet uses a simpler feature-fusion design than methods that concatenate feature grids from different layers.The paper contrasts its design with approaches requiring normalization, concatenation, and dimension reduction.
  • Multi-level Features: Unlike multi-scale input fusion, PANet uses information from all feature levels in a single-scale, in-network hierarchy and supports end-to-end training.Other methods extracted feature maps at different input scales and fused them with a max operation for feature selection.

3. Framework

PANet improves proposal-based instance segmentation by augmenting feature propagation with a bottom-up path, adaptive feature pooling, and a complementary mask-prediction path. The design strengthens localization and lets proposals access information across feature levels while remaining independent of the CNN structure.

  • Framework: PANet combines bottom-up path augmentation, adaptive feature pooling, and a complementary path for mask prediction to improve performance.The framework is designed with subtle computational overhead and is independent of the CNN structure, similar to FPN.
  • Framework: Low-level edge and instance-part responses strengthen localization by providing a shortcut of fewer than 10 layers to top feature levels, compared with 100+ layers through the FPN trunk.The shortcut uses clean lateral connections from low levels to top ones.
  • Augmented Bottom-up Structure: The augmented bottom-up path starts at P2 and progressively approaches P5 across FPN feature levels.Each feature level corresponds to a network stage, using ResNet as the basic structure.
  • Augmented Bottom-up Structure: Each bottom-up block downsamples Ni with a stride-2 3 × 3 convolution, adds it to Pi+1 through a lateral connection, and applies another 3 × 3 convolution to produce Ni+1.The resulting feature map is used by following subnetworks.
  • Adaptive Feature Pooling: Adaptive feature pooling addresses non-optimal FPN level assignment by allowing proposals to use richer high-level context and fine-detail low-level localization cues.FPN assigns proposals to levels according to proposal size, although similarly sized proposals can be separated by a 10-pixel difference.

FEATURE DISTRIBUTION

Adaptive feature pooling fuses proposal feature grids pooled from multiple FPN levels, allowing proposals of all sizes to exploit information across the feature hierarchy. The method uses element-wise fusion and places it within subsequent prediction subnetworks so the network can adapt the pooled features.

  • Feature distribution: Proposals of different sizes all exploit features pooled from several FPN levels rather than relying on a single assigned level.Figure 3 groups proposals by their original FPN-assigned levels and reports the feature distribution across source levels.
  • Feature distribution: Using element-wise max fusion lets the network select useful information independently at each feature-grid element.The analysis of pooled-feature ratios uses max fusion across levels.
  • Adaptive Feature Pooling Structure: Adaptive feature pooling maps each proposal to multiple feature levels, applies ROIAlign, and fuses the resulting feature grids with an element-wise max or sum.The pooled grids are then used for classification, box regression, and mask prediction.
  • Adaptive Feature Pooling Structure: Pooled feature grids pass through one parameter layer independently before fusion, with fusion inserted after the first fully connected or convolutional layer in the relevant branch.The box branch fuses after its first of two fc layers, while the mask branch fuses between its first and second convolutional layers.
  • Adaptive Feature Pooling Structure: The design fuses information within the in-network feature hierarchy and is simpler than approaches requiring L-2 normalization, concatenation, and dimension reduction.It does not fuse features from different input-image-pyramid feature maps.

4. Experiments

PANet is evaluated against state-of-the-art methods on COCO, Cityscapes, and MVD, with comprehensive COCO ablations. Across benchmarks and challenge settings, PANet achieves strong gains in instance segmentation and object detection.

  • Experimental Setup: PANet is compared with state-of-the-art methods on COCO, Cityscapes, and MVD, with results described as top ranked across all three datasets.The experiments also include COCO 2017 Instance Segmentation and Object Detection Challenge results.
  • Object Detection Results: PANet with ResNet-50 outperforms all other single-model object detectors by a large margin and surpasses the 2016 Champion with multi-scale training and single-scale testing.The comparison includes models using the much larger ResNeXt-101 as an initial model.
  • Ablation Study: 4.4 mask AP and 4.2 box ap APbb are the gains of the full PANet over baselines, with small- and medium-size instances contributing most.Half the improvement comes from multi-scale training and multi-GPU synchronized batch normalization.
  • COCO 2017 Challenge: 1st place in Instance Segmentation and 2nd place in Object Detection are achieved in the COCO 2017 Challenges without large-batch training.Compared with the previous champions, the gains are 9.1% absolute and 24% relative for instance segmentation, and 9.4% absolute and 23% relative for object detection.

5. Conclusion

PANet enhances information propagation for instance segmentation through multilevel feature pooling, shortened paths, and complementary proposal features. The method produces impressive results, with future work extending it to video and RGBD data.

  • PANet is presented as an instance segmentation method with simple components that enhance information propagation in representative pipelines.
  • Features are pooled from all feature levels, while distances between lower and topmost levels are shortened for reliable information passing.
  • A complementary path enriches the feature representation for each proposal.
  • The method produces impressive results, and future work will extend it to video and RGBD data.

Appendix · A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.

The appendix specifies dataset-specific training settings for Cityscapes and MVD, including inherited hyper-parameters, anchor configurations, optimization schedules, batch size, and input dimensions.

  • A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.: Cityscapes training hyper-parameters are adopted from Mask R-CNN and described in Section 4.3.
  • A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.: On Cityscapes, RPN anchors use 5 scales and 3 aspect ratios following Mask R-CNN and [35].
  • A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.: MVD training hyper-parameters are adopted from the winning entry.
  • A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.: 0.02 for 60k iterations and 0.002 for another 20k iterations are the MVD learning-rate schedule.
  • A. Training Details and Strategy of Generating Anchors on Cityscapes and MVD.: 16 images form one image batch during training.

B. Details on Implementing Multi-GPU Synchronized Batch Normalization.

The implementation synchronizes batch normalization statistics across multiple GPUs by aggregating per-GPU values with AllReduce and broadcasting batch-wide results. The same cross-GPU aggregation is used for gradient calculation during backpropagation.

  • Forward synchronization: Training samples are split evenly across n GPUs, each GPU computes local means, and AllReduce aggregates them into the full-batch mean µB.The full-batch mean is then broadcast to all GPUs.
  • Forward synchronization: Each GPU uses statistics calculated from all training samples in B to normalize activations with ym = γ xm−µB √.Batch-wide statistics are made available on every GPU before normalization.
  • Backward synchronization: During backpropagation, AllReduce similarly gathers information across GPUs for gradient calculation.
Loading 1803.01534v4…