PaperlayerView exact revision on arXiv

Source-linked AI summary

FaPN: Feature-aligned Pyramid Network for Dense Image Prediction

Shihua Huang, Zhichao Lu, Ran Cheng, Cheng He

arXiv:2108.07058v2cs.CV

TL;DR

Dense prediction requires spatial detail and semantic context, yet conventional feature fusion can misalign contexts and hurt boundary predictions. FaPN learns offsets to align upsampled features, selects informative lower-level detail, and integrates both modules into a top-down pyramid. It improves AP / mIoU over FPN across four tasks and reports 56.7% mIoU on ADE20K with MaskFormer.

  • Problem

    Feature fusion in dense prediction can produce misaligned contexts because upsampled and local features are combined directly, causing mis-classifications especially around object boundaries.

  • Method

    FaPN integrates a feature alignment module that learns offsets for upsampled higher-level features and a feature selection module that emphasizes lower-level spatial details in a top-down pyramid.

  • Results

    FaPN improves AP / mIoU by 1.2% - 2.6% over FPN across four dense prediction tasks and achieves 56.7% mIoU on ADE20K with MaskFormer.

  • Takeaways & Limitations

    FaPN provides a drop-in multi-scale feature replacement for FPN and improves dense prediction performance across tasks, datasets, backbones, object scales, and boundaries.

Abstract

from arXiv · show

Recent advancements in deep neural networks have made remarkable leap-forwards in dense image prediction. However, the issue of feature alignment remains as neglected by most existing approaches for simplicity. Direct pixel addition between upsampled and local features leads to feature maps with misaligned contexts that, in turn, translate to mis-classifications in prediction, especially on object boundaries. In this paper, we propose a feature alignment module that learns transformation offsets of pixels to contextually align upsampled higher-level features; and another feature selection module to emphasize the lower-level features with rich spatial details. We then integrate these two modules in a top-down pyramidal architecture and present the Feature-aligned Pyramid Network (FaPN). Extensive experimental evaluations on four dense prediction tasks and four datasets have demonstrated the efficacy of FaPN, yielding an overall improvement of 1.2 - 2.6 points in AP / mIoU over FPN when paired with Faster / Mask R-CNN. In particular, our FaPN achieves the state-of-the-art of 56.7% mIoU on ADE20K when integrated within Mask-Former. The code is available from https://github.com/EMI-Group/FaPN.

1. Introduction

Dense prediction must combine spatial detail for locating objects with semantic context for classifying them, but repeated downsampling and upsampling can misalign features, especially at boundaries. FaPN addresses this with feature alignment and feature selection modules integrated into a top-down pyramid, improving dense prediction across tasks and settings.

  • Motivation: Dense prediction labels every image pixel and supports tasks including object detection, semantic, instance, and panoptic segmentation.These tasks contribute to scene understanding and applications such as autonomous driving, medical imaging, and augmented reality.
  • Motivation: Feature hierarchies must combine rich spatial detail for object location with strong semantics for object classification across resolution levels.Existing approaches use atrous convolutions or top-down feature pyramids to address this multi-scale requirement.
  • Problem: Repeated downsampling and upsampling can misalign features, harming subsequent learning and causing mis-classifications, especially around object boundaries.FaPN aligns upsampled features to reference maps by learning offsets for convolutional sampling locations.
  • Approach: FaPN combines a feature alignment module for higher-level features with a feature selection module that emphasizes lower-level spatial details.Together, the modules form an enhanced drop-in replacement for FPN that generates multi-scale features.
  • Evaluation: FaPN was evaluated across four dense prediction tasks and three benchmark datasets, including MS COCO, Cityscapes, and COCO-Stuff-10K.The framework was implemented with Faster R-CNN, Mask R-CNN, PointRend, MaskFormer, PanopticFPN, and PanopticFCN.
  • Results: 1.2% - 2.6% improvement in AP / mIoU over FPN was reported, while FaPN achieved 56.7% mIoU on ADE20K with MaskFormer.The reported improvements cover the evaluated dense prediction tasks, and the ADE20K result is identified as state-of-the-art.

2. Related Work

Prior dense prediction methods use atrous convolutions or encoder-decoder feature pyramids to capture context and combine representations across scales. Related alignment methods preserve or learn spatial guidance, while RoIAlign addresses feature–RoI quantization misalignment.

  • Atrous convolutions: Atrous-convolution methods enlarge receptive fields and aggregate multi-scale context without spatially reducing feature-map resolution.DeepLab uses Atrous Spatial Pyramid Pooling with different atrous rates for semantic segmentation.
  • Atrous convolutions: The lack of multi-scale feature maps restricts atrous-convolution methods primarily to semantic segmentation beyond that task.This limitation motivates methods that build explicit feature hierarchies.
  • Encoder-decoder pyramids: Encoder-decoder methods combine bottom-up and top-down pathways, using step-by-step upsampling to propagate high-level semantic context into low-level features.DeconvNet, DSSD, FPN, Mask R-CNN, SOLOs, and Panoptic FPN exemplify this family across dense prediction tasks.
  • Feature alignment: SegNet preserves spatial information with stored max-pooling indices, whereas GUN learns guidance offsets before decoder upsampling.These methods address boundary-detail loss caused by repeated downsampling.
  • Feature alignment: RoIAlign avoids the extracted-feature and RoI misalignment caused by RoIPool quantization.It does so instead of relying on quantized RoI feature alignment.

3. Feature-aligned Pyramid Network

FaPN combines feature alignment and feature selection in a top-down pyramid to address spatial misalignment and emphasize detailed lower-level features before fusion.

  • Framework: FaPN integrates a Feature Selection Module and Feature Alignment Module as an enhanced replacement for FPN.The framework generates multi-scale features through a bottom-up network and top-down pathway.
  • Feature Alignment Module: Recursive downsampling and upsampling create spatial misalignment between upsampled and bottom-up features, harming predictions near object boundaries.Element-wise addition or channel-wise concatenation can therefore combine features with inconsistent spatial contexts.
  • Feature Alignment Module: The Feature Alignment Module learns offsets from paired upsampled and reference features, then uses them to adjust convolutional sampling locations.The alignment and offset functions are implemented with deformable convolutions followed by activation and standard convolutions.
  • Feature Alignment Module: The offset field preserves the input resolution and contains 2N channels representing N two-dimensional offsets for convolutional sample locations.For a 3 × 3 convolution, N equals 9, and each offset specifies a horizontal or vertical displacement.
  • Feature Selection Module: The Feature Selection Module models feature-map importance from global average-pooled information and recalibrates input maps before channel reduction.Its rescaling uses an importance vector and a skip connection, which helps avoid over-amplifying or over-suppressing channel responses.

4. Experiments

Experiments evaluate FaPN across four dense prediction tasks and multiple datasets, showing consistent gains over FPN, stronger boundary performance, and benefits for small objects and real-time segmentation.

  • Experimental Setup: FaPN was evaluated on object detection, semantic, instance, and panoptic segmentation using benchmark datasets including MS COCO, Cityscapes, COCO-Stuff-10K, and ADE20K.The experiments connect FaPN to Faster R-CNN, Mask R-CNN, PointRend, MaskFormer, and other dense prediction frameworks.
  • Ablation Study: 2.3 points: the feature alignment module alone improved FPN mIoU on Cityscapes with only marginal model-size growth.FaPN achieved 80.0 mIoU with 33.1M parameters, compared with 77.5@33.4M for an extra convolution and 78.9@47.6M for a larger backbone.
  • Main Results: FaPN improves boundary segmentation, with aligned features described as smoother and containing more precise object boundaries than raw upsampled features.The boundary results and visualizations support better predictions around object boundaries.
  • Main Results: 1.2–2.6 points: FaPN improved the primary evaluation metrics over FPN across all four tasks with ResNet50.The gains also held for stronger backbones and more sophisticated mask heads.
  • Main Results: 2.1 points and 1.8 points: FaPN improved small-object bounding-box AP over FPN on MS COCO object detection and instance segmentation, respectively.The paper links this pattern to small objects occupying fewer pixels and having many pixels near boundaries.
  • Main Results: FaPN achieves state-of-the-art performance on ADE20K and COCO-Stuff-10K when integrated with MaskFormer.The comparison reports both single-scale and multi-scale semantic segmentation performance.
  • Real-Time Segmentation: FaPN-R18 runs 2× faster than SwiftNet while maintaining similar mIoU, whereas FaPN-R34 reaches 78.1 mIoU at 30 FPS on Cityscapes.On COCO-Stuff-10K, FaPN-R34 outperforms BiSeNetV2 in both mIoU and inference speed.

5. Conclusion

FaPN is a simple top-down pyramid that aligns higher-level features and selects spatially detailed lower-level features. Across dense prediction settings, it consistently improves FPN and extends to real-time segmentation.

  • Conclusion: FaPN combines feature alignment and feature selection modules in a top-down architecture for generating multi-scale features.The alignment module adjusts higher-level features, while the selection module emphasizes lower-level spatial details.
  • Conclusion: FaPN substantially and consistently improves the original FPN across four dense prediction tasks and three datasets.The method also improves state-of-the-art segmentation performance when integrated with strong baselines.
  • Conclusion: FaPN can be extended to real-time segmentation with lightweight backbones and performs favorably against dedicated real-time methods.The conclusion presents FaPN as a potential baseline or module for dense image prediction.

Appendix

The supplementary material adds training details, real-time semantic segmentation details, and qualitative visualizations of FaPN.

  • Appendix: The supplementary material includes additional training details, real-time semantic segmentation experiment details, and qualitative visualizations.These additions are organized in Sections A, B, and C, respectively.

A. Training Settings

The experiments use SGD with standard momentum, weight decay, augmentation, frozen ImageNet batch-normalization weights, and no test-time augmentation.

  • Training Settings: Training uses SGD with 0.9 momentum and 0.0001 weight decay, together with horizontal flipping and scaling augmentation.ImageNet-pretrained batch-normalization weights remain frozen.
  • Training Settings: The experiments do not incorporate testing-time augmentation tricks, following prior work for consistency.

B. Real-time Semantic Segmentation Continued

FaPN builds multi-scale semantic features by progressively aligning upsampled features with lower-level maps and selecting spatially detailed information. On Cityscapes, ablations show gains from FAM, FSM, and concatenation-based fusion, while visual comparisons illustrate the resulting segmentation quality.

  • Architecture: FaPN attaches FSM to C5 and progressively upsamples, aligns, concatenates, and compresses features until generating the finest-resolution map P3 for semantic prediction.The final prediction layer operates on P3.
  • Experimental Setup: The ablation table compares FaPN components on Cityscapes validation using accuracy, parameters, and FLOPs.The analysis uses an ImageNet-pretrained ResNet18 backbone.
  • Ablation Study: The ablation baseline improves from 68.6% to 73.8% after incorporating FAM.This comparison is reported on the Cityscapes validation set using ResNet18.
  • Ablation Study: FSM raises performance to 74.2% with only 0.4M additional parameters.The result follows adding FSM after FAM in the ablation study.
  • Ablation Study: Replacing element-wise summation with concatenation improves performance by another 1.4% with few extra FLOPs.The concatenation merges detailed and aligned semantic features before channel reduction.
  • Visualization: Figure I compares ground truth, baseline, baseline with FAM, and FaPN outputs on Cityscapes validation using ResNet18.The comparison is presented visually across the four output conditions.

C. Additional Visualization

On MS COCO, FaPN visualizations show more accurate dense predictions at object boundaries and for small objects.

  • Additional Visualization: FaPN achieves more accurate segmentation on object boundaries and small objects in MS COCO visualizations.The comparison is shown in Figures II and III.
Loading 2108.07058v2…