Source-linked AI summary
Panoptic Feature Pyramid Networks
Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár
TL;DR
Panoptic segmentation requires jointly handling instance and semantic segmentation, yet prior top methods used separate networks. Panoptic FPN adds a lightweight semantic branch to Mask R-CNN over a shared FPN backbone, achieving strong single-task results and panoptic accuracy equivalent to two separate FPNs at roughly half the compute. The study also examines the training choices needed for effective joint prediction and presents the method as a simple, efficient baseline.
Problem
Top-performing instance- and semantic-segmentation methods use different architectures, leaving the joint panoptic task without a shared single-network baseline.
Method
Panoptic FPN adds a lightweight semantic segmentation branch in parallel with Mask R-CNN’s instance branch while sharing an unchanged FPN backbone.
Results
A single jointly trained FPN reaches accuracy equivalent to two separate FPNs for panoptic segmentation with roughly half the compute, while also performing strongly on instance and semantic segmentation.
Takeaways & Limitations
Panoptic FPN provides a simple, flexible, and computationally efficient baseline for unified panoptic segmentation.
Takeaways & Limitations
The grouped-FPN variant gives mixed gains, indicating that better multi-task training strategies may improve results.
Abstract
from arXiv · showhide
The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-of-the-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, top-performing method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.
1. Introduction
Panoptic FPN unifies instance and semantic segmentation in one network by adding a lightweight dense-prediction branch to Mask R-CNN with FPN. It achieves strong performance across both tasks while remaining computationally efficient.
- Motivation: Panoptic segmentation combines semantic segmentation for stuff classes with instance segmentation for thing classes, but designing one accurate network is challenging.Existing top-performing methods use substantially different architectures for the two tasks.
- Approach: Panoptic FPN adds a parallel semantic segmentation branch to Mask R-CNN with FPN without changing the FPN backbone.The region-based branch produces instance outputs, while the added dense-prediction branch produces semantic outputs.
- Approach: Proper joint training requires balancing branch losses, constructing minibatches, adjusting learning-rate schedules, and selecting data augmentation and semantic-branch designs.The method is robust to exact design choices, but these training decisions are important for good results.
- Results: Panoptic FPN achieves excellent independently trained results for instance and semantic segmentation on COCO and Cityscapes.Its instance-segmentation behavior is equivalent to Mask R-CNN, while its semantic branch matches the accuracy of recent dilation-based methods such as DeepLabV3+.
- Results: With proper training, one FPN solves both panoptic tasks at the accuracy of two separate FPNs using roughly half the compute.At equal compute, the joint network outperforms two independent networks by a healthy margin.
- Results: Panoptic FPN adds only slight overhead over Mask R-CNN and avoids dilation, preserving compatibility with standard high-performing backbones.The authors connect this efficiency and flexibility to fast training and inference.
2. Related Work
Prior panoptic-segmentation approaches generally relied on separate networks, while related semantic and instance-segmentation methods developed around distinct architectural traditions. Panoptic FPN instead uses a single shared-network baseline centered on Mask R-CNN with FPN.
- Panoptic segmentation: Panoptic segmentation renewed interest in jointly solving thing and stuff segmentation through a shared task specification and metrics.Its history includes scene parsing, image parsing, and holistic scene understanding.
- Panoptic segmentation: Competitive panoptic-segmentation entries used separate networks for instance and semantic segmentation without shared computation.This motivates designing a single effective network as a baseline for both tasks.
- Instance segmentation: Region-based methods such as Mask R-CNN with FPN have remained dominant on object-detection leaderboards.Panoptic FPN adopts this region-based foundation for instance segmentation while seeking a unified architecture.
- Instance segmentation: Alternative instance-segmentation methods begin with pixel-wise semantic segmentation and group predictions into instances, often using separate networks for instance-level information.Panoptic FPN instead targets a single network for the joint task.
- Semantic segmentation: Modern semantic-segmentation methods commonly use dilated convolutions to increase feature resolution, but this can substantially increase compute and memory.Panoptic FPN chooses a different approach to preserve backbone flexibility and compatibility with Mask R-CNN.
- Semantic segmentation: Panoptic FPN uses FPN as a lightweight encoder-decoder and shows that the unchanged FPN backbone can also be effective for semantic segmentation.FPN combines top-down upsampling with lateral connections across multiple feature resolutions.
- Multi-task learning: Multi-task learning can degrade performance for diverse tasks, although task relationships and mitigation strategies can produce gains.Examples include benefits from the mask branch for Mask R-CNN’s box branch and joint detection and semantic segmentation of thing classes.
3. Panoptic Feature Pyramid Network
Panoptic FPN extends Mask R-CNN with a lightweight semantic segmentation branch attached to a shared FPN backbone, producing instance and pixel-wise outputs in one network. Its design combines multi-scale FPN features, joint loss training, and efficient high-resolution prediction.
- Panoptic FPN: Panoptic FPN minimally extends Mask R-CNN with FPN by adding a parallel semantic segmentation branch while leaving the backbone unchanged.The shared architecture generates region-based instance outputs and dense-pixel semantic outputs.
- FPN backbone: FPN provides multi-resolution features through a top-down pathway with lateral connections, enabling a lightweight asymmetric decoder with shared channel dimensions.This contrasts with symmetric encoder-decoder designs while retaining progressively higher-resolution features.
- Semantic segmentation branch: The semantic branch upsamples each FPN level to 1/4 scale, sums the resulting feature maps, and produces full-resolution per-pixel class labels.It uses convolution, group normalization, ReLU, bilinear upsampling, a 1×1 convolution, and softmax; an “other” class covers object pixels.
- Panoptic inference: Panoptic inference resolves overlaps between instance and semantic outputs so each pixel receives a single class label and instance id or void.The post-processing is similar in spirit to non-maximum suppression.
- Efficiency: FPN at 1/4 output scale has computational cost similar to dilation-16 while producing 4× higher-resolution output, and is about 2× more efficient than a symmetric encoder-decoder.Increasing dilation-based resolution to 1/8 requires approximately 3× more compute and memory.
- Joint training: Joint training combines classification, box, mask, and semantic per-pixel cross-entropy losses, with separate weights needed because the branch losses differ in scale and normalization.The total loss is L = λi (Lc + Lb + Lm) + λsLs.
4. Experiments
Experiments evaluate Panoptic FPN as a lightweight single-network baseline across semantic, instance, and panoptic segmentation on COCO and Cityscapes. The results show competitive task-specific accuracy and strong joint-task efficiency and performance.
- Semantic segmentation: Semantic FPN achieves competitive semantic segmentation results on COCO and Cityscapes with a lightweight dense-prediction branch attached to FPN.The branch is designed to preserve compatibility with Mask R-CNN while avoiding the heavier dilation-based approach used by many semantic segmentation systems.
- Semantic segmentation: 128 channels in the semantic branch balance accuracy and efficiency, while sum aggregation is marginally more accurate and more efficient than concatenation.These design choices are supported by the semantic-branch ablations.
- Multi-task training: With properly tuned loss weights, adding one task’s branch can improve the other task over its single-task baseline.The experiments vary the semantic loss weight λs and instance loss weight λi to study these multi-task effects.
- Multi-task training: A single FPN achieves comparable accuracy to two independent FPNs with roughly half the compute, and combined-loss training performs much better than alternating losses.The backbone dominates compute, making the shared-backbone comparison especially consequential.
- Panoptic segmentation: At similar compute, Panoptic FPN significantly outperforms two separate networks and improves COCO single-network methods by an ∼9 point PQ margin.On Cityscapes, it also outperforms DIN by a 4.3 point PQ margin without extra coarse training data or additional bells and whistles.
- Panoptic segmentation: Panoptic FPN substantially outperforms single-model COCO challenge entries and establishes a baseline for panoptic segmentation, although grouped FPN channels produce mixed gains.The authors expect more sophisticated multi-task strategies could yield stronger improvements.
5. Conclusion
Panoptic FPN is a conceptually simple, effective baseline for panoptic segmentation.
- Panoptic FPN combines Mask R-CNN with a lightweight semantic segmentation branch for dense-pixel prediction.