Source-linked AI summary
UPSNet: A Unified Panoptic Segmentation Network
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun
TL;DR
Panoptic segmentation requires semantic understanding of amorphous regions and instance delineation of countable objects, but these subtasks traditionally use different modeling strategies. UPSNet unifies them with a shared backbone, specialized heads, and a parameter-free panoptic head that adds an unknown class. Across Cityscapes, COCO, and an internal dataset, it achieves state-of-the-art performance with significantly faster inference.
Problem
Panoptic segmentation must combine semantic segmentation of stuff with instance segmentation of things despite their different modeling strategies.
Method
UPSNet uses one residual backbone, deformable-convolution semantic and Mask R-CNN-style instance heads, and a parameter-free panoptic head that fuses logits with an unknown class.
Results
UPSNet achieves state-of-the-art performance with significantly faster inference on Cityscapes, COCO, and an internal dataset.
Takeaways & Limitations
The unified design supports end-to-end training, handles varying numbers of instances, and resolves conflicts between semantic and instance segmentation.
Abstract
from arXiv · showhide
In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolve the conflicts between semantic and instance segmentation. Additionally, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves state-of-the-art performance with much faster inference. Code has been made available at: https://github.com/uber-research/UPSNet
1. Introduction
Panoptic segmentation unifies semantic segmentation of amorphous regions with instance segmentation of countable objects. UPSNet uses shared representations, specialized heads, and a panoptic head to resolve their conflicts while achieving state-of-the-art performance with faster inference.
- Motivation: Semantic segmentation labels amorphous regions, whereas instance segmentation identifies countable objects and their instances.The tasks use different modeling strategies despite both operating at pixel level.
- Panoptic segmentation: Panoptic segmentation combines instance outputs for thing classes with semantic outputs for stuff classes.This formulation unifies the two complementary scene-understanding tasks.
- UPSNet: UPSNet shares a single backbone and adds semantic and Mask R-CNN-style instance heads to solve both subtasks simultaneously.The semantic head uses deformable convolution and multi-scale FPN information, while the instance head predicts masks, boxes, and classes.
- UPSNet: Its parameter-free panoptic head combines logits from both heads and adds an unknown class to resolve semantic–instance conflicts.The design also accommodates varying numbers of instances and supports end-to-end back propagation.
- Results: UPSNet achieves state-of-the-art performance with much faster inference on Cityscapes, COCO, and an internal driving dataset.The internal dataset is approximately three times larger than Cityscapes.
2. Related Work
Related work develops semantic and instance segmentation separately, while panoptic segmentation unifies their complementary outputs. UPSNet differs by using one shared backbone and a dedicated panoptic head.
- Semantic Segmentation: Semantic segmentation methods focus on classifying amorphous regions using multi-scale feature aggregation or structured prediction.The related work emphasizes global context and deep convolutional feature extraction.
- Instance Segmentation: Instance segmentation identifies both a pixel’s semantic class and the specific object instance to which it belongs.Earlier approaches commonly used region proposals followed by selection, while Mask R-CNN jointly predicts masks and recognition.
- Panoptic Segmentation: Panoptic segmentation addresses detection methods’ omission of stuff classes and semantic methods’ lack of instance boundaries.It defines thing outputs as instance segmentations and stuff outputs as semantic segmentation.
- UPSNet: UPSNet uses one backbone for both semantic and instance segmentation and adds a panoptic head for instance and class prediction.This contrasts with methods using separate branches for the two subtasks.
3. Unified Panoptic Sementation Network
UPSNet combines a shared residual-FPN backbone with semantic, instance, and panoptic heads. The panoptic head fuses per-pixel logits, supports variable instance counts, and introduces an unknown class to manage conflicts.
- Architecture: UPSNet uses a shared convolutional backbone with multiple task-specific heads.The backbone adopts a residual network with a feature pyramid network.
- Instance Segmentation Head: The instance head follows Mask R-CNN, producing bounding-box regression, classification, and mask outputs for instance-aware representations.These representations are passed to the panoptic head.
- Semantic Segmentation Head: The semantic head uses deformable convolutions and multi-scale FPN features to segment all semantic classes without distinguishing instances.It also uses pixel-wise cross-entropy and an RoI loss emphasizing foreground-object pixels.
- Panoptic Segmentation Head: The panoptic head combines semantic and instance logits into channels for fixed stuff classes and image-dependent instance classes.During inference, mask pruning determines the number of instances used to construct the panoptic logits.
- Unknown Prediction: The unknown class is preferred when an incorrect prediction is unavoidable because it increases one false negative without adding another class’s false positive.This follows the stated decomposition of panoptic quality into recognition and semantic quality.
- Unknown Prediction: UPSNet computes an unknown-class logit as Z_unknown = max(X_thing) − max(X_mask) to identify pixels likely missing an instance.Thirty percent of ground-truth masks are randomly treated as unknown during training, and unknown pixels are ignored during metric evaluation.
- Implementation: Training uses ground-truth boxes, masks, and labels to construct panoptic logits, with end-to-end RPN training and downsampled logits for high-resolution images.The implementation uses PyTorch, Horovod, and 16 GPUs.
4. Experiments
UPSNet is evaluated across COCO, Cityscapes, and an internal driving dataset using panoptic, segmentation, accuracy, and runtime metrics. Experiments report strong performance, faster inference than the combined method, and ablations supporting the panoptic head, instance-class assignment, loss balancing, and unknown-class prediction.
- Experimental setup: Experiments cover COCO, Cityscapes, and an internal ego-centric driving dataset, reporting validation-set PQ, RQ, SQ, mask AP, semantic mIoU, and inference runtime.COCO uses approximately 118k training and 5k validation images; Cityscapes uses 2,975 training, 500 validation, and 1,525 test images; the internal dataset uses 10,235 training, 1,139 validation, and 1,186 test images.
- COCO: On COCO, UPSNet performs better than the compared methods on all reported metrics except SQ.The paper notes that higher RQ can coincide with slightly lower SQ because more true-positive segments may have lower IoU.
- Cityscapes: On Cityscapes, UPSNet outperforms the combined method under single-scale testing and remains slightly better after multi-scale testing.Both methods improve with multi-scale testing; comparisons also include methods using different backbones or pretraining configurations.
- Our Dataset: On the internal dataset, UPSNet performs significantly better than the combined method on all metrics except SQ.The paper reports visual examples with observations similar to those on COCO and Cityscapes.
- Runtime: UPSNet is faster as image size increases; the combined method takes about 3× the inference time of UPSNet.Runtime measurements average 100 single-scale runs on one GTX 1080 Ti GPU and one Intel Xeon E5-2687W CPU.
- Ablation Study: Ablations show gains from training the panoptic head, the proposed instance-class assignment, loss balancing, and unknown-class prediction.Training the panoptic head improves PQ; the paper also reports improvements from loss weights of 0.2 and 0.1 for semantic and panoptic losses, respectively, and from unknown prediction with RoI loss.
- Ablation Study: Oracle experiments identify semantic segmentation as the largest improvement opportunity, with ground-truth semantic segmentation providing a +29.5 PQ gain.Ground-truth boxes and class assignments also improve PQ, while imperfect PQTh indicates room to improve mask segmentation.
5. Conclusion
UPSNet unifies panoptic segmentation with one backbone and lightweight heads, including a parameter-free panoptic head that handles varying classes and enables end-to-end learning. Across three datasets, it achieves state-of-the-art performance with significantly faster inference.
- UPSNet uses a single backbone and two lightweight heads to predict semantic and instance segmentation in one shot.
- The parameter-free panoptic head combines semantic and instance logits while adding an unknown-class channel.
- The panoptic head handles varying numbers of classes per image and enables back propagation into bottom representation learning.
- UPSNet achieves state-of-the-art performance with significantly faster inference on three large datasets.
6. Supplementary Material
The supplementary material specifies training, testing, and evaluation procedures, compares UPSNet variants, and presents panoptic results and visual examples across COCO, Cityscapes, and an internal dataset.
- All experiments use a 1500-iteration warm-up, increasing the learning rate from 0.002 to 0.02, with ImageNet-pretrained MSRA weights.
- COCO uses fixed-scale training, multi-scale testing across eleven image scales, left-right flipping, and averaged semantic logits.
- Cityscapes uses multi-scale training with shorter edges sampled from [800, 1024] and multi-scale testing across twelve scales.
- The internal dataset uses multi-scale training with shorter edges sampled from [800, 1200] but no multi-scale testing.
- UPSNet-C removes the panoptic head and uses semantic and instance losses with combine heuristics at test time.
- UPSNet-CP retains the UPSNet model and training but uses combine heuristics instead of the panoptic head during testing.
- Tables 7–9 report panoptic segmentation results for COCO, Cityscapes, and the internal dataset, while Figures 5–7 show visual examples.