PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen

arXiv:1911.10194v3cs.CV

TL;DR

Panoptic segmentation needs to combine semantic labeling with instance delineation, while existing top-down methods are slow and bottom-up methods had weaker performance. Panoptic-DeepLab uses separate semantic and instance branches with simple center-based grouping, achieving state-of-the-art bottom-up results across benchmarks. Its main limitation is weaker handling of large scale variation than top-down approaches.

  • Problem

    Existing top-down panoptic methods are slow, while bottom-up approaches had demonstrated inferior performance, motivating a strong and fast bottom-up baseline.

  • Method

    Panoptic-DeepLab uses dual-ASPP and dual-decoder branches, with a semantic segmentation branch and class-agnostic instance center regression for grouping.

  • Results

    65.5% PQ, 39.0% AP, and 84.2% mIoU let a single model rank first on all three Cityscapes test-set tasks, while also achieving state-of-the-art results on Mapillary Vistas and COCO.

  • Takeaways & Limitations

    Panoptic-DeepLab establishes a solid bottom-up baseline with state-of-the-art benchmark performance and near real-time end-to-end inference speed.

  • Takeaways & Limitations

    Panoptic-DeepLab handles large-scale variation less effectively than top-down approaches, and combining their strengths remains beyond this work's scope.

Abstract

from arXiv · show

In this work, we introduce Panoptic-DeepLab, a simple, strong, and fast system for panoptic segmentation, aiming to establish a solid baseline for bottom-up methods that can achieve comparable performance of two-stage methods while yielding fast inference speed. In particular, Panoptic-DeepLab adopts the dual-ASPP and dual-decoder structures specific to semantic, and instance segmentation, respectively. The semantic segmentation branch is the same as the typical design of any semantic segmentation model (e.g., DeepLab), while the instance segmentation branch is class-agnostic, involving a simple instance center regression. As a result, our single Panoptic-DeepLab simultaneously ranks first at all three Cityscapes benchmarks, setting the new state-of-art of 84.2% mIoU, 39.0% AP, and 65.5% PQ on test set. Additionally, equipped with MobileNetV3, Panoptic-DeepLab runs nearly in real-time with a single 1025x2049 image (15.8 frames per second), while achieving a competitive performance on Cityscapes (54.1 PQ% on test set). On Mapillary Vistas test set, our ensemble of six models attains 42.7% PQ, outperforming the challenge winner in 2018 by a healthy margin of 1.5%. Finally, our Panoptic-DeepLab also performs on par with several top-down approaches on the challenging COCO dataset. For the first time, we demonstrate a bottom-up approach could deliver state-of-the-art results on panoptic segmentation.

1. Introduction

Panoptic segmentation must combine semantic labels with individual thing instances, but prevailing top-down methods are slow and bottom-up methods had lagged in performance. Panoptic-DeepLab addresses this gap with a simple, fast bottom-up design that achieves state-of-the-art results across major benchmarks.

  • Panoptic segmentation assigns every pixel a value encoding both its semantic label and instance identity, covering individual things and stuff regions.
  • Top-down methods resolve overlapping predictions with confidence or category heuristics and are usually slow because their pipelines contain multiple sequential processes.
  • Bottom-up methods naturally produce non-overlapping segments and can merge predictions simply, but previously showed inferior performance to top-down approaches.
  • Panoptic-DeepLab uses dual-ASPP and dual-decoder branches, with class-agnostic center regression assigning pixels to their closest predicted centers.
  • 65.5% PQ, 39.0% AP, and 84.2% mIoU let a single model rank first on all three Cityscapes test-set tasks.
  • 42.7% PQ on the Mapillary Vistas test set from a six-model ensemble exceeded the 2018 challenge winner by 1.5% PQ, while COCO performance was on par with several top-down approaches.

2. Related Works

Panoptic segmentation methods are divided into top-down proposal-based systems and bottom-up proposal-free systems. Panoptic-DeepLab follows the bottom-up paradigm while simplifying instance grouping relative to earlier approaches.

  • Top-down: Top-down methods commonly use Mask R-CNN to extract overlapping instances, then apply post-processing and a stuff branch to complete the panoptic output.
  • Bottom-up: Bottom-up methods first predict semantic segmentation, then group thing pixels into instance clusters without generating proposals.
  • Panoptic-DeepLab: Panoptic-DeepLab uses dual-context and dual-decoder modules for semantic and instance predictions, followed by majority-vote fusion.
  • Panoptic-DeepLab: Panoptic-DeepLab predicts instance centers directly and assigns pixels to their closest centers, avoiding OPTICS and advanced clustering losses used by other methods.
  • Keypoint representation: Keypoint-based alternatives group pixels using embedding vectors, detected keypoints, paired corners, or extreme points relative to a center.

3. Panoptic-DeepLab

Panoptic-DeepLab is a bottom-up, single-shot architecture that separates semantic and instance processing while using center regression for class-agnostic grouping. Semantic and instance outputs are then merged efficiently into the final panoptic segmentation.

  • Panoptic-DeepLab is deployed in a bottom-up and single-shot manner.
  • The architecture combines a shared encoder, decoupled ASPP modules, decoupled task-specific decoders, and prediction heads.
  • The encoder uses an ImageNet-pretrained backbone with atrous convolution, while separate ASPP and decoder modules provide task-specific context and decoding.
  • Semantic segmentation: The semantic head predicts thing and stuff classes using weighted bootstrapped cross entropy.
  • Class-agnostic instance segmentation: Each object is represented by its center of mass, and foreground pixels predict offsets to their corresponding centers.
  • Simple instance grouping: Instance masks are formed by assigning each predicted thing pixel to the closest center after applying its predicted offset.
  • Simple instance grouping: Semantic predictions filter stuff pixels, whose instance ids are set to 0, before instance grouping and fusion.
  • Efficient merging: Majority-vote merging assigns each predicted instance the most common semantic label and runs in 3 ms for a 1025 × 2049 input.

4. Experiments

Experiments evaluate Panoptic-DeepLab across Cityscapes, Mapillary Vistas, and COCO using mIoU, AP, and PQ, with ablations, runtime measurements, and discussion of performance boundaries. The model achieves strong benchmark results while retaining a bottom-up design, but large-scale instance variation and post-processing remain relevant limitations.

  • Experimental setup: mIoU, AP, and PQ evaluate semantic, instance, and panoptic segmentation results, respectively.
  • Ablation studies: 0.7% PQ improvement comes from Adam over SGD momentum, while MSE heatmap training adds 0.8% PQ.
  • Ablation studies: 63.0% PQ, 35.3% AP, and 80.5% mIoU are achieved in the best Cityscapes validation ablation setting.
  • COCO: 41.2% PQ on COCO validation sets a state-of-the-art result for bottom-up methods, while 41.4% PQ on test-dev is comparable to most top-down methods.

5. Conclusion

Panoptic-DeepLab is presented as a simple, strong, and fast baseline for bottom-up panoptic segmentation. It uses three training losses, adds marginal parameters, and achieves state-of-the-art performance with near real-time end-to-end inference.

  • Panoptic-DeepLab is a simple, strong, and fast baseline for bottom-up panoptic segmentation.
  • The model requires only three loss functions during training and adds marginal parameters to a modern semantic segmentation model.
  • Panoptic-DeepLab is the first bottom-up and single-shot model reported to attain state-of-the-art performance on several public benchmarks.
  • The system delivers near real-time end-to-end inference speed.

A. HRNet Variant

The original HRNet segmentation head combines features from four resolutions at 1/4 resolution after upsampling. Its ImageNet classification head applies bottleneck residual modules to each output resolution while progressively downsampling feature maps.

  • HRNet’s original segmentation head upsamples features from all four resolutions to 1/4 resolution, concatenates them, and fuses them with a 1 × 1 convolution.
  • The ImageNet classification head applies a bottleneck residual module to every output resolution to increase channels.
  • The classification head gradually downsamples feature maps for ImageNet pre-training.

A.2. HRNet+

The HRNet+ variant retains the ImageNet classification head and replaces its final classifier with an ASPP and decoder. HRNet-Wider+ substitutes Xception modules to reduce parameters and computation at a marginal performance cost.

  • HRNet+ keeps the ImageNet classification head instead of discarding it after pre-training.The retained head contains around 20% of the total parameters.
  • HRNet+ replaces global average pooling and the linear classifier with an ASPP module and a decoder.Its encoder output stride is 32, and a stride-16 feature map is projected into the decoder.
  • HRNet-Wider+ replaces basic residual modules with Xception modules, reducing parameters and computation FLOPs at the cost of marginal performance degradation.

A.4. Atrous HRNet

The atrous HRNet variant removes downsampling that produces 1/32-resolution feature maps and uses rate-2 atrous convolution instead. This increases computation without improving performance, whereas removing the final Auto-DeepLab stride improves validation PQ by 1%.

  • A.4. Atrous HRNet: Table 10 compares the DeepLabV3+ and Panoptic-DeepLab decoder designs on the Cityscapes validation set.
  • A.4. Atrous HRNet: The atrous HRNet modification increases computation FLOPs but does not improve performance over HRNet+.
  • A.4. Atrous HRNet: Removing the stride that generates Auto-DeepLab’s 1/32 feature map improves Mapillary Vistas validation PQ by 1%.The modification keeps higher spatial resolution within the backbone.

C. Comparison with DeepLabV3+ decoder

Panoptic-DeepLab’s slightly modified decoder outperforms DeepLabv3+ on Cityscapes validation while also being marginally faster with only extra marginal parameters.

  • 0.5% PQ, 0.8% AP, and 0.3% mIOU improvements over DeepLabv3+ are reported on Cityscapes validation.
  • The decoder modification yields greater improvement for instance segmentation than for semantic segmentation.
  • Panoptic-DeepLab is slightly faster than DeepLabv3+ despite using extra marginal parameters.

D. Comparison with different instance scores

The confidence-score ablation affects instance AP but not the final mIoU or PQ, with the product of objectness and class scores performing best.

  • The confidence-score choice affects AP only, not final mIoU or PQ.
  • 35.3% AP is achieved by multiplying Score(Objectness) and Score(Class), compared with 28.9% and 35.1% using either score alone.
  • Panoptic-DeepLab avoids overlapping predictions, so confidence-score changes do not alter its final mIoU and PQ results.

F. More Visualization

Additional visualization results for Panoptic-DeepLab are provided in three figures.

  • More visualization results of Panoptic-DeepLab are provided.
  • The additional visualizations appear in Fig. 9, Fig. 10, and Fig. 11.
  • The visualization section presents further qualitative results for the proposed model.
Loading 1911.10194v3…