Source-linked AI summary
Panoptic-DeepLab
Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen
TL;DR
Panoptic segmentation calls for semantic and instance predictions, and Panoptic-DeepLab addresses it with a bottom-up, single-shot model using dual-ASPP and dual decoders. On the Cityscapes test set, its single model ranks first at all three benchmarks, reaching 65.5% PQ, 39.0% AP, and 84.2% mIoU; it also reports results on Mapillary Vistas.
Problem
The paper addresses panoptic segmentation by combining semantic segmentation with instance segmentation.
Source: §1 · p. 1Method
Results
65.5% PQ, 39.0% AP, and 84.2% mIoU on Cityscapes test rank the single model first at all three tasks against published works.
Source: §1 · p. 1Takeaways & Limitations
Panoptic-DeepLab is reported as the first bottom-up approach with state-of-the-art panoptic segmentation results on Cityscapes and Mapillary Vistas.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We present Panoptic-DeepLab, a bottom-up and single-shot approach for panoptic segmentation. Our Panoptic-DeepLab is conceptually simple and delivers state-of-the-art results. In particular, we adopt the dual-ASPP and dual-decoder structures specific to semantic, and instance segmentation, respectively. The semantic segmentation branch is the same as the typical design of any semantic segmentation model (e.g., DeepLab), while the instance segmentation branch is class-agnostic, involving a simple instance center regression. Our single Panoptic-DeepLab sets the new state-of-art at all three Cityscapes benchmarks, reaching 84.2% mIoU, 39.0% AP, and 65.5% PQ on test set, and advances results on the other challenging Mapillary Vistas.
1. Introduction
Panoptic-DeepLab is a bottom-up approach that delivers state-of-the-art panoptic segmentation results on Cityscapes and Mapillary Vistas. Its single model ranks first at all three Cityscapes benchmarks, with a simple design and marginal added parameters and computation.
1. Introduction
- 65.5% PQ, 39.0% AP, and 84.2% mIoU on the Cityscapes test set rank the single model first at all three tasks against published works.On Mapillary Vistas validation, the best single model reaches 40.6% PQ and an ensemble of six models reaches 42.2% PQ.Source: §1 · p. 1
- Panoptic-DeepLab is the first bottom-up approach demonstrating state-of-the-art panoptic segmentation results on Cityscapes and Mapillary Vistas.The approach is also described as conceptually simple.Source: §1 · p. 1
- The single model simultaneously ranks first at all three Cityscapes benchmarks without fine-tuning on different tasks.The paper also reports that it ranks second in the instance segmentation track when unpublished entries are included.Source: §1 · p. 1
- Panoptic-DeepLab requires three training loss functions and adds only marginal parameters and slight computation overhead over a modern semantic segmentation model.Source: §1 · p. 1
2. Methods
Panoptic-DeepLab shares an encoder while using separate context modules, decoders, and prediction heads for semantic and instance segmentation. It predicts class-agnostic instance centers and pixel offsets, then fuses the semantic and instance predictions by majority vote.
2. Methods
- A shared encoder, task-specific ASPP modules and decoders, and prediction heads form the bottom-up, single-shot architecture.The encoder uses atrous convolution in its last block; each lightweight decoder upsamples through stages with a single convolution.Source: §2 · p. 1
- Semantic segmentation uses the typical softmax cross-entropy loss.Source: §2 · p. 1
- Instance segmentation represents each object by a center-of-mass heatmap and predicts offsets from foreground pixels to their corresponding centers.The center is encoded as a 2-D Gaussian with an 8-pixel standard deviation; heatmaps use MSE loss and offsets use L1 loss.Source: §2 · p. 1
- At inference, predicted foreground pixels are grouped by their closest predicted center to form class-agnostic instances.Source: §2 · p. 2
- The semantic and class-agnostic instance predictions are fused, assigning each predicted instance the majority-vote semantic label.Source: §2 · p. 2, Figure 1 · p. 2
3. Experiments
Experiments evaluate Panoptic-DeepLab on Cityscapes and Mapillary Vistas, with ablations and comparisons to prior approaches. The single model ranks first on all three Cityscapes test tasks, while Mapillary Vistas results vary by backbone and model ensemble.
3. Experiments
- Experiments cover Cityscapes and Mapillary Vistas, report mIoU, AP, and PQ, and use dataset-specific training inputs with multiscale and flipped inference.The Cityscapes and Mapillary Vistas training crops, batch sizes, and iteration counts differ.Source: §3 · p. 2
3.1. Ablation Studies
- 63.0% PQ, 35.3% AP, and 80.5% mIoU are the best ablation results after increasing semantic-branch channels to 256.On Cityscapes validation, Adam adds 0.6% PQ, MSE center-heatmap loss adds 1.1% PQ, and dual decoder plus dual ASPP adds 0.7% PQ while maintaining similar AP and mIoU. The larger crop improves AP by 0.6% and mIoU by 0.9%.Source: §3.1 · pp. 2–3
3.2. Cityscapes Results
- On Cityscapes validation, the best model using only fine annotations exceeds SSAP by 3.0% PQ and 1.2% AP, and AdaptIS by 2.1% PQ, 2.2% AP, and 2.3% mIoU.With extra data, it exceeds UPSNet by 5.2% PQ, 3.5% AP, and 3.9% mIoU, and Seamless by 2.0% PQ and 2.4% mIoU; the authors use no COCO, coarse annotations, depth, or video.Source: §3.2 · p. 3
- The single unified Panoptic-DeepLab ranks first on all three Cityscapes test tasks against published works.It ranks second in the instance segmentation track when unpublished entries are also counted.Source: §3.2 · p. 3
3.3. Mapillary Vistas Challenge
- 40.6% PQ is the best Mapillary Vistas validation score for HRNet-W48+, while a six-model ensemble reaches 42.2% PQ, 18.2% AP, and 58.7% mIoU.The best single model exceeds DeeperLab by 8.3% PQ and Seamless by 2.6% PQ.Source: §3.3 · p. 3