Source-linked AI summary
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping Luo, Tong Lu
TL;DR
Panoptic segmentation exposes limitations in transformer-based DETR, including lengthy convergence, limited feature resolution, low-fidelity boundaries, and inadequate treatment of things and stuff. Panoptic SegFormer addresses these issues with deeply supervised mask decoding, query decoupling, and improved post-processing, achieving 56.2% PQ on COCO test-dev and improving DETR by 6.2% PQ.
Problem
DETR’s panoptic segmentation has lengthy convergence, limited feature resolution, low-fidelity boundaries, and insufficiently differentiated handling of things and stuff.
Method
Panoptic SegFormer combines a deeply supervised multi-scale mask decoder, separate thing and stuff query sets, and mask-wise post-processing that considers classification and mask qualities.
Results
Panoptic SegFormer reaches 56.2% PQ on COCO test-dev and improves over baseline DETR by 6.2% PQ.
Takeaways & Limitations
Task-specific treatment of things and stuff, together with efficient mask decoding and post-processing, supports strong panoptic segmentation performance with fewer training epochs.
Takeaways & Limitations
The method relies on deformable attention, has somewhat slow speed, struggles with larger spatial feature shapes, and performs poorly on small targets.
Abstract
from arXiv · showhide
Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework for panoptic segmentation with transformers. It contains three innovative components: an efficient deeply-supervised mask decoder, a query decoupling strategy, and an improved post-processing method. We also use Deformable DETR to efficiently process multi-scale features, which is a fast and efficient version of DETR. Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner. This deep supervision strategy lets the attention modules quickly focus on meaningful semantic regions. It improves performance and reduces the number of required training epochs by half compared to Deformable DETR. Our query decoupling strategy decouples the responsibilities of the query set and avoids mutual interference between things and stuff. In addition, our post-processing strategy improves performance without additional costs by jointly considering classification and segmentation qualities to resolve conflicting mask overlaps. Our approach increases the accuracy 6.2\% PQ over the baseline DETR model. Panoptic SegFormer achieves state-of-the-art results on COCO test-dev with 56.2\% PQ. It also shows stronger zero-shot robustness over existing methods. The code is released at \url{https://github.com/zhiqi-li/Panoptic-SegFormer}.
1. Introduction
Panoptic SegFormer addresses transformer-based panoptic segmentation by tailoring processing to things and stuff, improving mask quality, query specialization, and overlap resolution.
- Motivation: Panoptic segmentation unifies semantic and instance segmentation by assigning image content to things and stuff.Things are countable instances with unique ids, whereas stuff denotes amorphous regions without instance ids.
- Motivation: DETR-based methods face lengthy convergence, limited feature resolution, low-fidelity boundaries, and difficulty treating things and stuff appropriately.These issues arise from squared self-attention complexity and using bounding-box representations for both content types.
- Contributions: Panoptic SegFormer uses a deeply supervised mask decoder with multi-scale attention maps to produce higher-fidelity masks and faster convergence.Layer-wise supervision encourages discriminative intermediate attention representations.
- Contributions: Query decoupling assigns separate query sets to things and stuff, using bipartite matching for things and class-fixed assignment for stuff.The strategy avoids mutual interference and improves stuff segmentation quality.
- Contributions: Mask-wise post-processing considers classification probability and predicted mask quality when resolving overlaps, improving results without additional costs.The method replaces commonly used pixel-wise argmax processing.
2. Related Work
Prior panoptic segmentation methods evolved from separately combining instance and semantic models toward unified transformer-based query frameworks.
- Panoptic Segmentation: Panoptic segmentation was introduced as a joint task combining instance segmentation and semantic segmentation, with things and stuff handled separately.Early baselines combined outputs from individual instance and semantic segmentation models.
- Panoptic Segmentation: Panoptic FCN simplified the pipeline through a two-branch design that combines object- or region-level kernels with image-level features.Its design follows a “top-down meets bottom-up” structure similar to CondInst.
- Transformer-Based Methods: Transformer methods represent things and stuff as queries for end-to-end panoptic segmentation.DETR uses query-based detection and attention maps, Max-Deeplab predicts categories and masks directly, and MaskFormer adds a pixel decoder for higher-resolution features.
- End-to-End Object Detection: DETR simplified object detection by predicting bounding boxes with learnable queries and removing handcrafted components such as NMS and anchors.Its encoder-decoder transformer frames detection as a dictionary lookup problem.
3. Methods
Panoptic SegFormer refines multi-scale features, separates thing and stuff queries, predicts masks through attention-based decoding, and merges masks into panoptic outputs.
- Overall Architecture: The architecture comprises a transformer encoder, location decoder, and mask decoder operating on refined multi-scale backbone features.The encoder refines features, the location decoder captures thing location clues, and the mask decoder predicts categories and masks.
- Overall Architecture: The backbone provides multi-scale features that are projected to 256 channels, flattened, concatenated, and refined by the transformer encoder.The encoder processes the concatenated tokens from three feature scales.
- Query Decoupling: Separate thing and stuff queries prevent their responsibilities from interfering while retaining a shared processing and post-processing pipeline.Thing queries target instances, whereas stuff queries target stuff categories through class-fixed assignment.
- Location Decoder: The location decoder produces location-aware thing queries using refined feature tokens and can learn from bounding boxes or mask mass centers.An auxiliary detection head is used during training and discarded during inference.
- Mask Decoder: Deep supervision guides attention modules toward meaningful locations, addressing slow DETR convergence through an ultra-light mask-generation head.The overall training objective combines separate things and stuff losses weighted by hyperparameters.
- Post-Processing: Mask-wise merging resolves conflicts among predicted masks using confidence scores that combine classification probability and mask quality.This replaces pixel-wise argmax for panoptic inference and yields non-overlapping results.
4. Experiments
Experiments show that Panoptic SegFormer improves panoptic segmentation across COCO and ADE20K, while also supporting competitive instance segmentation. Ablations attribute gains to query decoupling, mask-wise merging, deeply supervised mask decoding, and location decoding.
- Main Results: 49.6% PQ is achieved on COCO val with ResNet-50 and single-scale input, surpassing DETR by 6.2% PQ and MaskFormer by 3.1% PQ.
- Main Results: 56.2% PQ is achieved on COCO test-dev with Swin-L, surpassing MaskFormer by 2.9% PQ and Innovation by 2.7% PQ.
- Main Results: Panoptic SegFormer achieves comparable COCO instance-segmentation performance to QueryInst and HTC, and exceeds K-Net by 1.8 AP.Discarding stuff queries converts the panoptic model into an instance-segmentation model; random crops add 1.3 AP.
- Ablation Studies: Adding location-decoder layers improves performance on things by helping queries learn location clues.
- Ablation Studies: Mask-wise merging consistently outperforms pixel-wise argmax in Mask PQ and Boundary PQ, while reducing post-processing time by 20% for DETR.For DETR, mask-wise merging produces a 1.3% PQ improvement over the original post-processing strategy.
- Ablation Studies: Query decoupling improves PQst and APseg over joint matching by separating thing queries from stuff queries.The experiments associate joint matching with interference between thing preference and stuff precision when one query set handles both types.
5. Conclusion
Panoptic SegFormer relies on deformable attention for multi-scale features, but remains limited in speed, larger spatial shapes, and small-target performance. The discussion argues that flexible, task-specific designs remain preferable to complete pipeline unification.
- Deformable attention enables multi-scale feature processing, but makes the model somewhat slow.
- The model struggles with larger spatial feature shapes and performs poorly on small targets.
- The authors argue that “seek common ground while reserving differences” is more suitable than complete unification for segmentation tasks.
- Task-specific designs currently deliver better performance, while Panoptic SegFormer is intended to inspire further unified-framework research.
A.1. Panoptic SegFormer.
This section details the Panoptic SegFormer implementation, including DETR-style architecture choices, training settings, post-processing, and multi-scale deformable attention integration.
- The model uses six deformable attention layers in the location decoder and six vanilla cross-attention layers in the mask decoder.
- COCO training uses 24 epochs, multi-scale augmentation, and 300 thing queries.
- ADE20K training uses 100 epochs, image sizes from 512 to 2048, and 50 stuff queries.
- The framework is slightly more efficient than DETR under the reported V100, batch-size-one FPS measurement.
- Mask-wise merging avoids incorrect pixel assignments by resolving overlaps among masks rather than selecting the largest per-pixel logit.
- Multi-scale deformable DETR generates attention maps by multiplying queries with C5 features while retaining other detection settings.
B. Discussion
The ablation discussion finds that multi-scale deformable attention improves panoptic quality, with multi-scale processing substantially outperforming single-scale processing.
- Effect of Deformation Attention: 2.9% PQ improvement is obtained from multi-scale deformable attention compared with DETR.
- Effect of Deformation Attention: 5.7% PQ separates multi-scale attention from single-scale attention, highlighting the role of multi-scale features for segmentation.
B.1. Post-processing Method
The post-processing analysis contrasts pixel-wise argmax with mask-wise merging and examines overlap resolution, confidence weighting, thresholds, query behavior, and mask-decoder attention.
- Defects of Pixel-wise Argmax: Pixel-wise argmax can produce false positives because it considers only per-pixel mask logits and may select extremely small or suboptimal values.
- Mask-wise Merging: Segmentation scores can remain high even when classification probability is low, motivating joint consideration of both qualities.
- Post-Processing Method: The heuristic procedure has the lowest PQst because it always prefers things over stuff during overlap resolution.
- Mask-wise Merging: Using both classification and segmentation scores performs better across multiple models, with α=1 and β=2 selected as defaults.
- Mask-wise Merging: The method is not very sensitive to its two filtering thresholds according to the reported threshold evaluation.
- Mask-wise Merging: Mask-wise merging assigns mask priorities using confidence scores that combine classification probability and predicted mask quality.
- Mask Decoder: The mask decoder’s attention heads show different regional preferences, whose collaboration supports accurate mask prediction.
B.2. Location Decoder
The location decoder filters low-quality thing queries and supports efficient panoptic segmentation, while the mask decoder uses multi-head attention to improve mask quality.
- Location Decoder: The workflow differs from box-based two-stage panoptic segmentation because segmentation is not performed by extracting regions from detected bounding boxes.The paper states that box-based methods depend heavily on detection quality, whereas its location decoder is used differently.
- Visualization Results: 50.6% PQ is reported for Panoptic SegFormer, compared with 45.1% PQ for DETR and 47.6% PQ for MaskFormer.These visualization results are presented for the original image, competing methods, and ground truth.
- Visualization Results: The method performs well in crowded or occluded scenes and produces fewer artifacts than DETR in the cited visual comparison.The paper attributes the reduced artifacts to mask-wise inference.
- Location Decoder: The location decoder filters low-quality thing queries during training and inference, reducing memory usage.Panoptic SegFormer can be trained with less than 12G memory, compared with more than 20G for MaskFormer under the stated setup.
- Mask Decoder: Multi-head attention improves performance over single-head attention by 0.4% PQ.Different attention heads focus on foreground, boundaries, or background regions.
B.4. Advantage of Query Decoupling Strategy
Query decoupling addresses the mismatch between things and stuff by assigning them separate query sets, while mask-wise merging combines their outputs during inference.
- Motivation: 52.5% and 9.2% are the mask-to-bounding-box area ratios for things and stuff, respectively, showing that bounding boxes represent stuff poorly.Stuff is described as amorphous and dispersed, unlike countable thing instances.
- Motivation: DETR’s shared query recipe can cause interference between things and stuff, with bbox AP dropping from 42.0 to 38.8 after panoptic training.The paper suggests that predicting stuff bounding boxes may require model readaptation and interfere with things.
- Query Analysis: Queries with the highest thing preference have stuff precision of 0.30, below the 0.60 average across queries.Queries with intermediate preference perform poorly on both stuff and things, supporting the claim that one shared query set is suboptimal.
- Query Decoupling: Query decoupling assigns separate query sets to things and stuff, adding a location decoder only for thing queries.Things use bipartite matching and an additional detection loss, while stuff uses class-fixed assignment; outputs retain the same format.
- Training Efficiency: The method’s training for 24 epochs can achieve comparable or higher results than training for 50 epochs with the stated ResNet-101 and Swin-L backbones.Figure D.1 reports this comparison for models trained with ResNet-101 and Swin-L separately.
E. Code and Data
The experiments use official implementations of DETR, MaskFormer, and Panoptic FCN, while Deformable DETR comes from MMDetection.
- Code and Data: Official implementations of DETR, MaskFormer, and Panoptic FCN were used for additional experiments.The paper states that these implementations reproduce the scores reported in their respective literature.
- Code and Data: Deformable DETR was obtained from MMDetection.