PaperlayerView exact revision on arXiv

Source-linked AI summary

CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation

Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, Liang-Chieh Chen

arXiv:2206.08948v1cs.CV

TL;DR

Panoptic segmentation transformers face sparse query updates and limited pixel-query communication despite needing dense pixel predictions. CMT-DeepLab treats object queries as cluster centers and iteratively updates assignments, centers, and pixel features. It improves on MaX-DeepLab by 4.4% PQ and achieves 55.7% PQ on COCO test-dev.

  • Problem

    Transformer segmentation can sparsely update object queries and give pixels only one chance to receive query information, despite requiring dense predictions.

    Source: §1 · p. 2

  • Method

    CMT-DeepLab treats object queries as cluster centers and uses feature-affinity assignments to update centers and pixel features, with dynamic position encoding for location-sensitive clustering.

    Source: §1 · p. 2

  • Results

    4.4% PQ over MaX-DeepLab [83], reaching 55.7% PQ on the COCO panoptic test-dev set [58].

    Source: §1 · p. 2

  • Takeaways & Limitations

    CMT-DeepLab achieves new state-of-the-art performance on COCO and sheds light on the working mechanism behind mask transformers for segmentation.

    Source: §5 · p. 8

    The clustering transformer improves segmentation quality more than recognition ability; scaling strategies boost recognition but cost parameters, computation, or training time.

    Source: §9 · p. 15

Abstract

from arXiv · show

We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering. It rethinks the existing transformer architectures used in segmentation and detection; CMT-DeepLab considers the object queries as cluster centers, which fill the role of grouping the pixels when applied to segmentation. The clustering is computed with an alternating procedure, by first assigning pixels to the clusters by their feature affinity, and then updating the cluster centers and pixel features. Together, these operations comprise the Clustering Mask Transformer (CMT) layer, which produces cross-attention that is denser and more consistent with the final segmentation task. CMT-DeepLab improves the performance over prior art significantly by 4.4% PQ, achieving a new state-of-the-art of 55.7% PQ on the COCO test-dev set.

1. Introduction

CMT-DeepLab reframes end-to-end panoptic segmentation as clustering, treating object queries as cluster centers and updating pixel features and centers through assignments. This addresses sparse query updates and infrequent pixel-query communication, and the method improves on MaX-DeepLab.

1. Introduction

  • Panoptic segmentation unifies semantic and instance segmentation, while many frameworks decompose it into proxy tasks such as box detection and semantic segmentation.Source: §1 · p. 1
  • End-to-end mask transformers predict class and mask pairs, but segmentation requires object queries and pixel features to interact when updating queries.Source: §1 · p. 1
  • Applying softmax over image positions sparsely updates queries, while final segmentation lets each pixel-query pair communicate only once.The resulting sparse attention is problematic for dense segmentation predictions.Source: Figure 1 · p. 1, §1 · p. 2
  • CMT treats object queries as cluster centers, assigns pixels by feature affinity, and uses assignments to update centers and pixel features across layers.The refined assignments produce the final panoptic mask and denser cross-attention maps than MaX-DeepLab [83].Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2
  • Dynamic position encoding uses a few-point reference mask per center and coordinate convolution to add location information to pixel features and centers.Source: §1 · p. 2
  • 4.4% PQ over MaX-DeepLab [83], reaching 55.7% PQ on the COCO panoptic test-dev set [58].Source: §1 · p. 2

2. Related Works

Prior panoptic segmentation work includes proxy-based systems and end-to-end mask-prediction systems. CMT-DeepLab builds on the latter by recasting pixel assignment as clustering.

Transformers

  • Transformer-based vision methods have advanced tasks including image classification, object detection, semantic segmentation, and panoptic segmentation.Source: §2 · p. 2

Proxy-based Panoptic Segmentation

  • Proxy-based panoptic methods use detection or semantic segmentation and group or fuse outputs into panoptic segments.Examples include box-based segmentation and center regression, Watershed, Hough-voting, and pixel-affinity tasks.Source: §2 · p. 2

End-to-end Panoptic Segmentation

  • MaX-DeepLab [83] directly predicts class-labeled masks with Hungarian matching; CMT-DeepLab instead approaches pixel assignment from a clustering perspective.Segmenter [78] and MaskFormer [20] also use end-to-end mask classification, extending to semantic segmentation.Source: §2 · p. 3

3. Method

CMT-DeepLab treats object queries as cluster centers and redesigns the transformer decoder around pixel-cluster assignment and updates to centers and pixel features. It also adds input-conditioned location information and instantiates the design on MaX-S variants.

3.1. Transformers for Panoptic Segmentation

  • End-to-end panoptic segmentation predicts a fixed number of masks and semantic classes from object queries that aggregate image features through a transformer decoder.Updated queries act as 1 × 1 dynamic convolution weights to produce binary masks.Source: §3.1 · p. 3

3.2. Current Issues and New Clustering Perspective

  • Transformer segmentation applies softmax over image locations for query updates but over queries for final masks, potentially yielding sparse query updates and infrequent pixel communication.The authors address these issues through clustering-based center and pixel-feature updates.Source: §3.2 · pp. 3–4

3.3. Clustering Mask Transformers

  • CMT treats object queries as cluster centers, assigns pixels by feature affinity, and updates cluster centers and pixel features within the decoder.The clustering assignment also supports residual connections between successive clustering results and the final segmentation output.Source: §3.2 · p. 4, §3.3 · p. 4
  • CMT adds an input-conditioned reference mask of points per cluster center and coordinate convolutions to incorporate location information during clustering.The reference mask provides a rough whole-object shape prior, while coordinate convolution is presented as a simpler way to incorporate location than the Deformable DETR reference point.Source: §3.3 · p. 5

3.4. Network Instantiation

  • CMT-DeepLab uses MaX-S-derived backbones with six dual-path axial-transformer blocks and trains with PQ-style and auxiliary losses, including pixel-wise instance discrimination.The authors also describe an RFN that stacks the model twice and averages stride-4 features; they report this fusion performed around 0.2% PQ better than their alternative.Source: §3.4 · pp. 5–6

4. Experimental Results

CMT-DeepLab improves COCO panoptic segmentation results over prior methods, while ablations examine clustering, location-sensitive clustering, decoder design, and scaling. Its clustering visualizations show assignments becoming more focused on target instances across transformer stages.

4.2. Ablation Studies

  • 1.3% PQ overall comes from clustering-style transformer redesign, including a 0.9% PQ gain from cluster-center updates and another 0.4% from pixel-wise contrastive loss.The ablation uses a MaX-DeepLab-S-GeLU baseline with Axial-R50, ImageNet-1K pretraining, 641×641 crops, and 100k iterations unless specified otherwise.Source: §4.2 · pp. 6–7
  • 0.7% PQ comes from location-sensitive clustering: reference-mask prediction adds 0.4% PQ and coordinate convolution adds another 0.3% PQ.Cluster centers need location information because they predict reference masks without pixel appearance features.Source: §4.2 · p. 7
  • 0.9% PQ improvement (47.1% vs. 46.2%) comes from the stronger decoder with six self-attention and six cross-attention modules.The decoder design aligns with DETR.Source: §4.2 · p. 8
  • 2.2% PQ over the MaX-DeepLab-S-GeLU baseline comes from combining the proposed components; the stronger decoder adds 29.1M parameters.Clustering update and location-sensitive clustering improve PQ by 1.3% and 0.7%, respectively, with negligible extra parameters.Source: Table 2 · p. 7, §4.2 · p. 8
  • Longer training adds 0.5% PQ, ImageNet-22K pretraining adds 0.9% PQ, mask-wise merging adds 0.5% PQ, and increasing input resolution from 641 to 1281 boosts performance by more than 2% PQ.A deeper backbone and RFN improve performance by 1.6% and 1.0%, respectively; visualizations show assignments refine from near-random to instance-focused across stages.Source: §4.2 · p. 8, Figure 5 · p. 8

5. Conclusion

CMT-DeepLab rethinks object queries as cluster centers for panoptic segmentation. Its cluster-center update enriches cross-attention and facilitates segmentation prediction, and the method achieves state-of-the-art performance on COCO.

5. Conclusion

  • CMT-DeepLab rethinks object queries in mask transformers for panoptic segmentation from a clustering perspective.The framework incorporates cluster-center updates in cross-attention, enriching learned attention maps and facilitating segmentation prediction.Source: §5 · p. 8
  • The proposed cluster-center update enriches learned cross-attention maps and further facilitates segmentation prediction.The framework considers object queries as cluster centers.Source: §5 · p. 8
  • CMT-DeepLab achieves new state-of-the-art performance on the COCO dataset and sheds light on mask transformers' working mechanism for segmentation.Source: §5 · p. 8

6. More Technical Details

The technical details describe an architectural comparison of MaX-DeepLab-S and CMT-DeepLab built on Axial-R50/104, including backbone scaling and Recursive Feature Network construction.

6. More Technical Details

  • CMT-DeepLab simplifies the MaX-DeepLab-S backbone by removing its transformer modules and stacking more blocks in the decoder.The comparison is built on Axial-R50/104.Source: §6 · p. 13
  • Axial-R104 scales up Axial-R50 with four times more layers in stage 4.Source: §6 · p. 13
  • The Recursive Feature Network stacks two models, averages stride-4 decoder and encoder features for fusion, and shares cluster centers updated sequentially between networks.The first network also receives supervision using Hungarian matching results based on the final output.Source: §6 · p. 13

7. More Results and In-depth Analysis

CMT-DeepLab improves over prior methods on COCO test-dev and Cityscapes, with reported gains in segmentation quality and accuracy-cost trade-offs. The analysis also identifies weaker recognition gains and the costs of scaling.

7. More Results and In-depth Analysis

  • 55.7% PQ on COCO test-dev is 4.4% above MaX-DeepLab-L, 2.4% above MaskFormer, and 0.5% above K-Net.The reported best model uses 200K iterations and RFN.Source: §7 · p. 13
  • On test-dev, CMT-DeepLab’s best model (200K iterations with RFN) achieves 55.7% PQ, exceeding MaX-DeepLab-L [83] by 4.4%, MaskFormer [20] by 2.4%, and K-Net [94] by 0.5%.Removing pixel-feature updates from the clustering transformer degrades PQ by 0.4%.Source: §7 · p. 13
  • CMT-DeepLab-L-RFN exceeds MaX-DeepLab-L by 4.2% PQ using 60% of its parameters and 33.6% of its FLOPs.CMT-DeepLab-S also outperforms MaskFormer-SwinB by 1.2% PQ with comparable model size and inference cost.Source: §7 · p. 13
  • On Cityscapes, CMT-DeepLab-S exceeds MaX-DeepLab-S by 2.9% PQ and 1.6% mIoU.The comparison uses the same pretraining weights, training hyperparameters, and post-processing scheme.Source: §7 · p. 13
  • With a ResNet-50 backbone on COCO val, CMT-DeepLab significantly outperforms MaX-DeepLab, MaskFormer, and K-Net.The paper presents this backbone-matched comparison to address differences in backbones across methods.Source: §7 · p. 13
  • Scaling model size, input size, or training iterations improves RQ, but costs parameters, computation, or training time.The authors identify efficient improvement of recognition ability as future work.Source: §7 · p. 13

8. Visual Comparison

The visual comparisons show progressive clustering refinement and denser attention maps for CMT-DeepLab, alongside an architecture comparison and benchmark tables.

8. Visual Comparison

  • Figure 6 compares MaX-DeepLab-S and CMT-DeepLab architectures, marking the pretrained backbone in blue; Tables 5–7 cover accuracy-cost, ResNet-50, and Cityscapes comparisons.The attention visualizations average across heads and normalize values to a 0–255 heatmap without smoothing.Source: §8 · p. 13, Figure 6 · p. 14, Table 5 · p. 14
  • 48.5 PQ for CMT-DeepLab exceeds 47.1 for K-Net and 46.5 for MaskFormer, while MaX-DeepLab scores 46.0.Table 4 is a COCO val and test-dev comparison.Source: §8 · p. 14, Table 4 · p. 14
  • Across the visualized stages, initially near-random clusterings gradually focus on objects, while some centers begin with semantic-class clustering.Examples include person, skis, zebras, elephant, skateboard, and grass.Source: §8 · p. 14
  • CMT-DeepLab shows denser attention maps than MaX-DeepLab in the visual comparisons.The figures compare attention maps from CMT-DeepLab and MaX-DeepLab.Source: §8 · p. 15

9. Limitations

The clustering transformer improves segmentation quality more than recognition ability, and scaling strategies that raise recognition performance incur resource costs.

9. Limitations

  • The clustering mask transformer improves SQ significantly but does not produce a comparable boost in RQ.The authors report that efficient improvement of recognition ability remains an open problem.Source: §9 · p. 15
  • Increasing model size, input size, or training iterations can improve RQ, but costs parameters, computation, or training time.Source: §9 · p. 15
  • The authors leave efficient and effective improvement of recognition ability for future exploration.Source: §9 · p. 15

10. Potential Negative Impacts

The paper presents CMT-DeepLab as a clustering-inspired panoptic segmentation framework with state-of-the-art performance, while noting that AI’s effects can vary by user.

10. Potential Negative Impacts

  • The authors state that advances to this computer-vision task may have beneficial or harmful influences depending on users.Source: §10 · p. 15
  • Visualizations show cluster assignments refining toward target instances, with some cluster centers beginning from semantic-class groupings.The examples include people, skis, zebras, elephants, grass, and a skateboard.Source: Figure 7 · p. 16, Figure 8 · p. 16, Figure 9 · p. 17
  • The figure captions describe denser CMT-DeepLab attention maps than MaX-DeepLab and associate them with superior dense-prediction performance.Source: Figure 11 · p. 18, Figure 12 · p. 18, Figure 13 · p. 18

Related papers

Loading 2206.08948v1…