PaperlayerView exact revision on arXiv

Source-linked AI summary

Per-Pixel Classification is Not All You Need for Semantic Segmentation

Bowen Cheng, Alexander G. Schwing, Alexander Kirillov

arXiv:2107.06278v2cs.CV

TL;DR

Semantic segmentation commonly uses per-pixel classification, while instance-level segmentation uses mask classification, leaving the tasks with different model paradigms. MaskFormer unifies them by predicting masks with class labels using the same model, losses, and training procedure, and achieves strong semantic and panoptic results. The method is especially competitive with many semantic classes and reaches 52.7 PQ on COCO panoptic segmentation.

  • Problem

    Semantic and instance-level segmentation commonly use different paradigms, creating a mismatch that hinders development of segmentation as a whole.

  • Method

    MaskFormer converts per-pixel models into mask-classification models that predict binary masks and class labels with unified model, losses, and training procedure.

  • Results

    MaskFormer outperforms state-of-the-art per-pixel semantic segmentation models especially with many categories and achieves 52.7 PQ on COCO panoptic segmentation.

  • Takeaways & Limitations

    Mask classification can serve as a competitive unified paradigm for semantic and panoptic segmentation without changing the model architecture, losses, or training procedure.

  • Takeaways & Limitations

    Speed comparisons across papers are not recommended because measurements use different environments.

Abstract

from arXiv · show

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

1 Introduction

MaskFormer reframes semantic and instance-level segmentation under one mask-classification paradigm. It predicts binary masks with global class labels and reports strong semantic and panoptic results.

  • Per-pixel classification applies a classification loss independently at each output location, naturally partitioning images into semantic regions.
  • Mask classification predicts binary masks paired with single class labels, separating image partitioning from classification.
  • Mask classification is sufficiently general for semantic- and instance-level segmentation, motivating one unified model and training procedure.
  • MaskFormer converts existing per-pixel classification models into mask-classification models using a Transformer decoder that predicts class labels and mask embeddings.Mask embeddings generate binary masks through dot products with per-pixel embeddings.
  • MaskFormer performs comparably on Cityscapes but surpasses per-pixel classification models on semantic datasets with larger class vocabularies.The evaluated datasets range from 19 to 847 categories.
  • 52.7 PQ on COCO establishes a new panoptic segmentation state of the art, exceeding prior state of the art by 1.6 PQ.

2 Related Works

Prior work established per-pixel classification as dominant in deep semantic segmentation, while mask classification remained central to instance-level tasks. MaskFormer connects these traditions by adapting per-pixel models to mask classification.

  • Early semantic segmentation methods used per-pixel Bayesian classifiers, while earlier mask-based methods achieved strong results by classifying mask proposals.
  • Fully Convolutional Networks made per-pixel classification the dominant formulation for deep semantic segmentation.
  • MaskFormer can convert any per-pixel classification model to mask classification, enabling adoption of advances in per-pixel architectures.
  • Instance-level tasks require a dynamic number of predictions, whereas per-pixel classification assumes a static number of outputs.

3 From Per-Pixel to Mask Classification

MaskFormer reformulates segmentation as predicting a set of class-labeled binary masks rather than classifying every pixel independently. Its architecture and inference procedures produce unified semantic and instance-level outputs from the same mask-classification framework.

  • Formulations: Per-pixel classification predicts category probabilities for every image pixel, whereas mask classification predicts N binary masks paired with class-probability distributions.Mask classification can use an auxiliary “no object” label and allows multiple masks to share a class.
  • Training: MaskFormer trains with a matching-based loss combining one classification loss and a binary mask loss for each predicted segment.The binary mask loss uses the same focal-and-dice formulation as DETR, while matching can use class and mask predictions directly.
  • MaskFormer architecture: MaskFormer computes N probability-mask pairs using pixel-level, Transformer, and segmentation modules.The pixel-level module produces per-pixel embeddings, the Transformer produces per-segment embeddings, and the segmentation module generates class and mask predictions.
  • MaskFormer architecture: Each mask prediction is generated by a dot product between a mask embedding and per-pixel embeddings, followed by sigmoid activation.The model predicts possibly overlapping binary masks rather than enforcing mutual exclusivity.
  • Inference: General inference assigns each pixel to the highest-scoring nonempty probability-mask pair, then merges same-class segments for semantic segmentation.Semantic inference instead marginalizes over mask pairs with a matrix multiplication; the choice of inference strategy depends largely on the evaluation metric.

4 Experiments

MaskFormer is evaluated across semantic and panoptic segmentation datasets, with ablations testing its formulation, query count, and category coverage. Results show stronger gains as semantic class vocabularies grow, competitive panoptic performance, and evidence that mask classification drives the semantic improvements.

  • Main results: MaskFormer achieves 55.6 mIoU on ADE20K, outperforming the prior state of the art by 2.1 mIoU while using fewer parameters and faster inference.The comparison uses compatible CNN and Swin Transformer backbones.
  • Main results: 3.5 mIoU is MaskFormer’s improvement over PerPixelBaseline+ on ADE20K-Full with 847 classes, while performance is similar on Cityscapes with 19 categories.The improvement increases as the number of classes grows.
  • Main results: 2.9 PQSt is MaskFormer’s gain on Cityscapes despite no mIoU improvement, reflecting better recognition quality but lower per-pixel segmentation quality.This suggests pixel-level mask quality remains challenging when class recognition is relatively easy.
  • Main results: 2.2 PQ is MaskFormer’s advantage over DETR under identical DETR post-processing on COCO panoptic segmentation.MaskFormer directly predicts masks rather than bounding boxes and obtains larger improvements for stuff classes.
  • Ablation studies: MaskFormer-fixed improves 1.8 mIoU over the per-pixel baseline, indicating that changing from per-pixel to mask classification is the main source of the semantic gains.Bipartite matching further improves results and permits predicting fewer masks than the total number of categories.
  • Ablation studies: 100 queries perform best across the studied datasets, while even 20 queries outperform the per-pixel classification baseline.The authors also find similar average numbers of classes per image across datasets with different total vocabularies, supporting queries that capture multiple categories.

5 Discussion

The discussion presents MaskFormer as a controlled exploration of mask classification, emphasizing mask-based matching and a computationally efficient mask head. It positions the approach as a simple alternative whose design choices support effective unified segmentation.

  • 5 Discussion: MaskFormer isolates mask classification by using DETR as a simple baseline and making few architectural changes.The authors describe it as a “box-free” version of DETR.
  • 5 Discussion: The discussion concludes that mask classification can unify semantic and instance-level segmentation without changing the model, losses, or training procedure.This framing focuses the comparison on the segmentation paradigm rather than unrelated architecture, loss, or augmentation changes.
  • 5 Discussion: Mask-based matching outperforms box-based matching for MaskFormer on COCO panoptic validation.Box-based matching requires an additional box prediction head, while direct mask matching has a clear advantage.
  • 5 Discussion: MaskFormer’s mask head matches DETR’s prediction quality under the same matching strategy while reducing computation and memory costs.Shared high-resolution per-pixel embeddings avoid DETR’s independent per-query upsampling modules, which are N times more expensive.

6 Conclusion

The conclusion argues that mask classification offers a unified alternative to task-specific segmentation models. MaskFormer is especially competitive when datasets contain many categories, while remaining competitive for panoptic segmentation without task-specific training changes.

  • 6 Conclusion: MaskFormer addresses the paradigm discrepancy that leads to different models for semantic- and instance-level segmentation.The authors suggest this unification could encourage joint progress across segmentation tasks.
  • 6 Conclusion: MaskFormer can outperform state-of-the-art per-pixel classification models, especially when the number of categories is large.The conclusion frames mask classification as a competitive alternative to per-pixel classification for semantic segmentation.
  • 6 Conclusion: MaskFormer remains competitive for panoptic segmentation without changing the model architecture, losses, or training procedure.

A Datasets description

The evaluation studies MaskFormer across five semantic segmentation datasets and two panoptic segmentation datasets. The appendix provides additional dataset information for these experiments.

  • A Datasets description: MaskFormer is evaluated on five semantic segmentation datasets and two panoptic segmentation datasets.

A.1 Semantic segmentation datasets

The semantic segmentation evaluation spans datasets with 19 to 847 categories and varied image resolutions, dataset sizes, and inference procedures. These datasets cover urban street views, general scenes, and open-vocabulary annotations.

  • A.1 Semantic segmentation datasets: ADE20K uses 20k training images and 2k validation images, with 150 semantic categories selected for evaluation.Images are resized so the shortest side is no greater than 512 pixels.
  • A.1 Semantic segmentation datasets: COCO-Stuff-10K contains 9k training images, 1k testing images, and 171 semantic-level categories.Its images are a subset of COCO.
  • A.1 Semantic segmentation datasets: ADE20K-Full contains 25k training images and 2k validation images, with 847 categories retained from an open-vocabulary annotation.The retained categories appear in both the training and validation sets.
  • A.1 Semantic segmentation datasets: Cityscapes provides 2,975 training, 500 validation, and 1,525 testing images across 19 classes.The dataset consists of high-resolution urban egocentric street-view images.
  • A.1 Semantic segmentation datasets: Mapillary Vistas contains 18k training, 2k validation, and 5k testing images spanning 65 categories and varied resolutions.Training uses 1280 × 1280 crops, while inference resizes the longer image side to 2048.

A.2 Panoptic segmentation datasets

The paper evaluates semantic segmentation across datasets spanning 19 to 847 categories, using standard metrics and reporting both single- and multi-scale results. These datasets provide comparisons across vocabulary size, resolution, and segmentation quality.

  • Metrics and settings: ADE20K evaluation reports pixel accuracy, mIoU, and the final test score, with training on combined train and validation data.The final test score is the average of pixel accuracy and mIoU.
  • Metrics and settings: Cityscapes results include standard mIoU and complementary PQSt, SQSt, and RQSt metrics under multi-scale inference.The analysis treats all categories as stuff for the PQSt breakdown.

B Semantic segmentation results

MaskFormer performs strongly across semantic segmentation benchmarks, with the largest advantages appearing on datasets with larger vocabularies and high-resolution images. On Cityscapes, which has fewer categories, its performance is on par with leading per-pixel methods.

  • ADE20K: MaskFormer outperforms previous state-of-the-art methods on all three ADE20K test metrics.The metrics are pixel accuracy, mIoU, and the final test score.
  • ADE20K-Full: On ADE20K-Full with 847 categories, MaskFormer performs better than per-pixel classification and requires less training memory.The paper attributes the memory benefit to decoupling the number of masks from the number of classes.
  • Cityscapes: MaskFormer performs on par with the best per-pixel classification methods on Cityscapes, which has 19 categories.The paper characterizes recognition as less challenging there than on the other datasets.
  • Cityscapes: On Cityscapes, MaskFormer has better recognition quality but lower per-pixel segmentation quality than reflected by the complementary PQSt analysis.The comparison is specifically between RQSt and SQSt.
  • Mapillary Vistas: On high-resolution Mapillary Vistas images, MaskFormer outperforms multi-scale per-pixel classification models without multi-scale inference.The paper links this result to global context captured by the Transformer decoder.

C Panoptic segmentation results

MaskFormer extends its semantic segmentation performance to panoptic segmentation. It outperforms prior state-of-the-art on COCO and remains competitive on ADE20K panoptic segmentation.

  • COCO: MaskFormer outperforms previous state-of-the-art by 2 PQ on the COCO panoptic test-dev set.The model is trained only on COCO train2017 with an ImageNet-22K pre-trained checkpoint.

D Additional ablation studies

The ablations examine inference, decoder depth, self-attention, and query behavior across semantic and panoptic settings. They show that MaskFormer’s task behavior changes with supervision and that decoder depth primarily improves recognition quality.

  • Inference strategies: General semantic inference filters masks below confidence 0.3 before assigning labels, while semantic inference uses the default semantic strategy.The ablation compares these strategies across ADE20K, COCO-Stuff, and ADE20K-Full.
  • Decoder depth: Stacking more decoder layers mainly improves recognition quality.This conclusion is stated in the decoder-layer ablation caption.
  • Semantic versus panoptic queries: Semantic supervision makes one query combine all instances of a class, whereas instance-level supervision uses different queries for each instance.The visualization uses cars to contrast semantic and panoptic queries.
  • Decoder depth: A single Transformer decoder layer already achieves reasonable semantic segmentation performance and can outperform the six-layer per-pixel baseline.The ablation studies semantic and panoptic segmentation on ADE20K.
  • Self-attention: Removing self-attention leaves semantic mIoU similar but slightly worsens PQSt, while panoptic performance worsens across all metrics.The effect is therefore stronger for panoptic segmentation than for semantic mIoU.
  • Task adaptation: The visualization and ADE20K examples support MaskFormer’s capacity to adapt to different tasks under different ground-truth annotations.The ADE20K visualization uses a Swin-L backbone and reports 55.6 mIoU.
Loading 2107.06278v2…