PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic Segmentation

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár

arXiv:1801.00868v3cs.CV

TL;DR

Existing vision tasks and metrics treat stuff and things separately, leaving unified, coherent scene segmentation insufficiently defined and studied. The paper proposes panoptic segmentation with a common output format and panoptic quality metric, then evaluates human and machine performance across three datasets. The task was also included in COCO and Mapillary Vistas recognition challenges at ECCV 2018.

  • Problem

    Semantic and instance segmentation use separate task formulations, methods, and metrics, while coherent unified recognition of stuff and things remains insufficiently established.

  • Method

    The paper defines panoptic segmentation with per-pixel semantic labels and instance ids, introduces the uniform panoptic quality metric, and studies human results plus a heuristic machine baseline.

  • Results

    The study analyzes human and machine panoptic performance on Cityscapes, ADE20k, and Mapillary Vistas, while PS also appeared in COCO and Mapillary Vistas challenges at ECCV 2018.

  • Takeaways & Limitations

    Panoptic segmentation offers a unified view of image segmentation intended to support broader study and adoption of coherent scene understanding.

  • Takeaways & Limitations

    The paper focuses on visible regions rather than amodal segmentation, and human–machine comparisons use different test images and class averages.

Abstract

from arXiv · show

We propose and study a task we name panoptic segmentation (PS). Panoptic segmentation unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance). The proposed task requires generating a coherent scene segmentation that is rich and complete, an important step toward real-world vision systems. While early work in computer vision addressed related image/scene parsing tasks, these are not currently popular, possibly due to lack of appropriate metrics or associated recognition challenges. To address this, we propose a novel panoptic quality (PQ) metric that captures performance for all classes (stuff and things) in an interpretable and unified manner. Using the proposed metric, we perform a rigorous study of both human and machine performance for PS on three existing datasets, revealing interesting insights about the task. The aim of our work is to revive the interest of the community in a more unified view of image segmentation.

1. Introduction

Panoptic segmentation unifies semantic and instance segmentation into a coherent scene representation covering stuff and things. The paper introduces a uniform metric and studies human and machine performance to support this joint task.

  • Stuff and thing recognition have traditionally been separated into semantic and instance segmentation, with differing datasets, details, metrics, and methods.
  • Panoptic segmentation assigns every pixel a semantic label and instance id, creating a unified representation for both amorphous regions and countable objects.
  • Panoptic quality (PQ) provides a simple, informative metric that evaluates stuff and things uniformly instead of using disjoint metrics.
  • The task introduces new challenges by requiring individual object differentiation, non-overlapping segments, and coherent resolution of stuff–thing inconsistencies.
  • The paper studies human and machine performance on Cityscapes, ADE20k, and Mapillary Vistas, using human consistency analysis and a heuristic machine baseline.
  • Panoptic segmentation was featured alongside semantic and instance segmentation in COCO and Mapillary Vistas recognition challenges at ECCV 2018.

2. Related Work

The paper revisits earlier joint scene-understanding work while distinguishing panoptic segmentation from multitask learning and prior task formulations. It emphasizes a unified task definition, simple output format, and common metric for stuff and things.

  • Existing datasets such as Cityscapes, ADE20k, and Mapillary Vistas contain information sufficient to bootstrap panoptic segmentation, despite separate semantic and instance tracks.
  • Panoptic segmentation is a single unified view of image segmentation, not a multitask problem combining independent tasks in one framework.
  • Earlier scene-parsing and holistic-understanding work jointly modeled segmentation, detection, and recognition but lacked an agreed task definition.
  • Compared with earlier joint segmentation work, PS addresses both stuff and things with a simple format and a uniform metric.
  • The paper focuses on visible regions rather than full amodal object extents, leaving amodal panoptic segmentation for future work.

3. Panoptic Segmentation Format

Panoptic segmentation represents each pixel with a semantic class and instance id, while distinguishing stuff from thing labels. This format generalizes semantic segmentation, constrains overlaps unlike instance segmentation, and aligns human and machine annotations.

  • Each pixel i maps to a pair (l_i, z_i), where l_i is its semantic class and z_i groups pixels into instances.
  • For stuff classes, instance ids are ignored; for thing classes, pixels sharing a class and instance id form one object.
  • Panoptic segmentation strictly generalizes semantic segmentation by adding instance distinctions when thing classes can contain multiple objects.
  • Unlike instance segmentation, panoptic segmentation assigns only one semantic label and instance id per pixel, so overlaps are disallowed by construction.
  • PS does not require segment confidence scores, giving humans and machines the same annotation format and simplifying human-consistency evaluation.

4. Panoptic Segmentation Metric

Panoptic quality (PQ) provides a unified, simple metric for evaluating stuff and thing classes in panoptic segmentation. It combines segment matching with class-wise assessment of recognition and segmentation quality while handling void and group regions explicitly.

  • Motivation: Existing semantic- and instance-segmentation metrics are specialized for different class types, motivating a unified metric for the joint task.Using disjoint metrics complicates algorithm development, comparisons, and communication.
  • Segment Matching: PQ matches predicted and ground-truth segments only when IoU is strictly greater than 0.5, yielding a unique correspondence under non-overlapping panoptic segmentation.The uniqueness theorem makes matching simple, efficient, and interpretable, without requiring a complex matching algorithm.
  • PQ Computation: For each class, PQ partitions segments into true positives, false positives, and false negatives before averaging performance across classes.Class-wise averaging makes PQ insensitive to class imbalance, while unmatched predictions and ground-truth segments are penalized.
  • PQ Computation: PQ decomposes into segmentation quality (SQ), the average IoU of matched segments, and recognition quality (RQ), an F1-based measure of matching.The decomposition provides an interpretable analysis of segmentation and recognition performance, although SQ and RQ are not independent.
  • Metric Properties: PQ evaluates all classes uniformly with a simple, interpretable formula and excludes void and eligible group-region predictions from inappropriate matching or false-positive penalties.Void pixels are removed during IoU computation, while unmatched predictions substantially overlapping same-class group regions are not counted as false positives.

5. Panoptic Segmentation Datasets

The study uses three public datasets with dense semantic and instance annotations: Cityscapes, ADE20k, and Mapillary Vistas. They cover urban driving, broad scene imagery, and varied street-view settings with different class vocabularies and annotation scopes.

  • Dataset Selection: The study uses Cityscapes, ADE20k, and Mapillary Vistas because they provide both dense semantic and instance segmentation annotations.The paper identifies these as the three public datasets meeting this annotation requirement.
  • Cityscapes: Cityscapes contains 5000 urban driving images with 19 classes, 97% pixel coverage, and instance-level segmentations for 8 classes.Its splits are 2975 training, 500 validation, and 1525 test images.
  • ADE20k: ADE20k contains over 25k densely annotated images and uses a selected vocabulary of 100 thing and 50 stuff classes covering 89% of pixels.The study uses the closed vocabulary selected for the 2017 Places Challenge.
  • Mapillary Vistas: Mapillary Vistas contains 25k street-view images with 98% pixel coverage across 28 stuff and 37 thing classes.The research edition includes 18k training, 2k validation, and 5k test images spanning a wide range of resolutions.

6. Human Consistency Study

The study uses human annotations to examine panoptic quality across datasets, error types, object scales, and metric settings. Human consistency is broadly balanced between stuff and things, but small objects and certain segmentation or classification ambiguities remain challenging.

  • Error analysis: Human annotators make both segmentation and classification errors, including splitting one car into two instances and misclassifying difficult scenes.Some Cityscapes examples are genuinely ambiguous, while some ADE20k errors are simple misclassifications.
  • Experimental setup: Human consistency is measured from doubly annotated images, treating one annotation as ground truth and the other as prediction.The study includes Cityscapes, ADE20k, and Vistas annotations, with different annotation protocols across datasets.
  • Object scale: On large objects, both SQ and RQ exceed 80 across datasets, whereas small-object RQ drops sharply while small-object SQ remains reasonably strong.Objects are partitioned into small, medium, and large groups using the smallest 25%, middle 50%, and largest 25% of objects.
  • Stuff vs. things: Human consistency is relatively similar for stuff and things, although thing classes are somewhat harder and PQ distributes the two class types fairly evenly.The study interprets this distribution as evidence that PQ balances stuff and thing performance without either dominating the error.
  • Metric analysis: Less than 16% of matched segments have IoU below 0.5, and changing the threshold from 0.25 to 0.5 has relatively little effect on PQ compared with raising it to 0.75.These findings support the default IoU threshold of 0.5 as a simple and reasonable choice.
  • Metric analysis: Lowering α increases RQ without affecting SQ, allowing the metric to change the relative influence of recognition and segmentation errors on PQ.The study uses α=0.5 by default and describes it as a balance between SQ and RQ.

7. Machine Performance Baselines

The baselines combine outputs from strong semantic and instance segmentation systems to evaluate panoptic segmentation, with PQ exposing differences between pixel- and instance-level performance. Machines remain substantially behind humans, especially in recognition quality, while merging predictions slightly degrades stuff performance.

  • Instance segmentation: Removing overlaps harms AP, but methods with stronger AP retain better APNO and PQ on instance segmentation.APNO measures AP after converting predictions into non-overlapping segments.
  • Algorithms and data: Heuristic panoptic baselines combine outputs from top-performing instance and semantic segmentation systems across Cityscapes, ADE20k, and Vistas.The study uses challenge-organizer outputs, including PSPNet and Mask R-CNN for Cityscapes and winning entries for ADE20k.
  • Semantic segmentation: Semantic methods with higher mean IoU also achieve higher PQ, while small hallucinated patches can sharply reduce PQ despite modest pixel-level IoU effects.PQ penalizes instance errors more strongly than IoU when many spurious class patches are produced.
  • Human vs. machine panoptic segmentation: Machines trail humans only slightly in SQ but have dramatically lower RQ, especially on ADE20k and Vistas.The reported comparison uses different test images and class averages, so it provides useful but imperfect evidence.
  • Human vs. machine panoptic segmentation: Figure 9 visualizes Cityscapes and ADE20k panoptic predictions produced by merging state-of-the-art instance and semantic segmentation outputs.Matched segments share colors, while unmatched and unlabeled regions use distinct visual markings.
  • Panoptic segmentation: Merging independent predictions leaves PQTh constant but slightly lowers PQSt because overlaps are resolved in favor of things.The panoptic output is formed from non-overlapping thing and stuff predictions using a merging heuristic.

8. Future of Panoptic Segmentation

The authors position panoptic segmentation as a platform for new research directions, including integrated models and higher-level reasoning for non-overlapping scene interpretations. Its challenge-track adoption indicates early community uptake.

  • Research direction: The authors aim for panoptic segmentation to drive research in novel directions and invite the community to explore the task.They frame new datasets and tasks as mechanisms that can challenge communities and help measure progress.
  • Integrated models: Future systems may use deeply integrated end-to-end models that jointly address the dual stuff-and-thing nature of panoptic segmentation.Non-overlapping instance-prediction methods are suggested as possible foundations for such systems.
  • Higher-level reasoning: Because panoptic outputs cannot contain overlapping segments, higher-level reasoning such as learnable NMS may be beneficial.The proposed direction extends learnable non-maximum suppression to panoptic segmentation.
  • Community uptake: Panoptic segmentation has already appeared as a challenge track in COCO and Mapillary Vistas recognition challenges.The authors cite this participation as evidence that the task is beginning to gain community traction.
Loading 1801.00868v3…