PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Networks for Panoptic Segmentation

Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, Jiaya Jia

arXiv:2012.00720v2cs.CV

TL;DR

Panoptic segmentation requires unified handling of things and stuff despite their conflicting representation needs. Panoptic FCN generates a specific kernel for each instance or stuff category and convolves high-resolution features directly, achieving strong performance across COCO, Cityscapes, and Mapillary Vistas.

  • Problem

    Panoptic segmentation must represent instance-aware things and semantically consistent stuff within one workflow, but prior approaches commonly separate them into branches or require extra predictions and postprocessing.

  • Method

    Panoptic FCN uses a kernel generator and feature encoder to represent each object instance or stuff category with a generated kernel and produce predictions by direct convolution.

  • Results

    Panoptic FCN outperforms previous box-based and box-free methods with single-scale input, attaining 44.3% PQ on COCO val, 47.5% PQ on COCO test-dev, 61.4% PQ on Cityscapes, and 36.9% PQ on Mapillary Vistas.

  • Takeaways & Limitations

    A unified generate-kernel-then-segment workflow satisfies instance awareness for things and semantic consistency for stuff without extra boxes for localization or instance separation.

Abstract

from arXiv · show

In this paper, we present a conceptually simple, strong, and efficient framework for panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline. In particular, Panoptic FCN encodes each object instance or stuff category into a specific kernel weight with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent properties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms previous box-based and -free models with high efficiency on COCO, Cityscapes, and Mapillary Vistas datasets with single scale input. Our code is made publicly available at https://github.com/Jia-Research-Lab/PanopticFCN.

1. Introduction

Panoptic FCN addresses the conflicting representation needs of things and stuff with a unified fully convolutional pipeline. It generates instance- or category-specific kernels and directly convolves high-resolution features, achieving strong results across three datasets.

  • Motivation: Panoptic segmentation must assign every pixel a semantic label and unique identity while unifying countable things and uncountable stuff.Things require instance-aware, object-varying features, whereas stuff benefits from semantically consistent representations.
  • Motivation: Prior methods commonly separate things and stuff into tailored branches, using boxes or auxiliary predictions for things and pixel-wise FCN predictions for stuff.These designs include box-based and box-free instance streams, with box-free methods still requiring extra predictions and postprocessing.
  • Panoptic FCN: Panoptic FCN uniformly represents things and stuff with generated kernels in a fully convolutional pipeline.A kernel generator produces instance-specific or category-specific weights, while a feature encoder supplies shared high-resolution features.
  • Panoptic FCN: Unlike dense prediction followed by NMS, Panoptic FCN generates instance-aware kernels directly for each instance.For stuff, it aggregates global context into semantically consistent kernels and predicts existing semantic classes in a whole-instance manner.
  • Results: 44.3% PQ on COCO val and 47.5% PQ on COCO test-dev accompany leading results of 61.4% PQ on Cityscapes and 36.9% PQ on Mapillary Vistas.These results use single-scale input and are reported without bells-and-whistles.

2. Related Work

Related work handles panoptic segmentation through separate or partially unified representations for things and stuff. Panoptic FCN instead uses a box-free kernel-based representation to produce object results directly.

  • Panoptic segmentation: Traditional panoptic systems separately segment things and stuff, then combine predictions from different models, creating computational overhead.Single-model approaches such as Panoptic FPN, AUNet, and UPSNet still use separate branches.
  • Panoptic segmentation: Foreground things are commonly represented with boxes or center offsets, while background stuff is predicted through a parallel FCN branch.Uniform pixel-level affinity approaches do not fully resolve the inherent ambiguity between instances.
  • Instance segmentation: Box-free instance segmentation avoids object boxes, but prior methods still require instance aggregation or object-level duplicate removal.Panoptic FCN generates one kernel per object and convolves detail-rich features directly without object-level duplicate removal.
  • Semantic segmentation: Semantic segmentation assigns each pixel a category without distinguishing diverse object identities.Related methods improve semantic consistency by capturing wider contextual cues or modeling long-range pixel relationships.

3. Panoptic FCN

Panoptic FCN unifies things and stuff through generated kernels, shared high-resolution features, and direct convolutional prediction. Its kernel generator localizes and classifies candidates, kernel fusion merges identities across stages, and feature encoding preserves spatial detail.

  • Framework: Panoptic FCN uses a kernel generator, kernel fusion, and feature encoder in a unified fully convolutional pipeline.The generator creates kernels, fusion merges matching identities across stages, and the encoder produces high-resolution features.
  • Kernel Generator: The position head represents things by object centers and stuff by semantic regions, treating same-category background as one instance.Convolutional heads predict position maps and category channels for things and stuff separately.
  • Kernel Generator: The kernel head concatenates relative coordinates with stage features and selects position-specific weights to represent predicted things and stuff.Each selected kernel is a C_e×1×1 weight associated with a predicted category and location.
  • Kernel Fusion: Kernel fusion averages weights assigned the same identity, preserving separate object embeddings for things and category-consistent embeddings for stuff.Object identity uses cosine similarity above a threshold, whereas stuff kernels sharing a category receive one identity.
  • Feature Encoder: The feature encoder generates a high-resolution encoded feature, and M+N kernels produce M+N predictions at W/4×H/4 resolution.The encoded feature incorporates positional cues before direct convolution generates whole-image predictions.
  • Training and Inference: Training uses object centers and stuff-region points to generate kernels, while inference aggregates candidates and follows a generate-kernel-then-segment workflow.Multiple high-scoring points may represent each object, but stuff uses one factor k=1 treatment across points of the same category.

4. Experiments

Experiments evaluate Panoptic FCN’s architecture, component choices, and efficiency across COCO, Cityscapes, and Mapillary Vistas. The method achieves strong panoptic quality with single-scale inputs and favorable speed-accuracy trade-offs.

  • Experimental Setting: Panoptic FCN uses ResNet-FPN, with P3–P7 features for kernel generation and P2–P5 features for high-resolution feature generation.The kernel generator is shared across FPN stages and uses GroupNorm, ReLU, and final projection convolutions.
  • Component-wise Analysis: Increasing kernel-generator depth improves performance, while deformable convolutions in the position head add 1.4% PQ, especially for stuff regions.The best kernel-generator setting uses 3 stacked Conv3 × 3 layers with 256 channels.
  • Component-wise Analysis: Position cues improve performance by up to 1.4% PQ when attached to both the kernel head and feature encoder.Attaching coordinates to either component alone yields improvements of up to 0.3% PQ.
  • Component-wise Analysis: Weighted dice loss improves performance by 1.1% PQ, and extending training from 1× to 2× and 3× adds 1.9% and 0.4% PQ, respectively.The weighted loss samples 7 top-scoring kernels per instance and optimizes them together.

5. Conclusion

Panoptic FCN represents and predicts things and stuff uniformly in a fully convolutional manner. Kernel generation, fusion, and high-resolution features produce predictions directly for instances and semantic categories.

  • 5. Conclusion: Panoptic FCN represents things and stuff uniformly through a fully convolutional pipeline.The framework is designed as a unified alternative to separate processing branches.
  • 5. Conclusion: Kernel generator and kernel fusion generate unique kernel weights for each object instance or semantic category.These components provide the representation used for both foreground instances and background stuff.
  • 5. Conclusion: Predictions are produced by convolving high-resolution features directly.The feature encoder supplies the high-resolution feature used by the convolutional prediction process.

A. Experimental Details

This section provides technical details for the training and inference processes of Panoptic FCN.

  • A. Experimental Details: The section presents technical details of the Panoptic FCN training process.It focuses on how the proposed network is optimized during training.
  • A. Experimental Details: The section presents technical details of the Panoptic FCN inference process.It also describes how predictions are produced during inference.
  • A. Experimental Details: Training and inference are both covered as parts of the proposed Panoptic FCN workflow.The section addresses both stages rather than introducing a separate method.

A.1. Training Details

Training uses position-based kernel selection for things and stuff, with object-center assignment and multiple top-scoring kernels supporting instance predictions.

  • A.1. Training Details: Mass centers generate object coordinates for ground-truth assignment in the position head.Mass centers are reported to be more robust than box centers, especially with weighted dice loss.
  • A.1. Training Details: Weighted dice loss selects k top-scoring positions within each object region to represent the same object with multiple kernels.The selected kernels generate k predictions optimized against the same segmentation target.
  • A.1. Training Details: Stuff regions merge positions sharing a category at each stage using AvgCluster to produce one specific kernel.For stuff regions, the factor k is treated as 1.
  • A.1. Training Details: Table 15 compares weighted dice loss and center type settings on the COCO validation set.The table defines weighted and center type as the loss and ground-truth-generation settings, respectively.

A.2. Inference Details.

Inference settings vary by dataset, changing the number of retained object kernels and the thresholds used to convert and compare predictions.

  • A.2. Inference Details.: COCO inference retains 100 top-scoring object kernels and uses a 0.4 threshold for soft-mask binarization.These settings are used for object prediction and converting soft masks to binary results.
  • A.2. Inference Details.: Cityscapes and Mapillary Vistas inference retain 200 top-scoring object kernels.Both traffic-related datasets use the same retained-kernel count.
  • A.2. Inference Details.: Cityscapes and Mapillary Vistas use a cosine threshold of 0.95 and a mask threshold of 0.5.These thresholds are specified for inference on both datasets.

B. Qualitative Results

Panoptic FCN produces qualitative panoptic results across COCO, Cityscapes, and Mapillary Vistas, preserving detailed, coherent predictions across varied scenes and object types.

  • The qualitative evaluation covers common-context and traffic-related scenarios across COCO, Cityscapes, and Mapillary Vistas.
  • COCO: On COCO, Panoptic FCN preserves foreground and background details while handling objects at various scales.The results use pixel-by-pixel processing and a unified representation.
  • Cityscapes: On Cityscapes, Panoptic FCN depicts cars and pedestrians and performs satisfactorily on slender objects such as street lamps and traffic lights.
  • Mapillary Vistas: On Mapillary Vistas, Panoptic FCN performs especially well on vehicles and traffic signs in larger-scale traffic scenes.The qualitative results also exhibit instance-awareness and semantic-consistency.
Loading 2012.00720v2…