Seamless Scene Segmentation

Lorenzo Porzi; Samuel Rota Bulò; Aleksander Colovic; Peter Kontschieder

Source-linked AI summary

Seamless Scene Segmentation

Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, Peter Kontschieder

arXiv:1905.01220v1May 3, 2019cs.CV

TL;DR

The paper tackles inconsistent and computationally costly combinations of separately trained semantic and instance segmentation models. It introduces an end-to-end CNN with a joint panoptic architecture and a segmentation head integrating FPN features with lightweight DeepLab-like context. The resulting model achieves state-of-the-art performance on three street-level datasets while using fewer computational resources than combined independent models.

Problem
Separately trained semantic and instance segmentation models can produce inconsistent results and require significant computational overhead.
Method
A single-backbone CNN jointly performs semantic and instance segmentation, using an FPN-integrated, lightweight DeepLab-like segmentation head and a refined panoptic metric for stuff classes.
Results
The architecture achieves state-of-the-art performance on Cityscapes, Indian Driving Dataset, and Mapillary Vistas.
Takeaways & Limitations
Joint operation on one backbone delivers high-quality seamless scene segmentation with substantially lower computational effort and model size than combined independent recognition models.
Takeaways & Limitations
Reported scores differ from the CVPR 2019 paper because of copied experiment-log numbers and improper handling of void labels in Vistas.

Abstract

from arXiv · show

In this work we introduce a novel, CNN-based architecture that can be trained end-to-end to deliver seamless scene segmentation results. Our goal is to predict consistent semantic segmentation and detection results by means of a panoptic output format, going beyond the simple combination of independently trained segmentation and detection models. The proposed architecture takes advantage of a novel segmentation head that seamlessly integrates multi-scale features generated by a Feature Pyramid Network with contextual information conveyed by a light-weight DeepLab-like module. As additional contribution we review the panoptic metric and propose an alternative that overcomes its limitations when evaluating non-instance categories. Our proposed network architecture yields state-of-the-art results on three challenging street-level datasets, i.e. Cityscapes, Indian Driving Dataset and Mapillary Vistas.

1. Introduction

The paper addresses the limitations of separately trained semantic and instance segmentation models by introducing a jointly trained, single-network architecture for seamless scene segmentation. It combines architectural integration with a revised panoptic metric and reports state-of-the-art results on three street-level datasets.

Semantic segmentation and instance-specific segmentation have largely been handled separately despite their shared role in scene understanding.The paper motivates joint treatment for applications including autonomous driving and augmented reality.
Panoptic evaluation jointly assesses semantic segmentation of stuff and instance-specific segmentation of things.Stuff denotes non-countable amorphous regions, whereas things are enumerable objects with defined shapes.
Winning panoptic challenge systems relied on fused individually trained models rather than streamlined architectures trainable from scratch.Reported results from early panoptic work remained significantly below those of fused individual models.
Separate models incur computational overhead and cannot enforce labeling consistency through cross-model information sharing.The paper also argues that separate models may redundantly model information.
The proposed single-network architecture jointly produces semantic and instance segmentation without pre-trained recognition models, at a fraction of fused-model computational cost.Its design interleaves segmentation and instance-segmentation modules.
The contributions include an FPN-plus-lightweight-DeepLab segmentation head, a refined panoptic metric for stuff classes, and evaluations on Cityscapes, Indian Driving Dataset, and Mapillary Vistas.The paper compares against individually trained fused models and analyzes parameters and computational requirements.

2. Related Works

Prior work developed semantic, instance, and joint scene-understanding methods using CNNs, graphical models, CRFs, and panoptic architectures. The paper situates its approach among these lines and notes contemporaneous single-network panoptic methods.

Semantic segmentation research progressed from encoder-decoder CNNs to architectures using dilated convolutions, multi-scale features, pooled context, and decoder refinement.Examples include DeepLab, pyramid scene parsing, and learned decoder approaches.
Instance segmentation methods evolved through region proposals, position-sensitive score maps, CRFs, multi-cut optimization, grouping networks, and ranked segmentation proposals.These approaches combine instance-agnostic or instance-aware cues in different ways.
Joint segmentation and instance-segmentation approaches predate deep learning, including Bayesian scene graphs, CRF refinement, and graphical-model structure prediction.These methods jointly represented stuff and things or alternated pixel labeling with instance prediction.
A unified architecture related to the paper's ideas reported results below state-of-the-art methods, while several contemporaneous works introduced single-network panoptic segmentation.The paper positions its approach within this emerging panoptic segmentation literature.

3. Proposed Architecture

The proposed architecture uses a shared feature-extraction backbone with separate semantic and instance segmentation branches. Its semantic head fuses multi-scale FPN features with lightweight contextual processing, while the instance branch follows Mask R-CNN components and both branches use distinct losses.

Shared Backbone: A modified ResNet-50 with an FPN serves as the shared feature extractor for both task-specific branches.The FPN receives ResNet outputs at downsampling factors ×4, ×8, ×16, and ×32, with optional ×64 and ×128 scales.
Instance Segmentation Branch: The instance segmentation branch follows Mask R-CNN and contains region proposal and region segmentation heads.The region proposal head transforms anchors into bounding-box proposals with objectness scores, while the region segmentation head predicts classes, box corrections, and masks.
Semantic Segmentation Branch: The architecture combines class-specific mask predictions from the instance branch with semantic-branch outputs for the final segmentation result.The mask prediction is explicitly combined with the output of the semantic segmentation branch.
Semantic Segmentation Branch: The semantic segmentation branch independently processes the first four FPN scales with Mini-DeepLab modules, upsamples them, concatenates the streams, and predicts pixel classes.A final 1 × 1 convolution produces class logits, which are bilinearly upsampled to the input-image size before softmax classification.
Semantic Segmentation Branch: Mini-DeepLab combines parallel convolutions with dilation factors 1 and 6 and large-kernel average pooling to provide contextual information.Boundary replication restores spatial resolution after pooling; unlike global pooling, the fixed-kernel design preserves translation equivariance at test time and limits contextual extent.
Training Losses: Training uses distinct losses for the two branches, including weighted per-pixel log-loss with hard negative mining in semantic segmentation.The weighting selects the 25% of pixels with the lowest ground-truth-class probabilities and assigns them weight τ = 4 W H.

4. Revisiting Panoptic Segmentation

The section reviews PQ, which averages class-specific scores but can over-penalize stuff classes by applying the same IoU threshold used for thing instances. It proposes PQ†, retaining PQ for things while matching stuff segments whenever their overlap is positive.

PQ metric: PQ evaluates panoptic segmentation by independently scoring each class and averaging the class-specific results, with separate thing and stuff aggregates.The metric excludes the void class from class averaging and is insensitive to imbalanced class distributions.
The issue with stuff classes: PQ over-penalizes stuff errors because it treats all pixels of a stuff class as one instance and requires IoU greater than 0.5 for a true positive.A large confusion within a stuff region can therefore reduce that class's segmentation quality to zero.
The issue with stuff classes: Figure 3 illustrates the threshold effect: pole at IoU 0.49 and traffic light at 0.46 approach PQ 0, whereas sidewalk at 0.62 contributes PQ 0.62.The caption notes that these classes can look qualitatively similar despite receiving sharply different metric outcomes.
Suggested alternative: Stuff segments can be matched directly without the IoU threshold because each image has at most one ground-truth and one predicted segment for a given stuff class.This contrasts with thing classes, for which the greater-than-0.5 IoU rule simplifies instance matching.
Suggested alternative: PQ† retains PQ for thing classes but changes stuff matching to pairs with IoU greater than 0, then averages the resulting class-specific scores.The proposed metric remains bounded in [0, 1] and continues to regard a stuff segment as a single instance.

5. Experimental Results

Experiments evaluate the proposed architecture on Cityscapes, IDD, and Mapillary Vistas under controlled comparisons. The combined model matches or improves panoptic performance while using substantially fewer parameters and FLOPs than independently trained fused models.

Experiments cover Cityscapes, IDD, and Mapillary Vistas with deliberately controlled model extensions, testing procedures, and pretraining conditions.The study omits test-time augmentation, ensembles, and several architectural extensions; all models use ImageNet pretraining.
Cityscapes: Cityscapes Ours Independent achieves PQ = 59.8%, PQSt = 64.5%, PQTh = 53.4% and PQ† = 59.0%.Ours Combined performs slightly better on PQ and PQ† despite using fewer parameters.
Cityscapes: All proposed variants outperform direct competitors by a considerable margin on the reported Cityscapes comparisons.The comparison includes baseline models and jointly trained architectures, with qualitative results shown for Cityscapes.
Indian Driving Dataset (IDD): IDD Ours Independent yields IoU = 68.0%, APM = 32.1%, PQ = 47.2%, PQSt = 46.6%, PQTh = 48.3% and PQ† = 48.8%.Ours Combined obtains PQ = 46.9%, PQSt = 45.9%, PQTh = 48.7%, PQ† = 48.6%, APM = 29.8% and IOU = 68.2%.
Mapillary Vistas: Mapillary Vistas Ours Combined improves over Ours Independent by +0.5% PQ and +0.4% PQ† while using significantly fewer parameters.It also exceeds DeeperLab by +5.7% PQ and TASCNet by +5.1% PQ.
Computational Aspects: The combined architecture uses ≈51.43M parameters and ≈0.514 TFLOP, versus ≈78.06M parameters and ≈0.864 TFLOP for the fused individual models.The individual models therefore require ≈51.8% more parameters and ≈50.4% more FLOPs.

6. Conclusions

The paper presents a CNN-based multi-task architecture that jointly performs semantic and instance segmentation on one backbone, achieving state-of-the-art recognition more efficiently than combined individual models.

A single backbone jointly supports semantic and instance segmentation through interleaved network components and a novel segmentation module.
The architecture departs from separately trained recognition models by targeting seamless scene segmentation in a unified multi-task design.
The paper revisits the panoptic metric and proposes a relaxed alternative for evaluating stuff segments.
The resulting models achieve state-of-the-art recognition results with significantly lower computational effort and smaller model sizes than combined individual models.