PaperlayerView exact revision on arXiv

Source-linked AI summary

DeeperLab: Single-Shot Image Parser

Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen

arXiv:1902.05093v2cs.CV

TL;DR

Whole image parsing must assign semantic and instance labels to every pixel while remaining efficient for complex, high-resolution scenes. DeeperLab addresses this with a single-shot, bottom-up parser and region-based evaluation alongside PQ, achieving reported accuracy-speed trade-offs on Mapillary Vistas.

  • Problem

    Multiple complicated networks and inference passes make whole image parsing computationally costly, especially for high-resolution inputs.

  • Method

    DeeperLab is a single-shot, bottom-up parser that uses an efficient fully convolutional design, shared decoder outputs, and fast prediction fusion.

  • Results

    31.95% PQ (val) / 31.6% (test) and 55.26% PC (val) were achieved at 3 fps on GPU, while a Wider MobileNetV2 version reached 22.61 fps with reduced accuracy.

  • Takeaways & Limitations

    DeeperLab demonstrates a reported trade-off between parsing accuracy and speed using a single-shot, bottom-up, single-inference paradigm.

  • Takeaways & Limitations

    Standard MobileNetV2 has a limited receptive field for high-resolution image parsing, while increasing depth adds feature maps that dominate memory usage.

Abstract

from arXiv · show

We present a single-shot, bottom-up approach for whole image parsing. Whole image parsing, also known as Panoptic Segmentation, generalizes the tasks of semantic segmentation for 'stuff' classes and instance segmentation for 'thing' classes, assigning both semantic and instance labels to every pixel in an image. Recent approaches to whole image parsing typically employ separate standalone modules for the constituent semantic and instance segmentation tasks and require multiple passes of inference. Instead, the proposed DeeperLab image parser performs whole image parsing with a significantly simpler, fully convolutional approach that jointly addresses the semantic and instance segmentation tasks in a single-shot manner, resulting in a streamlined system that better lends itself to fast processing. For quantitative evaluation, we use both the instance-based Panoptic Quality (PQ) metric and the proposed region-based Parsing Covering (PC) metric, which better captures the image parsing quality on 'stuff' classes and larger object instances. We report experimental results on the challenging Mapillary Vistas dataset, in which our single model achieves 31.95% (val) / 31.6% PQ (test) and 55.26% PC (val) with 3 frames per second (fps) on GPU or near real-time speed (22.6 fps on GPU) with reduced accuracy.

1. Introduction

The paper targets efficient whole image parsing, which must jointly assign semantic and instance labels across both 'thing' and 'stuff' classes. DeeperLab uses a single-shot bottom-up design and introduces efficiency-oriented network strategies and the Parsing Covering metric.

  • Whole image parsing combines semantic segmentation of 'thing' and 'stuff' regions with separating individual 'thing' instances.
  • High computational cost, multiple network passes, and high-resolution inputs make efficient deployment challenging.Mapillary Vistas images can reach 4000 × 6000 pixels.
  • DeeperLab generates per-pixel semantic and instance predictions in one fully convolutional pass, then fuses them with a fast algorithm.Its runtime is nearly independent of the number of detected object instances.
  • Parsing Covering evaluates image parsing from a region-based perspective alongside the instance-based Panoptic Quality metric.The paper motivates PC because PQ can emphasize small instances and 'thing' classes over 'stuff' classes.
  • The proposed design strategies reduce memory footprint for high-resolution inputs through architectural changes and hard data mining.Examples include depthwise separable convolution, shared decoder outputs, larger kernels, space-to-depth operations, and depth-to-space operations.
  • 31.95% PQ (val) / 31.6% (test) and 55.26% PC (val) were achieved at 3 fps on GPU with an Xception-71-based model.

2. Related Work

Related work frames image parsing as whole-scene understanding that unifies visual decomposition, semantic segmentation, and instance segmentation. It also reviews region-, contour-, and instance-based evaluation metrics, including limitations of PQ.

  • Image parsing decomposes images into visual patterns such as textures and object instances, unifying detection, segmentation, and recognition.
  • Early image-parsing methods used Bayesian frameworks, AND-OR graphs, exemplars, or conditional random fields and evaluated constituent tasks with separate metrics.
  • Modern whole-image parsing, also called Panoptic Segmentation, uses the unified instance-based Panoptic Quality metric.
  • Semantic segmentation: Semantic segmentation research commonly builds on fully convolutional networks and adds contextual modeling, multi-scale processing, or encoder-decoder structures.
  • Instance segmentation: Instance segmentation methods are broadly categorized as top-down detector-based approaches and bottom-up approaches.
  • Evaluation metrics: PQ treats all regions sharing a 'stuff' class as one instance and may overemphasize small instances and 'thing' classes.

3. Methodology

DeeperLab uses an efficient encoder-decoder network with shared prediction features for semantic and class-agnostic instance segmentation. Its design targets high-resolution image parsing by balancing accuracy with latency and memory usage.

  • Network Architecture: The encoder-decoder network generates semantic and instance segmentation from a shared decoder output before fusion.The architecture uses five two-layer prediction heads: one semantic head and four class-agnostic instance heads.
  • Network Architecture: High-resolution inputs create substantial memory usage and latency, motivating design choices that balance accuracy, computation, and memory footprint.The experiments resize Mapillary Vistas images to 1441 × 1441.
  • Encoder: Replacing MobileNetV2’s 3 × 3 convolutions with 5 × 5 convolutions expands the receptive field from 491 × 491 to 981 × 981 while maintaining feature-map memory.The change only mildly increases computation cost and forms the Wider MobileNetV2 backbone.
  • Encoder-Decoder: ASPP and large-kernel depthwise convolutions increase contextual coverage, while depth-to-space converts a 4096-channel feature map into 256 channels at stride 4.The encoder output has stride 16, and the decoder combines it with stride-4 low-level features to recover object boundaries.
  • Training Objective: Hard example mining backpropagates only the top 15% pixel losses, while pixels in instances smaller than 64 × 64 receive weight 3.The weighted bootstrapped cross-entropy loss emphasizes both difficult pixels and small instances.
  • Instance Segmentation Head: The instance representation uses five keypoints—four bounding-box corners and a center of mass—with heatmap and offset heads encoding their relationships to pixels.The four offset types are long-range, short-range, and middle-range maps alongside the keypoint heatmap.
  • Instance Segmentation Head: The middle-range offset map groups keypoints through a star-shaped directed relation graph linking the center to the four box corners.The graph has bidirectional connections between the mass center and each corner.

3.4. Prediction Fusion

DeeperLab fuses semantic predictions with keypoint-based instance predictions in a fast bottom-up pipeline. Keypoints are localized, clustered into instances, assigned to pixels through long-range offsets, and merged with semantic labels.

  • Prediction Fusion: The four instance-related maps are fused into a single class-agnostic instance segmentation map before final semantic-instance assignment.The maps are the keypoint heatmap and long-, short-, and middle-range offset maps.
  • Prediction Fusion: Recursive offset refinement improves predictions by repeatedly refining offset maps because offsets nearer corresponding keypoints are more accurate.The refinement follows the strategy used in PersonLab.
  • Keypoint Localization: Keypoints are localized by merging short-range and long-range Hough-voting score maps, weighted respectively by heatmap values and unit vote weights.The merged score map is then used for keypoint localization.
  • Instance Detection: A fast greedy algorithm clusters keypoints, follows middle-range offsets to form instances, and applies bounding-box non-maximum suppression to remove overlaps.Detected-instance confidence is the average of its keypoint scores.
  • Pixel Assignment: Each pixel is assigned to the instance whose keypoints are closest in L2 distance to the pixel’s predicted keypoints.Predicted keypoints are obtained from the pixel location plus its predicted long-range offset.
  • Semantic-Instance Merging: Stuff pixels receive one unique instance label, while other pixels retain instance labels from the instance segmentation result and semantic labels from semantic prediction.The method uses a simple merging procedure without additional post-processing such as removing small isolated regions.

3.5. Evaluation Metrics

The paper reviews Panoptic Quality (PQ) and introduces Parsing Covering (PC), a region-based metric designed to account for instance size and evaluate ‘thing’ and ‘stuff’ classes more evenly.

  • PQ matches ground-truth and predicted regions using a 0.5 Intersection-Over-Union threshold and combines true positives, false positives, false negatives, and matched-region quality.
  • PQ treats all regions of the same ‘stuff’ class as one instance and weights instances equally regardless of size.
  • PQ can therefore emphasize small instances and ‘thing’ classes, making it less suitable when applications prioritize large objects.
  • Parsing Covering (PC) extends the existing Covering metric to evaluate image parsing from a region-based perspective that accounts for instance sizes.
  • PC computes class-wise Covering using ground-truth and predicted regions of each semantic class, then averages the resulting Cov_i values across C classes.
  • PC includes every class and pixel, so false positives are not absorbed by omitting the background class.
  • Unlike PQ, PC uses no region matching or matching threshold and gives partially correct ‘stuff’ segmentation partial credit.

4. Experimental Results

Experiments evaluate DeeperLab on Mapillary Vistas using PQ, PC, accuracy-speed trade-offs, and ablations of backbone, decoder, pixel mining, and keypoint graphs.

  • Validation performance: 31.95% PQ and 55.26% PC are achieved by the Xception-71 model on the Mapillary Vistas validation set.The Wider MobileNetV2 model reaches 25.20% PQ and 49.80% PC with faster inference, while Light Wider MobileNetV2 reaches 9.37 fps before input downsampling.
  • Validation performance: 22.61 fps on GPU is reached by Light Wider MobileNetV2 after halving input resolution, with reduced accuracy.The reported results use single-scale inference and no heuristic post-processing; semantic-instance fusion adds 145 ms at 1441 × 1441 resolution and 45 ms at 721 × 721.
  • Backbone ablation: Larger kernels improve Wider MobileNetV2 accuracy on high-resolution Mapillary Vistas images, while ASPP further improves accuracy across settings.Using 5 × 5 kernels improves PQ by 1.75% and PC by 4.54%; 7 × 7 provides only marginal additional improvement.
  • Decoder and prediction heads: 23.48% PQ and 46.33% PC are obtained when S2D/D2S replaces bilinear upsampling in the decoder design ablation.The baseline reaches 19.85% PQ and 42.98% PC, while deeper heads and larger kernels provide additional gains before S2D/D2S.
  • Hard pixel mining: 0.92% PQ and 1% PC improvement over the baseline result from combining hard pixel mining with increased loss weights for small instances.Hard pixel mining alone increases PQ by 0.57%, while emphasizing instances smaller than 64×64 improves accuracy by 0.2% PQ.
  • Keypoint relation graph: The star keypoint graph yields 0.53% higher PQ and 1.89% higher PC than the rectangular graph.The star graph includes a mass-center keypoint connected to the other four keypoints.

5. Conclusion

DeeperLab is presented as an effective whole-image parser that balances accuracy and speed through a single-shot, bottom-up, single-inference design and multiple efficiency innovations.

  • Conclusion: DeeperLab achieves a reported trade-off between accuracy and speed for whole image parsing.The design combines single-shot, bottom-up, and single-inference processing with several network innovations.
  • Conclusion: The efficiency design integrates depthwise separable convolutions, shared decoder outputs, simple prediction heads, and enlarged kernels.These innovations are presented as enabling the proposed image parser’s overall design.
  • Conclusion: The paper demonstrates DeeperLab on the challenging whole image parsing task.The conclusion characterizes the model as effective within the reported experimental scope.

A. Performance on Cityscapes

On Cityscapes, DeeperLab is evaluated without extra coarse annotations, and the models show accuracy and speed trade-offs across backbones and input settings.

  • Experimental setting: DeeperLab is trained on 2,975 Cityscapes training images and evaluated on 500 validation images using 721 × 721 crops.The models do not use the dataset’s extra coarse annotations.
  • Validation performance: The Xception-71 model outperforms in both PQ and PC on Cityscapes.The Wider MobileNetV2 model achieves comparable accuracy at 6.71 fps on GPU.
  • Validation performance: 23.99 fps on GPU is achieved by Light Wider MobileNetV2 with downsampled inputs.The result is described as near real-time speed.

B. Performance on PASCAL VOC 2012

On PASCAL VOC 2012, DeeperLab is trained with augmented annotations, and the reported models combine competitive parsing accuracy with real-time inference.

  • Experimental setting: 10,582 augmented training images are used for PASCAL VOC 2012, with evaluation on 1,449 validation images.The augmented training set includes extra annotations provided by.
  • Validation performance: The Xception-71 model outperforms in both PQ and PC without COCO pretraining.The comparison is reported on the PASCAL VOC 2012 validation set.
  • Validation performance: 35.01 fps on GPU is achieved by Light Wider MobileNetV2 without downsampling inputs.The result is reported as real-time speed.

C. Performance on COCO

On COCO, DeeperLab reports a speed–accuracy trade-off across model variants, while test-dev performance remains close to validation performance.

  • Input images are not upsampled on COCO because of the speed consideration, and hard pixel mining is omitted because it hurts accuracy.
  • 33.79% Panoptic Quality (PQ) and 56.82% Parsing Covering (PC) are achieved at 10.59 fps on GPU by the Xception-71 model.These results are reported on the COCO validation set.
  • 17.19 fps on GPU is reached by the Wider MobileNetV2 model at the cost of accuracy.
  • 33.84 fps on GPU is reached by the Light Wider MobileNetV2 model with downsampled inputs.
  • The COCO test-dev results are reported in Table 11 and are very close to the validation-set results.The table notes that downsampled inputs use a 321 × 321 size.

D. Performance on Mapillary Vistas

The paper presents qualitative image-parsing results across several validation datasets, showing predictions overlaid on images and comparisons with ground-truth labels.

  • Figure 13 provides extra qualitative results for the Mapillary Vistas validation set.
  • Figures 7–9 show predicted semantic segmentation in the first row and predicted instance segmentation in the second row.These figures cover Cityscapes, Pascal VOC 2012, and COCO validation sets.
  • Figures 7–9 note that the model does not generate VOID labels.
  • Figures 10–13 compare image-parsing results with ground-truth labels across Cityscapes, Pascal VOC 2012, COCO, and Mapillary Vistas validation sets.
  • The last row in Figures 10–13 is identified as a failure case.
  • The Instance Detection column displays raw keypoints and middle-range offsets that are refined in a later stage.
Loading 1902.05093v2…