PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientPS: Efficient Panoptic Segmentation

Rohit Mohan, Abhinav Valada

arXiv:2004.02307v3cs.CVcs.LGcs.RO

TL;DR

Holistic robot scene understanding requires jointly recognizing semantic regions and object instances, but existing panoptic systems face efficiency, runtime, and fusion challenges. EfficientPS addresses these issues with an efficient shared multi-scale architecture, specialized heads, adaptive logit fusion, and KITTI panoptic annotations. It reports state-of-the-art performance across four benchmarks while being faster and more parameter efficient than existing state-of-the-art architectures.

  • Problem

    Panoptic segmentation is needed to unify semantic and instance segmentation for holistic scene understanding, while existing systems face computational-efficiency, runtime, and fusion challenges.

  • Method

    EfficientPS uses a modified EfficientNet with a bidirectional 2-way FPN, specialized semantic and Mask R-CNN-based instance heads, and a parameter-free adaptive panoptic fusion module.

  • Results

    EfficientPS sets the state of the art on Cityscapes, Mapillary Vistas, KITTI, and IDD while being faster and more parameter efficient than existing state-of-the-art architectures.

  • Takeaways & Limitations

    The architecture provides an efficient panoptic segmentation system and introduces KITTI panoptic annotations for future multi-task scene-understanding research.

  • Takeaways & Limitations

    The instance head retains Mask R-CNN’s IoU-based positive and negative matching strategy with predefined thresholds.

Abstract

from arXiv · show

Understanding the scene in which an autonomous robot operates is critical for its competent functioning. Such scene comprehension necessitates recognizing instances of traffic participants along with general scene semantics which can be effectively addressed by the panoptic segmentation task. In this paper, we introduce the Efficient Panoptic Segmentation (EfficientPS) architecture that consists of a shared backbone which efficiently encodes and fuses semantically rich multi-scale features. We incorporate a new semantic head that aggregates fine and contextual features coherently and a new variant of Mask R-CNN as the instance head. We also propose a novel panoptic fusion module that congruously integrates the output logits from both the heads of our EfficientPS architecture to yield the final panoptic segmentation output. Additionally, we introduce the KITTI panoptic segmentation dataset that contains panoptic annotations for the popularly challenging KITTI benchmark. Extensive evaluations on Cityscapes, KITTI, Mapillary Vistas and Indian Driving Dataset demonstrate that our proposed architecture consistently sets the new state-of-the-art on all these four benchmarks while being the most efficient and fast panoptic segmentation architecture to date.

1 Introduction

Panoptic segmentation unifies semantic and instance segmentation for holistic scene understanding, but existing systems face efficiency, runtime, fusion, and performance challenges. EfficientPS addresses these issues with an efficient multi-scale architecture, adaptive panoptic fusion, and new KITTI panoptic annotations.

  • Motivation: Existing panoptic methods still face computational-efficiency, runtime, and performance challenges relative to task-specific networks.
  • Motivation: Panoptic segmentation jointly predicts amorphous ‘stuff’ regions and countable ‘thing’ instances, unifying semantic and instance segmentation.This supports pixel- and instance-level scene understanding in applications such as autonomous driving, surveillance, and augmented reality.
  • EfficientPS Architecture: EfficientPS combines an efficient shared backbone and 2-way FPN with task-specific semantic and instance heads in an end-to-end architecture.The backbone uses mobile inverted bottleneck units, while the instance head uses a Mask R-CNN variant and separable convolutions.
  • EfficientPS Architecture: The semantic head independently aggregates fine and contextual multi-scale features before fusion, while the instance head is augmented with depthwise separable convolutions.This design targets semantically reinforced fine features and improved object-boundary refinement.
  • Panoptic Fusion: The parameter-free panoptic fusion module adaptively combines semantic and instance logits by attenuating or amplifying scores according to per-pixel agreement.Unlike standard fusion, it uses semantic-head ‘thing’ logits when resolving overlapping predictions.
  • Dataset: The work introduces KITTI panoptic segmentation annotations to complement KITTI’s existing perception and localization tasks.The authors intend these annotations to encourage multi-task learning for holistic scene understanding.

2 Related Works

Related work spans semantic segmentation, instance segmentation, and emerging panoptic methods. EfficientPS builds on these areas while targeting the efficiency and fusion limitations of existing top-down and bottom-up approaches.

  • Panoptic Segmentation: Panoptic segmentation unifies semantic and instance segmentation, but relatively few methods directly address this combined scene-understanding problem.
  • Semantic Segmentation: Semantic segmentation research progressed from handcrafted and patch-wise methods toward end-to-end CNNs with multi-scale feature aggregation and long-range context.Approaches such as PSPNet and ASPP emphasize multi-scale representations, while DPC and eASPP target greater efficiency.
  • Instance Segmentation: Instance segmentation methods are commonly divided into proposal-free and proposal-based categories, including CNN, CRF, recurrent, and Mask R-CNN approaches.
  • Panoptic Segmentation: Top-down panoptic systems address overlapping semantic and instance predictions using weak supervision, spatial ranking, or explicit mask constraints.
  • Panoptic Segmentation: Bottom-up panoptic methods group pixels or object centers, whereas EfficientPS adopts a top-down design for handling large instance-scale variation.The paper positions EfficientPS as a fast, efficient architecture with adaptive fusion of semantic and instance logits.

3 EfficientPS Architecture

EfficientPS combines an EfficientNet-based shared backbone and bidirectional 2-way FPN with specialized semantic and instance heads, whose predictions are adaptively fused into panoptic output.

  • Instance Segmentation Head: The architecture uses parallel semantic and instance segmentation heads, including a modified Mask R-CNN instance head and a specialized semantic head.Their outputs are passed to the panoptic fusion module rather than being used independently.
  • Network Backbone: EfficientPS uses a shared EfficientNet-based backbone with a 2-way FPN that fuses multiscale features through bidirectional information flow.The encoder uses compound scaling and is designed to balance representational capacity, parameter count, and computational complexity.
  • Network Backbone: The 2-way FPN has parallel top-down and bottom-up branches, with the latter downsampling higher-resolution features before adding them to lower-resolution encoder outputs.Both branches use 1 × 1 convolutions with 256 output filters at each scale for channel reduction.
  • Semantic Segmentation Head: The semantic head combines LSFE, DPC, and mismatch-correction modules to capture fine features, long-range context, and cross-scale correlations for boundary refinement.Large-scale inputs use LSFE modules, while small-scale inputs use DPC modules before correlation and feature aggregation.
  • Panoptic Fusion Module: The parameter-free panoptic fusion module adaptively combines semantic and instance predictions to produce the final panoptic segmentation output.It fuses head outputs into intermediate panoptic logits, selects predictions by channel-wise argmax, and fills remaining canvas regions with stuff predictions.

4 Experimental Results

The experiments evaluate EfficientPS on four challenging urban-scene datasets using standard panoptic metrics and complementary efficiency measures. The evaluation includes quantitative comparisons, qualitative visualizations, and benchmarking of computational cost.

  • Evaluation setup: EfficientPS is benchmarked on Cityscapes, KITTI, Mapillary Vistas, and Indian Driving Dataset using extensive quantitative comparisons and ablation studies.The section also includes qualitative comparisons and visualizations for each dataset.
  • Metrics: Panoptic Quality combines recognition and segmentation performance using true positives, false positives, false negatives, and intersection-over-union.SQ and RQ are also reported as complementary metrics.
  • Evaluation setup: The study reports PQ, SQ, and RQ across all, stuff, and thing classes, alongside AP, mIoU, inference time, and FLOPs.These metrics cover panoptic, semantic, instance, and efficiency performance.

4.1 Datasets

The paper benchmarks EfficientPS on four diverse and difficult urban-scene datasets, including a newly annotated KITTI panoptic dataset. Their challenges span clutter, occlusion, weather variation, and poorly delineated infrastructure.

  • Dataset suite: The benchmark covers Cityscapes, KITTI, Mapillary Vistas, and Indian Driving Dataset, with publicly released panoptic ground truth added for KITTI.The KITTI annotations extend an existing benchmark that already supports multiple perception and localization tasks.
  • Cityscapes: Cityscapes contains diverse European urban scenes with dynamic, clustered, and partially occluded objects that challenge thing segmentation.It provides pixel-level annotations for 19 classes across finely and coarsely annotated images.
  • KITTI: The KITTI panoptic dataset contains 1,055 images, 11 stuff classes, and 8 thing classes, with manually drawn instance masks combined with semantic annotations.The split includes 855 training images and 200 validation images.
  • Mapillary Vistas: Mapillary Vistas spans six continents and varied weather, seasons, times, cameras, and viewpoints, making distant-object recognition difficult under snowy conditions.Its images range from 1024×768 to 4000×6000 pixels.
  • Indian Driving Dataset: IDD represents unstructured urban environments with many thing instances and poorly delineated road infrastructure, especially at road-sidewalk boundaries.The dataset contains 10,003 images collected in two Indian cities and their outskirts.

4.2 Training Protocol

EfficientPS is trained on multiresolution crops with limited augmentation, ImageNet-initialized EfficientNet weights, and dataset-specific optimization schedules. Training uses SGD, warm-up, frozen synchronized normalization layers, and a combined semantic-instance loss.

  • Data preparation: Training uses crops at 1024×2048, 1024×1024, 384×1280, and 720×1280 pixels extracted from full-resolution images.Random augmentation is limited to flipping and scaling in [0.5,2.0].
  • Initialization: The EfficientPS backbone is initialized from EfficientNet weights pre-trained on ImageNet.This initialization is part of the stated training protocol.
  • Optimization: Optimization uses SGD with momentum 0.9 and a multi-step learning-rate schedule whose milestones and total iterations differ by dataset.The schedules are specified for Cityscapes, Mapillary Vistas, KITTI, and IDD.
  • Optimization: The protocol warms lrbase linearly from 1/3·lrbase to lrbase over 200 iterations, freezes iABN sync layers, and continues for 10 epochs at lr = 10^-4.The final optimized loss combines semantic and instance losses.
  • Hardware: Models are trained with batch size 16 on 16 NVIDIA Titan X GPUs, with one image assigned to each GPU.The paper states that the total loss contains semantic and instance components.

4.3 Benchmarking Results

Across Cityscapes, Mapillary Vistas, KITTI, and IDD, EfficientPS reports improvements over prior methods while using less computation. The results also expose a trade-off between semantic performance and efficiency associated with output stride and architectural design.

  • Cityscapes: On Cityscapes validation, single-scale EfficientPS exceeds AdaptIS by 1.9% PQ and 2.0% AP, and Panoptic-DeepLab by 0.9% PQ and 3.0% AP.With multi-scale evaluation, it improves over Panoptic-DeepLab by 1.0% PQ and 1.2% AP.
  • Cityscapes: On Cityscapes, EfficientPS achieves 64.1% PQ without extra-data pretraining and 66.4% PQ when pretrained on Mapillary Vistas.The latter sets the reported state of the art, while the model also ranks second in mIoU and AP on the benchmark.
  • Efficiency: EfficientPS runs in 166ms at 1024×2048 resolution and uses the fewest parameters and FLOPs among the compared state-of-the-art models.The comparison includes both top-down and bottom-up architectures.
  • Mapillary Vistas: On Mapillary Vistas, EfficientPS improves single-scale PQ by 0.6% over Seamless and Panoptic-DeepLab, and multi-scale PQ by 0.4% plus AP by 3.6% over Panoptic-DeepLab.These results use no model ensembles and no additional training data beyond ImageNet encoder pretraining.
  • KITTI: On KITTI, EfficientPS surpasses Seamless by 1.6% PQ, 1.2% AP, and 1.5% mIoU single-scale, and by 1.5% PQ, 1.3% AP, and 1.3% mIoU multi-scale.The paper attributes gains to multi-scale aggregation, long-range semantic context, and adaptive fusion.
  • Indian Driving Dataset: On IDD, EfficientPS reaches 50.1% single-scale PQ and 51.1% multi-scale PQ, improving multi-scale PQ by 2.6% over Seamless and 4% over UPSNet.Its single-scale PQSt is 49.8%, 2.7% above Seamless.

4.4 Ablation Studies

The ablations show that EfficientPS’s backbone, semantic head, and panoptic fusion module each contribute to performance, while several changes improve efficiency. The 2-way FPN and adaptive fusion provide especially clear gains over their alternatives.

  • Panoptic fusion: 0.6% PQ, 2.2% AP, and 0.8% mIoU improvements result from the panoptic fusion module without increasing parameters.The adaptive fusion uses semantic and instance outputs to address overlap conflicts.
  • EfficientPS architecture: Replacing standard instance-head convolutions with depthwise separable convolutions reduces parameters by 2.09M, with a 0.2% PQ drop.The authors adopt this trade-off for a more efficient model.
  • EfficientPS architecture: 59.7% PQ is achieved with the modified EfficientNet-B5 encoder, improving PQ by 1.1% over M3.The authors attribute the gain to EfficientNet-B5’s richer representational capacity.
  • 2-way FPN: The proposed 2-way FPN improves PQ by 1.8% and AP by 2.7% over the preceding model through bidirectional information flow.Its parallel bottom-up branch supplements the standard top-down pathway.
  • Semantic head: The semantic head’s multiscale feature fusion yields 63.9% PQ in M8, although its 2.4% gain cannot be attributed solely to that head.The proposed head processes FPN outputs at multiple scales and combines fine and contextual information.
  • 2-way FPN: The 2-way FPN reaches 62.2% PQ with a top-down FPN and 60.4% with a bottom-up FPN, while outperforming both by 1.4% and 3.2%.It also improves PQ by 0.5% over PANet FPN.

4.5 Qualitative Evaluations

Qualitative comparisons with Seamless show EfficientPS handling difficult object-instance and semantic-boundary cases across Cityscapes, Mapillary Vistas, KITTI, and IDD. The examples emphasize occlusion, misclassification, contextual ambiguity, and boundary precision.

  • Comparison protocol: The improvement/error map marks EfficientPS-only corrections in green, Seamless-only corrections in blue, and errors from both models in red.The visualization provides a direct pixel-level comparison against ground truth.
  • Cityscapes: EfficientPS more accurately segments heavily occluded thing instances than Seamless in Cityscapes examples.The cited cases include a distant truck behind cars and a partially visible parked car.
  • Mapillary Vistas: On Mapillary Vistas, EfficientPS detects people behind a fence and a heavily occluded van that Seamless misses.It also correctly labels a cyclist that Seamless misclassifies as a pedestrian.
  • KITTI: EfficientPS correctly handles a bus that Seamless labels as a truck in a KITTI scene containing poles and an advertisement board.The authors connect this example to bidirectional multiscale feature aggregation.
  • IDD: EfficientPS produces more precise stuff boundaries than Seamless for a bridge pillar and sidewalk extent in IDD examples.The authors attribute this to the semantic head’s object-boundary refinement capability.

4.6 Visualizations

EfficientPS visualizations span diverse urban scenes and perceptual conditions across four datasets. The examples include occlusion, viewpoint and lighting variation, thin structures, unstructured roads, and dense traffic.

  • Cityscapes: Cityscapes visualizations show complex road scenes with many traffic participants, including dynamic and static pedestrian groups in close proximity.The examples use single-scale evaluation with outputs overlaid on input images.
  • Mapillary Vistas: Mapillary Vistas examples include uncommon viewpoints and nighttime capture, yet EfficientPS demonstrates substantial robustness against these perceptual variations.The dataset scenes span diverse times of day and viewpoints.
  • KITTI: KITTI visualizations contain residential and highway scenes with parked and moving cars plus numerous thin structures such as poles.These examples test performance on both dynamic traffic and narrow structures.
  • IDD: IDD examples combine poorly delineated sidewalks, heavy traffic, motorcycles, pedestrians, and varied vehicles in unstructured environments.The authors report robust EfficientPS outputs in these challenging scenes.

5 Conclusions

The conclusion presents EfficientPS as a computationally efficient panoptic segmentation architecture combining multiscale backbone features, specialized heads, and adaptive fusion. It also introduces KITTI panoptic annotations and reports broad benchmark leadership.

  • Conclusions: EfficientPS combines a modified EfficientNet encoder, 2-way FPN, depthwise-separable Mask R-CNN instance head, semantic head, and adaptive panoptic fusion.The architecture is designed to capture fine, contextual, and instance-specific information efficiently.
  • Panoptic fusion: The parameter-free fusion module integrates instance-specific thing classes with stuff classes using logits from both heads.Its dynamic fusion is based on mask confidences.
  • Conclusions: The 2-way FPN establishes bidirectional multiscale information flow, while the semantic head aggregates scale-specific features and refines object boundaries.Together with the modified encoder, these components provide the paper’s stated performance–computational-complexity trade-off.
  • Conclusions: The architecture achieves state-of-the-art panoptic segmentation on Cityscapes, Mapillary Vistas, KITTI, and IDD while being faster and more parameter-efficient than existing state-of-the-art models.The paper reports first place on the Cityscapes panoptic leaderboard and second place on its semantic and instance leaderboards.
  • Dataset: The KITTI panoptic segmentation dataset adds panoptic ground-truth annotations to the challenging KITTI benchmark.The annotations complement KITTI’s existing perception tasks and are intended to encourage panoptic multi-task learning.
Loading 2004.02307v3…