Source-linked AI summary
Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling
Jonas Uhrig, Marius Cordts, Uwe Franke, Thomas Brox
TL;DR
The paper addresses instance-aware semantic labeling in street scenes without relying on complex architectures or graphical models. It predicts semantics, depth, and instance-center directions with an FCN from a monocular image, then uses standard computer-vision processing to generate instance segmentation. The method outperforms existing works on KITTI and Cityscapes while also providing per-instance depth and pixel-level semantic labeling.
Problem
Instance-aware semantic labeling requires combining pixel-level semantics with individual-instance localization, while prior approaches often use complex multi-task architectures or graphical-model post-processing.
Method
An FCN predicts semantic labels, absolute depth, and pixel directions toward visible instance centers, which are processed with template matching and proposal fusion.
Results
The method significantly outperforms existing works on KITTI and Cityscapes, including 37 % relative improvement averaged over all KITTI metrics and almost double performance on Cityscapes.
Takeaways & Limitations
The approach provides instance segmentation, pixel-level semantic labeling, and absolute per-instance depth from a single monocular image without region proposals.
Abstract
from arXiv · showhide
Recent approaches for instance-aware semantic labeling have augmented convolutional neural networks (CNNs) with complex multi-task architectures or computationally expensive graphical models. We present a method that leverages a fully convolutional network (FCN) to predict semantic labels, depth and an instance-based encoding using each pixel's direction towards its corresponding instance center. Subsequently, we apply low-level computer vision techniques to generate state-of-the-art instance segmentation on the street scene datasets KITTI and Cityscapes. Our approach outperforms existing works by a large margin and can additionally predict absolute distances of individual instances from a monocular image as well as a pixel-level semantic labeling.
1 Introduction
Instance-level semantic labeling combines semantic segmentation with instance detection to provide precise object localization for scene understanding. The paper uses one FCN to predict semantics, depth, and instance-center directions, then applies standard computer-vision post-processing to obtain holistic street-scene representations.
- Motivation: Instance-level semantic labeling combines detection with instance segmentation for precise localization of individual objects.Such representations support explicit occlusion reasoning, object tracking, motion estimation, and behavior modeling in robotics and autonomous vehicles.
- Related approaches: Existing methods commonly add CRF post-processing, region-proposal scoring, or multi-stage neural networks to FCN-based systems.
- Contribution: The proposed FCN predicts semantic class, object-center direction, and object distance from a single monocular image.Standard computer-vision post-processing converts these outputs into instance-aware semantic segmentation.
- Contribution: The method simultaneously produces pixel-level semantic labeling, instance-level segmentation of traffic participants, and 3D depth estimation for each instance.
- Results: The approach significantly outperforms state-of-the-art methods on KITTI and Cityscapes street-scene datasets.
2 Related Work
Prior instance-segmentation methods use proposals, sequential sampling, encoded instance IDs, or complex clustering architectures, each with important constraints. This paper instead uses proposal-free center-direction encoding together with depth for instance separation in urban scenes.
- Proposal-based methods: Proposal-based methods depend on proposal quality and can be slow because proposals are classified individually.
- Proposal-based methods: Proposal-based systems may fail when instances are missing during proposal generation, while proposal-free methods infer segmentation and class jointly.
- Proposal-free methods: Existing proposal-free approaches include boundary prediction, sequential sampling, depth-ordered instance IDs, and clustering from predicted box parameters.
- Limitations of prior work: Depth-ordered ID methods are limited to scenes with clear depth ordering and support relatively few instances.
- Limitations of prior work: PFN uses a complex architecture and depends heavily on correctly predicting the number of instances, which can be difficult in scenes containing hundreds of instances.
- Proposed representation: The proposed representation encodes each instance by pixel directions toward its center, independently of the number of instances, while depth supports separation in street scenes.
3 Method
The method extends an FCN with semantic, depth, and geometric direction channels trained as pixel-wise labeling tasks. Template matching, center detection, direction-based assignment, proposal fusion, and semantic merging produce instance masks with depth and background labels.
- FCN representation: The FCN jointly predicts semantic labels, object depth, and 2D directions toward instance centers using three output channels.The channels provide instance classification, scale and separation cues, and instance detection and segmentation cues.
- Semantics: Semantic predictions separate objects from background and distinguish objects belonging to different classes.
- Depth: Depth is constant within each object instance, discretized into classes, and used to guide scale handling and separate nearby objects.Close objects receive finer depth resolution than distant objects.
- Direction: Each object pixel is assigned a discretized direction toward its visible instance center, producing a region-based cue that separates neighboring instances and handles many occlusions.
- Direction: Softmax-weighted direction vectors recover continuous direction estimates from the discretized direction classes.
- Instance generation: Template matching detects instance centers using class-dependent aspect ratios and depth-dependent template sizes.
- Instance generation: Non-maximum suppression finds temporary centers, and pixels are assigned to centers when relative location and predicted direction agree.
- Instance generation: Proposal fusion merges neighboring hypotheses when direction bias, semantic class, and depth indicate that they belong to one instance.
4 Experiments
The method is evaluated on KITTI and Cityscapes for instance segmentation, depth prediction, and pixel-level semantic labeling. It outperforms baselines, while ablations show that depth and fusion are important for accurate instances.
- Datasets and Metrics: The evaluation uses KITTI and Cityscapes, with official splits and instance-level semantic, depth, and segmentation metrics.Depth metrics include MAE, RMSE, ARD, and relative inlier ratios; instances overlap more than 50% with ground truth.
- Ablation Studies: Removing depth substantially worsens most metrics, while additionally removing fusion causes a still larger performance drop on both datasets.Using proposals directly as final predictions is also slightly worse than the depth-removed variant, indicating that all components contribute to accurate instances.
- Instance Evaluation: 37 % relative improvement over Best is achieved on KITTI when averaged across all metrics.The method also outperforms the best individual baseline metric values represented by Mix.
- Instance Evaluation: Nearly double the main score AP is achieved against MCG+R-CNN on Cityscapes, with improvements across all proposed metrics.Performance is lower than on KITTI, reflecting the greater complexity of Cityscapes scenes.
- Depth Evaluation: 7.7 % average relative error and 1.7 m mean absolute error are obtained for instance depths on KITTI from single monocular images.On Cityscapes, the corresponding errors are 11.3 % and 7.7 m.
- Semantic Class Labels: The method achieves pixel-level semantic labeling on Cityscapes and is on par with the state of the art despite focusing on instance segmentation.The comparison includes FCN 8s and Dilation10.
5 Conclusion
The approach combines pixel-wise depth, semantics, and instance-level direction cues in an FCN, using standard computer vision techniques rather than complex architectures or graphical models. It outperforms existing methods on Cityscapes and KITTI while supporting absolute per-instance depth and semantic labeling.
- The FCN predicts pixel-wise depth, semantic labels, and instance-level direction cues for holistic scene understanding.
- Standard computer vision techniques applied to the three output channels replace complex architectures or graphical models for post-processing.
- The method does not depend on region proposals and scales to arbitrary numbers of object instances in an image.
- 37 % relative improvement averaged over all metrics is achieved on KITTI, while performance on Cityscapes is almost doubled.
- The method predicts absolute depth values per instance and achieves state-of-the-art pixel-level semantic labeling on Cityscapes.
Supplementary Material for Pixel-level Encoding and Depth Layering for
The supplementary material accompanies the work on instance-level semantic labeling and identifies its authors.
- The supplementary material is titled “Instance-level Semantic Labeling.”
- The listed authors are Jonas Uhrig, Marius Cordts, Uwe Franke, and Thomas Brox.
A Qualitative Results
Qualitative examples show that the approach segments many instances in urban scenes despite heavy occlusions and clutter.
- Figures 1 and 2 provide qualitative instance-segmentation examples from KITTI and Cityscapes urban scenes.
- The approach can segment high numbers of instances despite heavy occlusions and clutter.
B Depth Ranges
The method discretizes continuous instance depths into classes whose ranges are selected to improve template matching and balance depth resolution with training data coverage.
- Continuous instance depths are discretized into 19 depth classes with ranges chosen so object sizes are similar within each class.
- Similar object sizes within depth classes yield slightly better results because template matching receives more reliable scores.
- The depth-class values trade off the number of classes, depth resolution, and samples per class in the training data.
- Cityscapes requires different depth ranges from KITTI because it includes instances labeled beyond 200 meters.
C.1 Instance-level Evaluation
The approach outperforms the proposal-based MCG+R-CNN baseline by large margins for most classes, while cars and humans yield its best results. Bus, truck, and train remain comparatively difficult.
- The approach outperforms the MCG+R-CNN baseline by large margins for all other classes across every used metric.
- Cars and humans achieve the best results among the evaluated semantic classes.
- Bus, truck, and train are especially difficult because the FCN classifies them less reliably.
C.2 Pixel-level Evaluation
The method achieves competitive pixel-level semantic labeling across classes and outperforms most existing works on the instance-aware iIoU metric for person, car, and bicycle. Performance is lower for bus, truck, and train because weaker pixel-level semantic labeling degrades later processing stages.
- The method achieves competitive pixel-level semantic labeling for all classes compared with the listed baselines.
- On instance-aware iIoU, the method outperforms most existing works for person, car, and bicycle by a few percentage points.
- Bus, truck, and train perform comparatively poorly because weak pixel-level semantic labeling degrades template matching and instance generation.