Source-linked AI summary
Monocular Object Instance Segmentation and Depth Ordering with CNNs
Ziyu Zhang, Alexander G. Schwing, Sanja Fidler, Raquel Urtasun
TL;DR
The paper addresses instance segmentation and depth ordering from a single monocular image. It trains CNNs on overlapping multi-scale patches and fuses their predictions with an MRF, demonstrating effective performance on KITTI for both tasks.
Problem
The task is to jointly predict accurate pixel-level object-instance labeling and depth ordering from a single monocular image.
Method
CNNs predict instance segmentation and depth ordering on overlapping patches at multiple scales, while an MRF fuses the predictions into one coherent labeling.
Results
The approach demonstrates effective instance segmentation and depth ordering on KITTI, including 83.1% correctly ordered randomly sampled foreground pixel pairs.
Takeaways & Limitations
The method shows that weak 3D bounding-box annotations can support learning instance segmentation from monocular RGB images.
Takeaways & Limitations
Training uses 3D bounding boxes and stereo, although testing requires only a single RGB image.
Abstract
from arXiv · showhide
In this paper we tackle the problem of instance-level segmentation and depth ordering from a single monocular image. Towards this goal, we take advantage of convolutional neural nets and train them to directly predict instance-level segmentations where the instance ID encodes the depth ordering within image patches. To provide a coherent single explanation of an image we develop a Markov random field which takes as input the predictions of convolutional neural nets applied at overlapping patches of different resolutions, as well as the output of a connected component algorithm. It aims to predict accurate instance-level segmentation and depth ordering. We demonstrate the effectiveness of our approach on the challenging KITTI benchmark and show good performance on both tasks.
1. Introduction
The paper targets joint instance-level segmentation and depth ordering from a single monocular image, motivated by the need to represent individual objects and their spatial arrangements. It combines multi-scale CNN predictions with an MRF to produce coherent image labeling.
- Motivation: Accurate scene parsing should supplement object detection and semantic labeling with instance segmentation, generic classes, and 3D information such as depth ordering.
- Motivation: The task is to assign each pixel to an object instance while predicting the instances’ depth ordering from one monocular image.
- Approach: A CNN predicts instance segmentation and depth ordering on image patches sampled densely at several scales, whose outputs are merged through energy minimization.
- Motivation: The autonomous-driving setting is challenging because images contain many occluding cars, while shadows, reflectance, and saturation complicate recognition.
- Approach: The MRF jointly reasons about pixel-wise instance segmentation and depth ordering, using CNN outputs at multiple resolutions and connected-component ordering within patches.
- Training and evaluation: At test time, the method requires only a single RGB image, although 3D bounding boxes and stereo are used during training.
- Training and evaluation: On the KITTI car benchmark, the network predicts accurately for up to 5 object instances in an image patch.
2. Related Work
Prior work addresses depth ordering, instance segmentation, 3D reconstruction, or combinations of these tasks using geometric cues, detection, structured optimization, and learned models. The paper is most closely related to methods that jointly model class, instance, and depth information, but directly predicts instance ordering with a CNN and merges results through energy minimization.
- Depth ordering: Single-image depth-ordering methods use weak object information, junctions, or boundaries, but typically do not reason jointly about class and instance segmentation.
- Instance segmentation: Detection-and-segmentation systems localize objects first, then use top-down shape or appearance information to carve object regions within bounding boxes.
- 3D scene understanding: RGB-based 3D-object methods infer depth and instances through geometric cues, learned occlusion patterns, CAD models, or indoor Manhattan-world assumptions, but segmentation is often limited.
- Instance segmentation: Recent instance-segmentation methods use R-CNN detection followed by box-based segmentation or cuts through hierarchical segmentation trees.
- Joint formulations: The closest approaches jointly address class, instance, and depth segmentation, whereas this paper directly predicts instance labeling and depth ordering with a CNN before energy-based merging.
3. Instance Segmentation and Depth Ordering
The method jointly predicts instance segmentation and depth ordering from monocular images by combining CNN predictions over overlapping, multiscale patches with MRF-based global inference. CNN outputs are merged with connected-component structure to produce a coherent image-wide labeling.
- Problem: The task assigns each pixel to an object instance while ordering instances by distance from the camera.Each state encodes an instance, and lower state IDs represent objects closer to the camera.
- CNN prediction: A CNN predicts pixel-level instance labels and depth ordering for overlapping patches extracted at multiple scales.Multiscale patches address objects appearing at different sizes, while each patch produces depth-level and background probabilities.
- MRF patch merging: An MRF combines overlapping CNN predictions with connected components to infer a single coherent labeling across the image.CNN outputs are interpolated to their original resolution, merged across patches, and integrated with unary and pairwise energy terms.
- CNN prediction: The CNN is adapted from VGG by converting fully connected layers into convolutional units to produce multiple pixel-wise categorical outputs.The network uses enlarged 306×306 inputs and outputs a spatial map rather than a single categorical prediction.
- Label space: The system uses a maximum of 6 instance levels per patch, while the image-level MRF allows up to 9 cars.The CNN predicts up to 5 cars per patch; the MRF creates the coherent whole-image explanation.
- MRF patch merging: The MRF favors labels compatible with local CNN predictions, connected-component ordering, and spatial consistency between pixels.The CNN energy favors global states at least as large as local predictions, while component ordering uses vertical position as a depth cue.
4. Experimental Evaluation
The evaluation measures class-level segmentation, instance-level segmentation, and depth ordering on KITTI, including ablations of the MRF and post-processing. The full approach performs strongly overall, but post-processing affects tasks differently and small or merged cars remain failure cases.
- Experimental setup: The evaluation uses 301 annotated KITTI images with 1,229 cars for testing, while the remaining 6,744 images train the CNN.The dataset split includes 101 validation and 197 test images, with hyperparameters tuned only on validation data.
- Evaluation measures: The study evaluates class-level, instance-level, and depth-ordering performance using task-specific metrics.Instance metrics include coverage, precision, recall, false positives, and false negatives; depth metrics include recalled instances, recalled pairs, pair accuracy, and foreground pixel-pair correctness.
- Class-level segmentation: The full MRF formulation achieves the best average IoU for class-level segmentation, although post-processing generally harms binary prediction performance.The full formulation also performs well on other IoU metrics, accuracy, and recall, while the baseline has better precision but worse recall.
- Instance segmentation: The full MRF outperforms the baseline significantly on MWCov and MUCov, with post-processing improving instance-level performance by around 2%.After post-processing, the pairwise MRF clearly outperforms unary potentials alone; medium-sized patches perform very well.
- Depth ordering: 83.1% of randomly sampled foreground pixel pairs are ordered correctly, and post-processing is necessary for pairwise MRF connections to improve over raw CNN outputs.Without post-processing, the pairwise connections harm depth-ordering performance.
- Qualitative results: Qualitative results are strong for easily separable cars, but tiny cars may be missed and connected components may merge instances.The successful cases are attributed in large part to the connected component algorithm, while merged instances appear across several failure-case rows.
5. Conclusion
The paper presents a CNN and MRF approach for producing instance-level segmentations and depth orderings from a single monocular image. It demonstrates effectiveness on KITTI and learns segmentation from weak 3D bounding-box annotations.
- Conclusion: A convolutional neural network and Markov random field produce instance-level segmentations and depth orderings from a single monocular image.The method combines CNN predictions with MRF reasoning to produce the joint output.
- Conclusion: The approach is demonstrated on the challenging KITTI benchmark and learns segmentation from weak annotations in the form of 3D bounding boxes.The conclusion identifies weak 3D bounding-box annotations as the supervision source for learning segmentation.