Source-linked AI summary
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs
Ziyu Zhang, Sanja Fidler, Raquel Urtasun
TL;DR
The paper targets pixel-wise instance labeling from a single monocular autonomous-driving image, where globally consistent merging of local patch predictions is challenging. It introduces a densely connected MRF with efficient mean field inference and reports significant improvements over the prior patch-based method on KITTI, while failure cases remain associated with CNN confusion and heavy occlusion.
Problem
Globally consistent instance labeling from local predictions remains challenging for monocular autonomous-driving images.
Method
A densely connected pixel-level MRF merges overlapping patch CNN predictions using local consistency, long-range component connections, and contrast-sensitive smoothness with efficient mean field inference.
Results
The method achieves significant improvements over the prior patch-based approach on the challenging KITTI benchmark.
Takeaways & Limitations
Dense connectivity provides a globally consistent full-image instance labeling from local CNN predictions.
Takeaways & Limitations
Failure cases largely arise from CNN confusion between vans and cars and from heavily occluded cars.
Abstract
from arXiv · showhide
Our aim is to provide a pixel-wise instance-level labeling of a monocular image in the context of autonomous driving. We build on recent work [Zhang et al., ICCV15] that trained a convolutional neural net to predict instance labeling in local image patches, extracted exhaustively in a stride from an image. A simple Markov random field model using several heuristics was then proposed in [Zhang et al., ICCV15] to derive a globally consistent instance labeling of the image. In this paper, we formulate the global labeling problem with a novel densely connected Markov random field and show how to encode various intuitive potentials in a way that is amenable to efficient mean field inference [Krähenbühl et al., NIPS11]. Our potentials encode the compatibility between the global labeling and the patch-level predictions, contrast-sensitive smoothness as well as the fact that separate regions form different instances. Our experiments on the challenging KITTI benchmark [Geiger et al., CVPR12] demonstrate that our method achieves a significant performance boost over the baseline [Zhang et al., ICCV15].
1. Introduction
Instance-level masks provide richer object understanding than boxes or class labels, but globally consistent labeling from local patch predictions remains challenging. The paper addresses this with a densely connected MRF for monocular autonomous-driving images.
- Instance masks support reasoning about occlusion, depth layering, precise 3D localization, obstacle avoidance, route planning, grasping, and object counting.
- Existing patch-based CNNs show promising instance-labeling performance, but converting their local predictions into a globally consistent image labeling is challenging.
- The proposed method combines soft predictions from many overlapping patches into a global labeling using a densely connected MRF.
- The MRF encodes local-patch consistency, contrast-sensitive smoothness, and separation of disconnected regions into different instances.
- The formulation supports efficient mean field inference and achieves significant improvements over the prior method on the KITTI benchmark.
2. Related Work
Prior instance-segmentation methods use interaction, detection, CAD models, or direct instance reasoning, while this work builds on patch-based CNN predictions. Its contribution is a densely connected MRF for merging those predictions globally.
- Detection-based methods typically localize objects with bounding boxes before applying top-down shape and appearance information for segmentation.
- Some detection-based approaches jointly infer instance labels and depth layers, whereas this work operates directly at the pixel level.
- CAD-model methods provide image-aligned instance labels but are typically slow and not robust to differences between synthetic and real-object appearance.
- Interactive segmentation relies on user-supplied boxes or foreground/background scribbles to derive pixel-wise instance labels.
- Methods without object detection use hierarchical segmentation, multi-output CNNs, or clustering to obtain instance labels.
- The prior patch-based approach merges local CNN predictions with an MRF and connected components, while the proposed dense connectivity yields significantly better segmentations.
3. Object Instance Labeling
The paper builds a densely connected pixel-wise MRF to merge overlapping patch-level CNN predictions into one coherent image-wide instance labeling. Its potentials model smoothness, local CNN compatibility, and separation between disconnected components, with Gaussian formulations supporting efficient inference.
- Global formulation: The method extends local CNN instance predictions into a globally coherent labeling using a densely connected pixel-wise MRF.The CNN produces probabilistic instance labels within overlapping patches, while the MRF merges them across the full image.
- Global formulation: The global label space contains background plus up to nine instances, whereas each local CNN prediction contains six labels including background.The mismatch prevents directly using local CNN outputs as unary potentials, because identically numbered local instances may differ across patches.
- Smoothness: The smoothness term uses pixel position and CNN output features to encourage similar pixels to share labels while reducing influence between distant regions.A Potts compatibility function penalizes different labels for pixels with similar positions and CNN predictions; color is not used.
- Local CNN prediction: The local CNN prediction term preserves same-instance relations and relative instance ordering by comparing shifted local softmax vectors with Gaussian kernels.Zero-padding implements shifts, and matching without a shift favors equal global labels; the maximum shift is fixed to 2 in the experiments.
- Inter-connected components: The inter-connected component term discourages disconnected foreground components from receiving the same instance label.Foreground activation is thresholded into connected components, and cross-component pixel pairs are fully connected in the corresponding potential; each instance is assumed not to appear in two different components.
4. Experimental Evaluation
On KITTI, the model is evaluated against matched CNN-based baselines using class- and instance-level metrics. The full densely connected MRF improves instance separation and suppresses noisy boundary regions, while failures remain concentrated in difficult visual conditions.
- Dataset: The KITTI evaluation uses 3,524 images split by video into 3,260 training, 120 validation, and 144 test images, with 131 manually annotated images.The split keeps all frames from each video exclusively within one partition.
- Baselines: Baselines use the same CNN unaries and patch extraction as the proposed method, enabling comparison between their different MRF formulations.The baselines include ConnComp, Unary, and Unary+LongRange variants.
- Metrics: The evaluation reports class-level FIoU and instance-level coverage, precision, recall, false-positive, false-negative, and InsF1 metrics.MWCov and MUCov measure IoU overlap with ground-truth instances, whereas instance-level precision and recall address false-positive and missed-instance behavior.
- Quantitative Results: The full model significantly outperforms the baselines across multiple metrics after adding inter-connected-component and smoothness terms.The inter-connected-component term separates far-apart instances, while smoothness removes noisy regions near uncertain instance boundaries.
- Qualitative Results: Qualitatively, the full model more often separates neighboring instances and avoids visible patch boundaries than the ConnComp baseline.These examples are shown without post-processing.
- Failure Cases: Failures are largely caused by CNN errors involving large non-car vehicles, small objects, and severe occlusion.Vans that resemble cars can introduce false positives, while heavily occluded cars remain challenging.
5. Conclusions
The paper presents a densely connected pixel-wise MRF for globally consistent instance labeling from local CNN patch predictions. Experiments show significant improvements over the prior approach.
- Conclusions: The paper proposes a new approach for instance-level segmentation based on local CNN patch predictions.The method targets globally consistent labeling of the full image.
- Conclusions: The densely connected MRF combines local CNN predictions, long-range connections between far-apart instances, and contrast-sensitive smoothness.The formulation supports efficient inference and produces a globally consistent instance labeling.
- Conclusions: Experiments show significant improvements over the prior work [32].