Source-linked AI summary
Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer
Jun Xie, Martin Kiefel, Ming-Ting Sun, Andreas Geiger
TL;DR
Large-scale pixelwise semantic instance annotation is labor-intensive, especially for street scenes with limited labeled data. The paper moves annotation into 3D using bounding primitives and transfers labels into images, producing a large suburban dataset while improving efficiency, accuracy, and temporal consistency over 2D baselines.
Problem
Large-scale pixelwise annotation is labor-intensive, and street-scene datasets have limited semantic instance labels.
Method
The method annotates static 3D scene elements with bounding primitives and transfers their semantic and instance labels into images by jointly reasoning over 3D points and pixels.
Results
400k images are annotated, while the method yields better results than state-of-the-art 2D label-transfer baselines with lower annotation time and temporally consistent instance labels.
Takeaways & Limitations
3D annotation provides an efficient basis for generating large-scale semantic instance labels across street-scene video.
Takeaways & Limitations
The evaluation includes label ambiguities for porous objects and errors at complex boundaries, which remain difficult even for manual 2D annotation.
Abstract
from arXiv · showhide
Semantic annotations are vital for training models for object recognition, semantic segmentation or scene understanding. Unfortunately, pixelwise annotation of images at very large scale is labor-intensive and only little labeled data is available, particularly at instance level and for street scenes. In this paper, we propose to tackle this problem by lifting the semantic instance labeling task from 2D into 3D. Given reconstructions from stereo or laser data, we annotate static 3D scene elements with rough bounding primitives and develop a model which transfers this information into the image domain. We leverage our method to obtain 2D labels for a novel suburban video dataset which we have collected, resulting in 400k semantic and instance image annotations. A comparison of our method to state-of-the-art label transfer baselines reveals that 3D information enables more efficient annotation while at the same time resulting in improved accuracy and time-coherent labels.
1. Introduction
Large-scale street-scene annotation remains difficult because pixelwise labeling is labor-intensive and instance-level data are limited. The paper addresses this by annotating static scenes in 3D and transferring labels into images.
- Motivation: Semantic segmentation of street scenes lacks very large annotated datasets because human pixelwise annotation is labor-intensive.The paper calls this challenge the “curse of dataset annotation.”
- Motivation: Auxiliary-task transfer and synthetic datasets offer alternatives, but each has limitations motivating the use of additional 3D information.The paper notes representation bias from auxiliary tasks and insufficiently realistic generative models for synthetic imagery.
- Approach: The proposed workflow annotates static scene objects directly in 3D and transfers their labels into the image domain for semantic and instance video annotation.3D information can come from structure-from-motion, stereo, or laser data.
- Approach: The label-transfer model jointly reasons over semantic and instance labels for 3D points and image pixels using a non-local multi-field CRF.The model is designed to handle occlusion-related ambiguities, temporal issues, parameter learning, and label uncertainty.
- Contributions: The work contributes a geo-registered suburban dataset with over 400k images, over 100k laser scans, and semantic 3D annotations for static scene elements.The dataset was recorded from a moving platform.
- Contributions: The method transfers 3D labels into pixelwise semantic instance annotations and is evaluated against 2D and 3D baselines.The authors also make the code, dataset, and annotations publicly available.
2. Related Work
Prior work propagates labels mainly through 2D video or uses limited 3D information, while this paper directly annotates 3D scenes and projects labels into images. Its approach targets large-scale semantic instance annotation with occlusion-aware, temporally consistent outputs.
- Video annotation: 2D video annotation methods propagate labels from annotated keyframes using color, motion, Bayesian models, classifiers, or temporal association.These methods focus on exploiting temporal coherence between adjacent frames.
- 3D-to-2D transfer: The paper instead annotates scenes directly in 3D and projects those annotations into 2D, requiring a source such as SfM, stereo, or laser data.The authors report that 3D annotation can be more time efficient because scene elements may project into many video frames.
- 3D-to-2D transfer: Existing 3D-to-2D approaches include vehicle-specific figure-ground segmentation and interactive projection of annotated 3D points onto image superpixels.The paper distinguishes its approach by jointly reasoning about all objects and supporting categories without available CAD models or 3D point measurements.
- 3D-to-2D transfer: Some multimodal methods model image pixels and 3D points, but they primarily target classification performance or efficiency rather than transferring ambiguous primitive labels to every pixel.The paper positions dense semantic instance label transfer as its distinct objective.
- Datasets: Dense semantic datasets commonly contain only about 1k accurately annotated indoor or outdoor images, while larger resources may lack dense annotations or instance labels.The paper contrasts these limitations with its focus on large-scale street-scene semantic instance annotations.
3. Method
The method annotates static urban scene elements with 3D bounding primitives and transfers semantic and instance labels to image pixels using a joint CRF. It combines projected 3D constraints with appearance, geometric-boundary, and pairwise consistency cues.
- 3.1. Data Collection and Annotation: 3D cuboids and ellipsoids enclose annotated objects and carry their semantic labels for transfer into images.Annotators tightly enclose object points with primitives using a WebGL tool.
- 3.3. Model: The CRF jointly reasons over semantic labels for image pixels and sparse laser or stereo points.Semantic and instance inference use the same label framework, with unique IDs assigned to individual objects for instance segmentation.
- 3.3. Model: Projected 3D primitives determine admissible pixel labels, while pixel appearance probabilities provide additional semantic evidence.The pixel unary terms combine primitive-based constraints with learned pixel-wise label likelihoods.
- 3.3. Model: 3D curbs and folds encourage label changes near projected geometric boundaries between semantic regions.The model places the geometric constraint inside a 2D disc around each projected curb or fold, with penalty decay toward the disc boundary.
- 3.3. Model: Gaussian pairwise terms encourage coherence among image pixels, 3D points, and their cross-domain connections.For 3D points, surface-normal information is used alongside location; visible points are selected through ball-pivoting mesh reconstruction before projection.
- 3.4. Learning and Inference: The same regularized logistic-loss framework learns parameters for semantic and instance segmentation.Instance segmentation assigns distinct labels to separate objects, including mutually occluding objects of the same category.
4. Experimental Evaluation
The evaluation compares the proposed 3D-to-2D label transfer method with 2D and 3D baselines, ablates model components, and assesses semi-dense and instance segmentation. The method generally outperforms baselines, reduces annotation time, and achieves high accuracy when estimating only selected pixels.
- Quantitative evaluation: The evaluation measures semantic segmentation using frequency-weighted average Jaccard Index and average pixel accuracy, alongside instance segmentation results.The semantic evaluation uses 14 mapped categories, while instance evaluation includes Building, Car, Trailer, Caravan, and Box.
- Baseline comparison: 3D primitives or meshes with graph-cut smoothing perform poorly, while projecting visible 3D points followed by spatial propagation performs better.The results attribute the weaker primitive and mesh baselines to crude geometry approximations.
- Baseline comparison: The proposed method outperforms the evaluated label-transfer baselines in almost all categories.The comparison includes 2D-to-2D and 3D-to-2D baselines on consecutive images.
- Annotation efficiency: 20 hours of 2D annotation for 200 frames compares with less than 3 hours for the corresponding 3D annotations.The estimate assumes 60 minutes per annotated image and labeling every tenth frame; manual labeling of every frame would require 200 hours.
- Semi-dense inference: 94.9% JI and 97.4% accuracy are achieved at 90% pixel density by selecting pixels with the lowest predictive entropy.Entropy-based selection estimates the most certain regions and produces semi-dense label maps.
- Instance segmentation: Instance segmentation results are on par with semantic segmentation, demonstrating intra-class separation ability.The evaluation uses instance labels for five object classes, while also reporting jointly modeled semantic classes without unambiguous instances.
- Qualitative evaluation: Qualitative errors arise at low-contrast boundaries, sensor-occluded regions, porous objects, and difficult manual annotation boundaries.Examples include car-road, building-roof, tree, and hedge boundaries.
5. Conclusion
The paper presents 3D-to-2D semantic instance labeling for large datasets, improving results and reducing annotation time while maintaining temporal consistency.
- The method transfers semantic instance labels from annotated 3D primitives into image datasets.
- The approach outperforms several state-of-the-art 2D label-transfer baselines while lowering annotation time.
- The resulting instance labels are temporally consistent and explicitly expose label uncertainty.
- The authors introduce a publicly available dataset containing 400k images, laser point clouds, and annotations for all objects.