PaperlayerView exact revision on arXiv

Source-linked AI summary

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

Yiyi Liao, Jun Xie, Andreas Geiger

arXiv:2109.13410v2cs.CV

TL;DR

Existing driving datasets do not provide the dense, coherent multimodal annotations and cross-domain benchmarks needed for interdisciplinary autonomous-driving research. KITTI-360 constructs a georegistered suburban dataset with 3D annotation and 3D-to-2D label transfer, then evaluates tasks spanning vision, graphics, and robotics. The resulting benchmarks show that direct-observation segmentation is more tractable than tasks jointly recovering geometry, appearance, and semantics.

  • Problem

    Existing datasets lack dense, complete, and coherent semantic labels across 2D and 3D, limiting evaluation of interdisciplinary tasks such as semantic scene completion, novel view synthesis, and semantic SLAM.

  • Method

    The paper builds a georegistered 3D video dataset, labels static and dynamic objects with a WebGL-based 3D tool, and transfers labels to 2D using a joint semantic-instance model.

  • Results

    The resulting benchmarks show satisfactory performance on established 2D/3D segmentation tasks but substantially greater difficulty for jointly recovering geometry, appearance, and semantics.

  • Takeaways & Limitations

    KITTI-360 provides shared data, annotations, tools, and online benchmarks for research connecting computer vision, graphics, and robotics in autonomous driving.

  • Takeaways & Limitations

    Novel view semantic synthesis remains low-performing with simple baselines, leaving substantial room for future improvement through joint view and semantic synthesis.

Abstract

from arXiv · show

For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

1 INTRODUCTION

KITTI-360 addresses gaps in existing driving datasets by combining rich sensing, accurate localization, coherent 2D/3D annotations, and benchmarks spanning vision, graphics, and robotics. Its 3D annotation workflow and label-transfer model support efficient, dense scene understanding across modalities.

  • KITTI-360 targets limitations of KITTI, including incomplete semantic labels, partial perspective views, sparse 3D information, and localization insufficient for sub-pixel multi-frame fusion.The dataset adds comprehensive 2D/3D labels, richer 360° sensing, accurate poses, and new benchmarks.
  • Its sensor suite combines perspective stereo, fisheye cameras, Velodyne, and SICK laser scanning, while annotations provide consistent semantic and instance labels for 2D pixels and 3D points.The modalities together support 360° scene perception.
  • The non-local multifield CRF jointly reasons over semantic and instance labels for 3D points and 2D pixels while using learned 2D priors.Its probabilistic formulation also estimates label uncertainty for selective annotation refinement.
  • The paper establishes benchmarks for 2D/3D recognition, semantic scene completion, joint RGB-semantic novel view synthesis, and semantic SLAM.These tasks connect computer vision, graphics, and robotics through shared data and annotations.
  • KITTI-360 is a georegistered suburban driving dataset recorded from a moving platform, comprising over 300k images and 80k laser scans.
  • A WebGL-based tool labels static and dynamic scene elements in 3D, and a 3D-to-2D method produces pixel-wise semantic instance annotations.The approach establishes coherent labels across both domains and preserves object identity across sequences.

2 RELATED WORK

Related driving datasets differ in annotation domain, temporal and cross-domain consistency, sensing coverage, and benchmark support. KITTI-360 combines coherent 2D/3D labels with omnidirectional imagery, laser scans, and 3D reasoning, while targeting sparse outdoor observations.

  • Datasets: Unlike indoor RGB-D datasets with dense depth, KITTI-360 addresses outdoor street scenes where sparse 3D observations make 3D-to-2D label transfer challenging.
  • Outdoor Datasets: KITTI-360 differs from prior perspective-image datasets by providing temporally coherent instance annotations together with omnidirectional imagery, 3D laser scans, and 3D annotations.Its suburban focus complements datasets centered on inner-city scenes.
  • Datasets: KITTI lacks dense per-point 3D labels, while KITTI-360 provides labels for both 2D images and corresponding 3D points.
  • Synthetic Datasets: Synthetic urban datasets remain affected by a perceptual gap between virtual and real domains, making synthetic-to-real generalization difficult.
  • Efficient Annotation: Prior efficient-annotation methods focus on individual images or class-specific 3D shapes, whereas KITTI-360 annotates full 3D scenes and seeks coherent per-pixel 2D and per-point 3D labels.

3 ANNOTATION

KITTI-360 combines richer sensing, accurate georegistered poses, and a WebGL-based 3D annotation workflow for suburban driving scenes. Its primitive-based annotations cover static and dynamic objects, producing coherent labels efficiently across 2D and 3D.

  • Data Collection: The dataset uses fisheye and perspective cameras, Velodyne and SICK laser scanners, and structure-from-motion with GPS and LiDAR loop closures for accurate poses.The collection covers over 300k images, 80k laser scans, and 73.7 km of driving.
  • Annotation Tool: A WebGL annotation tool combines scene views, semantic label selection, and controllers for inserting labeled 3D primitives.The interface supports 2D images, 3D scenes, multiple views, and interactive point-cloud manipulation.
  • Static Annotation: Annotators label 3D point clouds with cuboids and ellipsoids across 37 classes, including 24 instance classes and 13 stuff classes.Ground stuff can also be annotated with bird’s-eye-view polygons that are automatically extruded into 3D.
  • Dynamic Annotation: Dynamic objects are annotated in 3D using a semi-automatic scheme that assumes fixed object size and smooth trajectories over time.This avoids placing a separate 3D bounding box in every frame.
  • Efficiency: 0.75 minutes per image is required for 3D annotation, compared with 7 minutes for coarse 2D annotation and 1.5 hours for pixel-accurate annotation.A full batch of approximately 240 frames takes about 3 hours on average.

4 LABEL TRANSFER METHOD

The label-transfer method jointly infers semantic and instance labels for image pixels and visible 3D points using a multifield CRF. It combines 3D primitive constraints, learned image priors, and pairwise consistency while performing inference independently per timestamp.

  • 3D-to-2D Transfer: Static primitives project into multiple frames, whereas dynamic primitives are included only at timestamps where the object is labeled to appear.This design preserves a unique instance ID across the sequence and across 2D and 3D.
  • Model Overview: The CRF reasons jointly over all image pixels and visible 3D points, resolving sparse and noisy 3D annotations into dense semantic and instance labels.Inference is applied independently at each timestamp, while shared 3D annotations preserve consistency across frames.
  • Unary Potentials: Pixel unary potentials combine learned per-pixel label probabilities with admissibility constraints obtained by projecting 3D bounding primitives into the image.The constraint excludes labels whose rays do not pass through a primitive of the corresponding class.
  • Pairwise Potentials: Pairwise potentials encourage coherence among pixels, 3D points, and pixel–point associations using Gaussian kernels based on image appearance, 3D geometry, and surface normals.The normal’s vertical component distinguishes horizontal surfaces such as roads from vertical surfaces such as walls and vehicle sides.
  • Inference: Mean-field variational inference with bilateral filtering estimates marginal label distributions and entropy-based uncertainty for every pixel and 3D point.The implementation uses a multiple-field extension because the model contains pixel, 3D-point, and pixel–3D variable connections.
  • Learning: Instance parameters use the same regularized logistic loss as semantic segmentation, assigning distinct labels to individual objects.The approximate marginals are computed by mean-field inference, and the regularization parameter is selected by cross-validation.

5 LABEL TRANSFER EVALUATION

The evaluation compares label-transfer baselines and ablates the proposed model for semantic and instance segmentation on static and dynamic objects. The method improves transfer quality while reducing annotation time and identifying uncertain regions.

  • Evaluation setup: The evaluation uses manually pixel-labeled images for training and testing, with semantic segmentation measured by mIoU and average pixel accuracy.The training set contains 125 diverse images, and evaluation follows the unweighted mean IoU convention of Cityscapes.
  • Baselines: The proposed method is compared against 2D-to-2D and 3D-to-2D label-transfer baselines using neighboring frames, Cityscapes images, 3D primitives, meshes, or visible 3D points.The benchmark includes methods based on label propagation, sparse tracking, fully connected CRFs, PSPNet, projected geometry, and spatial propagation.
  • Static objects: The full model outperforms all evaluated 3D-to-2D baselines on static objects, while projected primitives or meshes alone perform poorly because they crudely approximate geometry.Projecting visible 3D points followed by spatial propagation performs better than direct primitive or mesh projection with graph-cut smoothing.
  • Dynamic objects: The proposed method also outperforms all 2D label-transfer baselines on dynamic objects despite noisier accumulated 3D cues from motion and deformation.Dynamic-object evaluation treats static regions as one background class and excludes methods that cannot distinguish static and dynamic instances within a class.
  • Efficiency and qualitative analysis: 3 hours of 3D annotation replaces about 20 hours for labeling every tenth image in a 200-frame batch, while uncertainty-aware prediction reaches 94.3% mIoU and 98.4% accuracy at 70% pixel density.The confidence maps highlight less certain regions, including far buildings and ambiguous object boundaries, and instance results are reported as comparable to semantic segmentation.
  • Ablation study: Each model component improves semantic segmentation, with the largest gain coming from reasoning about relationships between 3D points and image pixels.The ablation adds local appearance, 3D terms, pixel pairwise terms, primitive constraints, and 3D pairwise constraints incrementally.

6 DATASET & BENCHMARKS

KITTI-360 establishes benchmarks across semantic scene understanding, novel view synthesis, and semantic SLAM using unified 2D/3D data. Baselines perform better on direct segmentation than on tasks requiring joint recovery of geometry, appearance, and semantics.

  • Benchmark scope: KITTI-360 benchmarks 2D/3D recognition, semantic scene completion, novel view synthesis, and semantic SLAM for autonomous driving.The benchmarks span computer vision, graphics, and robotics, with initial baselines and online evaluation.
  • Benchmark results: 2D and 3D segmentation baselines achieve satisfactory results when inference operates directly on observed data.The paper contrasts these established tasks with newly introduced tasks requiring joint recovery of multiple scene properties.
  • Semantic scene understanding: 3D instance segmentation outperforms naive clustering with PointGroup, and cars perform better in 3D than with 2D methods.The comparison covers Building and Car instances; building boundaries remain difficult in sparse point clouds.
  • Semantic scene understanding: Raw LiDAR provides accurate but incomplete geometry, whereas learning-based completion increases completeness at the cost of accuracy.The learning-based baseline also predicts a semantic label at each completed 3D point.
  • Semantic scene understanding: 3D semantic segmentation is harder than 2D semantic segmentation, while instance-segmentation difficulty varies by class.Cars are easier to segment in 3D than in 2D, whereas buildings are harder because spatially connected instances have indistinct boundaries in sparse point clouds.
  • Novel view synthesis and semantic SLAM: Novel-view and semantic-SLAM baselines remain challenging, with modality-specific trade-offs in fidelity, completeness, accuracy, and localization.ORB-SLAM2 achieves better overall localization than SUMA++, while ORB-SLAM2 + SGM improves completeness but reduces accuracy.

7 CONCLUSION

The paper presents KITTI-360 as a large-scale 3D video dataset with consistent 2D/3D semantics and benchmarks spanning vision, graphics, and robotics. Results are satisfactory for established segmentation tasks but show greater difficulty when geometry, appearance, and semantics must be recovered jointly.

  • Dataset: KITTI-360 contains 300k images and laser point clouds with consistent semantics in both 2D and 3D.The dataset uses a WebGL-based tool for annotating static and dynamic objects in 3D and transfers labels to obtain dense semantic instance annotations.
  • Benchmarks: The paper establishes online benchmarks for challenging tasks at the intersection of computer vision, graphics, and robotics.Several baselines are evaluated for each benchmark.
  • Conclusions: Existing methods perform satisfactorily on direct 2D/3D segmentation but struggle with semantic scene completion, novel-view synthesis, and semantic SLAM.These newer tasks require jointly recovering geometry, appearance, and semantics.

APPENDIX A ANNOTATION DATA PREPARATION

KITTI-360 combines accumulated 3D measurements with semi-automatic tools for detecting, annotating, and transferring dynamic and static scene information. The interface supports multiple views, editable primitives, efficient object labeling, and completeness checks.

  • Data preparation: 3D points are fused in a common world coordinate system, augmented with stereo points up to 15m, consistency-checked over 5 frames, and downsampled at 5cm.These steps reduce stereo outliers and redundant points before annotation.
  • Dynamic-object preparation: Dynamic points are detected by volumetric fusion, identifying points inside voxels whose accumulated evidence indicates free space.Moving objects occupy free space intermittently along their trajectories, producing this signal.
  • Dynamic-object preparation: Region Growing clusters noisy detections, and a cluster is labeled dynamic when the fraction of its points in free voxels exceeds a threshold.This second step filters false positives caused by independently classified voxels.
  • Annotation interface: The annotation tool provides normal and orthographic 3D viewports, fisheye and perspective camera views, and adjustable point size and brightness.These controls help annotators inspect boundaries and select informative viewpoints.
  • Annotation interface: Bounding primitives support translation, scaling, rotation, orientation assignment, copying, object-centric viewing, and a line-based fast annotation mode.The fast mode heuristically fits a primitive around selected objects, while object-centric mode hides distant scene content.
  • Dynamic-object annotation: Dynamic-object annotation assumes fixed object size and smooth trajectories, allowing keyframe primitives to generate remaining annotations automatically.Spline interpolation and occupancy templates support placement between annotated keyframes.

E.3 Qualitative Comparison of Ablation Study

The qualitative ablation comparison agrees with the quantitative analysis: the full label-transfer model performs best among the evaluated variants.

  • Ablation comparison: The full model achieves the best qualitative performance among the label-transfer variants.Figure 22 compares the variants and reports consistency with the quantitative analysis.

F.1 Statistical Analysis

This section characterizes KITTI-360’s label distributions, evaluation split, metrics, and semantic-scene-understanding baselines. Results compare transfer methods and segmentation models under confidence-aware evaluation.

  • Evaluation setup: Training and test sets are split without spatial overlap, with held-out test labels and task-specific released information.The test set is divided according to the label modalities required by different tasks.
  • Evaluation metrics: Confidence-weighted mIoU weights per-pixel intersections and unions by pseudo-ground-truth confidence before averaging across labels.Standard evaluation is recovered when every pixel has confidence 1.

G.2.3 Additional Results

Additional results evaluate instance segmentation and 3D bounding-box detection with class-specific metrics and baselines. The reported comparisons show similar performance for common classes across Mask R-CNN backbones.

  • 2D instance segmentation: Mask R-CNN backbones achieve similar performance on frequently observed classes such as Building and Car, but differ on less frequent classes.The comparison reports per-class AP and mean AP.
  • Instance segmentation: Static-object instance segmentation is compared with 2D and 3D-to-2D label-transfer baselines on 120 consecutive images.The comparison is summarized in Table 11.
  • Additional results: Additional quantitative results report individual-class performance alongside an overall metric in the final column.The table provides class-level and aggregate results for semantic scene understanding.
  • 3D detection: 3D bounding-box detection is evaluated at IoU thresholds 0.5 and 0.25, focusing on Building and Car because fewer-sample classes are harder to generalize to.VoteNet and BoxNet are used as baselines.

G.3.3 Additional Results

The evaluation reports baseline performance for 3D semantic and instance segmentation, including class-wise comparisons and metric-specific behavior.

  • Both 3D detection methods perform reasonably at IoU 0.25 but struggle at the higher IoU threshold.
  • PointNet++ achieves better 3D semantic segmentation performance than PointNet on all classes.

G.5.1 Evaluation Metric

The benchmark evaluates scene completion from a single LiDAR scan using accumulated corridor ground truth, geometric metrics, and learning-based reconstruction baselines.

  • The scene completion ground truth accumulates points within a 30m corridor along a 100m vehicle trajectory.
  • Neighboring poses are filtered when their heading deviates by more than 45° from the input scan's heading.
  • Geometric completion uses completeness and accuracy at a 20cm threshold, while separating observed and unobserved regions.
  • The learning-based baseline encodes input points, predicts a volumetric reconstruction with a 3D U-Net, and samples occupied voxels into a point cloud.
  • Joint geometry and semantic prediction is challenging, with this baseline performing worse than direct 3D semantic segmentation baselines.

H.1.2 Data Preparation

Additional evaluations cover novel-view synthesis and semantic synthesis under sparse-view settings, plus localization and semantic mapping with established baselines.

  • Novel View Synthesis: Novel-view synthesis uses five static scenes at a 50% frame drop rate and ten scenes at a 90% drop rate, with half the frames for training.
  • Novel View Synthesis: PCL, FVS, and PBNR provide point-cloud, free-view, and per-view attribute-optimization baselines for novel-view synthesis.
  • Novel View Synthesis: FVS and PBNR preserve fine details better but obtain lower PSNR, while mip-NeRF and DS-NeRF improve synthesis yet miss thin structures.
  • Semantic Synthesis: Semantic synthesis evaluates confidence-weighted IoU and applies PSPNet to synthesized images because the benchmark has no existing research baseline.
  • Semantic Synthesis: The naïve semantic-synthesis baseline substantially degrades performance on most classes, and small image changes can significantly alter semantic predictions.
  • Localization: Localization uses aligned Absolute Pose Error and one-meter Relative Pose Error, comparing ORB-SLAM2 with SUMA++.

I.2.3 Additional Results

Qualitative results compare localization trajectories and geometric mapping quality using completeness and accuracy, revealing complementary strengths among the methods.

  • Localization: Both localization methods achieve reasonable trajectory estimates, but SUMA++ has a larger maximum error than ORB-SLAM2.
  • Localization and Mapping: Mapping quality is evaluated at a 10cm threshold using completeness and accuracy visualizations.
  • Localization and Mapping: SUMA++ is more accurate, whereas ORB-SLAM2+SGM is more complete.
Loading 2109.13410v2…