PaperlayerView exact revision on arXiv

Source-linked AI summary

The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset

Richard Schwarzkopf, Fabian Immel, Alexander Blumberg, Jonas Merkert, Nils Rack, Kaiwen Wang, Fabian Konstantinidis, Julian Truetsch, Carlos Fernandez, Annika Bätz, Kevin Rösch, Marlon Steiner, Willi Poh, Yinzhe Shen, Royden Wagner, Felix Hauser, Dominik Strutz, Jaime Villa, Gleb Stepanov, Holger Caesar, Ömer Şahin Taş, Frank Bieder, Jan-Hendrik Pauls, Christoph Stiller

arXiv:2606.02956v1cs.CVcs.LGcs.RO

TL;DR

Existing autonomous-driving benchmarks provide limited high-fidelity sensing, complete geospatial context, dense 3D annotations, and geographic diversity for spatially aware urban driving. KITScenes Multimodal addresses these gaps with synchronized multimodal sensors, detailed HD maps, and four spatial-learning benchmarks, exposing systematic capability gaps in current methods.

  • Problem

    Existing benchmarks lack combinations of high-fidelity sensing, complete geospatial context, dense 3D annotations, and diverse urban environments needed for spatially aware driving.

  • Method

    KITScenes Multimodal provides a European driving dataset with synchronized multimodal sensors, complete Lanelet2 HD maps, dense 3D traffic-element annotations, and four spatial-learning benchmarks.

  • Results

    Across four benchmarks, current state-of-the-art methods exhibit systematic capability gaps in complete map prediction, long-range depth, geometrically consistent novel views, and map-grounded trajectory evaluation.

  • Takeaways & Limitations

    KITScenes Multimodal offers a controlled testbed for evaluating spatial-reasoning capabilities required on the path to L4 autonomy.

  • Takeaways & Limitations

    The current release lacks 3D bounding boxes, tracks, and instance segmentation for dynamic agents, which are planned for a future release.

Abstract

from arXiv · show

Existing autonomous driving datasets have enabled major progress, but fall short in sensor fidelity, map completeness, or geographic diversity. We present KITScenes Multimodal, a European dataset built around high-fidelity sensors and maps. Our fully synchronized sensor suite combines high-resolution global-shutter cameras, long-range lidar beyond 400m, 4D imaging radar, and redundant GNSS/INS localization. Our HD maps are, to our knowledge, the most complete of any sensor dataset, validated through autonomous driving trials on open-source software. For the first time in a public dataset, all driving-relevant traffic elements, such as traffic lights, are mapped in 3D to a reprojection-accurate level with full topological connectivity. Recorded in cities with irregular street layouts and mixed traffic modes, our dataset complements existing datasets by broadening the available geographic diversity. We also introduce four benchmarks, each advancing spatial learning for embodied AI: online HD map construction, long-range depth estimation, novel view synthesis, and end-to-end driving. Project page: https://kitscenes.com/

1 Introduction

Existing autonomous-driving datasets inadequately capture dense urban spatial complexity, geographic diversity, and complete geospatial context. KITScenes Multimodal addresses these gaps with a high-fidelity European robotaxi dataset, production-grade HD maps, and four challenging benchmarks.

  • Motivation: Existing datasets often lack public annotations or topology-aware maps, focus on simpler scenarios, and inadequately represent dense urban spatial complexity.These limitations hinder spatially aware driving research in complex urban environments.
  • Motivation: High-fidelity geospatial datasets have limited geographic coverage, leaving complex European urban environments underrepresented in current benchmarks.Existing coverage is heavily skewed toward North America and Asia.
  • Benchmarks: Four benchmarks evaluate online HD map perception, monocular depth beyond 200 m, novel view synthesis, and multimodal end-to-end driving.They target capabilities that existing datasets cannot benchmark at comparable fidelity.
  • Dataset: KITScenes records diverse European urban environments with synchronized sensors including 72.5 Mpx cameras, seven lidars, three 4D imaging radars, and redundant GNSS/INS.The lidars provide over 3× the point density and twice the effective range of the next closest dataset.
  • Dataset: Its production-grade Lanelet2 maps cover 62 km2 with 29 road-feature classes, 120 traffic-sign classes, and reprojection-accurate 3D traffic elements.The maps include regulatory elements for autonomous navigation and are validated in the open-source Autoware stack online and in simulation.

2 Related Work

Prior autonomous-driving datasets established multimodal sensing and broadened traffic coverage, but remain limited in map completeness, regulatory structure, sensor diversity, scale, or deployment-oriented support. Existing benchmarks also underrepresent long-range depth and face fidelity and lidar-density constraints in neural rendering.

  • Autonomous Driving Datasets for Perception: nuScenes, Waymo Open, and Argoverse 2 established the multimodal autonomous-driving dataset paradigm through complementary sensor configurations and annotation schemes.
  • Autonomous Driving Datasets for Perception: Additional datasets broaden traffic layouts and driving conditions, while KITTI, KITTI-360, ZOD, and MAN TruckScenes retain limitations in scale, sensor diversity, annotation scope, or deployment-oriented support.ZOD annotates only a single keyframe per scenario and mainly provides image-space labels.
  • HD Maps and Map Perception Benchmarks: nuScenes and Argoverse 2 expose lane geometry but omit traffic-light and sign regulatory structure, whereas OpenLaneV2 provides lane-topology links as image-space annotations rather than metric 3D maps.
  • Long-range Perception, Neural Rendering, and End-to-End Driving: Monocular-depth benchmarks rarely evaluate beyond 80–100 m, while neural-rendering methods are constrained by input image fidelity and lidar density.The passage identifies KITTI and DDAD as predominant depth benchmarks and notes strong near-range performance from recent foundation models.

3 The KITScenes Multimodal Dataset

KITScenes Multimodal combines a fully synchronized, precisely calibrated multimodal sensor suite with high-resolution cameras, dense long-range lidar, and pixel-accurate 3D HD maps usable in Autoware. Its release provides 1007 scenarios across 5.7 hours and 162 km in three cities, spanning distinct urban environments and seasons.

  • Sensor suite: The sensor suite is fully synchronized and calibrated across modalities, achieving subpixel intrinsic accuracy and 1 cm and 0.1° extrinsic accuracy.Calibration supports sensor fusion up to the maximum effective sensing range.
  • Cameras: Six 7.1 Mpx surround cameras, one 16.2 Mpx long-range camera, and a tilted stereo setup provide 72.5 Mpx per frame and full 360° coverage.The combined resolution is more than twice that of the next closest dataset.
  • Lidar: Seven lidar sensors provide overlapping 360° coverage, with over 900 k points per frame on average, peaks above 1.2 M points, and maximum range exceeding 400 m.These figures triple effective point density and nearly double the maximum range of the next-best dataset.
  • HD maps: The maps are pixel-accurate, directly usable in Autoware, annotated in Lanelet2, and encode the full regulatory structure required for autonomous driving.Road-level polylines use 29 classes, including road borders, dashed markings, and zebra crossings.
  • Dataset scope: The current release contains 1007 scenarios totaling 5.7 h and 162 km of synchronized multimodal recording at 10 Hz across Karlsruhe, Frankfurt, and Sindelfingen.The cities represent planned radial, metropolitan financial-district, and suburban-industrial environments; recordings span summer 2025 and winter 2025/26.

4 Benchmarks

KITScenes Multimodal introduces four benchmarks spanning map-level scene understanding, long-range depth, lateral novel view synthesis, and multimodal end-to-end driving. Together, they expose limitations of existing evaluations through richer map structure, long-range sensing, geometric fidelity, modality controls, and map-grounded safety metrics.

  • Online HD map construction: Online HD map construction predicts structured drivable maps with lane topology, regulatory elements, traffic signs, and traffic lights instead of simple geometric primitives.MapTRv2 and SDTagNet both suffer large performance drops on this complete formulation, revealing limitations hidden by narrower benchmarks.
  • Long-range monocular depth estimation: Long-range monocular depth evaluation stratifies AbsRel and δ1 across close, medium, far, and overall ranges to expose failures beyond 100 m.Existing models are dominated by close-range lidar training signals and fail to generalize reliably to long-range distances.
  • Long-range monocular depth estimation: MapAnything ranks first overall and at 0 m–100 m, whereas UniDAC ranks last overall but is strongest at long range; no method performs reliably beyond 200 m.The benchmark demonstrates that aggregate metrics can mask severe ranking inversions across depth ranges.
  • Lateral novel view synthesis: The map-based novel view synthesis benchmark measures traffic sign recall across seven lateral offsets and two resolutions, using real photographs as per-scale upper bounds.Ground-truth signs are projected from HD maps into shifted viewpoints, with lidar-based occlusion filtering retaining only unoccluded signs.
  • Lateral novel view synthesis: 27.8% relative recall drop occurs for ReconDrive at Δy=0 after upsampling to cropped sensor resolution, versus 7.6% at low resolution; loss exceeds 80% at ±3 m.These results indicate missing fine-grained structural detail and poor geometric fidelity under lateral translation.
  • End-to-end driving: End-to-end driving evaluation supports front-camera, 360° surround-view, and complete lidar-radar multimodal inputs, enabling controlled modality ablations and map-grounded safety assessment.Metrics include ADE, FDE, drivable-surface survival, collision-free rate, centerline distance, and Multi-Maneuver Score; zero-shot baselines reveal a substantial domain gap.

5 Limitations

KITScenes Multimodal currently lacks dynamic-object annotations, contains 5.7 h of recorded data, and evaluates end-to-end driving only through open-loop trajectory prediction. Closed-loop evaluation is enabled by the released artifacts but deferred to future work.

  • Dynamic-object annotations: The current release lacks 3D bounding boxes, tracks, and instance segmentation for dynamic agents.These annotations are planned for a future release.
  • Dataset scale: 5.7 h of recorded data makes KITScenes Multimodal smaller in raw volume than nuPlan Sensor (≈120 h) and Nvidia Physical AI AV (≈1700 h).The paper notes that these datasets target different tasks and do not provide the same annotation types or comparable sensor fidelity.
  • Open-loop end-to-end evaluation: The end-to-end benchmark evaluates open-loop trajectory prediction, despite maps being validated through closed-loop Autoware driving trials.The released artifacts enable closed-loop evaluation in the Autoware simulator, but such experiments are left for future work.

6 Conclusion · A Details on the Sensor Setup

KITScenes Multimodal combines a state-of-the-art synchronized sensor suite with the most complete public HD maps, covering 62 km^2 and validated through closed-loop autonomous-driving trials. Its four benchmarks expose systematic capability gaps in current methods, while the appendix details the camera, lidar, radar, and GNSS/INS configurations.

  • 6 Conclusion: Four benchmarks target online HD map construction, long-range depth estimation, novel view synthesis, and end-to-end driving.Together, they surface capability gaps involving complete Lanelet2-fidelity map prediction, long-range depth, geometrically consistent views, and map-grounded trajectory evaluation.
  • 6 Conclusion: Current state-of-the-art methods leave systematic capability gaps that prior datasets cannot expose in cluttered European urban scenes.The gaps span map prediction, long-range depth, novel-view geometry, and map-grounded trajectory evaluation.
  • A Details on the Sensor Setup: The sensor-setup appendix describes the recording vehicle through Tables 7–10 and illustrates it with Figure 10.The tables cover cameras, lidar, radar, and GNSS/GNSS-INS components.
  • A Details on the Sensor Setup: The camera setup uses Lucid Vision Labs cameras with low-distortion Fujinon CF8ZA-1S-23M lenses and 23 Mpx maximum resolution.These specifications are summarized in Table 7.
  • A Details on the Sensor Setup: The lidar suite comprises four groups using Velodyne VLS128-AP, Hesai OT128, Hesai XT32, and Seyond Falcon K1 units.The groups provide 128, 128, 32, and 150 lines, respectively, with maximum ranges of 245 m, 230 m, 120 m, and 500 m.
  • A Details on the Sensor Setup: The lidar groups have distinct fields of view, resolutions, wavelengths, effective point rates, and return modes.The listed wavelengths are 905 nm for the first three groups and 1550 nm for the Seyond Falcon K1; the top lidar was improved in December 2025.
  • A Details on the Sensor Setup: The radar setup uses three Continental ARS548 RDI 4D imaging radars, while GNSS/GNSS-INS combines two receivers with independent antennas.These configurations are specified in Tables 9 and 10.

A.1 Sensor Data Processing and Privacy · B Calibration Details

The dataset preserves high-fidelity multimodal sensor information through raw-image processing, lidar echo and non-return retention, radar and redundant localization, while applying privacy-compliant processing. Calibration combines reference-based camera modeling, standstill-frame ICP registration, and reprojection-based lidar–camera alignment for accurate sensor geometry.

  • A.1 Sensor Data Processing and Privacy: Raw Bayer images undergo AMaZE debayering with chromatic aberration correction, then JPEGLI compression using 4:4:4 subsampling at Q95.The pipeline targets visually lossless quality at manageable file sizes.
  • A.1 Sensor Data Processing and Privacy: Privacy processing is applied to comply with European privacy regulations.The supplied passage introduces this compliance step but is truncated before specifying the processed content.
  • A.1 Sensor Data Processing and Privacy: The 360° main lidar preserves non-return information for occupancy tasks, while Hesai lidars provide dual echoes and Seyond lidars provide return elongation.Dual echoes support information about reflective surfaces and adverse-weather conditions.
  • A.1 Sensor Data Processing and Privacy: Three 4D imaging radars add Doppler velocity measurements and resilience under adverse weather, while redundant GNSS and GNSS-INS provide localization for map validation and SLAM.The localization reference is described as high-accuracy.
  • A.1 Sensor Data Processing and Privacy: Redundant RTK GNSS position data are fused in a modified KISS-SLAM to obtain reprojection-level accuracy for georeferenced 6-DoF poses.The modified KISS-SLAM version is planned for publication.
  • B Calibration Details: Hardware-triggered global-shutter cameras and low-distortion lenses facilitate calibration using checkerboard targets, a reference camera model, and a subpixel-accurate pinhole fit.The calibration estimates both intrinsic and extrinsic camera parameters.
  • B Calibration Details: Calibration refinement focuses on angular error, which dominates at long perception ranges and is best observed using far-range natural surroundings.Mechanical construction already constrains sensor translation and orientation to a few degrees and centimeters.
  • B Calibration Details: Standstill frames avoid motion artifacts during joint lidar-to-lidar ICP calibration and radar registration to the joint lidar point cloud.One reference frame is selected per standstill phase.

C Additional Sample Data Visualization

The section provides additional sample visualizations to offer further insight into the dataset’s sensor and annotation data.

  • C Additional Sample Data Visualization: Additional sample visualizations illustrate the provided sensor and annotation data.These visualizations are presented in Figure 11.

D Data Collection Routes and Conditions · E Annotation Protocol and Quality Control · F Closed-loop autonomous driving map verification trials

KITScenes uses human-driven, manually selected routes to capture diverse traffic and map conditions while prioritizing annotation and localization quality. Its maps are built through complementary annotation passes, worker rotation, automated and relational checks, temporal validation, and closed-loop Autoware trials.

  • D Data Collection Routes and Conditions: Trained operators human-drove the vehicle, while scenes were manually selected to balance annotation and localization quality with diverse traffic scenarios and map layouts.
  • E Annotation Protocol and Quality Control: Annotation was completed by an in-house team within 10 000 total working hours, approximately 160 hours per km2.
  • E Annotation Protocol and Quality Control: Sensor recordings from 2025/26 were validated frame-by-frame against 2023-2024 imagery to detect map changes and adapt or exclude affected content from the release.
  • E Annotation Protocol and Quality Control: Road-level geometry, markings, topology, crosswalks, and BEV traffic-light and sign positions were annotated from geo-referenced aerial imagery, with crowdsourced street-level imagery resolving ambiguities.
  • E Annotation Protocol and Quality Control: Annotators rotated between geometric drafting, attribute classification, and topology linking, providing implicit cross-validation and reducing systematic per-annotator artifacts.
  • E Annotation Protocol and Quality Control: Quality control combined Lanelet2 core-logic validators, geometric and point/line integrity checks, relational and topological completeness checks, and an aerial-image polyline-attribute class check.
  • F Closed-loop autonomous driving map verification trials: Closed-loop driving trials followed test-suite and simulation-based verification, validating HD-map compatibility with Autoware as the reference open-source platform.
  • F Closed-loop autonomous driving map verification trials: Lanelet2 map statistics separately cover road-surface elements represented as 3D polylines and 3D traffic lights, signs, and other lines.

G Extended Dataset Statistics … H.3 Novel View Synthesis

The paper defines geographically separated evaluation splits to prevent map-prior leakage, then specifies benchmark setups for online HD mapping, long-range depth estimation, and novel-view synthesis. The NVS evaluation combines lateral traffic-sign recall with photometric crossmount and temporal-interpolation protocols.

  • G.1 Splits: Geographic overlap between training and validation can enable map-prior memorization and inflate HD map construction performance.
  • G.1 Splits: Validation and test scenarios are assigned to manually selected complex-road polygons with no scenario-pair overlap within 100 m of test or 70 m of validation poses.
  • G.1 Splits: The test-e2e split provides only local-frame poses, without maps or post-keyframe future poses and sensor data, for held-out end-to-end driving evaluation.
  • H.1 Online HD Map Construction: The online HD map benchmark uses unseen map areas, converts Lanelet2 maps into polyline instance graphs, and evaluates 120 of 220 traffic-sign classes.
  • H.1 Online HD Map Construction: Map elements are scored with Average Precision at Chamfer thresholds of 0.5 m, 1.0 m, and 1.5 m, while topology uses Hungarian matching and edge-based topology AP.
  • H.2 Long-range Monocular Depth Estimation: Long-range monocular depth methods use the 16.2 Mpx front-facing camera, while ground-truth depth fuses motion-compensated lidar over a ±1 s temporal window and projects it at 2× super-resolution.
  • H.3 Novel View Synthesis: NVS evaluation measures visible traffic-sign recall at 280×518 and 1600×2844 resolutions, alongside held-out crossmount and ego-trajectory photometric benchmarks for spatial extrapolation and temporal interpolation.

H.4 End-to-End Driving

The end-to-end driving evaluation uses 200 non-overlapping 9-second samples with 4 seconds of past observation and up to 5 seconds of future trajectory, reporting 3-second headline results against map-grounded safety and maneuver-quality metrics. Zero-shot comparisons show navigation-conditioned models generally outperform navigation-free Epona under MMS, while Epona achieves lower positional errors but is penalized for instruction-following.

  • Evaluation setup: The benchmark contains 200 non-overlapping 9-second windows with 4 seconds of past observation and up to 5 seconds of future trajectory, anchored at keyframes.Samples come from validation and overlap-train-validation scenes; headline results use a 3-second horizon, with a full 5-second protocol offered as a long-horizon challenge.
  • Evaluation metrics: Map-grounded evaluation measures drivable-surface survival, centerline distance, and collision-free rate using Lanelet2 polygons, drivable centerlines, and lidar-derived occupancy plus dynamic-agent boxes.The trajectory is evaluated at 10 Hz with the ego footprint oriented along each predicted heading.
  • Models and protocol: The study zero-shot evaluates UniAD, DMAD, SSR, and Epona without KITScenes Multimodal fine-tuning, contrasting navigation-conditioned multi-camera models with navigation-free front-camera Epona.Predicted 3-second trajectories are linearly extrapolated to the 5-second MMS evaluation horizon.
  • Results: All map-grounded safety and lane-compliance metrics degrade sharply beyond the 3-second headline horizon.Figure 20 profiles drivable-surface survival, strict topological lane membership, collision-free rate, and centerline-tracking error over the prediction horizon.
  • Results: Under MMS, navigation-conditioned UniAD, DMAD, and SSR generally rank ahead of navigation-free Epona, whose instruction-following disadvantage offsets its lower positional errors.MMS rewards alignment with the best admissible maneuver under joint similarity, comfort, instruction-following, and collision criteria, rather than proximity to only the recorded trajectory.

I Compute Resources

All models were trained and evaluated using 16 Nvidia A6000 Ada GPUs.

  • I Compute Resources: 16 Nvidia A6000 Ada GPUs were used to train and evaluate all models.
Loading 2606.02956v1…