Source-linked AI summary
Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task
Xiaoqing Ye, Mao Shu, Hanyu Li, Yifeng Shi, Yingying Li, Guangjie Wang, Xiao Tan, Errui Ding
TL;DR
Existing autonomous-driving perception datasets mainly use vehicle-mounted sensors and frontal views, leaving roadside perception underrepresented despite roadside cameras’ occlusion and long-time prediction advantages. This paper introduces Rope3D, a high-diversity roadside 3D dataset and benchmark, and adapts monocular 3D detection methods to its viewpoint and camera ambiguities.
Problem
Existing autonomous-driving perception datasets mainly use vehicle-mounted sensors and frontal views, leaving roadside perception underrepresented despite roadside cameras’ strengths against occlusion and for long-time event prediction.
Method
The paper constructs Rope3D with joint 2D-3D annotations, establishes roadside 3D detection metrics and an evaluation benchmark, and adapts monocular 3D detection methods using ground-plane geometry constraints.
Results
Rope3D contains 50k images and over 1.5M 3D objects across varied lighting, weather, road scenes, camera specifications, and viewpoints.
Takeaways & Limitations
Rope3D provides a dedicated high-diversity benchmark for roadside monocular 3D perception and draws attention to roadside views for autonomous driving.
Abstract
from arXiv · showhide
Concurrent perception datasets for autonomous driving are mainly limited to frontal view with sensors mounted on the vehicle. None of them is designed for the overlooked roadside perception tasks. On the other hand, the data captured from roadside cameras have strengths over frontal-view data, which is believed to facilitate a safer and more intelligent autonomous driving system. To accelerate the progress of roadside perception, we present the first high-diversity challenging Roadside Perception 3D dataset- Rope3D from a novel view. The dataset consists of 50k images and over 1.5M 3D objects in various scenes, which are captured under different settings including various cameras with ambiguous mounting positions, camera specifications, viewpoints, and different environmental conditions. We conduct strict 2D-3D joint annotation and comprehensive data analysis, as well as set up a new 3D roadside perception benchmark with metrics and evaluation devkit. Furthermore, we tailor the existing frontal-view monocular 3D object detection approaches and propose to leverage the geometry constraint to solve the inherent ambiguities caused by various sensors, viewpoints. Our dataset is available on https://thudair.baai.ac.cn/rope.
1. Introduction
Roadside perception addresses blind spots and limited range in vehicle-mounted sensing, but existing roadside research has largely focused on 2D tasks. Rope3D introduces a large, diverse roadside 3D dataset and adapts monocular 3D detection methods for this setting.
- Motivation: Roadside cameras mounted above ground provide longer-range, longer-duration observations with greater robustness to occlusion than vehicle-mounted sensors.Vehicle-mounted systems have limited perceptual range and are vulnerable to occlusion, whereas roadside cameras observe scenes from elevated positions.
- Motivation: Roadside perception can extend autonomous vehicles’ spatial and temporal field of view, expose otherwise hidden regions, and support global trajectory prediction.The introduction identifies cooperative safety, global perception, and cost efficiency as motivations for roadside sensing.
- Research gap: Existing roadside perception research mainly addresses 2D detection and tracking, leaving roadside 3D localization underexplored.Vehicle-view datasets such as KITTI, nuScenes, A*3D, and Waymo were not designed specifically for roadside 3D perception.
- Contributions: Rope3D contains 50k images and over 1.5M 3D objects spanning varied lighting, weather, road scenes, camera specifications, and viewpoints.The dataset is described as the first challenging, high-diversity roadside dataset for this task.
- Contributions: The authors tailor frontal-view monocular 3D detection methods to roadside data and introduce metrics designed for roadside 3D detection.The study aims to address viewpoint and camera-related ambiguities in roadside perception.
2. Related work
Prior autonomous-driving datasets primarily support vehicle-view perception, while roadside data introduce a distinct viewpoint and broader environmental variation. Monocular 3D detection methods are likewise mainly designed for vehicle-view data, motivating roadside-specific benchmarks and adaptations.
- Frontal-view autonomous-driving datasets: Established autonomous-driving datasets provide large-scale multimodal or 3D annotations, but are primarily organized around vehicle-view sensing.Examples include KITTI, H3D, nuScenes, Argoverse, and Waymo, covering tasks such as detection, tracking, and motion forecasting.
- Monocular 3D object detection: Monocular 3D detection predicts 3D bounding boxes from a single image and includes anchor-based methods that estimate offsets from predefined 3D anchors.M3D-RPN is cited as an anchor-based method using geometric constraints between 2D and 3D perspectives.
- Roadside data diversity: Rope3D samples include clear or sunny, rainy, night, and dawn/dusk conditions, reflecting environmental diversity in roadside data.Figure 2 presents these conditions in separate columns.
- Dataset comparison: Table 1 compares front-view and roadside 3D autonomous-driving datasets using attributes such as LiDAR channels and 2D-box annotation counts.The caption also distinguishes datasets with unknown information and datasets providing projected corners rather than complete 3D boxes.
- Monocular 3D object detection: Existing monocular 3D detection methods are mainly designed for vehicle-view data, leaving a domain gap for roadside settings.The related-work discussion identifies roadside data as a different distribution from the data used by existing methods.
3. The Roadside Perception Dataset
Rope3D combines roadside cameras with vehicle-mounted LiDAR to build a diverse, jointly annotated 3D perception dataset. Its collection spans varied sensors, environments, categories, depths, and difficult occlusion conditions.
- Data acquisition and calibration: Roadside images are paired with LiDAR point clouds through sensor synchronization, calibration, and transformations among world, camera, and LiDAR coordinates.The pipeline also estimates camera intrinsics and the ground-plane equation for reliable 2D-3D annotation.
- Data acquisition and calibration: More than 50k image frames are collected across lighting, weather, traffic-density, scene, viewpoint, and camera-setting variations.The dataset uses roadside cameras while a LiDAR-equipped vehicle parks or drives around the same scene.
- Joint annotation: 3D boxes are annotated on point clouds, projected into images, and adjusted to align with 2D instances; objects lacking LiDAR points receive complementary 2D-only labels.For scanned objects, 2D boxes are derived from the amodal projections of the eight 3D box corners.
- Category design: The dataset organizes traffic elements into coarse and fine-grained categories, including Car, Big Vehicle, Pedestrian, Cyclist, and their subdivisions.The fine-grained scheme distinguishes car and van, truck and bus, and several cyclist-related classes.
- Statistics and analysis: More than half of the objects are partially or heavily occluded, compared with 5% to 30% occlusion in KITTI.This distribution is reported as evidence of the dataset’s difficult 3D perception setting.
- Statistics and analysis: Roadside-camera diversity introduces inherent ambiguity through differing camera specifications, mounting heights, and viewpoints.The dataset analyzes camera mounting heights, pitch angles, and focal lengths as dimensions of this diversity.
4. Task and Metrics
The paper defines roadside monocular 3D detection as localization under ambiguous camera settings and introduces metrics that separately assess localization, orientation, size, and ground geometry. It also incorporates camera-ground geometry through ground-plane depth features to reduce ambiguity.
- Task Overview: Roadside monocular 3D detection localizes objects from images captured with varying camera specifications, viewpoints, and mounting positions.
- Evaluation Metrics: The benchmark uses AP and similarity metrics for ground-center distance, orientation, projected area, and four ground-point distances.
- Evaluation Metrics: AGD averages distances between four 3D-box ground vertices, consolidating location, orientation, and width/length.
- Evaluation Metrics: AGS measures ground-point similarity using the average normalized distance between four predicted and ground-truth ground points.
- Evaluation Metrics: RopeScore combines 3D AP with the averaged similarity score S using weights ω1 =8 and ω2 =2.
- Geometry-Aware Adaptation: Ground-plane equations derive depth from image coordinates and camera geometry, enabling early or deep fusion of ground depth with RGB features.
5. Experiments
Experiments evaluate monocular 3D detectors under homologous and heterologous splits, showing that camera-domain variation harms generalization while ground-plane and depth adaptations improve performance. Far-range detection remains difficult, especially at 90–120 meters.
- Experimental Setup: The dataset uses an 8:2 training-validation split and homologous or heterologous validation to assess generalization across scenes and camera specifications.
- Main Results and Analysis: Most methods show an obvious performance decline from homologous to heterologous validation, indicating that camera specifications and mounting positions create a significant domain gap.
- Main Results and Analysis: Ground-plane depth maps produce obvious improvements for most methods, including on the heterologous validation set.
- Main Results and Analysis: Detection performance decreases most at 90–120 meters because distant objects occupy small image regions and have fewer 3D annotations due to occlusion.
- Main Results and Analysis: MonoFlex-(G) performs better on cyclists and pedestrians but worse on motor vehicles.
6. Conclusion
The paper presents Rope3D as a high-diversity roadside monocular 3D perception dataset and adapts existing detectors to its viewpoint and camera ambiguities. It also restricts use to protect privacy and notes that further adaptation study is needed.
- Rope3D is a high-diversity roadside dataset with joint 2D–3D annotations designed specifically for roadside 3D perception.
- Existing monocular 3D detectors are tailored to Rope3D because its viewpoint and camera specifications create inherent ambiguity.
- Images are time-discrete and cannot be used for tracking, while sensitive information is masked to prevent illegal surveillance.
- Further study is needed to adapt concurrent vehicle-view 3D detection approaches.
A.1. Detailed Dataset Analisis
Rope3D has high object density and diverse object geometry across roadside scenes. Its size statistics support category-specific initialization and size prediction, while orientation distributions remain broad because of camera and scene diversity.
- Object Density: Rope3D has high average 3D-object density across all coarse-level classes compared with other datasets.
- Size and Orientation: Size analysis covers cars and big vehicles because non-motor categories usually have similar sizes.
- Size and Orientation: High-frequency orientations are not concentrated at a single peak because camera specifications and scenes vary.
- Fine-Grained Categories: Mean and standard-deviation statistics for fine-grained categories can define mean sizes and disturbance ranges for size prediction.
A.2. More samples of the Rope3D Dataset.
The section presents additional Rope3D roadside samples spanning varied weather, collection times, and object densities.
- Additional roadside samples are visualized in Figure 10.
- The examples cover different weather conditions.
- The examples also vary in collection time and object density.
B. Additional Experiments
Additional experiments evaluate pedestrian and cyclist performance across homologous and heterologous splits, fine-grained categories, and depth-based improvements. Incorporating ground-plane depth yields consistent gains across the evaluated baselines.
- Additional Experiments: The experiments use homologous and heterologous validation sets, with the latter containing unseen scenes and possibly different camera specifications.
- Additional Experiments: The fine-grained evaluation distinguishes cars, vans, trucks, buses, pedestrians, cyclists, motorcyclists, and tricyclists.
- Additional Experiments: Ground-plane depth is used to alleviate ambiguity caused by different camera specifications.
- Additional Experiments: Depth information is incorporated either by RGB-depth concatenation or by siamese depth-feature extraction with weighted fusion.
- Additional Experiments: Consistent performance gains are observed across all baselines when comparing vanilla and improved approaches.