PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection

Lei Yang, Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Li Wang, Xinyu Zhang, Peng Chen

arXiv:2303.08498v2cs.CV

TL;DR

Roadside cameras can extend perception beyond ego-vehicle sensors, but depth-based vision-centric methods face shrinking car–ground depth differences with distance. BEVHeight predicts per-pixel height instead, and reports gains over BEVDepth on clean roadside benchmarks and under camera-parameter changes.

  • Problem

    Roadside perception is less explored than ego-vehicle sensing, while depth-based methods face shrinking car–ground depth differences with distance and challenges from variable camera poses.

    Source: §1 · p. 2

  • Method

    BEVHeight predicts categorical per-pixel height instead of depth, projects contextual features into height intervals, and uses voxel pooling and a detection head.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    BEVHeight surpasses BEVDepth by 4.85% on DAIR-V2X-I and 4.43% on Rope3D under clean settings, and by 26.88% under robust settings.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Height-based prediction provides a roadside-camera alternative that retains an advantage over BEVDepth under external camera-parameter changes.

    Source: §1 · p. 2

    On ego-vehicle settings, BEVHeight falls behind BEVDepth by around 0.02 in mAP metrics.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

While most recent autonomous driving system focuses on developing perception methods on ego-vehicle sensors, people tend to overlook an alternative approach to leverage intelligent roadside cameras to extend the perception ability beyond the visual range. We discover that the state-of-the-art vision-centric bird's eye view detection methods have inferior performances on roadside cameras. This is because these methods mainly focus on recovering the depth regarding the camera center, where the depth difference between the car and the ground quickly shrinks while the distance increases. In this paper, we propose a simple yet effective approach, dubbed BEVHeight, to address this issue. In essence, instead of predicting the pixel-wise depth, we regress the height to the ground to achieve a distance-agnostic formulation to ease the optimization process of camera-only perception methods. On popular 3D detection benchmarks of roadside cameras, our method surpasses all previous vision-centric methods by a significant margin. The code is available at {\url{https://github.com/ADLab-AutoDrive/BEVHeight}}.

1. Introduction

Roadside cameras can extend perception beyond ego-vehicle sensors, but depth-based vision-centric methods face shrinking car–ground depth differences as distance increases. BEVHeight predicts per-pixel height instead, achieving strong roadside benchmark results and improved robustness to camera-parameter changes.

1. Introduction

  • Roadside cameras can extend perception range and help address occlusions that limit ego-vehicle sensors.Two large-scale roadside-camera benchmarks provide evaluation settings for this research.Source: §1 · p. 2
  • As a car moves farther from a roadside camera, the depth difference between its roof and nearby ground shrinks, complicating depth prediction.The authors conjecture that varied camera poses across datasets and sensitivity to changing extrinsic parameters make depth regression difficult.Source: §1 · p. 2
  • BEVHeight predicts categorical per-pixel height, projects contextual features into height intervals in wedgy voxel space, then applies voxel pooling and a detection head.Its height-sampling strategy has an adjustable hyperparameter, and the framework does not require explicit point-cloud supervision.Source: §1 · p. 2
  • BEVHeight surpasses previous vision-centric methods by 5% on traditional roadside settings and improves over BEVDepth by 26.88% under simulated extrinsic changes.Under those changes, BEVDepth drops from 41.97% to 5.49%.Source: §1 · p. 2

2. Related Work

Roadside perception remains less explored than ego-vehicle perception, despite roadside cameras’ longer range and relative robustness to occlusion. BEVHeight addresses roadside camera challenges by using height estimation rather than relying on depth estimation.

2. Related Work

  • Roadside cameras offer longer-range perception and relative robustness to occlusion, but ambiguous mounting positions and variable extrinsics challenge existing models.The section describes roadside perception as comparatively underexplored and notes datasets intended to facilitate it.Source: §2 · p. 2
  • Vision-centric BEV methods include transformer-based query approaches and depth-based methods that predict depth distributions to construct 3D volumetric features.Follow-up depth-based methods use LiDAR supervision or multi-view stereo to improve depth estimation.Source: §2 · p. 2
  • BEVHeight uses height estimation for roadside 3D detection, where complex camera mounting positions and variable extrinsics make depth prediction difficult.The section characterizes BEVHeight as achieving state-of-the-art performance and robustness for roadside detection.Source: §2 · p. 3

3. Method

BEVHeight replaces pixel-wise depth prediction with ground-height prediction for roadside camera-only 3D detection. Its method compares the two representations and describes a height-based pipeline that projects image features into BEV for object detection.

3.1. Problem Definition

  • The task is to predict foreground objects’ 3D bounding boxes from roadside-camera images, assuming camera intrinsic and extrinsic parameters are known after installation.Each box has seven degrees of freedom: location, cuboid dimensions, and yaw.Source: §3.1 · p. 3

3.2. Comparing the depth and height

  • Per-pixel depth spans 0–200 meters, while ground-height ranges from -1 to 2 meters; under extrinsic changes, height distributions overlap more with their clean setting.The authors note that height cannot be lifted to 3D directly like depth, motivating a dedicated projection module.Source: §3.2 · p. 3, Figure 2 · p. 4, Figure 3 · p. 4

3.3. BEVHeight

  • HeightNet predicts per-pixel height bins and context features; an outer product fuses them before projection into wedge-shaped 3D features, voxel pooling into BEV, and detection.Height is discretized into bins, with a dynamic discretization strategy rather than fixed depth-discretization strategies.Source: §3.3 · p. 4, Figure 4 · p. 5
  • The height-based projector uses a virtual coordinate system and a reference plane at fixed depth 1 to map fused features into wedge-shaped volume features.Voxel pooling then transforms the wedge-shaped features into BEV features for the detection head.Source: §3.3 · p. 5, Figure 4 · p. 5

4. Experiments

Experiments evaluate BEVHeight on roadside benchmarks under clean and perturbed camera settings, then examine its discretization, supervision, error, latency, and scope. The reported results show stronger benchmark performance and robustness, alongside a limitation on ego-vehicle settings.

4.1. Datasets

  • DAIR-V2X-I contains around 10,000 images and is evaluated on validation because test examples are unpublished; Rope3D uses a random 70%/30% train/test split.Rope3D contains over 500,000 images from seventeen intersections.Source: §4.1 · p. 6

4.3. Comparing with state-of-the-art

  • On DAIR-V2X-I, margins are 2.19%, 5.87%, and 4.61% for vehicle, pedestrian, and cyclist; on Rope3D, BEVHeight outperforms all listed BEV and monocular methods.The DAIR-V2X-I comparison is on the validation set; the Rope3D table reports vehicle results.Source: Table 1 · p. 6, §4.3 · p. 6, Table 2 · p. 7

4.2. Experimental Settings

  • Results use ResNet-101 for state-of-the-art comparisons and ResNet-50 for other ablations, with 864×1536 inputs and 150-epoch training.Training uses AdamW with an initial learning rate of 2e−4.Source: §4.2 · p. 6

4.3. Comparing with state-of-the-art

  • Under roll/pitch offsets, BEVHeight maintains 51.77% from 63.49%, while BEVDepth falls from 60.75% to 9.48% on vehicle.The evaluation applies random rotational offsets to roll and pitch; the table reports robustness settings.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Ablation Study

  • Dynamic discretization improves average precision by 1% on average over uniform discretization; α=2.0 underperforms α=1.5 in most cases.The table reports the discretization ablation on DAIR-V2X-I.Source: Table 4 · p. 7, §4.4 · p. 7
  • Point-cloud supervision does not help either BEVDepth or BEVHeight overall; stable roadside backgrounds provide inadequate supervision and increase fitting difficulty.BEVHeight shows only a slight improvement under IoU=0.5.Source: §4.4 · p. 7, Table 5 · p. 8
  • Height estimates yield less roadside distance error than depth, while BEVHeight runs at 16.1 FPS / 62ms versus BEVDepth’s 14.7 FPS / 68ms.The paper attributes the latency difference to fewer height bins than depth bins, a slimmer regression head, and fewer pseudo points for voxel pooling.Source: §4.4 · p. 8, Figure 6 · p. 8, Table 6 · p. 9
  • On nuScenes ego-vehicle settings, BEVHeight trails BEVDepth by 2%; truck-camera experiments attribute the behavior to camera height rather than a fixed ground plane.The truck study reports BEVHeight again surpassing the depth-based method.Source: Table 7 · p. 9, §4.4 · p. 9, Table 8 · p. 9

5. Conclusion

BEVHeight replaces depth prediction with height-to-ground estimation for roadside 3D detection. It surpasses BEVDepth on DAIR-V2X-I and Rope3D under clean settings and on robust settings with changed camera parameters.

  • BEVHeight surpasses BEVDepth by 4.85% on DAIR-V2X-I and 4.43% on Rope3D under traditional clean settings, and by 26.88% on robust settings with changed external camera parameters.Source: §5 · p. 9
  • BEVHeight predicts per-pixel height to the ground instead of depth, then projects 2D features into 3D space to improve detection.Source: §5 · p. 9
  • In roadside perception, foreground-background depth differences quickly shrink with camera distance, while per-pixel height remains distance-independent.Source: §5 · p. 9

A.1. Broader Impacts

The proposed approach targets vision-based 3D object detection for roadside perception. Its predictions may be inaccurate, and the authors identify combining height and depth estimation as a direction for extending the approach to autonomous driving scenarios.

  • The method targets vision-based 3D object detection for roadside perception.Source: §A.1 · p. 12
  • Inaccurate predictions may lead to incorrect cooperative-vehicle decisions and potential traffic accidents.Source: §A.1 · p. 12
  • The authors identify combining height and depth estimation as a direction for extending the approach to autonomous driving scenarios.Source: §A.1 · p. 12

A.2. Dynamic Discretization

Dynamic-increasing discretization (DID) adjusts height-bin spacing through a hyperparameter. Larger α places bins more densely near hmin.

  • DID adjusts height-bin spacing through α, with larger α placing bins more densely near hmin.The figure compares uniform, spacing-increasing, linear-increasing, and adjustable dynamic-increasing discretization across [hmin, hmax].Source: Figure 8 · p. 12

A.3. Results on V2X-Sim Dataset

BEVHeight is evaluated on V2X-Sim, a simulation dataset with four surround roadside cameras. It surpasses BEVDepth on vehicle and cyclist detection.

  • BEVHeight surpasses BEVDepth by more than 10.88% on vehicle and 21.15% on cyclist detection in V2X-Sim.Source: §A.3 · p. 12, Table 9 · p. 12
  • The V2X-Sim evaluation uses a simulation dataset containing four surround roadside cameras.Source: §A.3 · p. 12

A.4. Effectiveness on multi depth-based Detectors

BEVHeight improves performance over both BEVDepth and BEVDet across vehicle, pedestrian, and cyclist detection. The section also identifies the view-transformation variants evaluated and the reported object categories.

A.4. Effectiveness on multi depth-based Detectors

  • Against BEVDepth, BEVHeight improves performance by 2.19% on vehicle, 5.87% on pedestrian, and 4.61% on cyclist.Source: §A.4 · p. 12
  • Against BEVDet, BEVHeight improves performance by 8.56% on vehicle, 5.35% on pedestrian, and 8.60% on cyclist.Source: §A.4 · p. 12
  • Table 10 evaluates depth-based detector variants on the DAIR-V2X-I validation set for vehicle, pedestrian, and cyclist detection.Source: Table 10 · p. 12
  • VT denotes view transformation, while D and H denote depth-based and height-based variants, respectively.Source: §A.4 · p. 12

A.5. More Results on DAIR-V2X-I Dataset

On DAIR-V2X-I, BEVHeight outperforms BEVDepth under matched configurations. Additional visualizations report detection at middle and long distances and under extrinsic disturbance.

A.6. More Visualizations

  • Visualizations show BEVHeight detecting objects at middle and long distances on clean images and maintaining accuracy for cars, pedestrians, and cyclists under extrinsic disturbance.Figure 9 includes clean images, 1-degree roll, and simultaneous 1-degree roll-and-pitch disturbance; Figure 10 shows another disturbed scene.Source: §A.6 · pp. 12, 14, Figure 9 · p. 13, Figure 10 · p. 14

A.5. More Results on DAIR-V2X-I Dataset

  • BEVHeight surpasses BEVDepth by 2%–6% on the DAIR-V2X-I benchmark under the same configurations.Table 11 reports vehicle, pedestrian, and cyclist results across difficulty settings; the accompanying text identifies the validation set.Source: §A.5 · p. 12, Table 11 · p. 13
Loading 2303.08498v2…