Source-linked AI summary
Resource-Efficient 3D Shape and Orientation Estimation of Road Users using Roadside RGB Camera and 3D Radar
Shiva Agrawal, Savankumar Bhanderi, Gordon Elger
TL;DR
Roadside sensor geometry varies across locations, limiting end-to-end models that depend on site-specific spatial configurations. The paper proposes a two-stage camera-radar estimator using radar positions, size templates, and neural yaw prediction; it reports real-time operation and comparative gains in its evaluated setting.
Problem
Roadside installations vary in height, tilt, and field of view, while end-to-end fusion models depend on site-specific geometry and 3D ground truth.
Source: §I · p. 1Method
A two-stage pipeline uses camera detections and associated radar points, then estimates 3D boxes with radar positions, predefined size templates, and a lightweight yaw network.
Source: §I · p. 1, 2) Category-dependent processing · pp. 4–5Results
The system runs at 25 FPS on an NVIDIA Jetson AGX Orin and reports significant performance gains over camera-only homography baselines.
Source: §VI · p. 7Takeaways & Limitations
The evaluated framework provides a practical edge-oriented approach for estimating 3D shape and orientation of pedestrians, bicycles, and cars in roadside camera-radar setups.
Source: §VI · p. 7Evaluation is limited to a single roadside setup, and fixed template bounds cannot capture fine-grained size variation across vehicle subclasses.
Source: §VI · p. 7
Abstract
from arXiv · showhide
End-to-end deep learning pipelines for camera-radar 3D object detection are well established in driver assistance systems and autonomous vehicles; however, adapting them to roadside-mounted infrastructure presents significant challenges. On vehicles, sensor heights, pitch angles, and relative geometries remain largely uniform, allowing end-to-end models to generalize effectively. On the contrary, roadside sensor installations vary drastically in height, tilt, and field of view across locations, rendering end-to-end models prone to failure and difficult to adapt without massive location-specific datasets. Hence, instead of designing a complex end-to-end pipeline for direct 3D bounding box estimation, a low-cost, resource-efficient two-stage approach is proposed for real-time 3D shape and orientation estimation. The first stage utilizes any off-the-shelf, state-of-the-art method to extract 2D camera bounding boxes and associated 3D radar points for each road user. The proposed second stage then processes these inputs to construct 3D oriented bounding boxes where radar data determines 3D position, predefined templates define size for each road user, and a tiny neural network predicts a site-specific and optimized yaw angle. The algorithm is successfully validated at a curved road with a junction for pedestrians, bicycles, and cars, where it achieves 25 frames per second on an NVIDIA Jetson ORIN for real-time applications.
I. INTRODUCTION
Sparse radar-camera detection often provides road users’ locations and categories without the shape and orientation needed for spatial understanding. The paper proposes a resource-efficient two-stage estimator and validates it for pedestrians, bicycles, and cars.
I. INTRODUCTION
- Sparse radar-camera fusion often detects road users as point objects, omitting shape and orientation needed for localization and tracking.Three-dimensional oriented boxes represent size, position, and orientation.Source: §I · p. 1
- Prior lidar methods offer precise 3D reconstruction but are costly, while monocular and end-to-end camera-radar methods rely on substantial computation or 3D labels.These requirements complicate roadside deployment and real-time operation.Source: §I · p. 1
- Roadside installations vary in height, tilt, and field of view, making end-to-end models tied to one site difficult to generalize across locations.The authors therefore favor lightweight algorithms that can adapt to new sensor mountings.Source: §I · p. 1
- The proposed two-stage approach uses camera boxes and associated radar points, then constructs 3D boxes using radar position, predefined size templates, and a tiny yaw-prediction network.The method is validated for pedestrians, bicycles, and cars at a curved junction with a traffic light crossing.Source: §I · pp. 1–2
A. Key Contributions
- The stated contributions include size-template ablations and quantitative evaluation against lidar ground truth for position, size, and yaw.The contribution list also names comparative benchmarking against a camera-only homography approach.Source: A. Key Contributions · p. 2
II. RELATED WORK
Prior camera, radar, and lidar methods offer useful geometric or spatial cues, but face limits including monocular depth, radar sparsity, and mismatches between lidar-oriented processing and radar data. This work combines camera geometry, templates, sparse radar, and site-specific yaw prediction; its measurement setup uses synchronized and calibrated roadside sensors.
II. RELATED WORK
- Camera-only methods use image geometry, calibration, vanishing points, orientation networks, or known-size templates, but positional accuracy deteriorates with distance.The cited approaches include,,,, and.Source: §II · p. 2
- Radar provides direct range and velocity, but sparsity, multipath, angular noise, clutter, and methods tailored to reflective cars constrain its use.Traditional geometric strategies include convex hull extraction, L-shape fitting, and template matching [25],.Source: §II · p. 2
- Lidar architectures achieve high 3D localization accuracy from dense point clouds, but their fine spatial structures do not directly transfer to sparse, noisy radar targets.The cited lidar methods are –.Source: §II · p. 2
- The proposed framework combines camera projection and templates with sparse radar for positioning and sizing, while a site-optimized tiny network predicts yaw for bicycles and cars on curved roads and junctions.The authors state that classical geometry struggles to estimate those yaw angles reliably.Source: §II · p. 2
III. MEASUREMENT SETUP
- The roadside setup includes an RGB camera, 3D automotive radar, and lidar; the camera and radar estimate boxes, while lidar validates outputs and trains yaw prediction.Nearest synchronized frames are collected at 10 Hz, with intrinsic and extrinsic sensor calibration.Source: §III · pp. 2–3
- The sensor diagram distinguishes the coordinate setup from its transformations, including radar-to-camera, lidar-to-camera, radar-to-lidar, and lidar-to-ground extrinsics.K denotes the camera intrinsic matrix; P and ˜P denote standard and homogeneous projection matrices.Source: Figure 1 · p. 2
IV. METHODOLOGY
The framework separates instance association from 3D geometry estimation to avoid relying on an end-to-end model for direct 3D boxes. Its objective is to estimate an oriented-box state for each detected road user from image and sparse radar inputs.
IV. METHODOLOGY
- The two-stage framework separates instance association from geometry estimation, using image detections and radar associations before estimating 3D boxes.Stage 1 is independent of ground-plane embeddings and location-specific camera-radar calibration priors; Stage 2 uses radar-derived positions, size templates, and site-specific geometric constraints.Source: §IV · p. 3, Figure 2 · p. 3
A. Problem Formulation
- The input consists of a synchronized monocular image and sparse radar points, and the objective is an oriented 3D box for each road-user instance.The box state includes a centroid, extent dimensions, yaw, radial Doppler velocity, detection confidence, and category.Source: A. Problem Formulation · p. 3
B. Stage 1: 2D Camera Detection and 3D Radar Association
Stage 1 produces per-instance 2D detections associated with radar point subsets. Stage 2 builds 3D boxes from these associations, combining radar-derived quantities with category-specific estimates.
B. Stage 1: 2D Camera Detection and 3D Radar Association
- In the evaluated Stage 1, YOLOv11 extracts 2D boxes and DBSCAN associates radar points; empty or unassociated clusters fall back to monocular homography depth estimates.The confidence threshold is si ≥ 0.5, and the output includes each instance’s box, category, score, and radar subset.Source: B. Stage 1: 2D Camera Detection and 3D… · p. 3
C. Stage 2: 3D Shape and Orientation Estimation
- Stage 2 derives centroid and mean Doppler velocity from radar, estimates height from the image box, and assigns ground extents from class-specific templates.Lightweight networks predict yaw for bicycles and cars; pedestrians receive a neutral 0° orientation because their spatial profiles are symmetric.Source: C. Stage 2: 3D Shape and Orientation… · pp. 3–4
1) Common Preprocessing:
Common preprocessing derives spatial and kinematic parameters from radar clusters and estimates object height by projecting image-box corners into ground coordinates using radar-provided depth.
Common Preprocessing:
- The 3D target centroid is the mean of its radar-point cluster, and mean radial Doppler velocity is computed across the same cluster.Source: 1) Common Preprocessing · p. 4
- Radar supplies the metric depth scale through the cluster’s minimum longitudinal coordinate, enabling inverse projection of image-box corners into 3D radar and ground coordinates.The camera projection matrix and radar-to-lidar and lidar-to-ground transforms are used for the coordinate conversions.Source: 1) Common Preprocessing · p. 4
- Object height is computed from the vertical difference between transformed top-edge and bottom-edge corners of its 2D image bounding box.The four image-box corners are represented in homogeneous pixel coordinates before projection.Source: 1) Common Preprocessing · p. 4
2) Category-dependent processing:
Category-dependent processing completes the object state with orientation and ground dimensions tailored to each road-user class. Pedestrians use a fixed orientation, while bicycles and cars use bounded dimensions and learned yaw estimation.
Category-dependent processing:
- Pedestrians are assigned zero yaw and an initial square footprint; separate empirical scaling factors calibrate length and width for perspective distortion.The initial side length is derived from the projected horizontal image-box corners.Source: 2) Category-dependent processing · p. 4
- Sparse bicycle radar returns make geometric size and yaw heuristics unreliable, so bicycle dimensions are bounded by templates and yaw is regressed by a lightweight multimodal network.For targets with one radar point, minimum length and width are used; with at least two, cluster extents are bounded by the template limits.Source: 2) Category-dependent processing · pp. 4–5, Figure 3 · p. 5
- Car dimensions use bounded template formulations, followed by a line-of-sight centroid shift toward visible facets to align boxes with sparse surface reflections.Source: 2) Category-dependent processing · p. 5
- Car yaw uses the YawPredictionNet architecture with its final 256-unit hidden linear layer omitted, while the general state vector includes class-specific orientation and ground dimensions.Source: 2) Category-dependent processing · pp. 4–5
V. EXPERIMENTAL RESULTS AND DISCUSSIONS
The final 3D oriented-box evaluation uses an independent, sequence-disjoint set of 500 frames, with more than 1,100 road-user instances across a 0–50 m sensor range.
V. EXPERIMENTAL RESULTS AND DISCUSSIONS
- 500 independent frames, excluded from parameter tuning, form the final 3D oriented-box evaluation set, containing 451 persons, 360 bicycles, and 299 cars.Source: §V · p. 5
- Sequence-disjoint segments, rather than random frame sampling, are used to prevent temporal data leakage between dataset splits.Source: §V · p. 5
- The evaluation covers road users 0 to 50 m from the sensor and includes over 1,100 dynamic instances, with frame count constrained by labor-intensive dense-point-cloud annotation.Source: §V · p. 5
A. Training of YawPredictionNet
YawPredictionNet models for bicycles and cars are trained with AdamW and a shared sine-cosine regression loss. Their small parameter counts and limited training samples support the paper’s description of the models as deployment-friendly.
A. Training of YawPredictionNet
- Both models use AdamW with initial learning rate 10^-3, batch size 1, gradient clipping at 0.5, and a ReduceLROnPlateau scheduler.Training optimizes the sum of mean-squared errors for sin ψ and cos ψ.Source: A. Training of YawPredictionNet · p. 5
- Bicycle yaw training runs for 400 epochs on 262 training and 66 validation samples, using 73,442 trainable parameters.The bicycle model uses weight decay 0.3.Source: A. Training of YawPredictionNet · p. 5
- Car yaw training runs for 100 epochs on 329 training and 76 validation samples, using 40,162 trainable parameters.The car model uses weight decay 0.2.Source: A. Training of YawPredictionNet · p. 5
B. Ablation Experiments
The ablations select dimension parameters for pedestrians, bicycles, and cars by evaluating errors against ground truth across the YawPredictionNet training dataset. The resulting configurations are class-specific, and the parameter search uses manual rather than alternative optimization methods.
B. Ablation Experiments
- Pedestrians: Exp. 4 (LSF = 1.75, WSF = 1.25, HSF = 1.00) is optimal, with length, width, and height errors of (0.11, 0.09) m, (0.07, 0.06) m, and (0.20, 0.11) m.The length comparison starts from a square footprint assumption; the selected height factor retains the best height error.Source: B. Ablation Experiments · p. 6, Table I · p. 6
- Bicycles: Exp. 6 is optimal, using a 2.0/2.4 m length range, a 0.8/1.0 m width range, and HSF = 1.00.These settings yield length, width, and height errors of (0.10, 0.09) m, (0.05, 0.04) m, and (0.10, 0.08) m; scaling height increases error.Source: B. Ablation Experiments · p. 6, Table II · p. 6
- Cars: Exp. 12 gives the best trade-off, using 4.4/4.9 m length limits and 1.8/2.0 m width limits, while HSF = 1.00 avoids the accuracy loss from height scaling.The selected length limits yield xc error (0.38, 0.29) m; the width limits yield width error (0.15, 0.07) m and yc error (0.55, 0.42) m, which absorbs the sensor-facing edge-alignment shift.Source: B. Ablation Experiments · p. 6
- The dimension-factor search is manual; Bayesian optimization, grid or random search, and multi-objective evolutionary algorithms are outside this work’s scope.Errors are computed against ground truth over the dataset used to train YawPredictionNet.Source: B. Ablation Experiments · p. 6
C. Comparison with only camera-based algorithm
Against two camera-only homography baselines, the proposed approach improves centroid and yaw estimation across the reported object classes. It runs at 25 FPS on an NVIDIA Jetson AGX Orin; comparisons with end-to-end deep learning frameworks are omitted.
C. Comparison with only camera-based algorithm
- B1 estimates yaw with axis-aligned boxes, while B2 uses road-geography lookup tables; both estimate centroid and height by projecting 2D box points with homography.The baselines use the 2D box bottom-center rather than radar returns.Source: C. Comparison with only camera-based… · p. 6
- Across object classes, the approach outperforms both camera-only baselines; for cars, xc error falls from (2.67, 2.36) m in B1/B2 to (0.38, 0.29) m, and yaw error falls from (5.88, 4.30)° in B2 to (3.68, 3.65)°.The paper attributes the gains to sparse radar point clouds providing spatial cues that address longitudinal depth ambiguity in homography-only vision.Source: C. Comparison with only camera-based… · p. 6, Table IV · p. 7
- The proposed pipeline runs at 25 FPS on an NVIDIA Jetson AGX Orin, targeting real-time edge deployment under constrained computational resources.The paper intentionally omits direct comparisons with end-to-end camera or fusion frameworks because they use different computational paradigms.Source: C. Comparison with only camera-based… · pp. 6–7
VI. CONCLUSION
The paper presents a two-stage roadside camera-radar framework for real-time 3D shape and orientation estimation, with reported gains over camera-only homography baselines. Its evaluation is limited to a single roadside setup, and fixed templates cannot capture fine-grained size variation across vehicle sub-classes.
VI. CONCLUSION
- The framework uses sparse radar points for 3D position, bounded dimension templates for object scale, and a lightweight neural network for site-specific yaw prediction for bicycles and cars.The reported system operates at 25 FPS on an NVIDIA Jetson AGX Orin and is evaluated against camera-only homography baselines.Source: §VI · p. 7
- The work covers a single roadside setup, and fixed template bounds cannot represent fine-grained size variation across vehicle sub-classes.Future work will assess cross-site transfer, dynamic templates, and support for heavy vehicles and motorbikes.Source: §VI · p. 7