RLG-TPV: Radar- and LiDAR-Guided Tri-Perspective View Fusion for Camera-Radar 3D Object Detection

1. Introduction

• Camera-only TPV lifting leaves image evidence depth-ambiguous along viewing rays, while radar offers range and occupancy cues but little direct elevation information.LiDAR can supervise camera depth and vertical structure during training without becoming an inference-time sensor.Source: §1 · p. 1
• The ray-guided deformable-attention lift weights sampled image features along projected rays using LiDAR-supervised camera depth probabilities and radar frustum occupancy.Radar also refines the depth distribution before lifting; LiDAR-derived class-occupancy targets supervise the side and front planes during training, with that supervision removed at inference.Source: §1 · pp. 1–2
• RLG-TPV uses RCS-aware spatial scatter to distribute sparse radar evidence over local neighborhoods.The framework incorporates radar range cues and training-time LiDAR geometry according to the spatial information each sensor provides.Source: §1 · p. 2
• Doppler-Guided Temporal Fusion uses measured radar radial velocity to initialize motion alignment for past fused features, with a learned residual and a gate that limits unsupported warping.This lets radar provide measured motion information during temporal fusion as well as geometric guidance during lifting.Source: §1 · p. 2
• 0.4981 mAP and 0.5959 NDS on nuScenes validation, with orientation error reduced 31.9% and velocity error reduced 30.7% versus the published CRN baseline.Controlled lifting experiments identify sample-level geometric guidance as the largest contributor among the investigated lifting variants.Source: §1 · p. 2

Camera-radar view transformation and TPV lifting

• Camera-radar view transformation ranges from associating radar with image-derived object hypotheses to incorporating radar occupancy or depth cues into spatial representations.CRN combines camera depth distribution with radar occupancy in Radar View Transform, while RaCFormer incorporates radar-guided depth estimation in a query-based detector.Source: §2 · p. 2
• TPVFormer uses top, side, and front planes for camera occupancy prediction; RLG-TPV combines multi-plane representation with radar occupancy and LiDAR-supervised depth in deformable-attention sampling.LiDAR-derived occupancy additionally supervises the side and front planes during training, where radar provides limited elevation information.Source: §2 · p. 2

Temporal fusion with radar motion cues

• Prior temporal methods include recurrent, long-term, and object-centric camera fusion, while camera-radar methods aggregate temporal inputs, use predicted velocity, or exploit Doppler implicitly.RLG-TPV uses measured radar Doppler explicitly to guide warping of past fused features, with a learned residual correcting motion not represented by sparse measurements.Source: §2 · p. 2

Training-time LiDAR supervision and radar encoding

• LiDAR supervision has been used for camera depth, camera occupancy, and camera-radar distillation; RLG-TPV instead applies direct depth and side/front class-occupancy targets, with inference using cameras and radar.Source: §2 · p. 2
• RCS-aware scatter extends radar evidence beyond isolated cells, following spatial-support approaches such as RCBEVDet and implementing residual multi-scale spreading controlled by RCS before the radar backbone.The overview places RCS-aware radar features alongside TPV lifting and Doppler temporal alignment in the detection pipeline.Source: §2 · pp. 2–3, Figure 1 · p. 3

3.1. Framework Overview

• Four temporally spaced camera-radar keyframes feed a ResNet50 camera backbone, PointPillars radar encoder, BEV fusion, and CenterPoint detection pipeline.The framework modifies representation construction through multimodal TPV lifting, radar-aware encoding, and Doppler-guided temporal fusion.Source: §3.1 · p. 3

3.2. Ray-Guided Multimodal TPV Lift

• Camera features are sampled along projected rays into top, side, and front planes, weighted by LiDAR-supervised depth probabilities and radar occupancy before attention aggregation.Radar also biases the camera depth distribution before lifting; the same weighting applies across all three planes.Source: §3.2 · p. 4

3.3. Radar Encoding and Doppler-Guided Temporal Fusion

• RCS-conditioned multiscale scatter spreads sparse radar features to local neighbors while retaining the original feature through a residual connection.For temporal fusion, measured radar motion anchors past-frame alignment; a learned residual corrects the field, and gating limits warping where motion lacks support.Source: §3.3 · p. 5

3.4. Cross-Modal Height-Plane Supervision

• LiDAR-derived class-occupancy targets supervise the side and front planes during training, addressing their vertical structure, and the auxiliary heads are removed at inference.Cells without labeled LiDAR returns are excluded from the plane-supervision loss.Source: §3.4 · p. 5

4.1. Experimental Setup

• Experiments use nuScenes’ 700 training and 150 validation scenes, standard mAP/NDS and five error metrics, and 24-epoch ImageNet-pretrained ResNet-50 training.The protocol evaluates mAP over four BEV center-distance thresholds and uses a single deterministic forward pass without test-time augmentation or ensembling.Source: §4.1 · pp. 5–6

4.2. Comparison with Prior Methods

• 0.596 NDS is second-highest overall and highest among ImageNet-initialized 24-epoch models; 0.498 mAP exceeds published CRN (0.490) and HyDRa (0.494).Table 1 compares published configurations on nuScenes validation.Source: Table 1 · p. 6, §4.2 · p. 6
• 0.369 mAOE is the lowest reported, 8.7% below RCBEVDet (0.404) and 31.9% below CRN; velocity error is 0.238 versus CRN's 0.344.Against released CRN, nine of ten classes improve, led by pedestrian (+7.8 AP); motorcycle is the sole regression (−1.3 AP).Source: §4.2 · p. 6

4.3. Analysis of Ray-Guided TPV Lifting

• Per-sample depth and radar occupancy recover approximately 87% of the difference between unguided attention and LSS+RVT, while post-BEV height supervision yields no resolved improvement.The unguided and per-ray variants use different heads, so the comparison bounds rather than isolates guidance granularity; keyframe-only lifting also degrades substantially.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4. Incremental Construction of RLG-TPV

• The xy-only ray-guided lift falls 0.0082 mAP below released CRN, while the final model gains +0.0256 mAP / +0.0342 NDS over CRN.The largest increase follows the combined multi-plane configuration, which changes several components at once; target-source changes show no resolved difference, and offline targets omit rotation and scaling augmentation.Source: Table 3 · p. 7, §4.4 · p. 7

4.5. Component Ablations

• Removing temporal context or ray-level guidance produces the largest architecture-specific degradations; lower sampling density has a smaller but resolved effect.RCS-aware scatter changes NDS more strongly than mAP, while removing RRD has little effect on mAP.Source: Table 4 · p. 7, §4.5 · p. 7
• Replacing learned TPV height collapse with a larger 3D-convolutional encoder adds approximately 463 K versus approximately 2 K parameters and reduces mAP by 0.0059 and NDS by 0.0031.The study suggests height is useful during representation construction, but explicit height resolution need not persist through the detection head.Source: §4.5 · pp. 7–8

4.6. Qualitative Comparison

• The daytime examples show differences in orientation consistency and isolated false boxes; the nighttime comparison is illustrative rather than conclusive.Both models predict in regions without annotated objects in the nighttime scene, where visible image evidence is limited.Source: Figure 3 · p. 8, §4.6 · p. 8

• 0.4981 mAP and 0.5959 NDS, with orientation error reduced 31.9% and velocity error reduced 30.7% versus the published CRN baseline.Source: §5 · p. 8
• LiDAR-supervised depth probabilities and radar occupancy weight individual samples along projected camera rays in the ray-guided deformable-attention lift.Source: §5 · p. 8
• Measured radar Doppler guides temporal alignment, while LiDAR-derived occupancy constrains the side and front planes during training.Source: §5 · p. 8
• The results indicate that geometric-cue placement within representation construction is as important as cue availability; rotation-aware height-plane targets and radar with elevation measurements remain future directions.Source: §5 · p. 8