Source-linked AI summary
Reflection-Aware Reasoning for Non-Line-of-Sight Pedestrian Localization
Byeonggyu Park, Mingu Jeon, Seong-Woo Kim
TL;DR
NLOS pedestrian localization is difficult in ego-dynamic outdoor settings because vehicle motion makes radar multipath complex and noisy. The framework fuses camera and radar to estimate reflection paths and surfaces, then uses physics-guided ray tracing; in testbed scenarios it detects NLOS pedestrians 0.77 s earlier than first-bounce observations and operates near real time.
Problem
Vehicle motion, noise, and clutter make NLOS radar points difficult to identify, while multi-bounce paths distort pedestrian observations and require reflective-surface estimates for localization.
Source: §1 · p. 2Method
The framework jointly learns reflection-type point segmentation and reflective-surface estimation from camera–radar inputs, then applies physics-guided ray tracing for multipath interpretation.
Source: §1 · p. 2, §7 · p. 15Results
The framework detects NLOS pedestrians on average 0.77 s earlier than first-bounce radar observations and achieves 72 ms end-to-end latency (13.85 FPS).
Source: §7 · p. 15Takeaways & Limitations
The reported results support reliable and early NLOS perception beyond direct line of sight in the evaluated settings.
Source: §7 · p. 15Severe errors occur when ambiguous radar points and complex intersection geometry disrupt multipath spatial reasoning.
Source: §2.5 · p. 28
Abstract
from arXiv · showhide
Reliable localization of non-line-of-sight (NLOS) pedestrians is critical for safe urban autonomous driving, yet it remains highly challenging in ego-dynamic outdoor environments, where ego-vehicle motion makes radar multipath propagation complex and noisy. In this paper, we present a reflection-aware framework for NLOS pedestrian localization with a moving ego-vehicle in outdoor testbed scenarios. Our framework fuses front-view camera images and 2D radar point clouds to infer reflection orders and reflective surface distributions in bird's-eye-view space. It then uses physics-guided ray tracing to reconstruct distorted reflection paths and localize the hidden pedestrian. We validate the framework in outdoor testbed scenarios under ego-dynamic conditions. The results demonstrate the effectiveness of the proposed framework for NLOS pedestrian localization with a moving ego-vehicle.
1 Introduction
The paper targets NLOS pedestrian localization in moving-vehicle settings, where noisy, motion-varying radar multipath complicates identifying pedestrian returns and reflective surfaces. It combines camera–radar learning with physics-guided ray tracing and reports localization results in an outdoor intersection testbed.
1 Introduction
- Ego-vehicle motion makes radar observations vary across frames, while noise, clutter, and multi-bounce paths complicate NLOS point identification and localization.Accurate reflective-surface estimation is needed to reconstruct paths and locate occluded objects.Source: §1 · p. 2
- The framework fuses front-view camera images and 2D radar point clouds to predict reflection-type point labels and reflective-surface heatmaps in BEV space.Physics-guided ray tracing uses the estimated structures and identified radar points to interpret paths and reconstruct propagation trajectories.Source: §1 · pp. 2–3
- 0.54 m and 1.23 m localization errors for LOS and NLOS pedestrians, respectively, under ego-dynamic conditions; the method outperforms existing approaches.Under ego-static conditions, the corresponding errors are 0.44 m and 1.01 m.Source: §1 · p. 3
- The paper presents a reflection-aware framework for localizing NLOS pedestrians with a moving ego-vehicle and validates it in an outdoor testbed intersection.The framework combines multimodal representation learning with physics-guided ray tracing.Source: §1 · p. 3
2 Related Work
Prior NLOS sensing work includes acoustic, radar, and camera–radar approaches, but the cited work describes constraints in environment, prior scene information, or dynamic validation. The paper addresses outdoor ego-dynamic pedestrian localization using a moving vehicle-mounted radar-camera platform.
Acoustic-based NLOS perception
- Acoustic NLOS methods exploit reflection and diffraction, but cited approaches are mainly indoor or static, or rely on predefined maps and coarse state inference outdoors.These constraints limit their applicability to dynamic urban driving scenarios.Source: §2 · pp. 3–4
Radar-based NLOS perception
- Many radar methods rely on predefined wall or corner geometry, prior scene knowledge, auxiliary structural information, or controlled settings, limiting applicability to unknown or dynamic outdoor environments.Learning-based radar methods also include indoor or short-range systems with static targets.Source: §2 · p. 4
Camera–radar fusion for NLOS perception
- Prior camera–radar fusion often targets LOS perception, while NLOS approaches use structural cues or particle filtering and largely retain heuristic radar interpretation.The paper characterizes its setting as outdoor ego-dynamic localization with a moving vehicle-mounted radar-camera platform and multi-target scenarios.Source: §2 · p. 4
Radar-based NLOS perception
- Radar point clouds can be sparse and noisy, and localization geometry varies across first-, second-, and third-bounce paths.The paper models up to third-order reflections, citing mmWave attenuation characteristics as the reason for this modeling scope.Source: Figure 2 · p. 5
3 Problem Definition
The problem is to infer an NLOS object's position from synchronized camera and radar observations by representing radar reflection paths and reflective surfaces. The model predicts these intermediate representations for a physics-constrained inference operator to produce the object position.
Problem Definition
- Each radar point is labeled by reflection order—up to third order—and whether its signal is object-originated or surface-originated.Together, these labels encode reflection order and physical signal origin.Source: §3 · p. 5
Input Representation
- At each time step, the inputs include a front-view image and a 2D radar point cloud whose points carry BEV coordinates, Doppler velocity, and radar cross section.The number of radar points varies across frames.Source: §3 · pp. 5–6
Problem Formulation
- Given synchronized image and radar inputs, the model predicts point-wise reflection-path labels and a reflective-surface probability map in BEV space.The proposed model is denoted fθ and maps the inputs to these two intermediate representations.Source: §3 · p. 6
- A physics-constrained inference operator combines the predicted labels and surface map to estimate the object position, with localization error defined against ground truth.The stated objective is to minimize that localization error.Source: §3 · p. 6
4 Method
The method fuses camera and radar features to identify reflection orders and reflective surfaces, then uses geometric ray tracing to infer hidden pedestrian locations. Its components progress from reflection-aware representation learning through point segmentation and surface estimation to physics-guided localization.
4.1 Overview
- Camera cues and radar measurements are fused to predict reflection types and reflective surfaces, which guide ray tracing to infer NLOS pedestrian locations.The camera supplies structural and semantic cues, while radar provides range measurements and reflection characteristics.Source: §4.1 · p. 6
4.2 Reflection-Aware Representation Learning
- Cross-attention fuses camera and radar BEV features into a shared representation for reflection-type segmentation and reflective-surface estimation.Camera features are projected into BEV space, and ego-motion-compensated radar observations from multiple frames are aggregated before encoding.Source: §4.2 · pp. 6–8
4.3 Reflection-Type Point Segmentation
- The point-segmentation head assigns each radar return a reflection-order label, separating multipath observations into interpretable paths for surface estimation and NLOS reconstruction.Labels are predicted independently for each point using the fused BEV representation.Source: §4.3 · p. 8
4.4 Reflective Surface Estimation
- A BEV probability map estimates reflective-surface locations, supervised by a pseudo-map built from aggregated first-order reflections and anisotropic Gaussian splatting.The splatting uses anisotropic smoothing to preserve elongated reflective structures while limiting diffusion across their normal direction.Source: §4.4 · p. 9
4.5 Physics-Guided Ray Tracing
- Physics-guided ray tracing clusters reflection paths, searches for reflectors along ego-rays, and mirrors third-order returns to estimate pedestrian positions.First-order cluster centers and mirrored third-order estimates are combined and nearby candidates merged; the procedure requires no prior scene structure information.Source: §4.5 · pp. 9–10
5 Dataset
The dataset captures NLOS pedestrian scenarios at a constructed urban intersection, with both stationary and moving ego-vehicle conditions. It includes multimodal sensing and overhead-camera ground truth.
5 Dataset
- The constructed intersection testbed uses a front-view RGB camera, 77 GHz radar, LiDAR, and wheel encoder, with an overhead camera providing occluded-pedestrian ground truth.Sensor data are synchronized using ROS.Source: §5 · p. 11
- Scenarios include one to three pedestrians emerging from occluded side roads, with the ego vehicle stationary or moving through two intersection configurations.B1 has a frontal wall blocking the opposite side; B2 has an open opposite road.Source: §5 · p. 11
- The dataset contains 44 training and 13 validation scenarios for ego-static conditions, and 57 training and 10 evaluation scenarios for ego-dynamic conditions.Dynamic trials reach 22.00 km/h and include NLOS observations up to 17.92 m.Source: §5 · p. 11
6 Experimental Results
Experiments evaluate reflection-type point segmentation, reflective-surface estimation, and pedestrian localization across ego-static and ego-dynamic settings. The framework reports improved NLOS localization against geometry-based baselines, alongside segmentation, surface-estimation, and runtime results.
6.1 Reflection-Type Point Segmentation Performance
- 94.6% accuracy and 0.915 macro-F1 ego-static; 89.9% accuracy and 0.814 macro-F1 ego-dynamic, with strong third-bounce classification in both settings.The method consistently improves overall segmentation and third-bounce accuracy over baseline methods.Source: §6.1 · p. 12, Table 2 · p. 13
6.2 Reflective Surface Estimation Performance
- 0.280 m (B1) and 0.415 m (B2) RPE, with 7.175° and 6.698° RAE ego-static; across both settings, RPE remains sub-meter and RAE below 8°.Under ego-dynamic conditions, RPE is 0.285 m in B1 and 0.357 m in B2, with corresponding RAEs of 7.207° and 5.997°.Source: Table 3 · p. 13, §6.2 · p. 14
6.3 Object Localization Performance
- The framework runs at 13.85 FPS on an RTX 3090 GPU, with 55.22 ms per frame for reflection-aware learning and 16.80 ms for ray tracing.It has 1.17M parameters and 55.04 GMACs per frame; the semantic auxiliary head is used only during training.Source: Table 4 · p. 13, §6.3 · pp. 14–15
- NLOS error fell from 1.69 m to 1.01 m (approximately 40%) ego-static and 3.23 m to 1.23 m (more than 60%) ego-dynamic versus geometry-based baselines.The proposed method's average localization errors were 1.01 m for NLOS pedestrians ego-static and 1.23 m ego-dynamic; corresponding LOS errors were 0.44 m and 0.54 m.Source: Table 5 · p. 14, §6.3 · p. 14
7 Conclusion & Discussion
The framework combines reflection-type segmentation and reflective-surface estimation with physics-guided ray tracing for structured multipath interpretation. It improves NLOS localization in static and dynamic settings, detects pedestrians earlier using third-order reflections, and shows that surface-estimation accuracy matters to localization.
7 Conclusion & Discussion
- The framework jointly learns reflection-type point segmentation and reflective-surface estimation, then uses physics-guided ray tracing to interpret multipath.This combines data-driven multipath decomposition with physical consistency.Source: §7 · p. 15
- NLOS pedestrian detection is reported 0.77 s earlier on average than first-bounce radar observations by leveraging third-order multipath reflections.The system achieves 72 ms end-to-end latency, or 13.85 FPS.Source: §7 · p. 15
- Localization error increases significantly when reflective-surface estimation is inaccurate, indicating that surface reasoning and reflection-type segmentation are interdependent.Source: §7 · p. 15
1 Dataset Details
The dataset uses a real-world T-junction testbed with varied pedestrian scenarios, driving conditions, and two intersection layouts. It also specifies a shared radar evaluation protocol and annotation procedures for image and radar-point labels.
1 Dataset Details
- The testbed dataset contains 120 scenarios—67 ego-dynamic and 53 ego-static—with 12,539 frames and 2,853,228 radar points.Scenarios vary in pedestrian count and motion, vehicle speed, and spatial configuration; layouts B1 and B2 represent a blocked opposite side and an open opposite road, respectively.Source: §1 · p. 19
1.1 Experiment Protocol
- All compared radar methods use consistently processed, ego-motion-compensated point clouds; ego-dynamic data have more clutter, lower surface coverage, and more concentrated returns.Clutter/unlabeled ratio rises from 13.8% to 31.6%, while surface coverage falls from 0.478 to 0.412 and surface concentration rises from 0.135 to 0.213.Source: §1.1 · pp. 21–22, Table 3 · p. 22
- Removing camera input lowers point accuracy from 0.898 to 0.884, Macro-F1 from 0.818 to 0.800, and surface IoU from 0.738 to 0.672 in ego-dynamic tests.Removing auxiliary loss reduces surface IoU from 0.738 to 0.726.Source: Table 4 · p. 22, §1.1 · p. 22
1.2 Image Segmentation Annotation
- Front-view images are annotated with pixel-level wall and pedestrian masks every two frames, corresponding to 5 Hz.Annotations were performed using CVAT and include challenging partial occlusions.Source: §1.2 · pp. 22–23
1.3 Point Class Annotation
- Radar point labels distinguish six classes, including pedestrian-originated bounce orders and wall-originated surface reflections, with remaining points labeled None.LiDAR-derived reflector segments and virtual ray tracing support the propagation-path labels.Source: §1.3 · pp. 23–24
2 Additional Experimental Results
The experiments examine qualitative localization examples, modality assignments, camera and geometry ablations, semantic auxiliary loss, and failure cases. Ablations show different modality choices favor point classification versus reflective-surface prediction, while camera context and accurate geometry support localization.
2.1 Qualitative Results
- Fig. 3 presents four example scenes each for ego-static and ego-dynamic localization, with predicted and ground-truth NLOS pedestrian boxes shown separately.Green boxes mark predictions and red boxes mark ground truth.Source: §2.1 · p. 24, Figure 3 · p. 25
2.2 Ablation Study on Q–K–V Modality Assignments
- The radar–radar–camera assignment achieves the highest point accuracy and recall, while radar–camera–camera achieves the lowest surface MAE and highest Dice score.The table compares point classification and reflective-surface heatmap quality across query–key–value modality assignments; lower MAE and higher Dice indicate better surface prediction.Source: §2.2 · p. 26, Table 6 · p. 26
2.3 Camera and Geometry Ablations
- Removing camera input lowers point accuracy from 0.898 to 0.884, Macro-F1 from 0.818 to 0.800, and surface IoU from 0.738 to 0.672.The camera-off results indicate that visual BEV context complements radar cues for reflection-order reasoning and surface estimation.Source: §2.3 · p. 27
- With LiDAR-derived ground-truth wall geometry, localization errors are 0.375 m and 0.535 m for LOS, and 0.840 m and 1.230 m for NLOS, in ego-static and ego-dynamic settings, respectively.Accurate geometry benefits NLOS localization, while the proposed method estimates reflective surfaces from camera-radar input without LiDAR-derived reflector geometry at test time.Source: §2.3 · p. 27, Table 7 · p. 27
2.4 Semantic Auxiliary Loss and Model Complexity
- Removing the semantic auxiliary loss changes point accuracy from 0.898 to 0.896 and Macro-F1 from 0.818 to 0.819, while surface IoU falls from 0.738 to 0.726.The loss mainly supports consistency and generalization in reflective-surface estimation, preserving surface geometry under degraded ego-dynamic radar conditions.Source: §2.4 · p. 27
2.5 Failure Case
- Severe localization errors occur when ambiguous radar points and complex intersection geometry disrupt multipath spatial reasoning.Fig. 4 illustrates an incorrectly inferred reflection path ending far from the true pedestrian location.Source: §2.5 · p. 28, Figure 4 · p. 28