Source-linked AI summary
Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar
Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Nils Appenrodt, Werner Ritter, Jürgen Dickmann, Klaus Dietmayer, Bernhard Sick, Felix Heide
TL;DR
Existing sensors lose occluded scene components, and prior NLOS methods struggle with large outdoor scenes and high-speed motion. This paper uses automotive Doppler radar, geometric modeling, and temporal fusion for NLOS detection and tracking, achieving accurate localization and collision-warning capability in real-world automotive scenarios.
Problem
Existing sensors capture only directly visible objects, while NLOS methods struggle to handle large-scale outdoor scenes and high-speed motion.
Method
The paper combines a Doppler-radar image-formation model with a joint NLOS detection and tracking network that fuses estimated and measured velocity over time.
Results
The system enables NLOS detection and tracking in large-scale dynamic automotive scenes, achieving 0.1 m mean absolute localization error despite clutter and small hidden objects.
Takeaways & Limitations
The approach supports collision warning for pedestrians and cyclists before existing direct line-of-sight sensors see them in real-world autonomous-driving scenarios.
Takeaways & Limitations
Velocity recovery assumes that the real object velocity is parallel to the relay wall.
Abstract
from arXiv · showhide
Conventional sensor systems record information about directly visible objects, whereas occluded scene components are considered lost in the measurement process. Non-line-of-sight (NLOS) methods try to recover such hidden objects from their indirect reflections - faint signal components, traditionally treated as measurement noise. Existing NLOS approaches struggle to record these low-signal components outside the lab, and do not scale to large-scale outdoor scenes and high-speed motion, typical in automotive scenarios. In particular, optical NLOS capture is fundamentally limited by the quartic intensity falloff of diffuse indirect reflections. In this work, we depart from visible-wavelength approaches and demonstrate detection, classification, and tracking of hidden objects in large-scale dynamic environments using Doppler radars that can be manufactured at low-cost in series production. To untangle noisy indirect and direct reflections, we learn from temporal sequences of Doppler velocity and position measurements, which we fuse in a joint NLOS detection and tracking network over time. We validate the approach on in-the-wild automotive scenes, including sequences of parked cars or house facades as relay surfaces, and demonstrate low-cost, real-time NLOS in dynamic automotive environments.
1. Introduction
The paper addresses the challenge of recovering occluded objects in large, dynamic automotive scenes, where existing NLOS methods struggle outside controlled environments. It introduces Doppler-radar NLOS sensing with joint geometric reconstruction, detection, classification, and tracking.
- Motivation: Existing sensors capture only directly visible scene parts, while NLOS methods recover occluded objects from indirect reflections on visible surfaces.The motivation includes detecting approaching traffic participants hidden from direct line of sight.
- Motivation: Existing NLOS approaches remain challenged by large outdoor scenes, high-speed motion, modulation bandwidth, ambient illumination, and low-reflectivity targets.Optical pulse-based methods require sub-10-ps timing and specialized equipment, while alternative approaches have additional limitations.
- Approach: Doppler radar uses specular relay-wall reflections to preserve angular resolution and emitted RF power, enabling longer-range NLOS sensing in automotive environments.The approach instead separates direct and indirect reflections using Doppler signatures, which suppress static objects.
- Approach: The method formulates a Doppler-radar image-formation model to derive the position and velocity of occluded objects.This model provides the geometric basis for NLOS reconstruction.
- Approach: A joint NLOS detection and tracking network fuses estimated and measured NLOS velocity over time, with automated positioning used to acquire object labels.The training labels are captured alongside the radar data using a separate positioning system.
- Validation: The system is validated on in-the-wild automotive scenarios and demonstrates collision warning for vulnerable road users before direct line-of-sight observation.The reported scenarios use static building facades or parked vehicles as relay walls.
2. Related Work
Prior NLOS work spans optical, intensity-based, acoustic, and RF approaches, but these methods face constraints in temporal precision, target reflectivity, scene scale, acquisition time, or relay-wall knowledge. The paper positions Doppler-radar NLOS tracking against these limitations.
- Optical NLOS Imaging: Optical NLOS methods recover hidden objects from time-resolved light transport, typically requiring detectors with temporal precision below 10 ps.Early systems used streak cameras, while later work used single-photon avalanche diodes; carrier-wave methods are limited to microscopic scenes.
- NLOS Tracking and Classification: Intensity-image NLOS methods are limited by the ill-posed inverse problem to settings with highly reflective targets, sparse dark backgrounds, or additional occluders.Acoustic approaches remain limited to meter-sized laboratory scenes and minutes-long acquisition times.
- RF NLOS Imaging: Prior RF NLOS methods address imaging, tracking, and pose estimation through walls, but radar approaches require multiple known hidden relay walls for ray-traced high-order-bounce reconstruction.This requirement constrains their applicability to in-the-wild scenes.
3. Observation Model
The observation model uses FMCW MIMO radar to recover range, angle, and radial velocity from indirect radar returns involving visible relay walls and occluded targets. It models wall and target reflections, then processes the resulting range-angle-velocity measurements into a sparse radar point cloud.
- Radar measurements: A relay wall redirects radar energy toward an occluded object, whose diffuse backscatter returns to the visible wall and receiver.The forward model integrates contributions from wall positions and the hidden object surface.
- Radar measurements: FMCW MIMO radar resolves targets in range, azimuthal angle, and radial Doppler velocity using linear frequency sweeps.The received signal is convolved with the scene impulse response before radar processing.
- Reflection model: Flat surfaces at automotive radar wavelengths produce predominantly specular wall responses, while hidden-target reflectance is modeled with diffuse and specular components.At 76–81 GHz, the wavelength is approximately 5 mm; large-standoff walls may not capture the specular component, making diffuse reflectance dominant.
- Range and velocity estimation: Range is estimated from the FMCW beat frequency, with different path lengths appearing in different Fourier-analysis bins.The beat frequency results from the time difference between transmitted and received chirps.
- Range and velocity estimation: Radial velocity is estimated from phase shifts between consecutive chirps, applying Fourier analysis to phasors within each range bin.The chirp-sequence phase shift corresponds to Doppler frequency and yields radial velocity.
- Angle estimation and processing: Antenna-array phase differences provide incident angle estimates, producing a 3D range-velocity-angle data cube.The raw measurement cube contains 1024×512×64 bins for range, angle, and velocity; a constant false alarm rate filter reduces clutter to fewer than 10^4 points for efficient inference.
4. Joint NLOS Detection and Tracking
The paper combines radar-based NLOS geometry and velocity estimation with a multi-frame detection and tracking network. Temporal fusion across multiple scales supports predictions for current and future frames.
- Non-Line-of-Sight Detection: The detection pipeline converts BEV radar points into a sparse pseudo-image, encodes representations with a 2D convolutional backbone, and regresses oriented 2D boxes.The detection head predicts background, cyclist, or pedestrian classes and bounding-box center, dimensions, orientation, and velocity.
- Non-Line-of-Sight Detection: The model estimates virtual detections’ real locations by testing relay-wall geometry, including opposite-side placement and intersection within the wall segment.The recovered point is then used for downstream NLOS detection and tracking.
- Non-Line-of-Sight Detection: Hidden velocity is reconstructed from measured radial velocity under the assumption that the road user moves parallel to the relay wall.The velocity sign encodes approaching or departing motion and the object’s left- or right-half-plane allocation.
- Non-Line-of-Sight Doppler Tracking: The network jointly processes the current and preceding radar frames, predicting the current frame and n future frames while fusing temporal features at multiple backbone levels.Multi-scale fusion combines per-frame details with higher-level motion features without the memory and computational cost of direct 3D convolutions.
- Loss Functions: The training objective combines localization and classification losses, with localization supervision extending from the current frame to n future frames.Localization residuals cover position, dimensions, orientation, and velocity relative to anchors.
- Loss Functions: Automated GNSS- and IMU-based pose estimation provides full egovehicle and hidden-road-user localization for NLOS training data.The setup uses an automated positioning system to acquire labels for occluded pedestrians and bicyclists.
5. Data Acquisition and Training
The study uses automotive Doppler radar prototypes and automated localization to collect in-the-wild NLOS training and validation data. The dataset spans diverse outdoor relay surfaces, trajectories, and hidden road users.
- Prototype Vehicle Setup: The prototype radar operates at 76–77 GHz with 153 m maximum range, 140° field of view, and 22.6 ms measurement time.Its reported resolutions are 0.15 m, 1.8°, and 0.087 m s−1; comparable development kits cost a few hundred USD.
- Automated Ground-Truth Estimation: Automated ground truth uses a synchronized GeneSys ADMA-Slim module carried by hidden pedestrians or bicyclists, combining GNSS and IMU pose estimation.The system is synchronized with the capture vehicle via Wi-Fi.
- Training and Validation Data Set: The dataset contains 100 sequences across 21 in-the-wild automotive scenarios and more than 32 million radar points.It is equally distributed between hidden pedestrians and cyclists.
- Training and Validation Data Set: Relay surfaces include residential and industrial walls, marble garden walls, a guard rail, parked cars, garages, a warehouse wall, and a concrete curbstone.These surfaces provide varied outdoor settings for NLOS training and evaluation.
6. Assessment
The evaluation tests hidden-object detection, localization, and tracking in large automotive scenes using qualitative scenarios and quantitative comparisons. Across sparse and noisy radar observations, temporal velocity-based tracking produces stable predictions and accurate localization.
- Evaluation setting: The evaluation covers a 60 m × 80 m region discretized into a 600×800 grid, using AP and center-distance error for classification and localization.Ground-truth boxes are assigned to their highest-overlapping predicted boxes during training.
- Qualitative validation: The qualitative results span automotive scenes with different relay-wall types and include pedestrians and bicycles under sparse radar observations.In one scenario, the hidden object produces no more than three detected points.
- Qualitative validation: Stable box predictions remain possible when the model relies on velocity and orientation from sparse points, including consistent size and orientation across frames.The paper also reports correct sequence-level detection when a single frame is affected by imperfect ground truth or radar invisibility.
- Qualitative validation: The figures visualize joint detection and tracking across relay-wall types and object classes, plus trajectories segmented by tracking identity.Radar and lidar point clouds are shown with ground-truth and predicted boxes, while velocity direction is encoded by red and green segments.
- Quantitative results: 0.1 m MAE is achieved for localization despite clutter and the small diffuse cross sections of hidden pedestrians and bicycles.The evaluation separately reports NLOS and line-of-sight tracking performance.
- Quantitative results: Velocity supervision improves both localization and tracking accuracy compared with models without velocity supervision.For challenging NLOS data, unmatched objects increase, while matched objects remain precisely localized according to the reported MOTA and MOTP discussion.
7. Conclusion
The paper introduces joint NLOS detection and tracking of occluded objects with automotive Doppler radar. It validates collision warning for pedestrians and cyclists in real-world autonomous-driving scenes before direct line-of-sight visibility.
- Conclusion: The proposed method jointly detects and tracks occluded objects using automotive Doppler radar.It is trained end-to-end on a realistic NLOS automotive radar dataset.
- Conclusion: The approach enables collision warning for pedestrians and cyclists before existing direct line-of-sight sensors see them.The paper identifies higher-order-bounce detection and joint optical-radar NLOS as future directions.