Source-linked AI summary
A Multi-Modal Perception Pipeline for Object Detection and Tracking in Autonomous Racing
Davide Malvezzi, Michele Pestarino, Vittoria Cavicchioli, Valentina La Gamba, Silvia Severi, Fabio Bagni, Luca Bartoli, Massimiliano Bosi, Francesco Gatti, Micaela Verucchi, Ayoub Raji, Marko Bertogna
TL;DR
Autonomous racing perception must handle high speeds, delayed asynchronous measurements, sensor stress, and small safety margins. This paper presents a multimodal late-fusion pipeline with delay-aware multi-object tracking and vehicle- and track-informed priors, validated on real racing data across critical scenarios. The experiments support its effectiveness for high-speed planning.
Problem
Extreme speed, asynchronous sensing, detection delays, vibration, and noise make robust perception difficult in autonomous racing, where latency affects state estimation and safety.
Method
The pipeline late-fuses camera, LiDAR, and RADAR detections and tracks objects with delay compensation and a racing-line-informed Extended Kalman Filter.
Results
Real-world experiments across high-speed overtaking, occlusion, and side-by-side scenarios validate the pipeline’s effectiveness for autonomous-racing perception.
Takeaways & Limitations
The architecture supports safe high-speed planning by combining heterogeneous sensing, delay-aware tracking, and dynamics-informed track priors.
Abstract
from arXiv · showhide
Object detection and tracking are fundamental components of perception systems for autonomous driving. Achieving robust performance under adverse conditions such as limited visibility, sensor noise, and failures remains an open challenge, particularly in autonomous racing, where vehicles operate at very high speeds, experience strong vibrations, and interact under small safety margins. This paper presents a multi-modal late-fusion perception pipeline for object detection and tracking in the autonomous racing domain. The proposed system extends previous work by exploiting all onboard sensors through a late-fusion approach and a dedicated multi-object tracking framework. Independent detections from cameras, LiDARs, and RADARs are combined to provide timely and robust state estimates of surrounding vehicles. The tracking method explicitly compensates for detection delays and embeds in its model prior knowledge of vehicle dynamics and track layout. Experimental evaluation on real-world data across diverse critical scenarios, representative of challenging edge cases also in urban driving, confirms the effectiveness of the proposed pipeline and its suitability to support safe and adaptive planning decisions.
I. INTRODUCTION
Autonomous racing stresses perception with extreme speeds, asynchronous sensing, vibration, noise, and tight safety margins. The paper addresses these conditions with a late-fusion, delay-aware detection and tracking pipeline using onboard sensors.
- Motivation: At speeds approaching 80 m/s, a 100 ms latency can produce nearly 8 m of spatial displacement.Asynchronous measurements, detection delays, and high relative velocities amplify tracking inaccuracies and affect collision avoidance and overtaking safety.
- Contribution: The UNIMORE Racing system was deployed during the 2025 A2RL event at Yas Marina Circuit.The platform operates with high differential velocities, vibration, noise, and latency-sensitive conditions.
- Contribution: The proposed solution combines all onboard sensors through late fusion and uses an object tracker with explicit delay compensation.The system targets perception in full-scale autonomous race cars operating at speeds up to 80 m/s.
- Related Work: Prior autonomous-racing research includes multimodal datasets, single-modality detectors, and partial sensor-fusion approaches.Existing datasets and fusion methods provide relevant benchmarks, but the supplied passages describe continuing needs for broader evaluation and delay handling.
III. SYSTEM OVERVIEW
The system uses a broad onboard sensor suite and late fusion to combine heterogeneous detections before multi-object tracking. Synchronization and calibration support consistent associations, while the tracker supplies continuous estimates for planning.
- Hardware Platform: The Dallara EAV-24 carries three LiDARs, seven cameras, four RADAR units, and GNSS/inertial measurement systems.This sensor configuration provides the hardware basis for multimodal perception on the race car.
- Fusion Architecture: Late fusion combines detections from individual sensors while allowing fusion-stage changes without redesigning the entire system.The architecture is intended to integrate heterogeneous modalities flexibly.
- Fusion Architecture: Late fusion preserves tracking continuity when one sensor fails because remaining sensors can continue providing object detections.The passage also identifies temporal synchronization and spatial alignment as requirements for consistent fusion.
- Pipeline: Sensor detections are calibrated and associated across overlapping fields of view before fused measurements update the multi-object tracker.The tracker’s outputs are subsequently used by the planning module.
IV. DETECTIONS
The camera detection subsystem uses a neural detector trained for road objects and optimized for batched GPU inference. The broader vehicle setup supplies the multimodal context in which these camera detections operate.
- Sensor Setup: The vehicle’s sensor setup includes seven cameras alongside three LiDARs and four RADAR units.These sensors support the multimodal perception architecture described elsewhere in the pipeline.
- Camera Detections: Each camera produces 2D bounding-box detections from its input image batch.The camera subsystem represents each box by pixel coordinates, width, and height, with the image timestamp assigned to the detection.
- Camera Detections: GPU optimization with tkDNN achieves 4.5 ms average inference time for a batch of seven images on an NVIDIA RTX 6000 Ada GPU.The reported timing applies to the batched camera inference process.
B. LiDAR Detections
LiDAR detections use PointPillars on merged, calibrated scans, while RADAR detections are filtered and clustered before cross-modal temporal matching. These modality-specific outputs feed the fusion and tracking pipeline.
- LiDAR Detections: PointPillars detects vehicles from a point cloud formed by merging three LiDAR scans with their extrinsic parameters.The model was pretrained on KITTI and adapted using nearly 25,000 manually labeled race-event point clouds.
- LiDAR Detections: Each LiDAR detection contains a 3D position, orientation, and box dimensions, with its timestamp averaged across contributing points.The averaged timestamp provides a temporal reference for tracker delay compensation.
- RADAR Detections: RADAR processing filters points using RCS, SNR, and sensor existence probability before removing points outside track boundaries and clustering the remainder.The filtering targets outliers caused by reflections and ghosting effects.
- Fusion: The pipeline temporally matches detections from overlapping fields of view when their timestamps differ by less than ΔTmax before merging sources.This matching step precedes fusion and downstream tracking.
A. Camera-LiDAR Matching
The camera-LiDAR matching process projects filtered LiDAR points into camera images and retains points inside 2D detection boxes to estimate object distance and create merged 3D detections. The merged detections combine camera, LiDAR, and timestamp information.
- LiDAR points are filtered and projected into camera images using sensor calibration parameters and transformation matrices.The process removes ground points and filters points by geometric criteria before projection.
- Only LiDAR points whose projected image locations fall inside a camera bounding box are retained for matching.
- The retained points’ range values form a histogram, whose highest-probability bin represents the object’s distance.
- The final 3D detection is obtained by projecting the camera bounding box into world coordinates using the selected distance.
- The merged detection timestamp is recomputed as the mean timestamp of all contributing LiDAR points.
C. LiDAR-RADAR Matching
LiDAR-RADAR matching associates radar clusters with existing 2D detections and augments the merged detections with radial velocity. The detection timestamp is assigned from the RADAR point cloud.
- Each 2D bounding box is matched with each RADAR point-cloud cluster without applying ground removal.
- Merged detections gain the object’s radial velocity from the associated RADAR points.
VI. OBJECT TRACKING
The tracker estimates each object with an Extended Kalman Filter in Frenet coordinates tied to a racing line. It incorporates vehicle-dynamics priors and synchronizes detections with ego-vehicle states during tracking.
- Each detected object is tracked with an Extended Kalman Filter using the state [s, d, vs, vd] in Frenet coordinates.The state represents longitudinal position, lateral displacement, and longitudinal and lateral velocities relative to an associated racing line.
- The racing-line representation supplies reference position, heading, and speed-profile information for the tracked object.
- The process model uses modified constant-velocity dynamics with control inputs and first-order velocity dynamics.This formulation allows predicted velocities to converge smoothly toward reference values.
- Detection timestamps are aligned with ego-vehicle states by linearly interpolating between the immediately preceding and following states.
- The tracker updates active objects, removes outdated ones, and creates new objects when required.
B. Racing Line Matching
The tracker uses precomputed racing lines as reference trajectories and can switch among them as an object’s estimated trajectory evolves. Experimental analysis examines this matching across critical racing scenarios and identifies conditions where accuracy is reduced.
- B. Racing Line Matching: Newly detected objects receive a default racing line so their states can be represented in Frenet coordinates.
- B. Racing Line Matching: At each state update, the tracker selects the racing line minimizing cumulative Euclidean distance to the object’s last N estimated states.
- B. Racing Line Matching: The selected racing line is used to recompute the object state in its associated Frenet coordinate frame.
- VII. Experimental Results: The evaluation uses onboard sensor data and shared high-quality localization from controlled sessions with multiple autonomous race cars.
- VII. Experimental Results: Tracking errors can be larger when the ego-vehicle is far from the opponent, indicating a less accurate operating condition.
- VII. Experimental Results: The study analyzes high-relative-speed overtaking, occluded multi-object tracking, and side-by-side overtaking as critical scenarios.
A. Ablation Study
The ablation study shows that LiDAR and RADAR provide complementary information, while the full multi-modal configuration offers the best overall balance across tracking metrics. In a high-speed stopped-vehicle scenario, tracking errors rapidly decrease after initialization and velocity converges near zero.
- LiDAR-RADAR achieves the lowest longitudinal RMSE while maintaining low lateral and velocity errors, demonstrating complementary spatial and Doppler information.Camera-RADAR increases longitudinal and lateral errors, while camera-LiDAR increases velocity RMSE.
- The full multi-modal configuration provides the best overall balance across tracking metrics despite a slightly higher longitudinal RMSE.It achieves the lowest velocity error among the evaluated configurations.
- The results validate late fusion and the unified multi-modal pipeline for exploiting heterogeneous sensor information.
- At 57.35 m, the estimated velocity converges to almost 0 m/s, supporting early tracking of a stopped object approaching at high relative speed.The object is initially detected at approximately 80 m and begins active tracking after sufficient detections are associated.
- After stopped-object initialization, longitudinal, lateral, and orientation errors rapidly drop toward zero and the state remains stable.The maximum initialization errors are 1.03 m longitudinally, 0.25 m laterally, and 0.90 degrees in orientation.
C. Multi-object Tracking Under Occlusions
The occlusion scenario evaluates whether tracking can quickly recover an opponent hidden by a nearer vehicle during an overtake. Once the ego-vehicle begins overtaking, the previously occluded opponent becomes visible and can be tracked with time for trajectory adaptation.
- The blue opponent occludes the orange opponent until the ego-vehicle initiates the overtaking maneuver.
- The orange opponent becomes visible 51.74 m ahead, giving the planning module sufficient time to adapt the overtaking trajectory.
- The pipeline immediately initializes tracking for newly visible objects in the occlusion scenario.
D. Side-by-side Overtaking
The side-by-side overtaking case examines tracking accuracy at close range, where the minimum lateral distance reaches 1.75 m. Errors are linked to vehicle alignment, visibility constraints, velocity estimation, and the racing-line prior.
- The side-by-side maneuver reaches a minimum lateral distance of 1.75 m, making accurate tracking important for the overtaking assessment.
- Tracking errors correlate with the opponent’s relative position across front, side-by-side, and behind phases of the maneuver.
- Maximum longitudinal and lateral errors reach 0.84 m and 0.61 m, respectively, while rear visibility is limited to the rear camera and RADAR.
- Velocity estimation peaks at a 5 m/s error during abrupt braking and converges to the correct value in approximately 1.75 seconds.
- The conclusion identifies semantic-aware vehicle geometry as future work to reduce estimation biases caused by uncertainty about which vehicle portion is observed.