Source-linked AI summary
ReasonNet: End-to-End Driving with Temporal and Global Reasoning
Hao Shao, Letian Wang, Ruobing Chen, Steven L. Waslander, Hongsheng Li, Yu Liu
TL;DR
Large-scale autonomous-vehicle deployment remains difficult in dense urban traffic because systems must predict scene evolution and handle rare adverse events involving occluded objects. ReasonNet combines temporal and global scene reasoning, and the paper evaluates it across CARLA benchmarks while releasing DOS for occlusion evaluation. The model ranks first on the sensor track of the CARLA autonomous driving leaderboard.
Problem
Dense urban driving requires high-fidelity prediction of future scene evolution and robust handling of rare adverse events such as relevant objects hidden by occlusion.
Method
ReasonNet is an end-to-end framework with temporal reasoning over historic scene information and global reasoning over object-environment interactions.
Results
ReasonNet ranks first on the sensor track of the CARLA autonomous driving leaderboard across evaluations on complex urban driving benchmarks.
Takeaways & Limitations
The framework supports future-behavior forecasting, perception under occlusion, and systematic occlusion evaluation through the publicly released DOS benchmark.
Takeaways & Limitations
The CARLA simulator data used in the global reasoning module contain active objects in only 7% of the ego-vehicle-centered BEV map area.
Abstract
from arXiv · showhide
The large-scale deployment of autonomous vehicles is yet to come, and one of the major remaining challenges lies in urban dense traffic scenarios. In such cases, it remains challenging to predict the future evolution of the scene and future behaviors of objects, and to deal with rare adverse events such as the sudden appearance of occluded objects. In this paper, we present ReasonNet, a novel end-to-end driving framework that extensively exploits both temporal and global information of the driving scene. By reasoning on the temporal behavior of objects, our method can effectively process the interactions and relationships among features in different frames. Reasoning about the global information of the scene can also improve overall perception performance and benefit the detection of adverse events, especially the anticipation of potential danger from occluded objects. For comprehensive evaluation on occlusion events, we also release publicly a driving simulation benchmark DriveOcclusionSim consisting of diverse occlusion events. We conduct extensive experiments on multiple CARLA benchmarks, where our model outperforms all prior methods, ranking first on the sensor track of the public CARLA Leaderboard.
1. Introduction
ReasonNet addresses dense urban driving challenges by combining temporal reasoning about scene evolution with global reasoning about interactions, environments, and occluded objects. The paper introduces the ReasonNet framework and DOS benchmark, reporting first place on the CARLA sensor track.
- Motivation: Dense urban traffic complicates scene understanding, future-evolution prediction, and handling of rare adverse events involving occluded objects.These challenges matter because incorrect decisions can result in severe accidents or traffic infractions.
- ReasonNet: Temporal reasoning models relationships among features across frames to improve prediction of scene evolution and future object behavior.The proposed module fuses information from different frames rather than simply concatenating historic features.
- ReasonNet: Global reasoning uses interactions among objects and the environment to infer unobservable space and anticipate danger from occluded objects.The motivation draws on reasoning from road geometry and driving interaction patterns when relevant objects are hidden.
- Contributions: ReasonNet enhances historic scene reasoning for high-fidelity future prediction and contextual perception under occlusion.The framework is presented as a temporal and global reasoning network for end-to-end driving.
- Contributions: The publicly released DOS benchmark contains diverse urban occlusion scenarios for systematic evaluation of occlusion events.DOS is introduced as a benchmark for driving in occlusion simulation.
- Results: The model ranks first on the sensor track of the CARLA autonomous driving leaderboard after evaluation on multiple complex and adversarial urban benchmarks.The reported validation covers multiple benchmarks with complex and adversarial urban scenarios.
2. Related work
Related work places end-to-end urban autonomous driving within reinforcement-learning and imitation-learning approaches, while attention mechanisms have also been applied to driving-related tasks. These lines of work span simulated-environment learning, latent representations, motion forecasting, and driver-attention prediction.
- End-to-end Autonomous Driving: Recent end-to-end autonomous-driving methods in urban scenarios mainly use reinforcement learning or imitation learning.The discussion links this research area to the CARLA simulator and leaderboard.
- End-to-end Autonomous Driving: Reinforcement-learning methods train agents through repeated interaction with simulated environments and learning from the resulting experiences.Latent DRL is described as first learning an embedding space for environment observations before reinforcement learning.
- Attention for Autonomous Driving: Attention mechanisms have been applied in driving domains including motion forecasting and driver-attention prediction.The related-work discussion presents these applications as extensions of attention-based architectures from language and vision.
3. Method
ReasonNet combines multimodal perception with temporal memory and global interaction reasoning to support driving decisions and adverse-event detection.
- Perception Module: The perception module fuses RGB and LiDAR inputs to produce BEV features, waypoints, and traffic-sign features.It uses four RGB cameras and one LiDAR sensor, followed by 2D and 3D backbones and a BEV decoder.
- Temporal Reasoning Module: The temporal reasoning module fuses current and historic features through attention and stores past information in short-term and long-term memory buffers.Historic values are aggregated into a readout and progressively fused with current features using a GRU.
- Control: Global reasoning supports adverse-event detection, while the control system uses predicted waypoints, traffic signs, and occupancy risks to issue driving commands.The controller brakes for detected red lights or stop signs and performs emergency stops when occupancy or waypoint overlaps indicate danger.
- Global Reasoning Module: The global reasoning module represents object and environment features as a graph to model their interactions and relationships.Object features are selected from locations with high object-existence probability, while downsampled features represent the environment.
4. Drive in Occlusion Sim (DOS) Benchmark
The DOS benchmark evaluates driving under diverse occlusion scenarios designed to test temporal and global reasoning about hidden objects.
- Benchmark Design: DOS is a CARLA-based benchmark providing diverse urban driving scenarios with occluded objects.It includes four types of challenging occlusion scenarios.
- Benchmark Results: ReasonNet ranks first overall on the public CARLA leaderboard, with the highest driving and infraction scores and second-highest route completion.The table compares driving score, infraction score, and route completion, for which higher is better.
- Benchmark Design: Each DOS scenario contains 25 cases varying in road environment and background traffic.The benchmark therefore contains 100 cases across four scenarios.
- Reasoning Coverage: DOS includes intermittent occlusions suited to temporal reasoning and constant occlusions with interaction clues suited to global reasoning.These scenarios are designed around occlusion structures that can potentially be resolved using the corresponding reasoning mechanisms.
5. Experiments
Experiments evaluate ReasonNet on CARLA benchmarks and ablations of its temporal memory and global reasoning components. The model achieves leading leaderboard performance, while both reasoning modules improve performance, particularly on occlusion scenarios.
- Temporal reasoning ablation: Increasing both short-term and long-term buffer sizes improves every reported metric except road completion on Town 05 Long and DOS.The ablation evaluates collision rate, red-light violation, blocked frequency, success rate, and road completion over three runs.
- Comparison to the state of the art: ReasonNet outperforms prior CARLA Leaderboard methods, improving driving score, route completion, and infraction score over InterFuser by 5%, 2%, and 6%, respectively.It achieves the highest driving and infraction scores and the second-highest route completion.
- Temporal reasoning ablation: The combined usage-and-object memory-selection strategy outperforms random selection and single-criterion alternatives, especially on DOS.The usage-based strategy performs worse than the object-based strategy, indicating that object features better capture historic behaviors.
- Temporal reasoning ablation: The attention visualization shows current-frame object queries consistently attending to corresponding regions in historic feature maps.This provides a visual indication of correspondence used for temporal reasoning.
- Global reasoning ablation: Removing global reasoning drops average DOS success rate from 71.5 to 29.5, demonstrating its effectiveness for occlusion events.Removing consistency loss lowers driving performance and DOS success, while excluding traffic-sign features increases red-light violations.
6. Conclusion
ReasonNet combines temporal and global reasoning for end-to-end autonomous driving. It forecasts road-user behavior, detects adverse events such as occlusions, improves perception, and is accompanied by the DOS benchmark.
- Conclusion: ReasonNet combines temporal reasoning for historic-scene forecasting with global reasoning for object-environment interactions and adverse-event detection.The framework includes a dynamically maintained temporal memory bank and global reasoning focused especially on occlusion.
- Conclusion: The authors release the DOS benchmark, which contains diverse occlusion scenarios for studying occlusion detection in end-to-end autonomous driving.The benchmark is intended to facilitate research on occlusion detection.
A. Implementation Details
The implementation uses specified feature dimensions, training schedules, sensor preprocessing, and hyperparameters for ReasonNet. The first-stage perception pipeline is summarized in Figure 6.
- Model details: All decoders use feature dimension 256, while the 2D and 3D backbones encode RGB and raw LiDAR features with specified architecture settings.The 2D backbone uses the fifth ResNet stage, and the 3D backbone uses a simplified PointNet-like encoder.
- Training: The first training stage uses AdamW, cosine learning-rate scheduling, 35 epochs, five warm-up epochs, and RGB augmentation; the second stage freezes perception.The first-stage overview is shown in Figure 6.
- Sensors: RGB inputs come from one front, two side, and one rear camera with 800 × 600 resolution and 100° horizontal field of view per camera.The front and focusing-view images receive separate resizing and cropping procedures.
- Hyperparameters: Additional hyperparameter values used in ReasonNet are listed in Table 8.The passage identifies Table 8 as the location of these values without specifying them.
B. Benchmark details
ReasonNet is evaluated on the public CARLA Leaderboard, Town05, and DOS benchmarks. These benchmarks test route completion without collisions or traffic-rule violations across realistic, complex, and occlusion-focused scenarios.
- Benchmarks: The evaluation covers the CARLA public leaderboard, Town05 Short and Long, and the proposed DOS benchmark.The leaderboard includes varied weather and secret evaluation routes, while Town05 contains multi-lane roads, bridges, highways, and exits.
- Evaluation setting: In all benchmarks, the ego vehicle must complete a route without collision or traffic-rule violations.The benchmarks include adversarial events, while DOS specifically includes occlusion events.
C. More Experimental results
This section reports additional experiments on the CARLA public Leaderboard. ReasonNet leads vehicle collision and offroad infraction metrics among the evaluated methods.
- ReasonNet leads the vehicle collision and offroad infraction numbers among all evaluated methods on the CARLA public Leaderboard.The comparison is reported in Table 5 against baseline methods.
C.2. Town05 and CARLA 42 routes
ReasonNet is evaluated on the Town05 and CARLA 42 routes benchmarks, including adversarial urban events. It outperforms strong prior methods across the reported benchmark metrics.
- Evaluation metrics: The reported CARLA comparisons use driving score, road completion, and infraction score alongside other metrics.Table 6 and Table 7 compare these measures with prior state-of-the-art methods.
- Adversarial events: Adversarial events include unexpected agents rushing into the road from occluded regions and vehicles running red traffic lights.These events are part of the evaluated urban-driving scenarios.
- Town05 benchmark: ReasonNet outperformed six state-of-the-art methods in all metrics and scenarios on the Town05 benchmark.The comparison is summarized in Table 6.
D. Data statistics
The experiments use CARLA data spanning multiple towns, routes, and weather conditions, with distinct town splits for leaderboard submission and ablation studies. The materials also document benchmark statistics and ReasonNet parameters.
- Dataset composition: The dataset includes six weather types, and the online-leaderboard model is trained across all eight towns.Ablation models instead use five specified towns.
- Training splits: Ablation studies train on Town01, Town03, Town04, Town06, and Town07.These towns differ from the all-town training used for the online leaderboard submission.
- Experimental platform: The experiments use the open-source CARLA driving simulator, with licensing information provided for CARLA, its assets, and the pretrained ResNet model.The source code is stated to be publicly available once accepted under a CC-BY-NC 4.0 license.
- Benchmark coverage: The CARLA 42 routes benchmark compares methods using driving score and infraction score.ReasonNet is reported to outperform other methods on these measures.
- Model configuration: Table 8 lists the parameters used for ReasonNet.The parameter settings are documented separately from the town statistics.
- Data statistics: Table 9 reports the number of frames and brief descriptions for each town.These statistics document the composition of the simulated driving data.