Source-linked AI summary
End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners
Simon Hecker, Dengxin Dai, Luc Van Gool
TL;DR
The paper addresses the limited information available to driving models trained with only a front-facing camera and no route planner. It collects data and learns a model combining surround-view cameras with planned routes, finding benefits from both inputs, especially for steering-angle prediction. The authors caution against relying on past human maneuvers and note limits in qualitative interpretation.
Problem
The paper addresses the limited information available to driving models trained with only a front-facing camera and no route planner.
Method
It collects 60 hours of data and learns a model integrating eight surround-view cameras with planned routes to predict future driving maneuvers.
Results
360-degree views help avoid single-camera failures, while route planners significantly improve driving, especially steering-angle prediction.
Takeaways & Limitations
The results support using broader visual context and route guidance for end-to-end driving in city and intersection scenarios.
Takeaways & Limitations
The model excludes past human maneuvers because autonomous-driving errors could be amplified through feedback, while qualitative comparisons may not reveal why predictions differ.
Abstract
from arXiv · showhide
For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: 1) by representing the planned routes on OpenStreetMap as a stack of GPS coordinates, and 2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: 1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and 2) route planners help the driving task significantly, especially for steering angle prediction.
1 Introduction
The paper argues that end-to-end driving models need richer information than a single front-facing camera, including views around the vehicle and route guidance. It introduces a dataset and model that integrate surround-view cameras with planned routes to predict driving maneuvers.
- Approach: The system synchronizes eight-camera videos and route-planner data, then uses CNNs, LSTMs, and fully connected networks to predict driving maneuvers.The architecture encodes visual features, integrates them over time, and fuses information from multiple sensors.
- Motivation: Human drivers combine broad spatial information from mirrors and maps with local road observations to control steering and speed.The proposed task is framed as learning from both global direction and local visual context.
- Motivation: A single front-view camera can produce unsafe, divergent travel directions at forks when the intended route is unclear.The paper motivates additional sensing and route information as a more realistic learning setting.
- Contributions: The work contributes a 60-hour dataset and a learning algorithm integrating surround-view cameras with planned routes.The dataset includes eight-camera videos, route representations, low-level maneuvers, and GPS-IMU odometry.
2 Related Work
Prior work advances perception, navigation, and end-to-end driving separately, but their integration for learning driving models remains underexplored. The paper positions surround-view sensing and route planning as richer inputs for driving models.
- Research landscape: Autonomous-driving research spans mediated perception, end-to-end mapping, reinforcement learning, assistive technologies, navigation, and scene understanding.These lines of work address complementary parts of the driving problem.
- Surround-view systems: Surround-view cameras support applications such as object detection, tracking, lane detection, maneuver estimation, and parking guidance.A panoramic view is presented as useful for several vehicle-assistance tasks.
- Maps and navigation: Route planning and digital maps are established research areas, including positioning, high-definition mapping, and routing services.The related work distinguishes map-based navigation from its use as input to learned driving models.
- Research gap: The integration of digital maps and route planning with learned driving models has not received due attention in the academic community.Existing map-and-image work has focused on vehicle localization rather than directly learning driving behavior.
3 The Driving Dataset
The Drive360 dataset combines eight cameras, route-planner data, CAN-bus maneuvers, and vehicle odometry to capture realistic driving inputs. Its design emphasizes synchronized, diverse data and planned routes that differ from actual trajectories.
- Sensor setup: The sensor setup uses eight cameras, a map-based route planner, and a USB reader for vehicle CAN-bus data.Cameras are mounted around the vehicle at eight evenly spaced heading angles.
- Route planners: Planned routes are represented either as rendered TomTom map videos or as 300 GPS coordinates sampled along the next 300 meters.The OSM representation uses a smoothed route sampled at one-meter intervals.
- Driving signals: The dataset records steering-wheel angle and vehicle speed from the CAN bus at 50Hz.These low-level maneuvers provide direct control targets for end-to-end driving models.
- Synchronization and collection: All sensor streams are synchronized to GPS, with video-frame error up to 8.3 milliseconds and map-to-video error up to 0.5 seconds.The authors regard the map delay as acceptable because routes provide global navigation context.
- Synchronization and collection: Drive360 contains 60 hours of driving collected around multiple Swiss cities across scenarios selected to reduce repetitive-scenario bias.Drivers followed planned routes while obeying Swiss driving rules.
- Dataset comparison: Unlike many comparison datasets, the dataset combines planned routes, surround-view videos, and low-level maneuvers from real driving data.The authors distinguish planned routes from recorded trajectories because lane changes, overtaking, and construction can make them differ.
4 Approach
The approach maps synchronized surround-view imagery, planned routes, and recent vehicle states directly to future steering and speed actions. It uses a multi-camera temporal architecture and jointly learns the two continuous control outputs.
- The model directly maps planned routes, historical vehicle states, and current road situations to desired driving actions.
- Inputs and outputs are synchronized and discretized at sampling rate f, with decisions made every 1/f seconds.
- The model predicts continuous future steering angles and speeds from recent historical samples.Speed is represented from 0 to 180 km/h and steering angle from −720 to 720 degrees.
- Training minimizes a joint cost combining steering-angle and speed losses, with λ = 1 balancing the two terms.The continuous regression task uses L2 loss, and the model learns from multiple previous frames to capture traffic dynamics.
- The architecture uses CNN feature encoders, four LSTMs for camera streams, fusion across cameras and map information, and separate outputs for speed and steering.Training uses four cameras for a panoramic view, while all eight cameras were recorded for future flexibility.
5 Experiments
The experiments compare single-camera baselines with models incorporating route planners and surround-view cameras, using global and situation-focused evaluations. Route planning improves prediction, while surround-view cameras are particularly useful for low-speed city and intersection scenarios, though their overall speed benefit is mixed.
- Experimental setup: 48 hours of driving data and around 1.7 million synchronized sequence samples are used for training, with six of 30 routes reserved for testing.The split is designed to reduce overfitting to particular roads or weather conditions.
- 5 Experiments: The evaluation compares existing single-front-camera methods with models using route planners, surround-view cameras, or both.The study uses MSE for speed and steering-angle prediction and evaluates both overall performance and selected situations.
- 5.1 Comparison to other single-camera methods: The method outperforms [9] significantly and is slightly better than on speed and steering-angle prediction.Both comparison methods are trained and evaluated on this dataset using a single front-facing camera and a 0.3-second prediction horizon.
- Evaluation caveat: The past maneuver baseline performs well because driving actions have inertia, but reliance on past ground-truth vehicle states may amplify errors through feedback in autonomous driving.The authors therefore remove previous human driving states from the model inputs.
- 5.2 Benefits of Route Planners: The visual TomTom route planner significantly improves prediction performance, whereas the numerical OSM route representation does not yield a clear improvement.The route-planner benefit is more noticeable for steering-angle prediction because speed prediction is easier.
- 5.3 Benefits of Surround-View Cameras: Surround-view cameras modestly improve steering-angle prediction overall but reduce overall speed-prediction performance, while helping in low-speed city and intersection situations.They provide views of approaching road users from multiple directions, which is useful when speed depends on whether the planned path is clear.
- Qualitative Evaluation: Qualitative comparisons show route planning resolving ambiguous intersection directions and surround-view cameras adjusting speed when pedestrians are visible outside the frontal view.The visualization compares TomTom, surround-view, and combined models against the front-camera-only model using speed and steering gauges.
6 Conclusion
The paper extends end-to-end driving to surround-view cameras and rendered maps, using a 60-hour dataset and a network that predicts future maneuvers from sensor inputs. Experiments show effective use of both information sources, with rendered videos outperforming raw GPS coordinates for planned-route representation.
- The work introduces end-to-end driving with surround-view cameras and rendered maps instead of only a single front-view camera.
- The Drive360 dataset contains 60 hours of eight-camera driving videos, CAN-bus maneuvers, two route representations, and GPS-IMU odometry.
- A novel deep network maps sensor inputs directly to future driving maneuvers.
- Rendered route videos outperform stacks of raw GPS coordinates for representing planned routes.
1 Introduction
The supplemental material provides detailed model architecture information, Drive360 dataset samples, qualitative evaluation examples, and an accompanying video resource.
- The supplement describes the Surround-View + TomTom Route Planner architecture and provides Drive360 dataset samples.
- It includes a brief qualitative evaluation study illustrated in Figure S3.
- The supplied video contains a visualization tool and a comparison between front-camera-only and Surround+TomTom models.
2 Architecture
The Surround-View + TomTom architecture processes temporal image sequences from four surround-view cameras and combines their features with route-planner information.
- Four temporal frames from each of four cameras—front, rear, left, and right—are processed by separate pretrained ResNet34 networks.
- Two fully connected layers of size 1024 transform camera features before a four-layer LSTM with hidden size 128 processes the temporal vectors.
- The architecture integrates surround-view visual features with a TomTom route-planner feature vector.
- The example route spans urban streets, mountainous roads, and highways, illustrating varied driving environments.
3 Video
The video presents a route-based visualization tool followed by a comparison between front-camera-only and Surround+TomTom driving models, with examples of turns and roundabouts.
- The visualization tool renders the traversed route on a map and displays local camera frames with model predictions at selected points.
- The model comparison contrasts front-camera-only and Surround+TomTom predictions.
- Figure S2 illustrates the architecture of the Surround-View and TomTom route-planner model.
- Figure S3 qualitatively evaluates right-turn and roundabout maneuvers using temporal-frame sequences.