Source-linked AI summary
End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
Zhengyuan Yang, Yixuan Zhang, Jerry Yu, Junjie Cai, Jiebo Luo
TL;DR
End-to-end steering alone does not provide complete vehicle control because speed must also be predicted. The paper introduces discrete speed commands and then a multi-modal model using feedback speeds, reporting accurate speed prediction, improved steering accuracy, and better real-road testing through failure-case synthesis.
Problem
Steering angle alone is insufficient for vehicle control, while accurate speed values are difficult to infer from visual inputs alone.
Method
The paper jointly predicts steering and speed, first with discrete commands from image sequences and then with a multi-modal network using visual recordings and previous feedback speeds.
Results
The models predict steering and speed accurately, with steering error improving from 2.84° to 1.85° and then 1.26° across the evaluated models.
Takeaways & Limitations
The framework extends end-to-end vehicle control beyond steering and, with improved failure-case synthesis, supports autonomous driving in similar real-road conditions.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNN) have been successfully applied to autonomous driving tasks, many in an end-to-end manner. Previous end-to-end steering control methods take an image or an image sequence as the input and directly predict the steering angle with CNN. Although single task learning on steering angles has reported good performances, the steering angle alone is not sufficient for vehicle control. In this work, we propose a multi-task learning framework to predict the steering angle and speed control simultaneously in an end-to-end manner. Since it is nontrivial to predict accurate speed values with only visual inputs, we first propose a network to predict discrete speed commands and steering angles with image sequences. Moreover, we propose a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs. Experiments are conducted on the public Udacity dataset and a newly collected SAIC dataset. Results show that the proposed model predicts steering angles and speed values accurately. Furthermore, we improve the failure data synthesis methods to solve the problem of error accumulation in real road tests.
I. INTRODUCTION
End-to-end steering control does not provide sufficient vehicle control because speed commands are also needed. The paper therefore proposes joint steering and speed prediction, adding feedback speeds and failure-case synthesis for more effective control.
- Steering angle alone is insufficient for vehicle control because end-to-end methods lack speed commands.
- The paper first predicts accelerating, decelerating, and maintaining-speed commands from front-view image sequences.These discrete commands are chosen because visual inputs can indicate obstacles, clear roads, and low vehicle speed.
- The proposed multi-modal multi-task network uses previous feedback speeds with visual inputs to predict speeds and steering angles simultaneously.
- The work contributes a new SAIC dataset containing driving records collected during both daytime and nighttime.
- The authors improve failure-case data synthesis to address error accumulation during real-car tests.
II. RELATED WORK
Autonomous-driving research progressed from early neural-network driving systems to CNN-based end-to-end steering and approaches that improve interpretation or structure perception and control.
- ALVINN was among the earliest successful neural-network-based self-driving vehicle projects.
- CNN-based behavior-reflex systems directly predict steering angles from visual inputs, while mediated-perception and privileged-training approaches use more structured information.Behavior-reflex CNNs offer low model complexity and generalization with enough training data, but performance can be limited in complicated environments.
- Visualization methods and attention mechanisms have been proposed to make end-to-end driving predictions more interpretable.
III. METHOD
The method develops an end-to-end CNN hierarchy from steering-only prediction to joint steering and discrete-speed prediction, then adds feedback speed as a second input modality.
- The paper first uses a base CNN for end-to-end steering-angle prediction, then extends it to multi-task vehicle control.
- The final multi-modal multi-task network takes previous feedback speeds as an extra modality and predicts speed and steering angle simultaneously.
- The improved CNN contains nine layers, including five convolutional and four fully connected layers, with AlexNet-based convolutional design and larger early kernels.
- Mean absolute error trains steering prediction, with larger loss weights assigned to turning data to address steering-data imbalance.
B. Discrete Speed Command Network
The discrete speed command network extends end-to-end steering control by classifying speed commands from image sequences. The later multi-modal design adds feedback-speed information because visual inputs alone are insufficient for reliable speed commands.
- Steering angle alone is insufficient for vehicle control because vehicle speed is another required control parameter.
- The speed command network uses image sequences to predict steering angles and three discrete commands: accelerating, decelerating, and maintaining speed.
- Visual inputs alone can produce incorrect commands when the vehicle is already fast, at the speed limit, or slow enough to continue without decelerating.
- The multi-modal network combines a visual encoder with a speed encoder, using visual features for steering and concatenated visual-speed features for speed prediction.The visual encoder processes one frame rather than CNN-plus-LSTM sequences to reduce computation and support real-time performance on lower-performance GPUs.
IV. DATASET
The paper uses the public Udacity dataset and a newly collected SAIC dataset for evaluation. SAIC contains urban driving data across day and night conditions, with filtered data split into training, validation, and testing sets.
- The Udacity dataset contains about 20 minutes across six video clips, with speeds, steering angles, and three front-view camera streams.
- The SAIC dataset records five hours of mostly urban driving in north San Jose during both day and night.
- After filtering parking, traffic-light waiting, and other noisy conditions, two hours of SAIC data are split into training, validation, and testing sets.
B. Data Pre-Processing
The preprocessing pipeline targets robustness and prediction accuracy under changing vehicle-camera conditions. It combines color-space conversion, geometric augmentation, image flips, and side-camera failure-case synthesis.
- Frames are converted from RGB to HSV to improve robustness toward lighting changes.
- Random rotations simulate camera vibrations, while horizontal flips augment the training data.
- Side-camera data synthesis generates simulated failure cases for training.
2) Speed Command Generating:
Discrete speed commands are generated from one-second speed intervals by thresholding calculated acceleration. The resulting classes are accelerating, decelerating, and maintaining speed.
- Acceleration is calculated from the speed difference between the interval’s end and start, using one-second intervals.The paper denotes these speeds as speede and speeds.
- Accelerations above 0.25m/s2 are labeled Accelerating, while values below -0.25m/s2 are labeled Decelerating.
- Remaining minor speed changes are labeled Maintaining Speed.
- The evaluation includes speed value prediction on both the Udacity and SAIC datasets using the multi-modal multi-task network.
A. Steering Angle Prediction
The proposed models improve steering-angle prediction through multi-task learning and an additional feedback-speed modality. On Udacity, steering error falls from 2.84° to 1.26°, while the model achieves 0.17° on SAIC.
- The proposed model is evaluated against PilotNet, the Cg Network, and an improved single-task steering model using MAE in degrees.
- The improved base steering model outperforms reimplemented PilotNet and Cg Network on Udacity.
- 1.85° steering MAE improves on the base model’s 2.84° when discrete speed prediction is added.
- 1.26° steering MAE is achieved when previous feedback speeds are added as an extra modality to the multi-task network.
- The multi-modal multi-task network achieves 0.17° steering-angle prediction accuracy on the SAIC dataset.
B. Discrete Speed Command Prediction
The model first formulates speed prediction as classification of accelerating, decelerating, and maintaining-speed commands, jointly with steering prediction. On Udacity, this achieves 65.0% command accuracy and improves steering-angle accuracy, though the results remain limited by noisy and visually ambiguous speed cues.
- B. Discrete Speed Command Prediction: The experiment converts acceleration sequences into three discrete commands—accelerating, decelerating, and maintaining speed—for multi-task prediction with steering angles.All command labels are represented as one-hot vectors.
- B. Discrete Speed Command Prediction: 65.0% speed command classification accuracy is achieved on the Udacity dataset.
- B. Discrete Speed Command Prediction: Steering-angle prediction accuracy improves from 2.84° to 1.85° with the multi-task model.
- B. Discrete Speed Command Prediction: The discrete-command results are limited by noisy human-factor-related speed changes and the difficulty of inferring speed commands from visual inputs alone.Increasing the acceleration interval can reduce noise but delays command generation.
C. Speed Control Value Prediction
The multi-modal multi-task network predicts the next-frame speed value from visual input and ten previous feedback speeds, while also predicting steering. It achieves low speed-prediction errors on both datasets, but road tests expose error accumulation that motivates synthesized failure cases.
- C. Speed Control Value Prediction: The multi-modal network predicts the next-frame speed value from one visual frame and feedback speeds from ten previous timestamps.Speed prediction is modeled as regression, using numerical speed labels in m/s.
- C. Speed Control Value Prediction: The model achieves an MAE of 0.19m/s on Udacity and 0.45m/s on SAIC.No baselines were available because the speed-prediction task was novel.
- D. Road Tests and Data Synthesis: Road-test error accumulates in steering and speed control, with feedback speeds contributing to future speed-prediction errors.Failure-case samples are therefore needed, but collecting them with human drivers is dangerous and infeasible.
- D. Road Tests and Data Synthesis: Side-camera data synthesis adds artificial recovering angles and extends synthesis to speed data.The experiments use a 20-inch camera offset and a one-second recovery time; synthesized failures enable autonomous driving under similar SAIC conditions.
VI. CONCLUSION
The paper proposes joint end-to-end prediction of speed and steering from front-view recordings and feedback-speed sequences. Experiments report accurate speed prediction, improved steering prediction, a new SAIC dataset, and extended failure-case synthesis for road-test error accumulation.
- VI. CONCLUSION: The proposed multi-modal multi-task framework jointly predicts speed values and steering angles from front-view recordings and feedback speed sequences.
- VI. CONCLUSION: Experiments show accurate speed prediction and improved steering-angle prediction with the proposed framework.
- VI. CONCLUSION: The study introduces a new SAIC dataset for evaluation and further studies.
- VI. CONCLUSION: Extended failure-case data synthesis is proposed to address error accumulation in real-vehicle road tests.