Source-linked AI summary
Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso Garcia, Davide Scaramuzza
TL;DR
The paper asks whether event cameras can support robust vehicle steering-angle prediction where traditional cameras are challenged by motion and illumination. It converts events into polarity-separated frames, applies adapted convolutional networks, and evaluates them at large scale. The resulting event-based approach outperforms traditional-camera systems and remains robust across difficult conditions.
Problem
The paper addresses the limited use of event cameras for continuous steering-angle estimation and the challenge of robust prediction under difficult visual conditions.
Method
The method converts asynchronous events into polarity-separated frames and processes them with adapted ResNet-based convolutional networks for steering-angle regression.
Results
The event-based approach outperforms state-of-the-art traditional-camera systems and is especially robust under challenging illumination, fast motion, and large input spaces.
Takeaways & Limitations
Event cameras’ motion-sensitive, high-dynamic-range response can support robust steering prediction, while transfer learning from traditional-image networks improves event-based vision.
Abstract
from arXiv · showhide
Event cameras are bio-inspired vision sensors that naturally capture the dynamics of a scene, filtering out redundant information. This paper presents a deep neural network approach that unlocks the potential of event cameras on a challenging motion-estimation task: prediction of a vehicle's steering angle. To make the best out of this sensor-algorithm combination, we adapt state-of-the-art convolutional architectures to the output of event sensors and extensively evaluate the performance of our approach on a publicly available large scale event-camera dataset (~1000 km). We present qualitative and quantitative explanations of why event cameras allow robust steering prediction even in cases where traditional cameras fail, e.g. challenging illumination conditions and fast motion. Finally, we demonstrate the advantages of leveraging transfer learning from traditional to event-based vision, and show that our approach outperforms state-of-the-art algorithms based on standard cameras.
Multimedia Material
An accompanying video is available to illustrate the paper’s approach and results.
- The paper provides an accompanying video at the listed YouTube link.
1. Introduction
The paper investigates whether event cameras and deep learning can improve steering-angle prediction by exploiting motion information that standard cameras miss. It introduces a large-scale event-based regression study and reports advantages over traditional-camera systems.
- Event cameras asynchronously record pixel-wise intensity changes rather than full images at a fixed frame rate.They offer high temporal resolution, high dynamic range, low power and bandwidth requirements, and natural motion detection.
- The paper proposes learning steering policies from event-camera data, targeting motion estimation rather than autonomous-vehicle control.The stated goal is to understand how event-camera dynamics, redundancy reduction, speed, and dynamic range benefit learning-based motion estimation.
- The proposed event-based approach outperforms state-of-the-art systems based on standard cameras.
- ≈1 million images covering over 1000 km support the paper’s large-scale deep-learning application to event-based regression.
- The study adapts convolutional architectures to event-camera outputs and evaluates transfer learning from traditional-camera classification networks.
2. Related Work
Prior event-based learning work mainly addressed classification or discretized continuous problems on small datasets. This paper instead performs principled continuous steering-angle regression using event-camera data.
- Robust autonomous-driving policies remain challenging because visual environments are diverse and unpredictable.
- The paper learns policies from asynchronous, pixel-wise brightness changes with very low latency and high dynamic range.
- Previous learning-based event-camera studies primarily used classification tasks and generally relied on limited-size datasets.
- Earlier continuous-estimation approaches discretized the solution space into finite classes rather than predicting continuous values.
- The proposed steering-angle task is continuous regression, allowing outputs anywhere in [−180°,180°] without partitioning the solution space.
3. Methodology
The method converts asynchronous events into polarity-separated synchronous frames and feeds them to adapted convolutional networks for steering-angle regression. It compares these event representations with grayscale and temporal-difference inputs.
- Event-to-Frame Conversion: Event streams are accumulated pixel-wise over a constant interval to create event frames that encode relative scene-camera motion.
- Event-to-Frame Conversion: Positive and negative events are stored in separate histogram channels to prevent opposite-polarity cancellation and information loss.
- Network Architecture: The network uses stacked event frames with ResNet18 or ResNet50 convolutional feature extractors adapted for regression.
- Input Representations: The DDD17 inputs include traditional grayscale frames, temporal differences of consecutive frames, and time-integrated event images.
4. Experimental Setup
The experiments use the DDD17 event-camera dataset and evaluate steering-angle regression across several visual representations under challenging driving conditions. The setup compares event-based inputs with grayscale and temporal-difference images using temporally separated training and testing sequences and RMSE/EVA metrics.
- Dataset: DDD17 contains approximately 12 hours of annotated driving recordings totaling 432 GB, with asynchronous events and synchronous grayscale frames captured concurrently by the DAVIS2 sensor.The recordings span different weather, road, and illumination conditions and are divided into day, day sun, evening, and night subsets.
- Dataset split: Training and testing use alternate, consecutive non-overlapping sequences rather than random frames, with 40 seconds for training and 20 seconds for testing.This split avoids over-optimistic estimates caused by the similarity of successive frames and steering angles.
- Visual inputs: Steering angles are predicted from grayscale images, grayscale-frame differences, and event-accumulation images.The grayscale frames come from the same DAVIS photodiodes as the events, enabling a fair comparison across visual inputs.
- Visual inputs: Event cameras encode pixel-wise log-brightness changes of predefined size C, while accumulated events quantify brightness increases or decreases over a time interval.For small intervals, consecutive grayscale-frame differences provide a first-order Taylor approximation to the same intensity change, explaining the visual similarity between the two representations.
- Evaluation: The regression target is the steering-wheel angle one-third of a second after the current event or grayscale frame, evaluated with RMSE and explained variance.RMSE measures average prediction-error magnitude, whereas EVA compares residual variance with observed-value variance; EVA can be zero or negative when residual variance is not smaller.
5. Experiments
The experiments examine integration time, input representation, illumination scenarios, and scalability against traditional-camera methods. Event-based networks perform best at an intermediate integration time and consistently outperform grayscale-based baselines across scenarios and the full dataset.
- Experimental design: The experiments vary event integration time, compare event frames with grayscale and grayscale-difference inputs, and evaluate scaling to the full dataset against traditional-camera methods.The study uses the same convolutional architectures for different input types and trains each network independently.
- Integration-time sensitivity: Larger integration times create longer event traces, while excessive accumulation can blur object contours and reduce discriminability; small times produce few, less informative events.The authors hypothesize that networks exploit motion cues in the event traces for steering prediction.
- Integration-time sensitivity: The ResNet50 network performs best with 50 ms integration time, with performance degrading for both smaller and larger integration times.The authors use 50 ms in subsequent experiments.
- Illumination scenarios: On the difficult day subset, event images outperform grayscale and grayscale-difference baselines, while grayscale models struggle with lighting artifacts and high-velocity blur.ResNet18 trained on grayscale converges to a quasi-constant solution with EVA ≈0, whereas event-based models outperform both baselines with both architectures.
- Illumination scenarios: Across day, day-sun, evening, and night scenarios, the event-camera solution largely outperforms the baselines, including with the simpler ResNet18 architecture.The smaller evening and night subsets make their statistics easier for networks to model, despite their challenging illumination conditions.
- Entire-dataset evaluation: On the entire dataset, event-image architectures outperform traditional-frame baselines, with ResNet50 performing best and ImageNet initialization improving training and estimation.ImageNet initialization transfers low-level features such as edges from traditional RGB images to time-integrated event images.
6. Discussion
Event representations improve steering prediction by preserving motion information, while their HDR response broadens robustness across illumination conditions. Transfer learning from traditional-image classifiers helps overcome the data demands of learning motion features.
- Event-based representations: Microsecond temporal resolution preserves edge details at high velocities, where grayscale frames suffer from motion blur.Positive and negative events are provided in separate channels to the network.
- Event-based representations: 140 dB dynamic range versus 55 dB for dataset grayscale frames supports robustness across bright, dark, and rapidly changing illumination.Event data represent HDR scene content without the long exposure times required by traditional cameras.
- Event-based representations: Moving-edge responses filter temporally redundant data and provide information about vehicle motion, while selecting an appropriate integration time further improves performance.The paper identifies adaptive integration-time policies as a direction for future work.
- Transfer learning: Transfer learning from networks trained on traditional images can simplify learning motion features and unlock event-camera performance despite data demands.The approach reuses knowledge from classification networks trained on traditional images.
7. Conclusion
The paper demonstrates that a deep-learning approach tailored to event-camera outputs can accurately predict steering angles across varied conditions. It is especially robust where grayscale frames fail and outperforms state-of-the-art traditional-camera systems.
- Conclusion: The event-based deep-learning approach accurately predicts steering angles under a wide range of conditions.It learns steering angles from motion cues contained in event frames.
- Conclusion: Robustness is especially evident with large input spaces, challenging illumination conditions, and fast motion, where grayscale frames fail.These conditions are identified as examples in the paper's conclusion.
- Conclusion: The proposed method outperforms other state-of-the-art systems based on traditional cameras.