Source-linked AI summary
Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion
Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, Kostas Daniilidis
TL;DR
Event cameras make conventional photoconsistency-based motion learning difficult, especially when events are temporally distributed or blurred. The paper uses discretized event volumes and differentiable motion-compensation losses to learn optical flow, egomotion, and depth from events alone. It evaluates the approach on MVSEC and challenging scenes, while finding limited egomotion generalization in some settings.
Problem
Photoconsistency-based motion estimation is not directly valid for event streams, while existing learned approaches depend on grayscale images or can fail in blurry scenes.
Method
The framework encodes events in a discretized volume, predicts motion, deblurs event images, and minimizes residual blur; stereo census losses support metric pose and depth learning.
Results
The framework learns optical flow and egomotion and depth from the event stream only, with evaluations on MVSEC and qualitative results across challenging scenes.
Takeaways & Limitations
Motion compensation and event-specific representations provide an unsupervised route to learning optical flow, egomotion, and depth without grayscale input.
Takeaways & Limitations
Egomotion generalized reasonably only to outdoor day1; it performed poorly on indoor flying sequences and failed with fluorescent-light events in outdoor night scenes.
Abstract
from arXiv · showhide
In this work, we propose a novel framework for unsupervised learning for event cameras that learns motion information from only the event stream. In particular, we propose an input representation of the events in the form of a discretized volume that maintains the temporal distribution of the events, which we pass through a neural network to predict the motion of the events. This motion is used to attempt to remove any motion blur in the event image. We then propose a loss function applied to the motion compensated event image that measures the motion blur in this image. We train two networks with this framework, one to predict optical flow, and one to predict egomotion and depths, and evaluate these networks on the Multi Vehicle Stereo Event Camera dataset, along with qualitative results from a variety of different scenes.
1. Introduction
The paper addresses event-camera motion estimation without relying on photoconsistency by learning from the events’ spatiotemporal distribution and motion blur. It proposes discretized event volumes, motion-compensation losses, and networks for optical flow or egomotion and depth.
- Event cameras challenge conventional motion estimation because their events indicate only local log-intensity changes, making direct photoconsistency assumptions invalid.
- Prior self- and unsupervised methods still rely on grayscale images or photoconsistency assumptions that may fail in very blurry scenes.
- The paper introduces a discretized event volume that preserves the full spatiotemporal distribution of events for neural-network input.
- The framework predicts optical flow or egomotion and depth, using deblurred event images and a census-transform stereo loss for metric poses and depths.
- The methods are evaluated on the Multi Vehicle Stereo Event Camera dataset and on challenging nighttime and other scenes, with comparisons to grayscale-based methods and prior work.
- A motion-blur loss trains networks by predicting motion, deblurring event images, and minimizing residual blur using only the event stream.
2. Related Work
Prior event-camera methods estimate motion and structure using geometric representations, motion compensation, or learning-based photoconsistency. This work extends motion-blur minimization into a differentiable unsupervised learning framework and reports qualitative generalization across difficult scenes.
- Early event-camera methods estimate normal flow from event planes, jointly optimize intensity and flow, or reconstruct pose and maps with filtering and feature tracking.
- Self- and unsupervised frame-based methods learn optical flow, egomotion, and depth from brightness constancy, census losses, reprojection, and photoconsistency.
- Recent event-based approaches propagate events along motion directions and minimize motion blur in event images to estimate optical flow and other motion information.
- Qualitative flow outputs cover fast spinning, rapidly moving objects, and outdoor water motion, including dark scenes where grayscale imagery may miss the object.
- This paper adapts motion-blur loss into a fully differentiable formulation for unsupervised neural learning of optical flow and structure from motion.
3. Method
The method represents events as a temporally distributed volume, predicts motion with neural networks, and uses motion compensation to minimize blur for unsupervised learning of flow, egomotion, and depth.
- Pipeline: The pipeline passes a volumetric event representation through a convolutional network to predict optical flow or egomotion and depth.Predicted motion is then used to deblur the events before applying task-specific losses.
- 3.1. Input: The Discretized Event Volume: The discretized event volume bins timestamps and accumulates events with linear weighting, retaining their spatial and temporal distribution.This interpolation-like accumulation improves temporal resolution beyond the number of bins; overlapping events can still cause information loss.
- 3.1. Input: The Discretized Event Volume: The method treats time bins as channels and applies 2D convolutions, because full 3D convolutions yielded negligible gains for substantially greater processing time.The reported comparison concerns processing efficiency rather than a change to the event representation.
- 3.2. Supervision through Motion Compensation: Motion compensation propagates events to a target time, producing less blur for correct motion and potentially more blur for incorrect motion.The resulting deblurring quality supplies the principal supervision signal for motion prediction.
- 3.2. Supervision through Motion Compensation: The temporal loss separates events by polarity, forms bilinearly interpolated average-timestamp images, and minimizes the sum of their squared values.Computing the loss at both temporal endpoints mitigates gradient scaling so events contribute more evenly.
- 3.3. Optical Flow Prediction Network: The flow network combines the temporal motion-compensation loss with local smoothness regularization, while the second network adds stereo disparity supervision.The egomotion-depth network predicts camera pose and disparities, and applies a census-transform similarity loss between warped stereo event images.
- 3.4.2 Stereo Disparity Loss: For egomotion and depth, deblurred left and right event images are compared through census transforms after warping with predicted disparities.A Charbonnier loss is applied in both directions to accommodate differences in event counts between cameras.
4. Experiments
Experiments evaluate optical flow, egomotion, and depth-related performance on MVSEC sequences against established baselines. The method performs well for several outdoor evaluations but has clear generalization and comparison-scope limitations.
- Optical Flow Evaluation: Optical flow is evaluated on indoor flying and outdoor MVSEC sequences using AEE and the percentage of valid points exceeding 3 pixels.Predictions are generated at grayscale-frame timestamps for durations of one and four grayscale frames.
- Egomotion Evaluation: Egomotion is evaluated on outdoor day1 using relative pose and rotation errors, with comparison to SFMLearner trained on grayscale images.The comparison acknowledges that the proposed method uses stereo images during training, whereas SFMLearner is monocular.
- Egomotion Evaluation: Egomotion generalization is limited for indoor flying and outdoor night sequences, where fluorescent lights produce spurious events that lead to erroneous estimates.The network was only trained on driving sequences, and the authors identify filtering such anomalies as future work.
- Depth Evaluation: Depth evaluation compares against Monodepth using cropped VI-Sensor images, while ECN is excluded because its results provide only relative depth up to an unknown scale.Depth errors are reported for ground-truth thresholds up to 10 m, 20 m, and 30 m with at least one event.
- Egomotion Evaluation: The outdoor day2-trained model generalizes to outdoor day1 despite a substantial scene change, and its relative pose and rotation errors are significantly better than SFMLearner's.A recovered trajectory is also shown for outdoor day1.
5. Results
The proposed networks perform well across several challenging event-camera scenes, but generalization is constrained by scene memorization and spurious events from flashing lights.
- 5.1. Optical Flow: Our optical flow method outperforms EV-FlowNet in almost all experiments and nears UnFlow on short one-frame sequences.
- 5.1. Optical Flow: The flow network generalizes to very fast motions and dark environments, but struggles when events arise from flashing lights rather than motion.
- 5.2. Egomotion: The outdoor day2-trained model generalizes well from residential roads to a closed office park, capturing trajectory rotation despite overestimating scale.
- 5.2. Egomotion: Egomotion and depth networks memorize more scene-specific information and therefore generalize poorly to sequences such as indoor flying.
- 5.2. Egomotion: Flashing fluorescent lamps generate spurious events that the egomotion network interprets as camera motion, producing erroneous estimates on outdoor night sequences.
- 5.3. Depth: The depth model performs well on driving sequences and outperforms Monodepth, but fails to generalize to flying scenes and predicts very close depths for flashing lights.
6. Conclusions
The work combines a volumetric event representation with motion-compensation-based learning to predict optical flow, egomotion, and depth from event streams. Qualitative outputs cover indoor flying, outdoor day, and outdoor night sequences.
- 6. Conclusions: A novel event representation and motion-compensation loss enable deep networks to predict optical flow, egomotion, and depth using only event streams.
- 6. Conclusions: Qualitative results present grayscale images, event images, predicted depths with heading direction, and ground truth across indoor flying and outdoor day and night sequences.