Source-linked AI summary
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, Kostas Daniilidis
TL;DR
Event-based cameras offer fine-grained temporal information, but deep learning for them is hindered by asynchronous inputs and limited labeled data. EV-FlowNet uses a fixed-channel image representation of events and trains optical flow with synchronized grayscale images as a self-supervisory signal. It predicts accurate flow from events alone and is competitive with image-based and traditional methods, while motivating broader transfer of self-supervised learning to event cameras.
Problem
Deep learning for event cameras lacks suitable image-like representations and sufficient labeled data for supervised training.
Method
EV-FlowNet represents events as fixed-channel images of counts and latest timestamps, then trains flow prediction with a grayscale-image photoconsistency loss.
Results
EV-FlowNet predicts accurate optical flow from events alone and is competitive with image-based self-supervised and standard non-learning methods.
Takeaways & Limitations
The method enables optical-flow learning without ground-truth-flow supervision or manual labeling and provides a framework for transferring self-supervised methods to event cameras.
Abstract
from arXiv · showhide
Event-based cameras have shown great promise in a variety of situations where frame based cameras suffer, such as high speed motions and high dynamic range scenes. However, developing algorithms for event measurements requires a new class of hand crafted algorithms. Deep learning has shown great success in providing model free solutions to many problems in the vision community, but existing networks have been developed with frame based images in mind, and there does not exist the wealth of labeled data for events as there does for images for supervised training. To these points, we present EV-FlowNet, a novel self-supervised deep learning pipeline for optical flow estimation for event based cameras. In particular, we introduce an image based representation of a given event stream, which is fed into a self-supervised neural network as the sole input. The corresponding grayscale images captured from the same camera at the same time as the events are then used as a supervisory signal to provide a loss function at training time, given the estimated flow from the network. We show that the resulting network is able to accurately predict optical flow from events only in a variety of different scenes, with performance competitive to image based networks. This method not only allows for accurate estimation of dense optical flow, but also provides a framework for the transfer of other self-supervised methods to the event-based domain.
I. INTRODUCTION
EV-FlowNet addresses event-camera deep learning’s asynchronous inputs and limited labeled data with an image-based event representation and self-supervised optical-flow training. The method learns flow from events alone and is competitive with image-based and traditional methods.
- Event cameras provide microsecond-accurate temporal information useful for optical-flow estimation but existing algorithms require hand-crafted outlier rejection.
- The proposed representation converts each event stream into fixed-channel images encoding event counts and latest timestamps at each polarity and pixel.It preserves spatial relationships and recent temporal information while fitting standard image-based neural networks.
- EV-FlowNet trains from event inputs using a self-supervised photoconsistency loss based on synchronized grayscale images available during training only.The grayscale images supervise predicted flow without being passed into the network as input.
- EV-FlowNet is competitive with UnFlow and standard non-learning optical-flow methods on a newly generated event-camera optical-flow dataset.The evaluation dataset uses ground-truth depths and poses from the Multi Vehicle Event Camera Dataset.
- The network can learn accurate optical flow from events alone without ground-truth-flow supervision or expensive manual labeling.Training uses event data and corresponding grayscale images generated by the same camera.
II. RELATED WORK
Related work develops event-based optical-flow estimators, datasets, and neural approaches, but event representations and labeled-data availability remain central challenges. Prior work spans geometric, hybrid, dataset-generation, and neuromorphic strategies.
- Prior optical-flow methods estimate motion from spatiotemporal event geometry, sometimes adding iterative outlier rejection or combining event flow with traditional images.
- Labeled event-camera data remain scarce, leading early learning-based studies to rely on small, hand-collected datasets.
- Later datasets provide event sequences with ground-truth pose, depth, or vehicle measurements for learning and evaluation.
- Other work generates event-camera equivalents of existing image-based datasets by recording those datasets with event-based cameras.
- Neural networks have also been implemented on spiking neuromorphic processors for tasks such as gesture recognition.
C. Self-supervised Optical Flow
EV-FlowNet converts asynchronous events into a fixed four-channel image representation so standard convolutional networks can estimate optical flow, using grayscale frames only during self-supervised training.
- Self-supervised training: Training uses event timestamp images with grayscale images captured immediately before and after each event window, while inference uses event images alone.The approach follows self-supervised optical-flow methods using photometric and smoothness constraints.
- Event representation: The first two channels count positive and negative events at each pixel.Event counts provide a compact image-like summary of polarity-specific activity.
- Event representation: The final two channels store the most recent positive and negative event timestamps at each pixel.This retains recent temporal information without stacking every timestamp into separate channels.
- Event representation: Dense-event regions with large motion can receive nearly identical recent timestamps, but shorter time windows reduce this deficiency.The representation is otherwise reported as sufficient for estimating correct flow in most regions.
B. Self-Supervised Loss
The self-supervised objective uses grayscale-image reconstruction to supervise event-based flow, combining a photometric warp loss with a smoothness regularizer.
- Photometric loss: The photometric loss warps the second grayscale image toward the first using the estimated flow and bilinear sampling.It minimizes the intensity difference between the warped second image and the first image.
- Photometric loss: The Charbonnier loss is used in the photometric objective for outlier rejection.It is described as a common loss function in optical-flow literature.
- Smoothness loss: The smoothness loss regularizes flow by minimizing differences between neighboring pixels horizontally, vertically, and diagonally.The neighborhood is denoted N around pixel (x, y).
- Limitation: Because supervision comes from frame-based images, the method remains susceptible to image-based issues such as the aperture problem.The smoothness term is introduced as a regularizer for this setting.
- Loss formulation: The total loss is a weighted sum of photometric and smoothness losses.This objective follows traditional variational optical-flow formulations.
C. Network Architecture
EV-FlowNet uses an encoder-decoder architecture that progressively downsamples and upsamples the event image to produce a final flow estimate.
- Network architecture: The network applies four strided convolution layers, two residual blocks, and four upsample convolution layers.Output channels double through the encoder, while nearest-neighbor resampling precedes convolution in the decoder.
- Network architecture: Skip connections link each upsample convolution layer to its corresponding encoder layer.This follows the encoder-decoder design used in architectures such as U-Net and stacked hourglass networks.
IV. OPTICAL FLOW DATASET
The paper constructs a ground-truth optical-flow dataset from MVSEC poses and depths, generating reference flow across indoor and outdoor scenes for evaluation and release.
- Dataset construction: The dataset derives ground-truth optical flow from MVSEC stereo event-camera data, using camera poses and depth maps.The source scenes include flying, driving, and handheld sequences.
- Motion estimation: Camera poses at two times are differentiated numerically under a linear velocity assumption to estimate linear and angular velocities.The pose contains rotation R and translation p.
- Motion estimation: A central moving-average filter is applied to the estimated velocities before motion-field computation.The filtered velocities are used with ground-truth depths at undistorted pixel positions.
- Flow generation: The motion field is scaled by the image-pair time interval, and the resulting displacement approximates optical flow for each pixel.The flow is transformed to distorted-image coordinates by shifting pixels and applying distortion.
- Dataset scope: Ground-truth flow is generated for indoor flying, outdoor day, and outdoor night sequences, with indoor flying and outdoor day used for evaluation.All generated sequences are intended for dataset release.
V. EMPIRICAL EVALUATION
The evaluation trains on outdoor driving sequences with varied temporal gaps and standard data augmentation, using fixed optimization settings and substantial training compute.
- Training Data: Training uses two outdoor driving sequences containing roughly 12,000 and 26,000 images from visually different environments.The scenes include industrial-complex and public-road driving, with mostly straight and turning motions plus occasional moving vehicles and pedestrians.
- Training Data: Temporal gaps are varied by selecting image pairs up to k images apart, with k ∈[2, 4, 6, 8, 10, 12].Inputs are also horizontally flipped and randomly cropped to the desired resolution.
- Optimization: The model uses λ=0.5 for smoothness, equal weighting for intermediate losses, Adam optimization, and a learning rate initialized at 1e-5.The learning rate decays by 0.8 every 4 epochs; training lasts 300,000 iterations and takes around 12 hours on an NVIDIA Tesla V100.
B. Ablation Studies
The ablation study compares input representations and model capacity, alongside event-based and frame-based reference methods.
- Ablation Design: The study trains EV-FlowNet variants to test the effects of changing the input representation and increasing network capacity.The described architecture is denoted EV-FlowNet2R.
- Input Ablation: EV-FlowNetC uses only event counts, while EV-FlowNetT uses only event timestamps.These variants isolate the contributions of the first two and last two input channels, respectively.
- Baselines: Comparisons include Event-based Visual Flow, an event-based optimization method, and UnFlow, a self-supervised method for traditional frames.The Event-based Visual Flow implementation returned valid flow values for fewer than 5% of points, whereas UnFlow produced reasonable testing results after fine-tuning on outdoor day2.
D. Test Sequences
Testing compares event-based and frame-based flow across outdoor and indoor sequences, using controlled frame gaps and endpoint-based metrics, while including a documented fast-motion failure case.
- Test Sequences: UnFlow is evaluated on 800 outdoor day1 frames and indoor flying sequences 1 to 3, with all intervening events used as event input.The outdoor day1 test segment was selected for consistently bright grayscale images and minimal camera shaking.
- Test Sequences: Indoor flying sequences total roughly 240s and include vertical and backward motions unseen in the driving scenes.Fine-tuning on these sequences was avoided because their similar appearance could make overfitting easy.
- Metrics: Average endpoint error measures the distance between predicted and ground-truth flow-vector endpoints.The evaluation also reports the percentage of pixels whose endpoint error exceeds 3 pixels and 5% of flow magnitude.
- Metrics: AEE is computed only where at least one event occurred and ground-truth flow is available, covering 20-30% of pixels per image.The same event mask is applied to event-based and frame-based methods.
- Evaluation Protocol: Each sequence is evaluated with image pairs one frame apart and four frames apart to vary the observed flow magnitude.Table I reports AEE and the percentage of outlier pixels for both settings across the evaluated methods.
F. Results
Qualitative evaluation finds that predicted flow closely follows ground truth, while event sparsity and timestamp overwriting define important boundaries of the predictions.
- Qualitative Results: Predicted flow closely follows ground truth across the qualitative results and test set.The examples include outdoor day1 and indoor flying scenes.
- Qualitative Results: The network tends to predict zero flow in areas without events because the photometric loss provides little intensity-change signal there.Useful flow can therefore be extracted by using predictions at points with events.
2) Ablation Study Results:
The ablations show that combining event counts with timestamps improves flow estimation, while performance depends on motion and temporal-window conditions. The comparison also identifies noise behavior and potential overfitting in larger models.
- Ablation Study Results: Timestamp-only models outperform count-only models in most tests but fail under significant image noise or when fast motion overwrites older timestamps.The full network also struggles in regions dominated by recent timestamps.
- Ablation Study Results: Combined event-count and timestamp models perform better because counts indicate pixel importance while timestamps encode event ordering and velocity.Pixels with few events may be noise, whereas many events are more likely to carry useful information.
- Ablation Study Results: The larger EV-FlowNet4R performs worse than the smaller EV-FlowNet2R, likely because its greater capacity overfits training sets that differ substantially from the test sets.An additional 200,000 training iterations produced no appreciable improvement.
- Comparison Results: Compared with UnFlow, EV-FlowNet is less noisy in low-texture regions because sparse events lead it to predict no flow rather than an incorrect output.UnFlow is roughly correct for most inputs but tends to be very noisy in low-texture areas.
- Comparison Results: EV-FlowNet performs better on dt=4 tests and worse on dt=1 tests, except for outdoor driving1 and indoor flying3.The authors attribute extreme-motion problems to windows with too few events or saturated recent timestamps, suggesting event-count-based windows as one possible solution.
- Conclusion: The self-supervised network predicts optical flow from events alone and generalizes from hand-designed laboratory scenes to natural scenes.Its training does not require manual labeling, relying on data recorded directly from the camera.