Source-linked AI summary
E-RAFT: Dense Optical Flow from Event Cameras
Mathias Gehrig, Mario Millhäusler, Daniel Gehrig, Davide Scaramuzza
TL;DR
Event-based optical flow lacks explicit matching costs and has typically been evaluated with U-Net-based methods for sparse prediction. The paper introduces dense estimation with feature correlation and sequential warm-starting, together with a more challenging real-world dataset. It reports 23% lower EPE on MVSEC and 66% lower EPE on the proposed dataset than existing approaches.
Problem
Event-based optical flow methods generally lack explicit matching costs, rely on U-Net architectures, and have been evaluated on datasets with limited displacement and resolution.
Method
The approach performs dense optical flow estimation using cost volumes and differentiable warm-starting to incorporate temporal information from sequential events.
Results
23% lower EPE on MVSEC (0.27 vs 0.35) and 66% lower EPE on the proposed dataset (0.79 vs 2.32) are reported against existing approaches.
Takeaways & Limitations
Cost volumes and temporal recurrency are reported as effective concepts for event-based optical flow, while the new dataset enables evaluation of larger displacements.
Takeaways & Limitations
Existing event-based datasets are constrained by small size and restricted camera motion, limiting their suitability for training and evaluation of neural-network approaches.
Abstract
from arXiv · showhide
We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Modern frame-based optical flow methods heavily rely on matching costs computed from feature correlation. In contrast, there exists no optical flow method for event cameras that explicitly computes matching costs. Instead, learning-based approaches using events usually resort to the U-Net architecture to estimate optical flow sparsely. Our key finding is that the introduction of correlation features significantly improves results compared to previous methods that solely rely on convolution layers. Compared to the state-of-the-art, our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC. Furthermore, we show that all existing optical flow methods developed so far for event cameras have been evaluated on datasets with very small displacement fields with a maximum flow magnitude of 10 pixels. Based on this observation, we introduce a new real-world dataset that exhibits displacement fields with magnitudes up to 210 pixels and 3 times higher camera resolution. Our proposed approach reduces the end-point error on this dataset by 66%.
1. Introduction
Optical flow is important for motion-aware vision, but event data is sparse, irregular, and asynchronous, while prior event-based methods largely use U-Nets for sparse prediction. E-RAFT addresses this gap with dense estimation, feature cost volumes, temporal warm-starting, and a higher-displacement dataset, improving results on both MVSEC and the proposed benchmark.
- Motivation: Event cameras offer microsecond temporal resolution, over 120 dB dynamic range, and no motion blur, but their asynchronous event format complicates motion extraction.These properties are valuable for optical flow in real-world applications, although event data is sparse and irregular rather than directly image-like.
- Prior work: Prior event-based optical flow methods use plane fitting, variational optimization, or convolutional architectures, with learning-based methods typically producing sparse predictions through U-Nets.Frame-based methods instead commonly use feature correlations summarized in cost volumes.
- Evaluation: 23% lower EPE on MVSEC (0.27 vs 0.35) and 66% lower EPE on the proposed dataset (0.79 vs 2.32) demonstrate improvements over existing approaches.The proposed dataset reaches flow magnitudes up to 210 pixels and has three times MVSEC’s event-camera resolution.
- Approach: E-RAFT is designed for dense event-based optical flow and introduces cost volumes plus differentiable warm-starting to exploit sequential event data.Warm-starting initializes the next time step’s flow estimate and supports fine-tuning the non-recurrent model without additional parameters.
- Dataset: The new real-world dataset targets shortcomings of existing event-based benchmarks, including small displacements, low resolution, impaired ground truth, and unclear train-test separation.The authors report that training on the new dataset enables high-quality predictions unattainable with previous event-based datasets.
2. Related Work
Prior event-based optical-flow work includes algorithmic and learning-based approaches, but existing datasets and image-based assumptions limit evaluation and robustness. The paper motivates cost-volume methods for events and a dataset with improved scale and quality.
- Image-based optical-flow methods rely on feature correlations, cost volumes, and iterative updates, motivating their adaptation to event data.
- Image-based methods are vulnerable to motion blur, limited dynamic range, and other image degradations in challenging scenarios.
- Event-flow algorithms span iterative asynchronous, plane-fitting, and variational optimization approaches.
- Learning-based event-flow methods mainly use self-supervision and U-Net architectures, often masking predictions where events are absent.
- MVSEC has impaired ground-truth quality, low resolution, small displacements, and no clear train-test split.The paper proposes addressing these shortcomings with improved ground truth, 3 times higher camera resolution, and a defined split.
3. Approach
The approach adapts RAFT-style correlation and recurrent updates to sequential event representations, then propagates prior flow through differentiable warm-starting. It trains with supervision across sequence times and recurrent iterations.
- Core architecture: Consecutive event packets are converted into tensor-like representations and encoded to produce an all-pairs correlation volume at 1/8-th resolution.A separate context encoder supplies features for the recurrent update module.
- Problem definition: Dense event optical flow estimates pixel correspondences between timestamps using the events occurring between those timestamps.
- Event representation: The model regresses current flow from two short consecutive event sequences rather than only the events between the current timestamps.
- Differentiable warm-starting: Differentiable warm-starting is trained end-to-end so temporal priors are incorporated and gradients backpropagate through time without adding parameters.The module can also be added to already trained networks for finetuning.
- Differentiable warm-starting: Previous flow is forward-warped before initializing the next estimate, resolving source-location changes and ambiguities through average splatting.The warped flow is normalized with bilinear-kernel weights.
- Supervision: Training supervises optical-flow predictions with L1 distance to ground truth across sequence times and recurrent iterations, using γ = 0.8.
4. DSEC-Flow Dataset
DSEC-Flow extends DSEC with optical-flow ground truth and targets the displacement and resolution limitations of MVSEC. Its displacement distribution is substantially larger, including after normalization by image width.
- Dataset construction: DSEC-Flow extends the DSEC driving dataset with optical-flow ground truth and VGA event cameras containing 3.4 times as many pixels as MVSEC’s Davis346.The dataset’s disparity ground truth is reprojected using calibration data and combined with odometry under a static-scene assumption.
- Motivation: Large displacement fields matter because large displacements are harder to predict accurately, while long-range correspondences support accurate, driftfree flow estimation.
- Displacement distribution: 80 % of MVSEC flow magnitudes are below 4 pixels, compared with 22 pixels at the 80 % percentile for DSEC-Flow.For temporally upsampled MVSEC, the corresponding value is 1.7 pixels.
- Displacement distribution: At the 80 % percentile, normalized flow magnitude is 3.4 % for DSEC-Flow versus 1.3 % for MVSEC and 0.5 % for its upsampled version.This indicates greater challenge even after accounting for camera resolution.
- Evaluation: The evaluation uses approximately 2100 ground-truth flow maps from seven sequences and reports EPE plus N-pixel errors for thresholds 3, 2, and 1.EPE is treated as the default metric because it measures flow accuracy most accurately.
5. Experiments
Experiments on MVSEC and DSEC-Flow evaluate dense optical flow, dataset difficulty, and warm-starting. The approach outperforms baselines on both datasets, with especially large gains on DSEC-Flow.
- MVSEC: 0.27 dense EPE and 0.24 sparse EPE show that the approach outperforms sparse and dense MVSEC baselines.Sparse EPE is 11% lower than dense EPE, making dense prediction the more difficult evaluation.
- MVSEC: Supervised methods generally outperform self-supervised methods on MVSEC, with supervised EV-FlowNet reaching 0.31 sparse EPE versus Zhu et al. at 0.32.The comparison includes sparse and dense evaluation settings.
- MVSEC: 23% lower EPE than supervised EV-FlowNet is achieved on both temporally upsampled and original-rate MVSEC ground truth.The original-rate evaluation does not provide additional insight into larger-displacement performance, while 1-pixel error increases for both methods.
- DSEC-Flow: 2.32 versus 0.79 EPE on DSEC-Flow gives the proposed method a 2.9-times lower error than EV-FlowNet.The baseline performs substantially worse on DSEC-Flow than in the MVSEC experiments, possibly because it was designed for small displacements.
- Ablation Study: 4.0% lower EPE on DSEC-Flow is obtained when warm-starting is incorporated during both training and evaluation.This configuration outperforms the ablated methods, although the no-warm-start baseline performs better than some inference-only variants.
- Ablation Study: 4.2% lower EPE on MVSEC is achieved by training and evaluating with warm-starting compared with the closest ablated method.Inference-only warm-starting performs better than no warm-starting on MVSEC, unlike the DSEC-Flow result.
6. Conclusion
The work adapts frame-based optical-flow design choices to event data and introduces a real-world dataset addressing shortcomings of existing event-based datasets.
- The proposed method adapts frame-based optical-flow design choices for event-based input.
- The method drastically outperforms baseline methods on two datasets.
- The results suggest that the commonly used U-Net architecture is suboptimal for event-based optical flow.
- The work introduces a novel real-world optical-flow dataset addressing major shortcomings of existing event-based datasets.
- Training on the new dataset leads to high-quality optical-flow predictions unattainable with previous event-based vision datasets.