Source-linked AI summary
Learning Dense and Continuous Optical Flow from an Event Camera
Zhexiong Wan, Yuchao Dai, Yuxin Mao
TL;DR
Two-frame methods estimate optical flow at fixed frame intervals, while event-only methods can estimate continuous flow but have difficulty producing reliable dense predictions in regions without events. DCEIFlow combines a first image and event streams using feature fusion, correlation, iterative updates, and bidirectional training. The authors report improved dense and continuous flow results over event-based state-of-the-art methods and baselines, while identifying failure cases involving weak texture and non-motion reflections.
Problem
Two-frame methods estimate flow only at discrete frame intervals, while event-only methods have difficulty producing reliable dense predictions in regions without triggered events.
Source: §I · p. 1Method
DCEIFlow fuses a first image with event streams through event-image correlation and iterative flow updates, and uses bidirectional training with reversed events.
Source: §I · p. 2Results
Experiments on synthetic and real-captured datasets report improved dense and continuous flow estimation over event-based state-of-the-art methods and designed baselines.
Source: §I · p. 2Takeaways & Limitations
The framework estimates dense flow like two-frame methods and continuous flow like event-based methods from a single image with event streams.
Source: §V · p. 13Weakly textured regions with few events can yield incorrect predictions, while reflections that generate non-motion events can seriously affect event-based predictions.
Source: E. Failure cases and limitation · pp. 12–13
Abstract
from arXiv · showhide
Event cameras such as DAVIS can simultaneously output high temporal resolution events and low frame-rate intensity images, which own great potential in capturing scene motion, such as optical flow estimation. Most of the existing optical flow estimation methods are based on two consecutive image frames and can only estimate discrete flow at a fixed time interval. Previous work has shown that continuous flow estimation can be achieved by changing the quantities or time intervals of events. However, they are difficult to estimate reliable dense flow , especially in the regions without any triggered events. In this paper, we propose a novel deep learning-based dense and continuous optical flow estimation framework from a single image with event streams, which facilitates the accurate perception of high-speed motion. Specifically, we first propose an event-image fusion and correlation module to effectively exploit the internal motion from two different modalities of data. Then we propose an iterative update network structure with bidirectional training for optical flow prediction. Therefore, our model can estimate reliable dense flow as two-frame-based methods, as well as estimate temporal continuous flow as event-based methods. Extensive experimental results on both synthetic and real captured datasets demonstrate that our model outperforms existing event-based state-of-the-art methods and our designed baselines for accurate dense and continuous optical flow estimation.
I. INTRODUCTION
The paper combines a single intensity image with event streams to estimate dense flow continuously from a frame-anchored start time. It introduces event-image correlation and bidirectional training, and reports improvements over event-based baselines and state-of-the-art methods.
I. INTRODUCTION
- Event-only methods can estimate continuous flow but struggle to produce reliable dense predictions where no events are triggered.Two-frame methods estimate dense flow only between frames; fusing a first image with events supports dense flow at varying intervals from that frame.Source: §I · p. 1
- DCEIFlow fuses a first image with events using an event-image fusion and correlation module, then predicts flow with an iterative update network.The fusion module creates a pseudo second-image feature constrained by the real second-image feature during training.Source: §I · p. 2
- Because dense-flow event data are unavailable, the model is pretrained on FlyingChairs2 with simulated events and evaluated on real-captured MVSEC.The paper also reports visual comparisons on EV-IMO.Source: §I · p. 2
- Bidirectional flow training uses reversed events and the second image to estimate backward flow with the same network.This training mechanism is intended to leverage the order of motion information in events.Source: §I · p. 2
- Experiments on MVSEC and EV-IMO report significant improvement over event-based state-of-the-art methods and designed baselines for dense and continuous flow estimation.The paper also reports comparisons on synthetic and real-captured datasets.Source: §I · p. 2
II. RELATED WORK
The related work reviews two-frame and event-based optical flow, including sparse, dense, and image-fused approaches. It then introduces a learning-based framework that combines events and a first image to estimate dense, continuous flow.
II. RELATED WORK
- The related-work section surveys learning-based two-frame and event-based optical-flow approaches, including dense and sparse estimates from events or a single image.Source: §II · p. 2
A. Two-Frame-based Optical Flow Estimation
- Two-frame optical-flow research progressed from FlowNet to designs using pyramids, warping, and correlation, with RAFT introducing an all-pairs iterative network.The review describes these as learning-based two-frame methods.Source: A. Two-Frame-based Optical Flow… · p. 2
B. Event-based Optical Flow Estimation
- Event-based methods face a density challenge because event streams encode discrete brightness changes rather than absolute brightness, complicating dense flow estimation.The review groups event-based methods by their inputs and outputs.Source: B. Event-based Optical Flow Estimation · p. 2
1) Sparse Flow from Events:
- Sparse event-flow methods include normal-flow estimation under brightness constancy, methods estimating full flow,, and an SNN approach.The cited SNN method estimates sparse flow efficiently, but the review notes its limited application.Source: 1) Sparse Flow from Events · p. 3
2) Dense Flow from Events:
- Dense event-only methods explore self-supervision, event representations, spiking networks, and recurrent structures, but their predictions are relatively unreliable in regions without triggered events.The review names EV-FlowNet, Zhu et al., and several representation and network variants.Source: 2) Dense Flow from Events · p. 3
3) Dense Flow from Single Image with Events:
- Prior single-image-plus-event methods include image reconstruction and optimization-based fusion, while Fusion-FlowNet uses a direct concatenation-fusion network.The review notes that reconstruction quality affects flow accuracy for one approach, and that optimization methods require time and post-processing.Source: 3) Dense Flow from Single Image with… · p. 3
III. APPROACH
- The proposed framework processes event volumes and image features through fusion, event-image all-pairs correlation, and iterative flow updating, with bidirectional training.It is designed to estimate dense and continuous optical flow from events and the first image.Source: §III · p. 3
A. Preliminaries
The preliminaries describe event-camera measurements and distinguish frame-based discrete flow from event-based continuous flow. They also review feature correlation as a way to represent matching between image features.
1) Event camera model:
- An event records a pixel’s space-time location and brightness-change polarity; an event is triggered when the log-brightness change reaches threshold c.The text identifies polarity p = ±1 with the direction of brightness change.Source: 1) Event camera model · p. 3
- Two-frame methods estimate flow at discrete integer frame intervals, whereas events support continuous flow at high temporal resolution within frames.The paper distinguishes its temporal meaning of discrete flow from discrete optimization in prior work,.Source: 2) Discrete Flow and Continuous Flow · p. 3
3) Feature Correlation for Matching:
- Feature correlation represents matching similarity between image features by dot products, using the second-image feature warped by coarse flow and a local search range.The search range is set by horizontal and vertical hyperparameters, and warping uses bilinear interpolation.Source: 3) Feature Correlation for Matching · p. 3
C. Dense Iterative Event-Image Flow Network
DCEIFlow combines event and image features to construct a matching representation, then iteratively refines dense optical flow. Its design also uses a polarity-separated event volume and compares against event-only and image-event baselines.
4) Baselines:
- The experiments use EV-FlowNet as an event-only baseline and Baseline-EI, which concatenates image and event features as decoder input.Baseline-EI is a modified EV-FlowNet with an added image encoder.Source: 4) Baselines · p. 4
B. Event Representation
- The event representation aggregates events into temporal bins for both polarities, concatenated into a 2B-channel event volume that preserves spatial-temporal information.The framework extracts event and image features before fusion and correlation construction.Source: B. Event Representation · p. 4, C. Dense Iterative Event-Image Flow… · p. 4
- Event-image fusion creates a pseudo second-image feature, which is correlated with the first-image feature to provide matching information for flow estimation.The convolution-based fusion separately encodes image and event features, concatenates them, and uses residual addition; the paper also describes direct addition as an ablation.Source: B. Event Representation · pp. 4–5
- The all-pairs event-image correlation spans feature-map locations, uses pooled pyramid levels for varied search ranges, and supplies local correlation lookups to the iterative updater.The correlation is built from the first-image feature and fused pseudo second-image feature.Source: B. Event Representation · p. 5
- The flow updater uses a ConvGRU to predict residual flow at each iteration and add it to the previous estimate; the final flow is upsampled to input resolution.The described network iteratively looks up correlation information while updating the estimate.Source: C. Dense Iterative Event-Image Flow… · p. 4, B. Event Representation · p. 5, Figure 4 · p. 5
D. Bidirectional Training
DCEIFlow trains on forward and backward motion by reversing event order and polarity for the backward pass. This training uses paired flow annotations and cannot produce backward flow from the first image alone at inference.
D. Bidirectional Training
- Reversing event timestamps and brightness-change polarity reverses the represented motion, allowing the event stream to support backward-flow estimation.The backward motion is treated as equivalent to inverse motion.Source: D. Bidirectional Training · pp. 5–6
- Bidirectional training feeds original events with the first image for forward flow and reversed events with the second image for backward flow, supervising both with ground truth.The mechanism is intended to improve flow accuracy and generalization without adding network parameters.Source: D. Bidirectional Training · p. 6, Figure 6 · p. 6
- Backward flow requires the second image, so the bidirectional training mechanism cannot produce it from the first image during inference.Source: D. Bidirectional Training · p. 6
1) Flow Loss:
Training combines flow supervision with a feature-similarity loss. The flow loss supervises iterative predictions, while the selected total loss depends on whether forward and backward flow annotations are available.
1) Flow Loss:
- Each iterative flow prediction is supervised against ground truth with a robust loss, and later predictions receive greater weight.The experiments use six iterations and set φ = 0.8 to weight later predictions more heavily.Source: 1) Flow Loss · p. 6
2) Feature similarity Loss:
- The feature-similarity loss uses L2 distance to make the fused pseudo second-image feature similar to the real second-image feature.Source: 2) Feature similarity Loss · p. 6
3) Total Training Loss:
- The total training loss uses bidirectional loss when both flow directions are annotated and unidirectional loss when only forward flow is annotated.The feature-similarity loss is also part of the weighted total loss.Source: 3) Total Training Loss · p. 6
IV. EXPERIMENTS
The experiments evaluate DCEIFlow on synthetic and real-captured data, using FlyingChairs2 for pretraining and additional datasets for different motion settings. Events for FlyingChairs2 are simulated after frame interpolation.
1) Datasets:
- FlyingChairs2 supplies full forward and backward flow annotations for pretraining, while MVSEC has sparse, low-resolution flow annotations.The authors simulate events on FlyingChairs2 with ESIM for pretraining before evaluating on real captured MVSEC data.Source: 1) Datasets: - · p. 6
- EV-IMO is included to assess non-rigid dynamic scenes, while its test set is used only for qualitative flow analysis because it lacks optical-flow annotations.MVSEC annotations are based on LIDAR depth and IMU ego-motion and cover rigid scenes.Source: 1) Datasets: - · pp. 6–7
- Sintel provides naturalistic sequences with challenging long-range non-rigid motion, but only its training set is used because test flow annotations are unavailable.Events are simulated for the Sintel training set to evaluate generalization.Source: 1) Datasets: - · p. 6
IV. EXPERIMENTS
- Figure 7 compares MVSEC indoor and outdoor scenes and reports better visual results for the proposed model in both.M denotes flow masked to pixels with events.Source: Figure 7 · p. 7
2) Event data simulation:
- FlyingChairs2 frames are interpolated with Super-SloMo before ESIM event simulation, with the interpolation amount determined by motion range.Source: 2) Event data simulation · p. 7
3) Model training details:
The section describes model training and evaluation on the MVSEC dataset, including metrics for endpoint accuracy, outliers, and prediction density. It also explains how the Dense Ratio distinguishes errors on event-triggered pixels from errors elsewhere.
3) Model training details:
- The model is pretrained on FlyingChairs2 for 100 epochs, then evaluated on MVSEC with the same pretrained model at dt=1 and dt=4.Training uses random crops of [368, 496], a batch size of 8, and two NVIDIA 2080Ti GPUs; the training process takes about 42 hours.Source: 3) Model training details · p. 7
- The MVSEC baseline comparison also trains on outdoor day2, with models evaluated at dt=1 and dt=4 frames.The described MVSEC training uses 200 epochs and a batch size of 16.Source: 3) Model training details · p. 7
4) Evaluation metrics:
- EPE measures the Euclidean distance between predicted and ground-truth flow, while the outlier metric counts points exceeding 3 pixels and 5% of flow magnitude.Both outlier measures are smaller when better.Source: 4) Evaluation metrics · p. 7
- Dense Ratio measures output density by comparing EPE on event-triggered pixels with EPE on pixels without triggered events.A ratio below 1 means lower error at event edges than elsewhere; the text notes that unsupervised training tends to make models fit edges.Source: 4) Evaluation metrics · pp. 7–8
B. Results on the MVSEC dataset
On MVSEC, DCEIFlow achieves strong indoor results, improves at the longer time interval, and is comparable to two-frame methods. Visual comparisons on MVSEC and EV-IMO also assess the density and accuracy of its predictions, with EV-IMO lacking public ground-truth flow annotations.
B. Results on the MVSEC dataset
- For MVSEC comparisons, DCEIFlow uses one pretrained model across dt=1 and dt=4, whereas existing event-based methods usually train separate models for each interval.Source: B. Results on the MVSEC dataset · p. 8
- DCEIFlow achieves state-of-the-art EPE and outlier performance on MVSEC indoor sequences, improves at dt=4, and is comparable to two-frame methods.It does not achieve the best EPE on outdoor day1, though its outlier metric there is significantly lower than others; E-RAFT is comparable outdoors while DCEIFlow performs significantly better indoors.Source: B. Results on the MVSEC dataset · p. 9
- DCEIFlow processes MVSEC-sized data in 28 ms on one 2080ti GPU, compared with 62 ms for E-RAFT.Source: B. Results on the MVSEC dataset · p. 9
- Visual comparisons on MVSEC show DCEIFlow producing fewer errors and denser estimates than event-based methods, despite spatial mismatches in some ground-truth labels.The labels are calculated from sparse depth and camera motion, and the text notes that these mismatches make visual comparison more difficult.Source: B. Results on the MVSEC dataset · p. 9
C. Results on the EV-IMO dataset
- On EV-IMO, where ground-truth flow is unavailable, DCEIFlow’s predictions are denser than event-based methods and mostly consistent with RAFT’s reference visualizations.The comparison highlights details such as edges and holes where RAFT differs from DCEIFlow.Source: C. Results on the EV-IMO dataset · p. 10, Figure 8 · p. 10
D. Model analysis
The model analysis compares DCEIFlow with baselines, examines supervised and unsupervised performance, and ablates its components. Results favor the full model and show trade-offs between generalization and dense prediction across event-poor regions.
1) Baselines for comparison:
- DCEIFlow improves significantly over the compared baselines, including when its model size is smaller, and performs better than them when trained on one interval setting.Training with two interval settings performs better than training with only one; the authors suggest the larger data size may help supervised learning.Source: 1) Baselines for comparison · p. 10
- Without pretraining, DCEIFlow is comparable to or better than existing event-based unsupervised methods, while pretraining on FlyingChairs2 yields a significant improvement.The authors conclude that unsupervised methods usually generalize better, while supervised methods may not excel on limited training data.Source: 1) Baselines for comparison · p. 10
2) Supervised and unsupervised training:
- Supervised methods generally provide better dense flow in weakly textured regions, while unsupervised methods generally generalize better under the same training protocols.Weakly textured regions are described as lacking events; unsupervised methods usually do worse there than in richly textured regions.Source: 2) Supervised and unsupervised training · p. 11
3) Ablation study:
- Ablations report gains from polarity-separated event representations, correlation, convolutional fusion, and bidirectional training; the full model performs best.Bidirectional training improves performance without increasing parameters, and iterative models usually produce fewer outliers than pyramid-based models on MVSEC.Source: 3) Ablation study · pp. 11–12
4) Dense and Continuous:
The section reports stronger dense and continuous-flow predictions than competing methods, while noting reflection-driven errors and higher computation cost. The conclusion summarizes the framework and its evaluation across multiple datasets.
4) Dense and Continuous:
- On MVSEC, the two proposed models outperform others across dense and sparse metrics and frame intervals; on EV-IMO, the model gives more accurate, denser continuous predictions, including at non-integer intervals.Two-frame methods cannot handle non-integer intervals because no corresponding second-frame image exists.Source: 4) Dense and Continuous · p. 12, Figure 9 · p. 13
E. Failure cases and limitation
- Reflections that generate many events despite little object motion seriously affect predictions from event-based methods, including this model.The failure cases are shown on EV-IMO.Source: E. Failure cases and limitation · p. 13, Figure 10 · p. 13
- At 28ms per MVSEC-sized input on a 2080ti, the model meets the 30 fps standard but is slower than EV-FlowNet at 5ms and SpikeFlowNet at 15ms.The authors identify higher time consumption as a limitation to address.Source: E. Failure cases and limitation · p. 13
V. CONCLUSION
- The conclusion attributes dense and continuous-flow estimation to event-image fusion and correlation, iterative flow updates, and bidirectional training, with evaluation across multiple datasets.The comparison is with existing event-only or fused single-image methods.Source: §V · p. 13