Source-linked AI summary
E-WAVE: Event-based Continuous Optical Flow via Warping-Aligned Visual Encoding
Jiale Wu, Xiaoyang Bai, Haoming Yu, Yiwei Chen, Yifan Peng, Weiwei Xu
TL;DR
Frame-based optical flow faces a temporal-resolution and computational-cost tradeoff, while event methods often use memory-intensive correlation volumes. E-WAVE replaces all-pairs correlation with attention and B´ezier trajectory-guided warping, achieving strong benchmark results and robustness on head-mounted real-world data.
Problem
Frame-based methods trade temporal density against bandwidth and computation, while event-based methods commonly rely on costly all-pairs correlation volumes.
Source: §1 · p. 1Method
E-WAVE uses attention and B´ezier-parameterized trajectory-guided feature warping with iterative updates to estimate event-based flow at arbitrary timestamps.
Source: §1 · p. 2, §3.3.2 · p. 4Results
E-WAVE achieves state-of-the-art HTR motion estimation on MultiFlow and DSEC-Flow, and on MultiFlow beats ResFlow by 19% in EPE under the unimodal setting.
Source: §4.5 · p. 7, §5 · p. 10Takeaways & Limitations
Head-mounted prototype evaluations report robustness under severe underexposure and rapid, nonlinear motion, supporting E-WAVE as a candidate for immersive VR/AR perception.
Source: §5 · p. 10Dense event voxel encoding has computational cost that scales with input event frame rate, and real-world noise can degrade flow accuracy in unconstrained environments.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Temporally dense optical flow is essential for dynamic perception in immersive VR/AR systems, where rapid head, hand, and object motion must be continuously captured and tracked. Existing frame-based optical flow estimation methods are constrained by the tradeoff between temporal resolution and computational cost; while event cameras, with their high temporal resolution and energy efficiency, serve as a natural solution to the dilemma. However, event-based approaches commonly rely on correlation volumes to capture pairwise voxel correspondences, which incur substantial memory and computation overhead. We present E-WAVE, a correlation-free framework for high-temporal-resolution (HTR) optical flow estimation from event streams. Instead of constructing all-pairs correlation volumes, E-WAVE employs global attention mechanism to model long-range feature dependencies and performs trajectory guided feature warping using B\'ezier curve. Through iterative updates, it predicts trajectories that allow for querying at arbitrary timestamps without repeated inference. Experiments on MultiFlow and DSEC-Flow demonstrate a 25% lower trajectory error and comparable endpoint flow estimation accuracy relative to state-of-the art baselines. Additional evaluations on self-captured data using a head-mounted prototype validate that E-WAVE remains robust under challenging real-world conditions.
1 INTRODUCTION
E-WAVE targets the temporal-resolution and computational-cost tradeoff in optical flow by replacing correlation volumes with attention-driven, trajectory-guided warping. It reports strong benchmark performance and real-world robustness, with extensions for multiple input modalities.
Motivation
- Frame-based optical flow trades temporal density against bandwidth and computation, while lower frame rates can introduce large displacements and motion blur.Dense motion estimates matter for tracking fast head, hand, and object interactions in VR/AR.Source: §1 · p. 1
- Event cameras provide microsecond-level temporal resolution and continuous acquisition, but event-flow methods often incur quadratic memory and computation costs from all-pairs correlation volumes.Pairwise correlations also fail to exploit the continuous temporal trajectories encoded in event streams.Source: §1 · pp. 1–2
Contributions
- E-WAVE uses Vision Transformer attention for long-range dependencies and B´ezier-parameterized trajectories for continuous feature warping at arbitrary timestamps.Iterative warp-and-update steps refine the trajectory parameters.Source: §1 · p. 2
Contributions and evaluation
- E-WAVE achieves state-of-the-art HTR performance on MultiFlow while maintaining competitive LTR capability; extended models achieve state-of-the-art LTR performance on DSEC-Flow.The framework is also evaluated on self-captured sequences with challenging motion and underexposure.Source: §1 · p. 2
- A unified feature-construction, warping, and updating pipeline supports both event-only and event-RGB inputs across different motion dynamics.The work also evaluates a head-mounted acquisition prototype on real-world sequences.Source: §1 · p. 2
2 RELATED WORK
Prior optical-flow work developed warping, correlation, iterative refinement, and attention, while event-based methods used voxel representations and increasingly dense temporal matching. The related work identifies correlation scalability and linear-motion assumptions as limitations that motivate continuous trajectory warping.
2.1 Optical Flow Estimation
- Optical-flow methods progressed from pyramids, warping, and correlation volumes to iterative refinement and global attention for avoiding full 4D correlation costs.WAFT combines warping with attention for higher-resolution estimation.Source: §2.1 · p. 2
2.2 Event-based Optical Flow Estimation
- Event-flow methods evolved from iterative correlation volumes toward multi-volume, bidirectional, and deblurring approaches, but correlation memory limits spatial and temporal scalability.Existing HTR approaches also subsample temporally or accumulate velocity estimates whose drift degrades longer-sequence performance.Source: §2.2 · p. 2
2.3 Feature Alignment with Warping
- Prior warping-based event methods often assume short intervals and constant motion; E-WAVE instead represents continuous event-guided trajectories with B´ezier displacement fields to account for nonlinear motion.Related RGB methods combine iterative warping with attention, while Figure 2 contrasts queried dense correlation volumes with attention-derived warping parameters.Source: §2.3 · p. 3, Figure 2 · p. 3
3.1 Event Camera and Event Voxelization
Event cameras asynchronously record brightness changes as sparse streams rather than regularly sampled image tensors. E-WAVE converts events into compact, fixed-dimensional voxel representations, including polarity channels.
Event representation
- Event cameras asynchronously trigger events in response to changes in logarithmic brightness, producing sparse, variable-length streams instead of regularly sampled image tensors.An event fires when the brightness change approximately reaches the polarity-weighted contrast threshold pC.Source: §3.1 · p. 3
Event voxelization
- Raw events are converted into compact, fixed-dimensional voxel representations to avoid inconsistent input dimensions and large memory overhead.The construction accumulates event contributions at each pixel, separately for positive and negative activations.Source: §3.1 · p. 3
- Three-channel event voxels are formed to match RGB image dimensions by adding the difference between the clipped and normalized polarity channels.Source: §3.1 · p. 3
3.2 Problem Definition and Feature Matching
LTR estimation predicts flow between boundary timestamps, whereas HTR estimation predicts flow at intermediate timestamps. Feature matching can use dense correlation volumes or warp features into alignment.
Problem definition
- LTR estimation retrieves a pixelwise flow map between two boundary timestamps, while HTR estimation computes flow for any intermediate timestamp.Source: §3.2 · p. 3
Feature matching
- Correlation volumes store feature similarities over candidate displacements and are queried along estimated pixel trajectories for local correspondence information.Source: §3.2 · p. 3
- Warping aligns features with a per-pixel displacement, producing one aligned feature map for a sequence rather than separate matching hypotheses for each feature pair.The section associates this design with higher spatial and temporal resolution on densely voxelized event data.Source: §3.2 · p. 3
3.3 Warping-Aligned Visual Encoding
E-WAVE extracts event and optional RGB features, parameterizes continuous pixel motion with Bézier trajectories, and iteratively warps and refines features to produce flow at queried timestamps.
3.3.1 Feature extraction and fusion
- RGB inputs, when available, use a ResNet encoder complemented by frozen Depth Anything V2 depth features; event voxels use a matching ResNet encoder.A shared 2D positional encoding is also applied to each event feature map before warping.Source: §3.3.1 · pp. 3–4
3.3.2 Trajectory parameterization
- Each pixel’s displacement relative to τ0 is represented by a degree-L Bézier curve, with normalized time spanning the interval from τ0 to τK.The network predicts differences between adjacent control points, with the initial point fixed at zero so displacement at τ0 is zero.Source: §3.3.2 · p. 4
3.3.3 Feature warping and residual update
- At iteration r, the current U(r) warps event features and available RGB references; aligned features, hidden state, and control parameters form z, which ViT-DPT refines into a dense output.The method operates at a 2×2 downsampled scale; a learned convex 3×3 kernel upsamples ΘR to input resolution before outputting Ψt.Source: §3.3.3 · pp. 4–5
- Across iterations, the warp-and-update process improves pixelwise trajectory alignment and object-level flow coherence in the first few steps, then converges.The pipeline also predicts an uncertainty vector at each iteration and computes endpoint NLL and intermediate-trajectory losses for training.Source: Figure 3 · p. 4, Figure 4 · p. 5
3.4 Analysis on Iterative Warp-and-Update
E-WAVE’s linear update components make the first r refinements equivalent to one refinement-network inference predicting the accumulated updates, but do not guarantee convergence. Empirically, quality improves in early iterations and then stabilizes.
3.4 Analysis on Iterative Warp-and-Update
- Because the trajectory and warping operations are linear, the first r iterations equal one refinement-network inference predicting the sum of their updates.This equivalence means increasing iteration count does not inherently make the output diverge.Source: §3.4 · p. 5
- The update equivalence does not prove convergence, which is difficult to establish theoretically because the refinement network is nonconvex.The authors assess iteration behavior empirically instead.Source: §3.4 · p. 5
- Figure 4 shows estimation quality improving significantly in the initial iterations and stabilizing afterward.The visualization indicates improved trajectory alignment and flow coherence during the early warp-and-update steps.Source: §3.4 · p. 5
3.5 Training Objectives
E-WAVE supervises each refinement with endpoint uncertainty loss and, when dense annotations exist, trajectory loss, then combines losses across iterations.
3.5 Training Objectives
- Each refinement iteration uses a mixture-of-Laplace endpoint uncertainty loss, with two mixture logits and two log-scales predicted per pixel.The uncertainty-aware NLL is described as respecting standard L1 behavior while accounting for ambiguous or heavily occluded regions.Source: §3.5 · p. 5
- When temporally dense annotations are available, a vector Charbonnier penalty supervises predicted Bézier trajectories at valid timestamps.Averaging by valid pixels at each timestamp prevents timestamps with more valid pixels from dominating the loss.Source: §3.5 · p. 5
- The overall objective exponentially weights both loss terms across all refinement iterations.The per-iteration endpoint NLL and trajectory losses are combined for training.Source: §3.5 · p. 5
3.6 Extensions for Improved Performance
E-WAVE’s warping-based design supports bidirectional feature matching and cross-dataset finetuning without changing its core framework. The extensions improve performance, while the DSEC-Flow visualization compares estimation detail at different spatial resolutions.
3.6 Extensions for Improved Performance
- Bidirectional matching and cross-dataset finetuning extend E-WAVE without altering its core warping-based framework.Finetuning pretrains on a larger synthetic dataset such as MultiFlow, then adapts to real-world data such as DSEC-Flow.Source: §3.6 · pp. 5–6
- Figure 5 compares DSEC-Flow estimates; E-WAVE’s higher spatial resolution produces more detailed flow, while the bottom 60 pixels are omitted for lack of supervision.Source: Figure 5 · p. 6
- Forward and backward endpoint flows construct a degree-2 trajectory used to warp event voxel features in both temporal directions.The flows are jointly optimized using bidirectional ground truth available in datasets such as DSEC-Flow.Source: §3.6 · p. 6
- Cross-dataset finetuning complements sparse real-world annotations and mitigates overfitting observed in the experiments.The strategy pretrains on a larger synthetic dataset before finetuning on real-world data.Source: §3.6 · p. 6
4 EXPERIMENTS AND RESULTS
Experiments evaluate endpoint and continuous-trajectory flow on DSEC-Flow and MultiFlow, then test ablations and zero-shot real-world sequences. E-WAVE performs strongly on MultiFlow, while its extensions improve DSEC-Flow results and its qualitative predictions remain robust in challenging captures.
4.1 Datasets
- DSEC-Flow is an event-RGB driving dataset with 10 Hz ground truth for LTR evaluation; MultiFlow supplies dense trajectories for HTR evaluation.Training uses only 2,000 MultiFlow samples, about one-fifth of its training set, while evaluation uses the complete test set.Source: §4.1 · p. 6
4.2 Baselines; 4.3 Metrics
- Baselines span event-only, event-RGB, and RGB-only inputs, with HTR methods aligned to degree-10 trajectory representations; endpoint and trajectory metrics assess different prediction regimes.Endpoint metrics include EPE and AE; MultiFlow trajectory metrics include TEPE and TAE.Source: §4.2 · p. 6, §4.3 · p. 7
4.4 Implementation Details
- MultiFlow training uses five iterative updates and 20 event voxels; DSEC-Flow uses eight voxels and degree-2 Bézier curves because HTR supervision is unavailable.Source: §4.4 · p. 7
4.5 Experimental Results
- On DSEC-Flow, event-RGB E-WAVE has 7.2% lower EPE than BFlow, and its extensions reach state-of-the-art among event-only and event-RGB models.The benchmark also shows comparable performance to prior methods in event-only and event-RGB settings.Source: Table 1 · p. 6, §4.5 · p. 7
- On MultiFlow, E-WAVE beats ResFlow by 19% in EPE and 20% in TEPE under the unimodal setting.LTR models estimate endpoint flow reasonably well but fail significantly on HTR metrics.Source: §4.5 · p. 7, Table 2 · p. 7
4.6 Ablation Study
- Degree-10 Bézier warping outperforms linear and no-warp policies; removing trajectory loss degrades Bézier predictions, especially on trajectory metrics.With linear warping, removing trajectory loss improves endpoint prediction because intermediate supervision acts as noise in that setting.Source: Table 3 · p. 7, §4.6 · pp. 8–9
- Reducing spatial resolution degrades performance, while increasing event voxels from 5 to 20 consistently improves all metrics at a fixed 25 ms per voxel.Source: Table 4 · p. 8, §4.6 · p. 9
4.7 Real-world Assessment
- In zero-shot self-captured scenes, E-WAVE produces clearer motion boundaries, coherent fast-motion trajectories, and fine finger boundaries; under low light, events retain useful information when RGB detail fails.The low-light event stream is sparser and noisier, yet E-WAVE recovers object boundaries and plausible trajectories for pixels invisible to RGB cameras.Source: §4.7 · p. 9
5 DISCUSSIONS AND CONCLUSION
E-WAVE combines attention with warping-aligned features in a correlation-free framework for event-based optical flow, with benchmark and head-mounted evaluations reporting strong performance and robustness. Its limitations include event-rate-scaled encoding cost, reliance on ground-truth annotations, and sensitivity to real-world noise.
5 DISCUSSIONS AND CONCLUSION
- Dense event voxel encoding costs scale with event frame rate, training uses ground-truth annotations, and real-world flicker, reflections, and shadows can degrade flow accuracy.The authors identify noise-robustness mechanisms as necessary before deployment in VR/AR devices.Source: §5 · p. 9
- E-WAVE achieves state-of-the-art HTR motion estimation on DSEC-Flow and MultiFlow, and LTR estimation with minor modifications.Source: §5 · p. 10
- Head-mounted evaluations confirm robustness under severe underexposure and rapid, non-linear motion dynamics.Source: §5 · p. 10