Source-linked AI summary
VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li
TL;DR
Most optical-flow methods estimate from frame pairs despite video streams containing additional temporal cues. VideoFlow jointly estimates bidirectional flows across frames using triplet TROFs linked by MOP, and outperforms previous methods by large margins on all benchmarks.
Problem
Most optical-flow efforts estimate from frame pairs, leaving temporal cues from neighboring video frames unused.
Source: §1 · p. 1Method
VideoFlow jointly estimates bidirectional flows for three-frame triplets with TROF and links TROFs using MOP for multi-frame estimation.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
VideoFlow reaches subpixel average end-point-error on Sintel's clean pass for the first time.
Source: §4 · p. 5
Abstract
from arXiv · showhide
We introduce VideoFlow, a novel optical flow estimation framework for videos. In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. The information of the frame triplet is iteratively fused onto the center frame. To extend TROF for handling more frames, we further propose a MOtion Propagation (MOP) module that bridges multiple TROFs and propagates motion features between adjacent TROFs. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP. By effectively exploiting video information, VideoFlow presents extraordinary performance, ranking 1st on all public benchmarks. On the Sintel benchmark, VideoFlow achieves 1.649 and 0.991 average end-point-error (AEPE) on the final and clean passes, a 15.1% and 7.6% error reduction from the best-published results (1.943 and 1.073 from FlowFormer++). On the KITTI-2015 benchmark, VideoFlow achieves an F1-all error of 3.65%, a 19.2% error reduction from the best-published result (4.52% from FlowFormer++). Code is released at \url{https://github.com/XiaoyuShi97/VideoFlow}.
1. Introduction
VideoFlow addresses the gap between video availability and pairwise flow estimation by estimating bidirectional flows across consecutive frames. Its TROF and MOP modules fuse information within frame triplets and propagate motion features across them.
1. Introduction
- Most optical-flow methods estimate frame pairs despite available video context, while many downstream algorithms need multi-frame bidirectional flows.Source: §1 · pp. 1–2
- TROF uses a triplet as its basic unit and jointly estimates flows from the center frame to its adjacent frames.Its recurrent updates integrate bidirectional motion information; the two flows originate from the same pixel and belong to one continuous temporal trajectory.Source: §1 · p. 2
- MOP bridges TROFs by warping bidirectional motion features to adjacent units using current flow predictions, growing the temporal receptive field through recurrence.Source: §1 · p. 2
- Additional contextual frames can improve flow estimates for regions occluded or out of view in target images, which are challenging in the two-frame setting.Source: §1 · p. 2
- VideoFlow outperforms previous methods by large margins on all benchmarks.Source: §1 · p. 2
2. Related Work
Optical-flow research moved from learned regression networks to iterative and transformer-based architectures. Earlier multi-frame approaches used temporal initialization, feature fusion, or propagation, with designs differing in directionality and frame scope.
2. Related Work
- Learned flow methods progressed from FlowNet and FlowNet2.0 [27] to coarse-to-fine approaches such as SpyNet [48] and PWC-Net [55] [56].RAFT [59] introduced recurrent refinement, while FlowFormer [21] and FlowFormer++ [51, 22] represent transformer-based approaches.Source: §2 · p. 2
- Multi-frame methods include RAFT warm-start [59], PWC-Fusion, and one-directional warping approaches such as ContinualFlow, Unsupervised Flow, and Starflow.Source: §2 · p. 2
- SelfFlow [40] takes three-frame estimation as strong pretraining for two-frame estimation, whereas VideoFlow integrates motion features from both forward and backward directions during flow refinement.Point-tracking [9] [15] [1] [60] follows specified points across later frames rather than estimating dense correspondences between adjacent frames.Source: §2 · p. 3
3. Method
VideoFlow extends optical-flow estimation from frame pairs to multiple video frames using tri-frame estimation and motion propagation. Its TROF and MOP modules iteratively fuse temporal information to refine flows across a sequence.
3. Method
- VideoFlow combines tri-frame flow estimation with motion propagation to extend optical-flow estimation to multiple consecutive frames.TROF estimates bidirectional flows for three-frame clips; MOP bridges TROFs for longer sequences.Source: §3 · p. 3, §3.2 · p. 4
3.1. TROF for Tri-frame Optical Flow Estimation
- TROF jointly estimates bidirectional flows from the center frame to its neighbors, then fuses correlation and flow features to iteratively refine predictions.The flows share source pixels at the center frame; recurrent updates decode residual flows that are added to current predictions.Source: §3.1 · p. 3
3.2. Bridging TROFs with Motion Propagation
- MOP bridges overlapping TROFs by warping neighboring motion states into each unit, expanding temporal context as recurrent iterations proceed.The motion state is updated each iteration; inference predicts T frames’ flows from clips of length T + 2.Source: §3.2 · p. 4
3.3. Training Loss
- VideoFlow is trained with ℓ1 loss on predicted bidirectional flows, weighting later recurrent predictions more heavily.The experiments use 12 recurrent steps and γ = 0.85.Source: §3.3 · pp. 4–5
4. Experiments
Experiments evaluate VideoFlow on Sintel and KITTI-2015, report quantitative and qualitative comparisons, and ablate its three-frame and motion-propagation designs.
4.1. Quantitative Experiment
- 0.99 and 1.65 AEPE on Sintel clean/final are 7.6%/15.1% reductions from FlowFormer++; 3.65% Fl-all on KITTI-2015 is a 19.2% reduction from FlowFormer++.Source: Table 1 · p. 6, §4.1 · p. 7
4.2. Qualitative Experiment
- VideoFlow preserves barrier gaps and better handles shadows and artifacts than FlowFormer++ in Sintel and KITTI test visualizations.Source: §4.2 · p. 7, Figure 4 · p. 8
4.3. Ablation Study
- Center-originating bidirectional estimation improves most metrics over the unidirectional baseline, and recurrent fusion further boosts results.Source: §4.3 · p. 7, Table 3 · p. 8
- Maintaining motion state lets the temporal receptive field grow recurrently; passing it through a temporal GRU instead drops performance on FlyingThings and KITTI-2015.Source: §4.3 · p. 7, Table 4 · p. 8
- Backward-flow tests show similar accuracy to forward predictions for three- and five-frame models, and bidirectional flows suit downstream video processing.Source: Table 5 · p. 8, §4.3 · p. 8
5. Conclusion
VideoFlow uses a three-frame TROF module as its building block and connects TROF units with motion propagation for multi-frame estimation. It outperforms previous methods by large margins on all benchmarks.