Source-linked AI summary
SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow
Yihan Wang, Lahav Lipson, Jia Deng
TL;DR
Optical-flow methods face accuracy, efficiency, and generalization challenges, especially with ambiguous motion and costly recurrent refinement. SEA-RAFT addresses these through a new loss, direct initial-flow regression, rigid-flow pre-training, and simplifications, achieving strong cross-dataset accuracy and speed.
Problem
Optical flow contains ambiguous samples, and RAFT-style methods can require many recurrent refinement iterations, limiting efficiency and generalization.
Method
SEA-RAFT combines mixture-of-Laplace loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications in a RAFT variant.
Results
SEA-RAFT achieves state-of-the-art Spring accuracy and runs at least 2.3× faster than existing methods with comparable accuracy.
Takeaways & Limitations
SEA-RAFT provides strong accuracy, cross-dataset generalization, and accuracy-speed trade-offs for high-resolution optical flow.
Abstract
from arXiv · showhide
We introduce SEA-RAFT, a more simple, efficient, and accurate RAFT for optical flow. Compared with RAFT, SEA-RAFT is trained with a new loss (mixture of Laplace). It directly regresses an initial flow for faster convergence in iterative refinements and introduces rigid-motion pre-training to improve generalization. SEA-RAFT achieves state-of-the-art accuracy on the Spring benchmark with a 3.69 endpoint-error (EPE) and a 0.36 1-pixel outlier rate (1px), representing 22.9% and 17.8% error reduction from best published results. In addition, SEA-RAFT obtains the best cross-dataset generalization on KITTI and Spring. With its high efficiency, SEA-RAFT operates at least 2.3x faster than existing methods while maintaining competitive performance. The code is publicly available at https://github.com/princeton-vl/SEA-RAFT.
1 Introduction
SEA-RAFT is a simpler, more efficient RAFT variant for optical flow that combines architectural and training changes. It achieves strong accuracy and efficiency across benchmarks, including state-of-the-art Spring results.
- Optical flow estimates per-pixel 2D motion between video frames and supports downstream tasks including action recognition, frame interpolation, and 3D reconstruction.
- 18% error reduction on 1px-outlier rate (3.686 vs. 4.482) and 24% error reduction on endpoint-error (0.363 vs. 0.471) on Spring.
- At least 2.3× faster than existing methods with comparable accuracy, while the smallest model reaches 21fps on 1080p images using an RTX3090.
- The method combines mixture-of-Laplace loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications.The loss is intended to reduce overfitting to ambiguous cases, while direct regression reduces refinement iterations.
- These improvements are presented as novel and orthogonal to existing RAFT-style changes such as replacing convolutional blocks with transformers.
2 Related Works
Prior optical-flow research spans deep-learning architectures, synthetic and real-world datasets, and probabilistic prediction. SEA-RAFT extends RAFT with several distinct improvements, including a specialized mixture-of-Laplace formulation for optical flow.
- RAFT-style methods use recurrent networks and multi-scale cost volumes to iteratively refine optical-flow estimates.
- SEA-RAFT adds a new loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications to RAFT.Direct initial-flow regression is described as new relative to efficient RAFT variants and as reducing the required iterations.
- Optical-flow datasets trade off synthetic scale against realism, while real-world datasets remain limited in quantity and diversity.
- Probabilistic prediction has also been used in related matching tasks to represent uncertainty and filter or adapt processing for uncertain matches.
- SEA-RAFT's mixture-of-Laplace use is new for RAFT-style optical flow and requires one component with constant variance to align with L1.This formulation reflects optical flow's requirement for accurate correspondence at every pixel.
3 Method
SEA-RAFT modifies RAFT’s iterative optical-flow pipeline with uncertainty-aware training, fewer refinement iterations, rigid-flow pre-training, and simpler architecture. These changes target ambiguous occlusions, convergence speed, generalization, and implementation efficiency.
- Iterative refinement: RAFT-style refinement builds multi-scale correlations, looks up motion features around the current flow, and predicts residual updates through an RNN.Feature and context encoders produce dense features; the lookup radius is fixed at r = 4 in SEA-RAFT.
- Iterative refinement: 12 training iterations and up to 32 inference iterations are typical for RAFT-style methods, whereas SEA-RAFT uses 4 training and up to 12 inference iterations.The reduced iteration count addresses the latency bottleneck of recurrent refinement while maintaining competitive performance.
- Mixture-of-Laplace loss: SEA-RAFT models flow with a mixture of two Laplace distributions, assigning one component to ordinary cases and the other to ambiguous cases.Fixing β1 = 0 makes one component align with L1 behavior, while the predicted mixture parameters represent per-pixel uncertainty.
- Mixture-of-Laplace loss: The probabilistic sequence loss combines losses across refinement iterations and exponentially downweights early iterations with γ < 1.The authors report that N = 4 iterations suffice for SEA-RAFT to take first place on Spring.
- Rigid-flow pre-training: Rigid-flow pre-training uses TartanAir stereo-camera motion in static scenes to expose SEA-RAFT to more realistic and diverse training data.The authors report that this pre-training improves generalization despite limited motion diversity.
4 Experiments
Experiments across Spring, KITTI, and Sintel show that SEA-RAFT achieves strong accuracy and generalization with substantially improved efficiency. Ablations attribute these gains to rigid-flow pre-training, direct initial-flow regression, efficient recurrent design, and the mixture-of-Laplace loss.
- Spring results: SEA-RAFT outperforms existing methods on Spring without extra data and remains competitive when extra data are allowed.With extra data, it is slightly behind MS-RAFT+ but 24× faster and 11× smaller.
- Cross-dataset results: KITTI zero-shot Fl-epe improves from 4.09 to 3.62 and Fl-all from 13.7 to 12.9, while Sintel final-pass performance initially trails existing methods.Adding about 1.2k KITTI and HD1K image pairs remarkably reduces the Sintel final-pass gap.
- Cross-dataset results: 19.9% improvement on Sintel clean, 4.2% on Sintel final, and 15.7% on KITTI Fl-all accompany speedups of 1.8× and 4.6×.These comparisons are against RAFT for accuracy and against similarly performing methods for speed.
- Ablations and analysis: Rigid-flow pre-training improves performance, while the new RNN reduces computation without performance loss compared with RAFT’s GRU.Direct initial-flow regression significantly improves accuracy with little overhead; more refinements increase latency nearly linearly.
5 Conclusion
SEA-RAFT is presented as a simpler, more efficient, and more accurate RAFT variant with strong accuracy across datasets and cross-dataset generalization. Its accuracy-speed trade-offs support use in real-world high-resolution optical flow.
- SEA-RAFT combines simpler architecture, higher efficiency, and greater accuracy than RAFT across a diverse range of datasets.The paper reports strong cross-dataset generalization and state-of-the-art accuracy-speed trade-offs.