SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow

Yihan Wang; Lahav Lipson; Jia Deng

Source-linked AI summary

SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow

Yihan Wang, Lahav Lipson, Jia Deng

arXiv:2405.14793v1May 23, 2024cs.CV

TL;DR

Optical-flow methods face accuracy, efficiency, and generalization challenges, especially with ambiguous motion and costly recurrent refinement. SEA-RAFT addresses these through a new loss, direct initial-flow regression, rigid-flow pre-training, and simplifications, achieving strong cross-dataset accuracy and speed.

Problem
Optical flow contains ambiguous samples, and RAFT-style methods can require many recurrent refinement iterations, limiting efficiency and generalization.
Source: §3.2 · p. 5, §3.1 · p. 5
Method
SEA-RAFT combines mixture-of-Laplace loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications in a RAFT variant.
Source: §1 · p. 1, §3.2 · p. 6, §3.4 · p. 8
Results
SEA-RAFT achieves state-of-the-art Spring accuracy and runs at least 2.3× faster than existing methods with comparable accuracy.
Source: §1 · p. 1, §4.1 · p. 11
Takeaways & Limitations
SEA-RAFT provides strong accuracy, cross-dataset generalization, and accuracy-speed trade-offs for high-resolution optical flow.
Source: §4.2 · p. 12, §5 · p. 14

Abstract

from arXiv · show

We introduce SEA-RAFT, a more simple, efficient, and accurate RAFT for optical flow. Compared with RAFT, SEA-RAFT is trained with a new loss (mixture of Laplace). It directly regresses an initial flow for faster convergence in iterative refinements and introduces rigid-motion pre-training to improve generalization. SEA-RAFT achieves state-of-the-art accuracy on the Spring benchmark with a 3.69 endpoint-error (EPE) and a 0.36 1-pixel outlier rate (1px), representing 22.9% and 17.8% error reduction from best published results. In addition, SEA-RAFT obtains the best cross-dataset generalization on KITTI and Spring. With its high efficiency, SEA-RAFT operates at least 2.3x faster than existing methods while maintaining competitive performance. The code is publicly available at https://github.com/princeton-vl/SEA-RAFT.

1 Introduction

SEA-RAFT is a simpler, more efficient RAFT variant for optical flow that combines architectural and training changes. It achieves strong accuracy and efficiency across benchmarks, including state-of-the-art Spring results.

Optical flow estimates per-pixel 2D motion between video frames and supports downstream tasks including action recognition, frame interpolation, and 3D reconstruction.Source: §1 · p. 1
18% error reduction on 1px-outlier rate (3.686 vs. 4.482) and 24% error reduction on endpoint-error (0.363 vs. 0.471) on Spring.Source: §1 · p. 1
At least 2.3× faster than existing methods with comparable accuracy, while the smallest model reaches 21fps on 1080p images using an RTX3090.Source: §1 · pp. 1–2
The method combines mixture-of-Laplace loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications.The loss is intended to reduce overfitting to ambiguous cases, while direct regression reduces refinement iterations.Source: §1 · p. 2
These improvements are presented as novel and orthogonal to existing RAFT-style changes such as replacing convolutional blocks with transformers.Source: §1 · p. 2

2 Related Works

Prior optical-flow research spans deep-learning architectures, synthetic and real-world datasets, and probabilistic prediction. SEA-RAFT extends RAFT with several distinct improvements, including a specialized mixture-of-Laplace formulation for optical flow.

RAFT-style methods use recurrent networks and multi-scale cost volumes to iteratively refine optical-flow estimates.Source: §2 · p. 3
SEA-RAFT adds a new loss, direct initial-flow regression, rigid-flow pre-training, and architectural simplifications to RAFT.Direct initial-flow regression is described as new relative to efficient RAFT variants and as reducing the required iterations.Source: §2 · p. 3
Optical-flow datasets trade off synthetic scale against realism, while real-world datasets remain limited in quantity and diversity.Source: §2 · p. 3
Probabilistic prediction has also been used in related matching tasks to represent uncertainty and filter or adapt processing for uncertain matches.Source: §2 · pp. 3–4
SEA-RAFT's mixture-of-Laplace use is new for RAFT-style optical flow and requires one component with constant variance to align with L1.This formulation reflects optical flow's requirement for accurate correspondence at every pixel.Source: §2 · p. 4

3 Method

SEA-RAFT modifies RAFT’s iterative optical-flow pipeline with uncertainty-aware training, fewer refinement iterations, rigid-flow pre-training, and simpler architecture. These changes target ambiguous occlusions, convergence speed, generalization, and implementation efficiency.

Iterative refinement

RAFT-style refinement builds multi-scale correlations, looks up motion features around the current flow, and predicts residual updates through an RNN.Feature and context encoders produce dense features; the lookup radius is fixed at r = 4 in SEA-RAFT.Source: §3.1 · pp. 4–5
12 training iterations and up to 32 inference iterations are typical for RAFT-style methods, whereas SEA-RAFT uses 4 training and up to 12 inference iterations.The reduced iteration count addresses the latency bottleneck of recurrent refinement while maintaining competitive performance.Source: §3.1 · p. 5

Mixture-of-Laplace loss

SEA-RAFT models flow with a mixture of two Laplace distributions, assigning one component to ordinary cases and the other to ambiguous cases.Fixing β1 = 0 makes one component align with L1 behavior, while the predicted mixture parameters represent per-pixel uncertainty.Source: §3.2 · pp. 5, 7
The probabilistic sequence loss combines losses across refinement iterations and exponentially downweights early iterations with γ < 1.The authors report that N = 4 iterations suffice for SEA-RAFT to take first place on Spring.Source: §3.2 · p. 8

Rigid-flow pre-training

Rigid-flow pre-training uses TartanAir stereo-camera motion in static scenes to expose SEA-RAFT to more realistic and diverse training data.The authors report that this pre-training improves generalization despite limited motion diversity.Source: §3.4 · p. 8

4 Experiments

Experiments across Spring, KITTI, and Sintel show that SEA-RAFT achieves strong accuracy and generalization with substantially improved efficiency. Ablations attribute these gains to rigid-flow pre-training, direct initial-flow regression, efficient recurrent design, and the mixture-of-Laplace loss.

Spring results

SEA-RAFT outperforms existing methods on Spring without extra data and remains competitive when extra data are allowed.With extra data, it is slightly behind MS-RAFT+ but 24× faster and 11× smaller.Source: Table 1 · p. 10, §4.1 · p. 11

Cross-dataset results

KITTI zero-shot Fl-epe improves from 4.09 to 3.62 and Fl-all from 13.7 to 12.9, while Sintel final-pass performance initially trails existing methods.Adding about 1.2k KITTI and HD1K image pairs remarkably reduces the Sintel final-pass gap.Source: §4.2 · p. 12
19.9% improvement on Sintel clean, 4.2% on Sintel final, and 15.7% on KITTI Fl-all accompany speedups of 1.8× and 4.6×.These comparisons are against RAFT for accuracy and against similarly performing methods for speed.Source: §4.2 · p. 12, Table 3 · p. 12

Ablations and analysis

Rigid-flow pre-training improves performance, while the new RNN reduces computation without performance loss compared with RAFT’s GRU.Direct initial-flow regression significantly improves accuracy with little overhead; more refinements increase latency nearly linearly.Source: §4.3 · pp. 13–14, Figure 7 · p. 14

5 Conclusion

SEA-RAFT is presented as a simpler, more efficient, and more accurate RAFT variant with strong accuracy across datasets and cross-dataset generalization. Its accuracy-speed trade-offs support use in real-world high-resolution optical flow.

SEA-RAFT combines simpler architecture, higher efficiency, and greater accuracy than RAFT across a diverse range of datasets.The paper reports strong cross-dataset generalization and state-of-the-art accuracy-speed trade-offs.Source: §5 · p. 14