PaperlayerView exact revision on arXiv

Source-linked AI summary

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Zachary Teed, Jia Deng

arXiv:2003.12039v3cs.CV

TL;DR

Optical flow remains difficult to estimate accurately across challenging motion and scene conditions. RAFT introduces a single-resolution recurrent architecture with lightweight updates and achieves state-of-the-art accuracy, including a 5.10% KITTI F1-all error and strong generalization.

  • Problem

    Optical flow remains unsolved, with accuracy limited by fast motion, occlusions, motion blur, and textureless surfaces, motivating architectures that improve performance and generalization.

  • Method

    RAFT is an end-to-end optical-flow model that maintains a single-resolution flow field and applies many lightweight recurrent update operators.

  • Results

    5.10% F1-all error on KITTI represents a 16% reduction from the best published result, while RAFT also shows strong cross-dataset generalization and efficiency.

  • Takeaways & Limitations

    RAFT achieves state-of-the-art accuracy across diverse datasets while generalizing across datasets and remaining efficient in inference time, parameter count, and training iterations.

Abstract

from arXiv · show

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.

1 Introduction

RAFT is a recurrent deep architecture for optical flow that combines all-pairs multi-scale correlation volumes with iterative high-resolution flow updates. It achieves state-of-the-art accuracy on KITTI and Sintel, strong cross-dataset generalization, and efficient inference and training.

  • Motivation: RAFT addresses optical flow, which remains unsolved because of fast-moving objects, occlusions, motion blur, and textureless surfaces.Optical flow estimates per-pixel motion between video frames.
  • Architecture: RAFT extracts per-pixel features, constructs multi-scale 4D correlation volumes for all pixel pairs, and recurrently updates a high-resolution flow field.Its GRU-based update operator retrieves values from correlation volumes and iteratively updates flow initialized at zero.
  • Results: 5.10% F1-all error on KITTI represents a 16% error reduction from the best published result (6.10%), while 2.855 pixels end-point-error on Sintel final pass represents a 30% reduction from 4.098 pixels.Experiments report state-of-the-art performance on both datasets.
  • Results: 5.04 pixels end-point-error on KITTI, when trained only on synthetic data, is a 40% error reduction from the best prior deep network trained on the same data (8.36 pixels).This demonstrates strong cross-dataset generalization.
  • Efficiency: 10 frames per second on 1088×436 videos, 10X fewer training iterations, and a smaller model with 1/5 of the parameters demonstrate RAFT’s efficiency.The smaller version runs at 20 frames per second while still outperforming all prior methods on Sintel.
  • Design novelty: RAFT maintains one fixed high-resolution flow field, uses a recurrent lightweight update operator, and applies a convolutional GRU to multi-scale correlation-volume lookups.These choices differ from prevailing coarse-to-fine designs and prior refinement modules that typically use plain convolution or correlation layers.

2 Related Work

Prior optical-flow methods formulate estimation as continuous or discrete optimization, direct prediction, or iterative refinement, often relying on coarse-to-fine processing. RAFT instead maintains a high-resolution flow field and learns repeated updates using multi-resolution correlation volumes.

  • Optical Flow as Energy Minimization: Classical optical flow treats estimation as energy minimization, balancing data and regularization terms within a variational framework.Robust estimation frameworks address oversmoothing and noise sensitivity.
  • Optical Flow as Energy Minimization: Continuous formulations refine a single flow estimate but work well mainly for small displacements, motivating coarse-to-fine image pyramids for larger motion.Coarse-to-fine processing can miss small fast-moving objects and fail to recover from early mistakes.
  • Discrete Optimization: Discrete optimization methods use global objectives, but each pixel may be paired with thousands of points in the other frame, creating a massive search space.Prior work reduces or approximates this search using feature descriptors, message passing, or distance transforms.
  • Direct Flow Prediction: Direct flow-prediction networks avoid explicit optimization, while many recent methods use coarse-to-fine processing; RAFT instead updates a single high-resolution flow field.RAFT’s related iterative-refinement methods commonly use coarse-to-fine pyramids, whereas RAFT maintains high-resolution updates.
  • Iterative Refinement for Optical Flow: RAFT uses a simpler 2.7M-parameter refinement module that can run for 100+ iterations, contrasting with prior refinement systems constrained by network size or pyramid levels.IRR’s FlowNetS variant has 38M parameters and is applied for up to 5 iterations; its PWC-Net variant is limited by pyramid levels.
  • Learning to Optimize: RAFT can be viewed as learning to optimize through many update blocks, but it does not explicitly define an objective gradient; correlation-volume retrieval proposes the descent direction.This differs from approaches that backpropagate through an explicitly embedded solver or mimic first-order optimization updates.

3 Approach

RAFT is an end-to-end differentiable architecture that extracts image and context features, constructs a multi-scale all-pairs correlation representation, and iteratively updates optical flow. Correlation lookups provide local, multi-scale motion evidence while recurrent updates refine estimates toward a fixed point.

  • 3 Approach: RAFT comprises three differentiable stages: feature extraction, visual-similarity computation, and iterative flow updates.The stages are composed into an end-to-end trainable architecture.
  • Feature Extraction: The feature encoder produces 1/8-resolution features with D = 256 using six residual blocks, while a matching context network processes only I1.Both networks are run once and form the first stage.
  • Correlation Pyramid: The method computes an all-pairs correlation volume by dot products between feature vectors and organizes it into a four-layer pyramid with pooling kernels 1, 2, 4, and 8.The pyramid preserves high-resolution I1 dimensions while representing both large and small displacements.
  • Correlation Lookup: Correlation lookup bilinearly samples a radius-r local neighborhood around each current correspondence at every pyramid level, concatenating the retrieved features.At k = 4 with radius 4, the lowest level covers a 256-pixel range at original resolution.
  • Efficient Computation: 17% of total inference time is required to precompute all-pairs correlations for 1088x1920 videos, and an alternative implementation scales as O(NM).The precomputed approach is reportedly not a bottleneck because of optimized GPU matrix routines.
  • Iterative Updates: Starting from f0 = 0, the recurrent update operator uses flow, correlation, and hidden-state inputs to predict successive flow updates and hidden states.Tied weights and bounded activations encourage convergence toward a fixed point; a separable ConvGRU variant expands the receptive field.

4 Experiments

RAFT achieves state-of-the-art optical-flow performance on Sintel and KITTI, including strong cross-dataset generalization after FlyingChairs and FlyingThings training. Ablations show that its recurrent architecture, multiscale correlation, and learned upsampling contribute to performance, while the method also scales to high-resolution video and efficient inference.

  • Benchmark Evaluation: RAFT achieves state-of-the-art performance on Sintel and KITTI after pretraining on FlyingChairs and FlyingThings followed by dataset-specific finetuning.It also scales to 1080p video from DAVIS.
  • Sintel: 1.43 average EPE on Sintel(train) clean after C+T training, a 29% lower error than FlowNet2.The result demonstrates cross-dataset generalization despite a significantly shorter training schedule.
  • Sintel: 36% and 30% lower errors than prior work on the Sintel(test) clean and final passes, respectively, with RAFT ranking first on both.The reported absolute improvements are 0.9 pixels on clean and 1.2 pixels on final.
  • KITTI: 5.04 EPE on KITTI-15(train) after C+T training, improving on prior work’s 8.36 and ranking first on the KITTI leaderboard.This result supports strong cross-dataset generalization.
  • Ablations: The GRU update operator, context network, correlation pooling, all-pairs correlation, correlation-based refinement, and learned upsampling each improve performance in ablations.Pooling captures large and small displacements, while learned upsampling helps particularly near motion boundaries.
  • Inference: RAFT quickly converges during inference, surpassing PWC-Net after 3 updates and FlowNet2 after 6 updates while continuing to improve with more updates.The model can apply an arbitrary number of inference updates, including an evaluated extreme case of 200.

5 Conclusions

RAFT is an end-to-end trainable optical-flow model that operates at a single resolution with many lightweight recurrent update operators. It achieves state-of-the-art accuracy across diverse datasets while providing strong cross-dataset generalization and efficient inference, parameter count, and training iterations.

  • RAFT is a new end-to-end trainable model for optical flow.
  • The model operates at a single resolution using many lightweight, recurrent update operators.
  • RAFT achieves state-of-the-art accuracy across a diverse range of datasets.
  • The method provides strong cross-dataset generalization and efficiency in inference time, parameter count, and training iterations.

A Network Architecture

RAFT’s architecture includes full and small model variants, distinct normalization choices for its encoders, and an update block that combines context, correlation, and flow features. The full model has 4.8M parameters, or 5.3M with the upsampling module, while the small model has 1.0M parameters.

  • Model variants: The full RAFT model has 4.8M parameters, increasing to 5.3M with the upsampling module, while the small model has 1.0M parameters.These are the parameter counts specified for the full and small models.
  • Encoders: The context and feature encoders share the same architecture, but the feature encoder uses instance normalization and the context encoder uses batch normalization.The normalization scheme is the sole stated architectural difference between the two encoders.
  • Update block: RAFT-S replaces the residual units with bottleneck residual units, and the update block takes context, correlation, and flow features as inputs.The update block uses these feature types to update the latent flow representation.

B Upsampling Module

RAFT’s upsampling module predicts weights that combine each coarse-resolution flow pixel’s 9 neighbors into a high-resolution flow field. It improves accuracy near motion boundaries and recovers flow for small, fast-moving objects.

  • Upsampling mechanism: Each high-resolution flow pixel is computed as a convex combination of its 9 coarse-resolution neighbors using network-predicted weights.This describes the upsampling module’s mechanism.
  • Accuracy improvements: The upsampling module improves accuracy near motion boundaries and recovers flow for small, fast-moving objects such as birds.The figure illustrates recovery of flow for small fast-moving birds.

C Training Details

RAFT training uses staged schedules across FlyingChairs, FlyingThings, Sintel, KITTI-2015, and HD1K, with Sintel finetuning sampling specified dataset proportions. Augmentation perturbs photometric properties and applies dataset-dependent spatial rescaling and stretching.

  • Training schedule: Sintel finetuning samples S(.71), T(.135), K(.135), and H(.02).The abbreviations denote Sintel, FlyingThings, KITTI-2015, and HD1K, respectively.
  • Photometric Augmentation: Photometric augmentation randomly perturbs brightness, contrast, saturation, and hue using Torchvision ColorJitter.The default settings are brightness 0.4, contrast 0.4, saturation 0.4, and hue 0.5/π.
  • Photometric Augmentation: On KITTI, photometric augmentation uses brightness, contrast, saturation, and hue values of 0.3, 0.3, 0.3, and 0.3/π.Color augmentation is applied independently to each image with probability 0.2.
  • Spatial Augmentation: Spatial augmentation randomly rescales and stretches images with dataset-dependent scaling ranges for FlyingChairs, FlyingThings, Sintel, and KITTI.The passage specifies ranges 2^[−0.2,1.0], 2^[−0.4,0.8], 2^[−0.2,0.6], and 2^[−0

D Timing, Parameters, and Training Iterations

RAFT’s inference behavior is evaluated across update iterations, with convergence indicated by diminishing flow-update magnitudes. Its timing, parameter count, training iterations, and Sintel accuracy are reported after 10 updates, including a mixed-precision training variant.

  • Inference-time convergence: RAFT’s Sintel EPE is evaluated as a function of inference-time iteration count, while update magnitudes indicate convergence toward a fixed point.The update sequence is denoted f_k → f* as the updates diminish.
  • Timing and parameters: 10 updates are used to report RAFT’s inference timing and Sintel (train) final-pass accuracy on a GTX 1080Ti GPU.The table also reports parameter counts and training iterations.
  • Training iterations: Mixed-precision training produces similar results for RAFT while reducing training cost.The passage identifies this variant as Ours(mixed), but does not provide the truncated training-cost value.
Loading 2003.12039v3…