Source-linked AI summary
FlowNet: Learning Optical Flow with Convolutional Networks
Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox
TL;DR
Optical flow estimation requires both precise per-pixel localization and correspondence matching, challenging standard CNN approaches. FlowNet trains end-to-end CNNs on synthetic Flying Chairs data and achieves competitive accuracy on natural-scene datasets at up to 10 image pairs per second.
Problem
Optical flow requires precise per-pixel localization and matching correspondences between two images, creating an architectural challenge for CNNs.
Method
FlowNet trains CNNs end-to-end to predict optical flow from image pairs, comparing generic and correlation-based architectures with synthetic Flying Chairs training data.
Results
Synthetic Flying Chairs training generalizes to natural scenes with competitive accuracy, while the networks run at up to 10 image pairs per second.
Takeaways & Limitations
Unrealistic synthetic training data can suffice for CNN-based optical flow prediction in natural scenes.
Takeaways & Limitations
The study did not examine in detail how Flying Chairs dataset parameters affect FlowNet performance.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks where CNNs were successful. In this paper we construct appropriate CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
1. Introduction
The paper formulates optical-flow estimation from image pairs as end-to-end CNN learning, addressing the task’s need for precise localization and cross-image correspondence. It introduces a correlation-based architecture and a synthetic training dataset, achieving real-time accuracy on Sintel.
- Architecture: The authors propose end-to-end CNNs for predicting optical-flow fields from pairs of images, including an architecture with a correlation layer for explicit matching.The correlation layer supports correspondence finding while higher layers refine the prediction.
- Motivation and Problem: Optical flow requires both precise per-pixel localization and learned correspondences between two input images, unlike earlier CNN applications.The task requires learning image features and matching them across different locations.
- Training Data: The synthetic Flying Chairs dataset provides arbitrarily large training data by overlaying segmented chair images on random Flickr backgrounds.The dataset trades realism for quantity because existing optical-flow datasets are too small for competitive CNN training.
- Results: 10 image pairs per second is the reported maximum full-resolution Sintel prediction rate, with state-of-the-art accuracy among real-time methods.The speed is enabled by an efficient GPU implementation of CNNs.
2. Related Work
Prior optical-flow research has centered on variational methods, with related machine-learning and unsupervised neural approaches exploring learned statistics, representations, and motion inference. However, unsupervised neural methods were not competitive with classical methods on realistic videos.
- Optical Flow: Variational approaches have dominated optical-flow estimation since Horn and Schunck, with later work targeting improvements and large displacements.Combinatorial matching was integrated into variational approaches for large-displacement estimation.
- Optical Flow: Deep-Matching and DeepFlow aggregate feature information from fine to coarse using sparse convolutions and max-pooling.The paper identifies this feature-aggregation strategy as related to its own approach.
- Machine Learning for Optical Flow: Earlier machine-learning approaches learned optical-flow regularizers or statistics, principal-component basis flows, or classifiers for flow prediction.Examples include Gaussian scale mixtures, Gaussian mixture models, and linear combinations of learned basis flows.
- Unsupervised Learning: Unsupervised neural methods inferred disparity or optical flow from latent variables modeling relations between image pairs, often through multiplicative interactions.Examples include factored gated restricted Boltzmann machines and synchrony autoencoders.
- Unsupervised Learning: Unsupervised neural approaches worked well in controlled setups and learned features useful for activity recognition, but were not competitive with classical methods on realistic videos.This limitation distinguishes their controlled-setting performance from realistic-video optical-flow performance.
3. Network Architectures
The paper presents two end-to-end CNN architectures for predicting optical flow directly from image pairs: a generic stacked-image network and a two-stream network with a correlation layer. Both architectures refine coarse pooled representations into dense flow predictions using upconvolutional layers and related refinement mechanisms.
- FlowNetSimple: FlowNetSimple stacks both input images and processes them with a generic network consisting only of convolutional layers.The network learns how to extract motion information from the image pair end to end.
- FlowNetCorr: FlowNetCorr uses separate identical processing streams, then combines the image representations at a later stage to support matching.This design first produces meaningful representations separately before combining them at a higher level.
- FlowNetCorr: The correlation layer performs multiplicative patch comparisons between two feature maps, comparing each patch from one map with each patch from the other.It provides the network with an explicit mechanism for finding correspondences between feature representations.
- FlowNetCorr: For computational reasons, correlation comparisons use a limited maximum displacement and strides in both feature maps, producing an output of size (w × h × D^2).Comparing all patch combinations would require w^2 · h^2 computations and make efficient forward and backward passes intractable.
- Refinement: Upconvolutional refinement combines expanded coarse feature maps with corresponding contractive features and, when available, an upsampled coarser flow prediction.Upconvolutional layers perform unpooling followed by convolution to recover dense per-pixel predictions from pooled representations.
- Refinement: The alternative variational refinement starts at 4 times downsampled resolution, uses 20 coarse-to-fine iterations, then applies 5 more iterations at full image resolution.Image boundaries are computed separately, and the smoothness coefficient is replaced by α = exp(−λ.
4. Training Data
Existing optical-flow datasets are too limited or specialized to provide sufficient training data for large CNNs. The paper therefore introduces Flying Chairs, a synthetic dataset with randomized motion and online augmentation to reduce overfitting.
- Existing datasets: Existing datasets provide limited training supervision: Middlebury has 8 image pairs with typically sub-10-pixel displacements, while KITTI has 194 pairs but specialized, sparse ground truth.KITTI assumes rigid scenes and motion from a moving observer, and cannot capture distant-object motion such as the sky.
- Existing datasets: Sintel is the largest available dataset, with 1,041 training image pairs per version and dense ground truth spanning small and large displacements, but remains too small for large CNNs.Its Final version includes motion blur and atmospheric effects, whereas Clean omits them.
- Flying Chairs dataset: Flying Chairs is built by applying affine transformations to Flickr images and rendered 3D chair models, using 964 source images split into 4 quadrants at 512 × 384 resolution.The Flickr images come from the city, landscape, and mountain categories.
- Flying Chairs dataset: Randomly sampled background and chair transformations generate the second image, optical flow, and occlusion regions, with parameter distributions adjusted to match Sintel’s displacement histogram.Chair transformations are relative to the background transformation, representing camera and object motion.
- Augmentation: 22,872 image pairs and flow fields are generated, and online augmentation is crucial to avoid overfitting despite the dataset’s size.Augmentation includes geometric transformations, Gaussian noise, and changes in brightness, contrast, gamma, and color; paired-image transformations also diversify flow fields.
5. Experiments
Experiments evaluate FlowNet on synthetic and real optical-flow datasets, including fine-tuning, variational refinement, and runtime comparisons. Networks trained on Flying Chairs generalize well, while results vary by architecture, dataset, and training strategy.
- Experimental setup: The experiments measure endpoint error on Sintel, KITTI, Middlebury, and Flying Chairs, alongside fine-tuning, variational refinement, and runtime comparisons.EPE is the Euclidean distance between predicted and ground-truth flow vectors, averaged over pixels.
- Overall results: Networks trained only on Flying Chairs beat LDOF, while Sintel fine-tuning outperforms EPPM on Sintel Final and KITTI at twice its speed.The paper reports that Flying Chairs training transfers well to real optical-flow datasets despite the dataset being non-realistic.
- Dataset and architecture comparisons: FlowNetC outperforms FlowNetS on Flying Chairs and Sintel Clean, whereas FlowNetS performs better on Sintel Final and KITTI.On Sintel Final, FlowNetS+ft+v is on par with DeepFlow.
- Training data: On Sintel, training exclusively on Sintel yields EPE roughly 1 pixel higher than Flying Chairs training followed by Sintel fine-tuning.Aggressive data augmentation makes Sintel alone sufficient to learn optical flow fairly well.
- Training data: Removing data augmentation from Flying Chairs training increases Sintel test EPE by roughly 2 pixels.The result indicates that data augmentation remains useful even with the fairly large Flying Chairs dataset.
- Large-displacement performance: FlowNetC+ft has an s40+ error of 48px versus 43.3px for FlowNetS+ft, consistent with greater difficulty handling large displacements.The paper attributes this partly to the correlation layer’s limited maximum displacement range.
6. Conclusion
The paper shows that convolutional networks can directly predict optical flow from two input images. Training on the synthetic Flying Chairs dataset generalizes to natural scenes with competitive accuracy, despite its unrealistic data.
- CNNs can be trained to directly predict optical flow from two input images.
- Synthetic training data with affine motions of rigid objects is sufficient for competitive optical-flow prediction in natural scenes.The data come from the artificial Flying Chairs dataset, despite being unrealistic.
- On the Flying Chairs test set, the CNNs outperform DeepFlow and EpicFlow.The passage characterizes DeepFlow and EpicFlow as state-of-the-art methods.
Supplementary Material for ‘FlowNet: Learning Optical Flow with · 1. Flow field color coding
The supplementary material explains how optical-flow fields are visualized using Sintel’s color-coding tool. Direction is encoded by hue, magnitude by intensity, and each image pair is independently normalized for maximum intensity.
- 1. Flow field color coding: Flow direction is encoded with color, while flow magnitude is encoded with color intensity.White represents no motion.
- 1. Flow field color coding: The color-coding diagram represents each pixel’s flow vector from the square’s center to that pixel.This illustrates how vector direction maps to the displayed color.
- 1. Flow field color coding: Because flow magnitudes differ substantially across image pairs, maximum color intensity is independently normalized for each pair.The normalization is performed consistently across the visualization of each image pair.
2. Details of generating Flying Chairs
Flying Chairs is generated by compositing randomly selected rendered chairs onto Flickr backgrounds, then applying sampled transformations to produce image pairs, flow fields, and occlusion maps. The transformation distributions are designed to match Sintel’s displacement statistics, and simpler Gaussian sampling reduces accuracy.
- Dataset construction: The dataset uses 964 Flickr backgrounds and 809 chair models rendered from 62 views, compositing a random set of chairs onto each background.Each background image has resolution 1024 × 768 pixels; chair views combine 31 azimuth angles with 2 elevation angles.
- Pair and flow generation: Each image pair and its flow field are generated by applying composed zooming, rotation, and translation transformations to the chairs and background.The process also produces an occlusion map and splits each image into four 512 × 384-pixel quarters, yielding four image pairs.
- Displacement matching: The sampling distributions aim to match Sintel’s displacement distribution because Gaussian-only sampling produces too few small displacements.Networks trained with the simpler Gaussian strategy still work but are less accurate than networks trained with the described distributions.
- Transformation distributions: Transformation parameters are sampled from mixtures combining a constant value with a powered, clamped Gaussian, with parameters represented by G(k, µ, σ, a, b, p).The mixture selects the constant with probability 1 −p and the transformed Gaussian with probability p; parameter settings are given in Table 1.
3. Convolutional Filters
FlowNet filters become more structured in higher layers: first-layer filters remain noisy but show coarse gradients, while filters after the correlation layer exhibit visible structure and selectivity for flow direction and magnitude.
- Filter structure: Higher-layer FlowNet filters are more structured than lower-layer filters.Lower-layer filters have little structure, whereas higher-layer filters are more structured.
- Filter structure: Filters applied after the correlation layer show visible structure and selectivity for different flow directions and magnitudes.Their structure is more apparent than in the first-layer filters.
- Filter structure: First-layer FlowNetCorr filters are noisy but retain visible coarse-gradient structure.The filters have not completely converged, although coarse gradients are visible.
4. Video
The supplementary video demonstrates real-time FlowNet operation on 640 × 480 webcam images, showing flow fields from FlowNetSimple and FlowNetCorr in indoor and outdoor scenes.
- Video: Real-time FlowNet operation is demonstrated on a GeForce GTX 980M using 640 × 480 webcam images.The supplementary video shows example flow fields from FlowNetSimple and FlowNetCorr for indoor and outdoor real-life scenes.
- Video: FlowNetCorr uses 256 correlation-layer filters representing 21 × 21 displacement-vector patches that favor unique displacement patterns.Each patch’s center pixel corresponds to zero displacement.