Source-linked AI summary
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz
TL;DR
Optical flow methods need accurate estimates without the computational expense of complex energy optimization, while CNN models face size and training challenges. PWC-Net combines learnable feature pyramids, warping, and cost volumes to deliver compact, fast, and high-performing optical flow estimation.
Problem
Optical flow estimation is important for computer vision applications, but complex energy-function optimization is computationally expensive and large CNN models can be difficult to train and deploy.
Method
PWC-Net combines learnable feature pyramids, warping, and cost volumes, using warped features and CNN processing to estimate optical flow.
Results
PWC-Net outperforms all published optical flow methods on the Sintel final pass and KITTI 2015 benchmarks while being about 17 times smaller and running at about 35 fps on Sintel-resolution images.
Takeaways & Limitations
Combining deep learning with domain knowledge reduces model size and improves performance, yielding a compact, fast, and effective optical flow model.
Takeaways & Limitations
Training can get stuck at poor local minima, requiring validation checks and a different random initialization to fix it.
Abstract
from arXiv · showhide
We present a compact but effective CNN model for optical flow, called PWC-Net. PWC-Net has been designed according to simple and well-established principles: pyramidal processing, warping, and the use of a cost volume. Cast in a learnable feature pyramid, PWC-Net uses the cur- rent optical flow estimate to warp the CNN features of the second image. It then uses the warped features and features of the first image to construct a cost volume, which is processed by a CNN to estimate the optical flow. PWC-Net is 17 times smaller in size and easier to train than the recent FlowNet2 model. Moreover, it outperforms all published optical flow methods on the MPI Sintel final pass and KITTI 2015 benchmarks, running at about 35 fps on Sintel resolution (1024x436) images. Our models are available on https://github.com/NVlabs/PWC-Net.
1. Introduction
Optical flow is important but traditional high-performing methods are computationally expensive, while CNN models expose a size–accuracy trade-off. PWC-Net addresses this trade-off by embedding pyramids, warping, and partial cost volumes into an end-to-end network.
- Optical flow supports applications including action recognition, autonomous driving, and video editing, but traditional energy minimization is usually too expensive for real-time use.
- FlowNet2 approaches state-of-the-art performance and runs quickly, but its large 640MB footprint requires sequential training and limits use on mobile and embedded devices.
- SpyNet reduces model size using spatial pyramids and warping, but performs below FlowNetS and FlowNet2, exposing a trade-off between accuracy and size.
- The paper asks whether CNN optical flow models can become both more accurate and smaller, proposing that domain knowledge combined with deep learning can achieve both goals.
- PWC-Net uses learnable feature pyramids, warping, and a partial cost volume to estimate large-displacement flow while reducing model size.
- PWC-Net outperforms published methods on MPI Sintel final pass and KITTI 2015, is about 17 times smaller than FlowNet2, and runs at about 35 fps on Sintel-resolution images.
2. Previous Work
Previous optical-flow research spans variational optimization, matching and interpolation, CNN-based systems, unsupervised learning, and benchmark development. PWC-Net incorporates classical principles into an end-to-end CNN and reports leading benchmark performance.
- Variational approach: Variational optical-flow methods combine brightness constancy and spatial smoothness, often using coarse-to-fine warping, but solving their optimization problems is computationally expensive.
- Matching and interpolation: Feature matching and sparse-to-dense interpolation methods address small or fast-moving objects that can disappear at coarse pyramid levels.
- CNN-based methods: CNN-based systems learn features for cost volumes, sparse matching, or scene classification, often combining them with sophisticated post-processing such as EpicFlow.
- Learning-based methods: Early and recent learning approaches use synthetic data, U-Net-style CNNs, stacked FlowNet models, spatial pyramids, or unsupervised objectives for optical flow.
- Cost volume: A full cost volume records matching costs across candidate correspondences but is computationally expensive and memory intensive when built at one scale.
- Datasets and benchmarks: KITTI 2015 and Sintel final are challenging benchmarks featuring large motion, illumination changes, occlusions, or atmospheric effects, and PWC-Net outperforms published methods on both.
3. Approach
PWC-Net combines learnable feature pyramids, warping, and partial cost volumes in a CNN for efficient coarse-to-fine optical-flow estimation. Its estimator processes these representations across pyramid levels, with an optional context network refining the predicted flow.
- Feature pyramid: PWC-Net replaces fixed image pyramids with learnable feature pyramids and uses CNN layers to estimate flow from feature representations.The pyramids are built by convolutional downsampling across levels.
- Training and efficiency: Warping and cost-volume layers have no learnable parameters, while the network uses multi-scale training losses with robust fine-tuning.The robust loss uses an L1-based penalty with q < 1 to reduce the penalty for outliers.
- Warping layer: At each pyramid level, PWC-Net warps second-image features toward the first image using the 2× upsampled flow from the next level.The warping uses bilinear interpolation and supports backpropagation through the input features and flow.
- Cost volume: The network constructs a partial cost volume from correlations between first-image features and warped second-image features within a limited search range.Limiting the range makes the cost volume computationally manageable while linking pyramid levels supports large-displacement flow.
- Optical flow estimator: A multi-layer CNN receives the cost volume, first-image features, and upsampled flow, then predicts the flow at the current pyramid level.This estimation process repeats across levels, with separate estimator parameters at different levels.
- Context network: An optional context network enlarges each output unit’s receptive field and refines the estimated flow using the estimate and preceding estimator features.Its design uses dilated convolutions to incorporate contextual information.
4. Experimental Results
Experiments evaluate PWC-Net across Sintel and KITTI, showing strong benchmark accuracy, efficiency, and the contributions of its architectural components and training choices.
- Main Results: PWC-Net outperforms all published methods on the MPI Sintel final pass and all published two-frame methods on KITTI 2015.
- Main Results: PWC-Net is less accurate than traditional approaches on Sintel clean, while results suggest better suitability for real images with corrupted edges.
- Main Results: PWC-Net has lower Sintel test-set errors but higher training-set errors than FlowNet2, suggesting more appropriate capacity for the task.
- Ablation Experiments: Warping, context processing, DenseNet connections, residual estimation, and larger-capacity feature pyramids improve performance in the reported ablations.
- Ablation Experiments: Sequential training on FlyingChairs, FlyingThings3D, and Sintel gradually improves performance, whereas training directly on test data produces dataset-specific overfitting.
5. Conclusions
PWC-Net combines pyramidal processing, warping, and cost volumes into a compact optical-flow CNN. It improves efficiency and benchmark performance while supporting practical video-processing applications.
- PWC-Net uses pyramidal processing, warping, and a cost volume to build a compact optical-flow CNN.
- PWC-Net is about 17 times smaller, 2 times faster in inference, and easier to train than FlowNet2.
- PWC-Net outperforms all published optical-flow methods on the Sintel final pass and KITTI 2015 benchmarks.
- PWC-Net runs at about 35 fps on Sintel-resolution images of 1024×436 pixels.
- The authors expect PWC-Net to be useful in video-processing systems and release the models for comparison and further innovation.
1. More Ablation and Visual Results
Additional experiments show strong benchmark performance, robustness to difficult motion, and sensitivity to architectural choices. Thin structures remain a recognized failure case when they are rare in training data.
- PWC-Net outperforms all published methods on MPI Sintel final in both accuracy and running time.
- PWC-Net achieves the best balance between model size and accuracy among existing end-to-end CNN models.
- Replacing dilated context-network convolutions with plain convolutions worsens performance on Chairs and Sintel but slightly improves KITTI performance.
- PWC-Net recovers sharp motion boundaries under large motion, severe occlusions, shadows, and atmospheric effects.
- PWC-Net tends to produce errors on thin structures that rarely occur in the training set, including bicycle wheels.
2. Network Details
The experiments use a seven-level feature pyramid and level-specific optical-flow estimator networks. The top level differs by directly constructing its cost volume without upsampled flow.
- The feature-pyramid extractor network has seven levels, with the bottom level containing the original input images.
- At pyramid level 2, the optical-flow estimator network uses the illustrated architecture, which is shared by other non-top levels.
- The top pyramid level directly computes the cost volume from both images’ features without upsampled optical flow.
3. Screenshots of MPI Sintel and KITTI Public Table
At submission time, PWC-Net ranked first on MPI Sintel final and KITTI 2015, and second on KITTI 2012. Its KITTI 2015 advantage over SDF is notable because KITTI 2015 includes dynamic objects.
- Figures 7–9 show screenshots of the MPI Sintel final, KITTI 2015, and KITTI 2012 public tables at submission.
- PWC-Net ranked 1st among optical-flow methods on MPI Sintel final and KITTI 2015, and 2nd on KITTI 2012.
- PWC-Net performed better than SDF on KITTI 2015, which contains dynamic objects and is more challenging.
4. Learned Features
PWC-Net’s first-layer filters focus on input regions with different properties, and later fine-tuning changes them little.
- The first convolution layer learns filters that focus on regions with different properties in the input image.
- Fine-tuning on FlyingThings3D and Sintel changes the first-layer filters little after training on FlyingChairs.
- Figure 10 presents both the learned first-layer filters and their feature responses to an input image.