PaperlayerView exact revision on arXiv

Source-linked AI summary

Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz

arXiv:1809.05571v1cs.CV

TL;DR

The paper asks whether optical-flow CNNs can become both more accurate and smaller, and examines how model design and training contribute to performance. It introduces PWC-Net, combining pyramidal processing, warping, and cost volumes, and reports substantial efficiency and accuracy gains alongside improved training results.

  • Problem

    The paper investigates whether optical-flow CNN accuracy can increase while model size decreases, despite the usual size–accuracy trade-off.

  • Method

    The authors design PWC-Net using pyramidal processing, warping, and cost-volume processing, then experimentally retrain FlowNetC and refine training procedures.

  • Results

    PWC-Net is 17 times smaller, 2 times faster in inference, and 11% more accurate on Sintel final than FlowNet2, while training changes further improve accuracy across Sintel and KITTI.

  • Takeaways & Limitations

    Performance gains in optical-flow CNNs reflect both network architecture and training procedures, so experiments must distinguish their contributions.

  • Takeaways & Limitations

    An input/output bug initially excluded 44 KITTI 2015 sequences, requiring cropping and retraining with the correct 200 sequences.

Abstract

from arXiv · show

We investigate two crucial and closely related aspects of CNNs for optical flow estimation: models and training. First, we design a compact but effective CNN model, called PWC-Net, according to simple and well-established principles: pyramidal processing, warping, and cost volume processing. PWC-Net is 17 times smaller in size, 2 times faster in inference, and 11\% more accurate on Sintel final than the recent FlowNet2 model. It is the winning entry in the optical flow competition of the robust vision challenge. Next, we experimentally analyze the sources of our performance gains. In particular, we use the same training procedure of PWC-Net to retrain FlowNetC, a sub-network of FlowNet2. The retrained FlowNetC is 56\% more accurate on Sintel final than the previously trained one and even 5\% more accurate than the FlowNet2 model. We further improve the training procedure and increase the accuracy of PWC-Net on Sintel by 10\% and on KITTI 2012 and 2015 by 20\%. Our newly trained model parameters and training protocols will be available on https://github.com/NVlabs/PWC-Net

1 INTRODUCTION

Optical-flow CNNs face a trade-off between accuracy and model size, while performance also depends strongly on training procedures. PWC-Net combines pyramidal processing, warping, and cost volumes to improve both efficiency and accuracy, and controlled retraining shows that models and training both matter.

  • Models matter: PWC-Net combines pyramidal processing, warping, and cost volume processing to target both accuracy and model size.The design uses a partial cost volume at each pyramid level and warping to estimate large displacements.
  • Training matters: Direct comparisons between models can be misleading when their training procedures differ, so fair evaluation requires matched training protocols.The authors report that improperly trained models may perform poorly and call for careful experimental designs to identify performance sources.
  • Training matters: 56% more accurate on Sintel final than the previously trained FlowNetC, retrained FlowNetC is 5% more accurate than reported FlowNet2.These comparisons use the same training protocol as PWC-Net, isolating the contribution of training procedures.
  • Training matters: 10% higher Sintel accuracy and 20% higher KITTI 2012 and 2015 accuracy result from further training improvements for PWC-Net.The changes include adding KITTI and HD1K data during fine-tuning and fixing an I/O bug that omitted part of the KITTI training data.
  • Contributions: PWC-Net performs robustly across four major flow benchmarks and won the optical-flow category of the robust vision challenge.The authors make newly trained parameters and training protocols available for comparison and further innovation.

2 PREVIOUS WORK

Previous optical-flow methods combine variational optimization, feature or sparse matching, CNNs, and post-processing, while dataset limitations complicate real-world evaluation. PWC-Net builds on these directions with a fully end-to-end CNN using multi-scale cost volumes.

  • Variational optical-flow methods impose brightness constancy and spatial smoothness through energy minimization, but their optimization is computationally expensive for real-time applications.
  • Many strong methods combine learned features or matches with full cost volumes and sophisticated post-processing such as EpicFlow.
  • PWC-Net instead constructs partial cost volumes at multiple pyramid levels, reducing computational and memory demands relative to full single-scale volumes.
  • Sintel and KITTI provide challenging benchmarks, but synthetic data, semi-dense labels, dynamic scenes, illumination changes, motion blur, and occlusions complicate evaluation.

3 APPROACH

PWC-Net combines feature pyramids, warping, cost-volume processing, multi-level flow estimation, and optional contextual refinement in an end-to-end CNN. These components incorporate classical coarse-to-fine principles while using learnable features and CNN-based estimation.

  • Feature pyramid extractor: PWC-Net builds feature pyramids from both input images and estimates flow from coarse to fine pyramid levels.Features are downsampled by a factor of 2 across levels.
  • Warping layer: At each level, PWC-Net upsamples and rescales the coarser flow, then warps second-image features toward the first image.Bilinear interpolation supports backpropagation through the warping operation.
  • Comparison with traditional methods: PWC-Net replaces fixed image pyramids with learnable feature pyramids, embeds warping and cost-volume layers, and uses contextual refinement.Warping and cost-volume layers have no learnable parameters, helping reduce model size.
  • Cost volume layer: The network constructs a correlation-based cost volume between first-image features and warped second-image features using a limited search range.At pyramid level l, the partial cost volume has dimension d^2×H^l×W^l.
  • Optical flow estimator: A multi-layer CNN receives the cost volume, first-image features, and upsampled flow, then predicts flow at the current pyramid level.Estimators at different pyramid levels use separate parameters.
  • Context network: A context network uses dilated convolutions to enlarge receptive fields and refine the estimated flow at the desired level.It is a seven-layer CNN with 3×3 spatial kernels and varying dilation constants.
  • Training: Training uses a multi-scale loss with parameter regularization, followed by a robust fine-tuning loss based on an L1 penalty with reduced outlier weight.The robust loss uses q < 1 and a small constant ϵ.

4 EXPERIMENTAL RESULTS

Experiments show that PWC-Net’s accuracy depends on both architecture and training, with strong results on Sintel and KITTI but persistent challenges for small, rapidly moving objects and data-handling errors.

  • Implementation details: PWC-Net uses a seven-level pyramid, outputs quarter-resolution flow, and scales flow before warping features at each level.The model uses a search range of 4 pixels for cost-volume construction.
  • MPI Sintel: PWC-Net is about 5% less accurate but 40% faster than the full model after dropping DenseNet connections.The smaller variant is intended for applications with tighter memory or runtime constraints.
  • MPI Sintel: PWC-Net is less accurate than traditional methods on Sintel clean but may be better suited to real images with corrupted edges, as in the final pass.The final pass includes motion blur, atmospheric changes, and noise that corrupt image edges.
  • MPI Sintel: PWC-Net recovers sharp motion boundaries, but small and rapidly moving objects such as the left arm in “Market 5” remain challenging.Its errors are lower than FlowNet2 on the test set despite higher training-set errors.
  • KITTI: Correcting an I/O bug and retraining on all 200 KITTI 2015 sequences improves KITTI 2012 Fl-Noc from 4.22% to 3.41% and KITTI 2015 Fl-all from 9.60% to 7.90%.The bug caused the last 44 sequences to be read improperly because they had different resolutions.
  • Training: Mixed datasets reduce MPI Sintel final AEPE from 5.04 to 4.9 but increase KITTI 2015 F-all from 9.60% to 11.63%.The authors attribute the contrast to a larger training-test mismatch for Sintel than for KITTI 2015.
  • Efficiency: PWC-Net is about 17 times smaller than FlowNet2, about 2 times faster in forward inference, and at least 3 times faster in training.Timing comparisons used the same desktop and excluded image reading and writing.

5 COMPARISON WITH CLOSELY-RELATED WORK

PWC-Net differs from closely related models by combining deeper feature pyramids, feature warping, and multi-resolution cost volumes, while retaining end-to-end training. The comparison emphasizes architectural choices and reports a trade-off between accuracy and training complexity across alternatives.

  • Architectural differences: PWC-Net learns deeper feature pyramids and warps features, enabling information to propagate throughout the feature pyramids.SpyNet uses image pyramids and warps images, whereas FlowNet2 uses shallower feature pyramids and also warps input images.
  • Architectural differences: PWC-Net feeds CNNs a cost volume and constructs it at multiple resolutions, unlike SpyNet’s image inputs and FlowNet2’s single-resolution cost volume.The cost volume is described as a more discriminative representation of the optical-flow search space.
  • Performance and training: PWC-Net outperforms SpyNet and can be trained end-to-end from scratch, whereas SpyNet and LiteFlowNet use sequential or stage-wise training.PWC-Net’s smaller architecture achieves similar or better performance than FlowNet2’s stacked large-capacity design.
  • Related architectures: LiteFlowNet incorporates feature pyramids, feature warping, and multi-resolution cost volumes, but requires sequential training.The cited comparison reports LiteFlowNet’s CVPR final version as about 8% less accurate on Sintel final than the first arXiv version of PWC-Net.

6 CONCLUSIONS

The paper concludes that compact optical-flow CNNs can combine pyramidal processing, warping, and cost volumes with strong accuracy, and that both architecture and training determine performance. PWC-Net is smaller and faster than FlowNet2, performs robustly across benchmarks, and improved further under revised training procedures.

  • Conclusions: PWC-Net performs robustly across four benchmarks using the same parameters and wins the optical-flow category of the robust vision challenge.The conclusion presents this as evidence of performance across multiple evaluation settings.
  • Conclusions: Retrained FlowNetC is 5% more accurate on Sintel final than published FlowNet2, showing that performance gains reflect both architecture and training procedures.The paper further reports training-procedure gains of 10% on Sintel and 20% on KITTI 2012 and 2015 for PWC-Net.

APPENDIX

The appendix details PWC-Net’s feature-pyramid extractor, optical-flow estimator, and context network. The extractor uses a seven-level pyramid with shared Siamese encoding for the two input images, while estimator and context layers use leaky ReLU units except at the flow-output layer.

  • Feature pyramid extractor: PWC-Net’s experiment uses a 7-level feature pyramid extractor whose bottom level contains the original input images.The two images are encoded using the same Siamese network.
  • Feature pyramid extractor: At each pyramid level, convolution and ×2 downsampling are implemented by one convolutional layer with stride 2.Each convolution is followed by a leaky ReLU unit.
  • Optical-flow estimator: The optical-flow estimator uses leaky ReLU after each convolution except the final layer, which outputs optical flow.The cited architecture is shown at pyramid level 2, with analogous estimator structure at other levels except the top level.
  • Context network: The context network similarly applies leaky ReLU after each convolution except the final flow-output layer, and its convolution labels include dilation constants.The appendix identifies the displayed context architecture at pyramid level 2.
Loading 1809.05571v1…