Source-linked AI summary
Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness
Jason J. Yu, Adam W. Harley, Konstantinos G. Derpanis
TL;DR
Supervised optical-flow convnets require large labeled datasets, but real-scene groundtruth flow is unavailable and existing datasets are limited. This paper trains a convnet end-to-end from image pairs using photometric constancy and flow smoothness losses, achieving competitive or superior performance to supervised training on KITTI.
Problem
Real-scene optical-flow groundtruth is unavailable, while supervised convnet success depends on large annotated datasets that are limited or laborious to create.
Method
The paper trains a convnet end-to-end from image pairs using a variational loss combining photometric constancy and spatial smoothness.
Results
The unsupervised approach outperforms the supervised one on KITTI’s non-occluded metric and remains competitive when all pixels are considered.
Takeaways & Limitations
Unsupervised training can use image data without dense groundtruth and supports competitive or superior optical-flow performance on KITTI.
Abstract
from arXiv · showhide
Recently, convolutional networks (convnets) have proven useful for predicting optical flow. Much of this success is predicated on the availability of large datasets that require expensive and involved data acquisition and laborious la- beling. To bypass these challenges, we propose an unsuper- vised approach (i.e., without leveraging groundtruth flow) to train a convnet end-to-end for predicting optical flow be- tween two images. We use a loss function that combines a data term that measures photometric constancy over time with a spatial term that models the expected variation of flow across the image. Together these losses form a proxy measure for losses based on the groundtruth flow. Empiri- cally, we show that a strong convnet baseline trained with the proposed unsupervised approach outperforms the same network trained with supervision on the KITTI dataset.
1 Introduction
Optical-flow convnets have largely depended on supervised training, but real-scene groundtruth is unavailable and labeled datasets are limited or laborious to create. The paper proposes end-to-end unsupervised training from image pairs using photometric and spatial losses, and reports superior performance to supervised training on KITTI.
- Optical flow estimates pixelwise velocities between two temporal images and has traditionally been studied within variational frameworks.
- Real-scene optical-flow groundtruth cannot be directly obtained, making conventional supervised convnet training problematic.
- The proposed method trains a convnet end-to-end from images alone using a standard variational loss rather than groundtruth flow.
- Its loss combines photometric constancy from inverse warping with spatial smoothness between neighboring flow predictions.
- Existing real-image datasets are too small for accurate network training, while synthetic data generation and large diverse imagery remain laborious.
- The approach differs from related unsupervised methods by using a simpler feedforward convnet for flow prediction and enabling end-to-end learning without a Taylor approximation.
- On KITTI, the unsupervised model outperforms the same supervised network despite limited available groundtruth flow.
2 Technical approach
The approach trains a convnet to predict optical flow from image pairs using an unsupervised variational loss, combining photometric consistency with flow smoothness. Differentiable warping enables end-to-end learning, while FlowNet Simple uses intermediate losses to refine predictions across its expanding stages.
- Unsupervised objective: The model learns a nonlinear mapping from two RGB images to a two-channel optical-flow field.The input has shape H×W×6 and the output has shape H×W×2.
- Unsupervised loss: The unsupervised loss combines photometric error between the first image and an inverse-warped second image with spatial smoothness of neighboring flow predictions.The smoothness weight λ controls the relative importance of the regularization term.
- Robust regularization: The robust Charbonnier penalty is used for the photometric term to mitigate outlier effects and for the smoothness term to regularize ambiguous regions.Smoothness addresses regions with insufficient image structure, where multiple velocities can receive similar photometric scores.
- Differentiable warping: A spatial transformer performs the backward warp with a differentiable image sampler, allowing standard backpropagation through the photometric loss.The sampler uses bilinear interpolation based on coordinates generated from the predicted flow.
- Reference architecture: FlowNet Simple combines a contractive encoder with an expanding skip-layer architecture that iteratively refines coarse flow predictions.Intermediate losses at multiple expansion stages guide earlier layers toward the final objective.
3 Empirical evaluation
The evaluation uses synthetic Flying Chairs and real-world KITTI data, emphasizing KITTI’s limited dense groundtruth and testing the unsupervised model against supervised FlowNet.
- Datasets: Flying Chairs contains 22,232 training and 640 test image pairs with groundtruth flow, with 2,000 training pairs reserved for validation and overfitting monitoring.
- Datasets: KITTI provides 194 training and 195 testing image pairs with sparse groundtruth flow, while raw city, residential, and road imagery supports unsupervised training.
- Training setup: The experiments use FlowNet Simple, Adam optimization, dataset-specific Charbonnier parameters, and smoothness weights of 1 for Flying Chairs and 0.53 for KITTI.
- Results: On Flying Chairs, FlowNet trained with groundtruth flow outperforms the unsupervised model, reflecting the advantage of sufficient dense supervision in synthetic imagery.
- Qualitative result: Figure 3 compares overlaid input frames, groundtruth flow, and unsupervised predicted flow on a KITTI example.
- Evaluation metric: Table 1 reports average endpoint error over all labeled pixels and over non-occluded labeled pixels, with supervised FlowNet results using FlowNet Simple without variational smoothing post-processing.
- Results: On KITTI’s NOC metric, the unsupervised approach improves over supervised FlowNet on both the training and official test sets, while remaining competitive across all pixels.
4 Discussion and summary
The paper concludes that end-to-end unsupervised training can match or exceed supervised performance for optical flow. It points to video from commodity cameras and improved losses as supported directions for extending the framework.
- The paper presents an end-to-end unsupervised approach for training convnets to predict optical flow.
- The proposed approach achieves competitive and sometimes superior performance relative to supervised training.
- Commodity-camera video in target domains such as automotive applications offers a potential source for further improvement.
- The framework can be extended with more sophisticated losses to improve convnet mappings between temporal imagery and flow.