Source-linked AI summary
Occlusion Aware Unsupervised Learning of Optical Flow
Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, Wei Xu
TL;DR
Unsupervised optical-flow methods lag supervised approaches, partly because occlusion and large motion undermine photometric reconstruction. The paper introduces explicit occlusion modeling and larger-search-space warping, achieving competitive benchmark results and surpassing supervised FlowNetS on KITTI, while learning capability remains a limitation.
Problem
Unsupervised optical-flow estimation still has a relatively large performance gap versus supervised methods, with occlusion and large motion among its major limitations.
Method
The method uses an end-to-end architecture that explicitly predicts occlusion and employs a larger-search-space warping strategy for large motion.
Results
The method significantly improves unsupervised estimation on Flying Chairs, MPI-Sintel, and KITTI, and outperforms supervised FlowNetS on KITTI.
Takeaways & Limitations
Modeling occlusion and large motion yields competitive unsupervised optical-flow results across multiple benchmarks and supports training from abundant unlabeled videos.
Takeaways & Limitations
Additional KITTI raw-video samples and a SpyNet-based structure did not improve results, suggesting the model’s learning capability remains limiting.
Abstract
from arXiv · showhide
It has been recently shown that a convolutional neural network can learn optical flow estimation with unsupervised learning. However, the performance of the unsupervised methods still has a relatively large gap compared to its supervised counterpart. Occlusion and large motion are some of the major factors that limit the current unsupervised learning of optical flow methods. In this work we introduce a new method which models occlusion explicitly and a new warping way that facilitates the learning of large motion. Our method shows promising results on Flying Chairs, MPI-Sintel and KITTI benchmark datasets. Especially on KITTI dataset where abundant unlabeled samples exist, our unsupervised method outperforms its counterpart trained with supervised learning.
1. Introduction
Optical flow is a fundamental computer-vision problem, but unsupervised learning still trails supervised methods because occlusion and large motion limit current approaches. The paper addresses these issues with explicit occlusion modeling and new network components, improving results across major benchmarks.
- Optical flow supports 3D scene estimation, motion-based object segmentation, object tracking, action recognition, and video object detection.
- Unsupervised optical-flow methods lag behind supervised counterparts, with occlusion and large motion identified as major limiting factors.
- Photometric reconstruction penalizes occluded regions because target pixels may lack source correspondences, encouraging incorrect flow that fills gaps with nearby pixels.
- The method predicts occlusion from backward flow and computes reconstruction loss only over non-occluded regions.
- The framework adds a new large-motion warping method, extra warped inputs, histogram equalization, and channel representation to improve unsupervised flow learning.
- The method significantly improves unsupervised flow estimation on Flying Chairs, MPI-Sintel, and KITTI, outperforming supervised FlowNetS on KITTI.
2. Related Work
Prior optical-flow research spans classical optimization, deep supervised prediction, occlusion-aware modeling, and unsupervised photometric reconstruction. The paper builds on these directions while targeting the remaining accuracy gap of unsupervised methods.
- Optical flow estimation: Classical optical-flow methods commonly optimize energy functions using brightness constancy and spatial smoothness, often with coarse-to-fine frameworks for large motion.
- Occlusion-aware optical flow estimation: Occlusion-aware methods jointly estimate flow and occlusion, treating occluded pixels as outliers, interpolating them, or ignoring their loss through flow symmetry.
- Deep learning for optical flow: Deep methods evolved from patch-matching feature extraction to fully convolutional networks that directly predict dense flow fields with supervised end-to-end training.
- Unsupervised learning for optical flow: Unsupervised methods warp images with differentiable spatial transformers and use photometric loss instead of direct flow supervision.
- Unsupervised learning for optical flow: The proposed architecture uses shared-parameter FlowNetS copies to estimate forward and backward flow, with occlusion-aware warping and smoothness regularization.
3. Network Structure and Method
The method uses bidirectional FlowNetS networks, explicit occlusion modeling, enlarged-search-space warping, and multi-scale refinement to train optical flow without direct flow labels.
- Network structure: Two FlowNetS copies with shared parameters estimate forward flow F12 from (I1, I2) and backward flow F21 from (I2, I1).The forward flow reconstructs I1 by warping I2, while the backward flow generates the occlusion map.
- Loss function: Photometric loss compares the warped image with I1 only in non-occluded regions, while edge-aware smoothness regularizes F12, including occluded areas.The smoothness term uses first- and second-order flow derivatives, and the photometric loss combines image brightness and image gradients.
- Occlusion modeling: The backward flow is forward-warped into a range map, whose thresholded values form a soft occlusion map O for regions in I1 lacking correspondence in I2.Reversed bilinear sampling distributes translated pixels to neighboring grid locations, and O(x, y) = min(1, V(x, y)).
- Large-motion warping: Enlarged-neighbor backward warping searches beyond four nearest neighbors and uses symmetric weighted samples to provide gradients toward the correct location during large motion.The original local sampler can give incorrect gradient directions when the true displacement is far from the estimated position.
- Multi-scale refinement: The FlowNetS decoder is modified across scales by adding coarser-flow warped images and photometric-error maps, so finer stages estimate residual flow.A multi-scale scheme guides unsupervised learning, and histogram equalization plus augmented RGB channels improve moving-object contrast in downsampled images.
4. Experimental Results
The method is evaluated on Flying Chairs, MPI-Sintel, and KITTI using endpoint-error metrics, qualitative examples, occlusion estimation, and component ablations. It improves substantially over unsupervised baselines and, on KITTI, surpasses a supervised counterpart while remaining behind FlowNet2.
- Evaluation setup: The evaluation uses average endpoint error (EPE) across benchmarks, with Fl-all additionally reported for KITTI2015; smaller values indicate better performance.Table 1 separates supervised and unsupervised methods, and results trained on the same data are not directly comparable to others.
- Flying Chairs: 3.30 EPE on Flying Chairs improves over the previous unsupervised result of 5.11 and approaches the supervised result of 2.71.The model is trained without ground-truth flow on a 95%/5% train-test split.
- MPI-Sintel: 7.95 EPE on Sintel Clean test improves over the previous unsupervised result of 10.40, although fine-tuning provides little additional benefit.Fine-tuning uses clean and final training data without ground-truth labels; the dataset contains only around a thousand image pairs.
- KITTI: 4.2 EPE on KITTI 2012 test improves from 9.9 and outperforms the supervised FlowNetS+ft counterpart, while remaining worse than FlowNet2.KITTI provides more unlabeled than labeled data, and the qualitative results capture motion-induced occluded areas without DSTFlow-like artifacts.
- Occlusion estimation: Occlusion estimation reaches maximum F-measure values of 0.95 on KITTI2012 and 0.88 on KITTI2015.On MPI-Sintel, the result is comparable to previous non-neural-network methods; S2D uses ground-truth occlusion labels.
- Ablation study: Adding occlusion handling lowers EPE from 5.11 to 4.51 on Flying Chairs and from 7.82 to 7.32 on MPI-Sintel Final.The ablation starts from a FlowNetS baseline without occlusion handling.
5. Conclusion
The paper presents an end-to-end unsupervised optical-flow framework that models occlusion and large motion, achieving competitive results across multiple benchmarks.
- The framework performs optical-flow prediction through end-to-end unsupervised learning.
- Explicitly modeling occlusion and large motion produces competitive results on multiple benchmark datasets.
- The approach supports using abundant unlabeled videos to train flow predictors for higher-level computer-vision tasks.