Source-linked AI summary
Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation
Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, Feiyue Huang
TL;DR
Unsupervised optical-flow learning lacks reliable supervision in challenging scenes because view synthesis can fail under occlusion, brightness changes, and ambiguous textures. The paper adds transformation-based self-supervision through an extra forward pass and introduces a lightweight multi-frame network, achieving leading unsupervised accuracy and competitive supervised performance with fewer parameters.
Problem
View-synthesis supervision for unsupervised optical flow is unreliable in challenging scenes, while supervised methods depend on synthetic ground truth that can degrade real-world transfer.
Method
The framework runs an additional forward pass on transformed images and trains consistency against transformed predictions from the original images, using a lightweight multi-frame network.
Results
The method consistently improves accuracy across benchmarks, leads deep unsupervised methods, and achieves comparable supervised performance with much fewer parameters.
Takeaways & Limitations
Transformation-based self-supervision provides a general framework applicable to diverse augmentations, while the lightweight design supports strong accuracy with reduced model size.
Takeaways & Limitations
Photometric loss remains invalid for occluded or out-of-view pixels and ambiguous in textureless or repetitive regions, requiring masking or regularization.
Abstract
from arXiv · showhide
Unsupervised learning of optical flow, which leverages the supervision from view synthesis, has emerged as a promising alternative to supervised methods. However, the objective of unsupervised learning is likely to be unreliable in challenging scenes. In this work, we present a framework to use more reliable supervision from transformations. It simply twists the general unsupervised learning pipeline by running another forward pass with transformed data from augmentation, along with using transformed predictions of original data as the self-supervision signal. Besides, we further introduce a lightweight network with multiple frames by a highly-shared flow decoder. Our method consistently gets a leap of performance on several benchmarks with the best accuracy among deep unsupervised methods. Also, our method achieves competitive results to recent fully supervised methods while with much fewer parameters.
1. Introduction
Unsupervised optical flow avoids costly ground truth but view-synthesis supervision becomes unreliable in challenging scenes. The proposed framework uses transformation-based self-supervision and achieves strong accuracy with a single network.
- Ground-truth collection is difficult, so supervised optical-flow methods rely heavily on synthetic datasets whose domain gap can degrade real-world transfer.
- View-synthesis objectives assume brightness constancy, which fails under extreme brightness, partial occlusion, and other challenging conditions.
- Heavy augmentation is avoided because it shifts data distributions and creates challenging samples where view-synthesis supervision is unreliable.
- The framework adds a forward pass on transformed images and uses transformed predictions from the original pass as reliable self-supervision.
- The method leads deep unsupervised approaches, matches previous supervised performance, and uses far fewer parameters with strong cross-dataset generalization.
2. Related Work
Related work contrasts supervised methods' dependence on synthetic data with increasingly sophisticated unsupervised alternatives. Augmentation and consistency-based learning provide additional directions for improving training.
- Supervised optical-flow methods offer accurate and efficient inference but depend heavily on large-scale synthetic datasets, limiting real-world transfer.
- Unsupervised optical-flow research removes the need for dense ground truth and improves accuracy through occlusion reasoning, multi-frame inputs, and geometric constraints.
- Prior augmentation research includes learned spatial transformations, consistency training, and augmentation optimization for complex or semi-supervised learning settings.
3. Preliminaries
The basic unsupervised framework predicts dense flow between consecutive frames and trains implicitly by reconstructing one view from the other. Photometric supervision is masked or regularized where correspondence is unreliable or ambiguous.
- Given consecutive RGB frames, a network predicts dense optical flow U12 using learnable parameters Θ.
- View synthesis warps image I2 with predicted flow U12 to reconstruct the viewpoint of image I1, producing a photometric training objective.
- The photometric loss compares pixels using a similarity measure such as ℓ1 distance or structural similarity, with bilinear sampling for continuous coordinates.
- Photometric supervision fails for occluded or out-of-view pixels, which are identified using forward-backward flow checking and excluded from the loss.
- Photometric supervision is ambiguous in textureless or repetitive regions, motivating smooth regularization that constrains flow toward neighboring predictions.
4. Method
The method regularizes unsupervised optical-flow learning by adding a transformed-data forward pass supervised by transformed predictions from original data. It also supports occlusion-focused transformations and a lightweight multi-frame architecture with shared flow decoding.
- 4.1. Augmentation as a Regularization: The framework applies corresponding transformations to images, flow, and occlusion maps so augmentations remain consistent with optical-flow geometry.Spatial transformations require warping an intermediate flow field, while transformed occlusions combine warped old occlusions with newly out-of-boundary occlusions.
- 4.1. Augmentation as a Regularization: A complete training step adds a second forward on transformed images, using transformed original flow predictions as supervision.The original and transformed samples are processed separately, with the transformed prediction trained for consistency.
- 4.1. Augmentation as a Regularization: Occlusion transformation combines random cropping with random superpixel masking to create additional occlusions within one-stage end-to-end training.Unlike prior two-stage distillation, the framework optimizes a single model and avoids offline transformed-sample creation.
- 4.2. Overall Objective and Convergence Analysis: The training objective combines photometric, smoothness, and augmentation-consistency losses, with augmentation loss focused on non-occluded original pixels.A minor augmentation weight keeps original data dominant, while consistency supervision becomes important for newly occluded pixels.
- 4.3. Lightweight Network Architecture: The lightweight multi-frame extension reduces decoder connections, shares the flow decoder across pyramid levels, and reuses it for forward and backward flow.Additional convolutions align feature maps, while repeated warping and correlation extend the model to multiple frames.
5. Experimental Results
Experiments on Sintel and KITTI show that ARFlow improves unsupervised optical-flow accuracy, while its lightweight multi-frame design reduces parameter costs and generalizes across datasets.
- Comparison with State-of-the-art: ARFlow outperforms previous unsupervised methods on MPI Sintel, improving AEPE from 6.18 to 4.49 on Clean and from 6.57 to 5.67 on Final.These correspond to 27.3% and 13.7% improvements, respectively.
- Comparison with State-of-the-art: On KITTI, ARFlow achieves AEPE=1.26 on 2012 and AEPE=2.85 on 2015, with 25.4% and 41.2% relative improvements over the previous best unsupervised method.On the test set, it reaches AEPE=1.5 and F1-all=11.79% among unsupervised methods.
- Comparison with State-of-the-art: ARFlow reaches or approaches supervised LiteFlowNet and PWC-Net performance using 27.1% of PWC-Net’s parameters.The comparison includes both supervised and unsupervised methods on optical-flow benchmarks.
- Ablation Study: The augmentation-regularization framework improves optical-flow accuracy by over 10% across model architectures for both occluded and non-occluded pixels.The ablation evaluates AEPE over all, non-occluded, and occluded regions, as well as motion-speed ranges.
- Ablation Study: Sharing the flow decoder across pyramid levels achieves improvement with only 25.6% of the original model’s parameters, while the multi-frame extension gives the best performance with minimal parameter overhead.The reduced dense variant also maintains performance while reducing parameters by 39.2%.
- Ablation Study: Spatial transformation is most helpful across measurements, while occlusion or spatial transformations substantially improve accuracy in occluded regions.Each transformation improves performance individually, especially for large-displacement estimation with spatial transformation.
- Ablation Study: Direct augmentation worsens all metrics, whereas the proposed single-stage framework avoids unreliable photometric loss on transformed samples and achieves the best results.Distillation alleviates unreliable supervision but remains limited by a frozen teacher and multi-stage training.
- Cross Dataset Generalization: Cross-dataset evaluation shows supervised PWC-Net performs best on its training dataset, while ARFlow transfers better to other datasets.A model trained without ground truth on CityScapes performs best on KITTI 2012 and 2015 among the compared models trained on synthetic data.
6. Conclusion
The framework learns optical flow from unlabeled image sequences using augmentation-based self-supervision and extends this with lightweight, multi-frame networks. Experiments report improved accuracy, compatibility, and generalization ability.
- The proposed framework learns optical flow from unlabeled image sequences using self-supervision from augmentations.It adds a forward pass for transformed images and supervises them with transformed predictions of the original images.
- The method avoids relying on view-synthesis objectives for transformed data by using transformed predictions of original images as supervision.
- A lightweight network and a multi-frame extension with a highly shared flow decoder are presented.
- Extensive experiments show significant accuracy improvements together with high compatibility and generalization ability.
- The framework may be combined with geometrical constraints or transferred to depth and scene-flow estimation.