Source-linked AI summary
SelFlow: Self-Supervised Learning of Optical Flow
Pengpeng Liu, Michael Lyu, Irwin King, Jia Xu
TL;DR
Optical-flow learning is hindered by occlusions, scarce real-world annotations, and the gap between synthetic and natural data. SelFlow distills reliable non-occluded flow to supervise hallucinated occlusions and aggregates multiple-frame information. It achieves leading unsupervised benchmark performance, while fine-tuning reaches state-of-the-art results, including EPE=4.26 on Sintel.
Problem
Optical-flow learning lacks abundant real-world labels, while occlusions make photometric supervision unreliable and synthetic pre-training differs from natural scenes.
Method
SelFlow distills reliable flow estimates from non-occluded pixels to supervise hallucinated occlusions and uses a CNN to aggregate temporal information from multiple frames.
Results
The method achieves the highest unsupervised accuracy on Sintel and KITTI benchmarks, while fine-tuning reaches state-of-the-art results on all three datasets, including EPE=4.26 on Sintel.
Takeaways & Limitations
Self-supervised pre-training can reduce reliance on synthetic labeled datasets while providing an effective initialization for supervised fine-tuning.
Abstract
from arXiv · showhide
We present a self-supervised learning approach for optical flow. Our method distills reliable flow estimations from non-occluded pixels, and uses these predictions as ground truth to learn optical flow for hallucinated occlusions. We further design a simple CNN to utilize temporal information from multiple frames for better flow estimation. These two principles lead to an approach that yields the best performance for unsupervised optical flow learning on the challenging benchmarks including MPI Sintel, KITTI 2012 and 2015. More notably, our self-supervised pre-trained model provides an excellent initialization for supervised fine-tuning. Our fine-tuned models achieve state-of-the-art results on all three datasets. At the time of writing, we achieve EPE=4.26 on the Sintel benchmark, outperforming all submitted methods.
1. Introduction
Optical flow remains difficult to learn accurately because occlusions mislead photometric supervision, while labeled real-world data are scarce and synthetic data differ from natural scenes. SelFlow addresses these constraints by distilling reliable non-occluded predictions, hallucinating occlusions, and incorporating multiple-frame information, achieving strong unsupervised and fine-tuned results.
- Motivation: Supervised CNN training is constrained by the difficulty of obtaining large amounts of labeled optical-flow data, especially for occluded regions.Synthetic pre-training helps but introduces a distribution gap with natural scenes and requires dataset-specific learning schedules.
- Motivation: Occlusions remain a central challenge because photometric losses work for non-occluded pixels but provide misleading information for occluded pixels.Existing methods exclude occluded pixels or add spatial and temporal regularization.
- Approach: SelFlow distills reliable flow from non-occluded pixels to guide learning for hallucinated occlusions created by perturbing target-image superpixels.The method trains a non-occlusion model and an occlusion model, using the former's reliable predictions as supervision for newly occluded pixels.
- Approach: SelFlow also uses temporal information from multiple frames through a simple CNN architecture to improve flow prediction accuracy.The paper describes this temporal extension as a second principle of the approach.
- Results: EPE=4.26 is reported for the fine-tuned model on the Sintel benchmark, with the highest reported accuracy at the time of writing.The paper also reports that fine-tuning achieves state-of-the-art results on all three datasets.
2. Related Work
Classical and supervised optical-flow methods face limitations involving computation, large displacements, and dependence on synthetic pre-training. SelFlow relates to these approaches through a lightweight CNN-based, unsupervised strategy that uses multi-frame information and non-occluded predictions for self-supervision.
- Classical Optical Flow Estimation: Classical variational methods use brightness constancy and spatial smoothness but can fail under large displacements and may be computationally expensive.Later approaches add feature matching and pyramidal coarse-to-fine interpolation to obtain dense flow.
- Temporal Information: Multi-frame optical-flow methods commonly impose temporal constraints such as constant velocity, constant acceleration, low-dimensional motion, or rigid/non-rigid segmentation.The paper contrasts these assumptions with its direct learning approach.
- Supervised Learning of Optical Flow: CNN optical-flow systems such as FlowNet, FlowNet 2.0, SpyNet, PWC-Net, and LiteFlowNet learn dense flow with end-to-end or lightweight architectures.PWC-Net and LiteFlowNet warp CNN features and report state-of-the-art results within lightweight frameworks.
- Supervised Learning of Optical Flow: High-accuracy supervised CNN methods generally require pre-training on multiple synthetic datasets and specific training schedules.SelFlow is presented as reducing this reliance through self-supervised training with unlabeled data.
- Unsupervised Learning of Optical Flow: SelFlow uses reliable non-occluded predictions as self-supervision for occluded pixels, extending domain-specific self-supervision beyond generic representation-learning tasks.This design is positioned as an unsupervised optical-flow learning strategy.
3. Method
The method trains separate non-occlusion and all-pixel flow models, distilling reliable non-occluded predictions to supervise hallucinated occlusions while incorporating temporal information from three frames. It estimates occlusions through forward-backward consistency and trains with photometric and self-supervision losses, followed by supervised fine-tuning when annotations are available.
- Self-supervised flow learning: The NOC-Model learns accurate flow for non-occluded pixels, while the OCC-Model predicts flow for all pixels using distilled NOC-Model estimates as guidance.Only the OCC-Model is needed at test time.
- Multi-frame flow estimation: Three consecutive frames provide temporal information, with the network simultaneously estimating forward and backward flows and cost volumes.The architecture is built on PWC-Net and shares forward and backward network structure and weights.
- Occlusion estimation: The method uses forward-backward consistency to mark pixels as occluded when forward flow and reversed forward flow disagree beyond a constraint.The same procedure computes other occlusion maps, with α1 = 0.01 and α2 = 0.05.
- Occlusion hallucination: Synthetic occlusions are created by injecting random noise into selected superpixels, whose irregular shapes and object-boundary edges better match real-world occlusions than rectangles.Noise can be injected into one or multiple consecutive frames.
- Loss functions: Photometric loss trains the NOC-Model on non-occluded pixels, while the OCC-Model combines photometric loss with self-supervision loss on synthetic occlusions.The losses do not rely on spatial or temporal consistency assumptions and support both two-frame and multi-frame estimation.
- Supervised fine-tuning: After pre-training on raw data, the model is initialized with the pre-trained OCC-Model and optimized using supervised loss on available forward-flow annotations.The supervised loss uses a validity mask indicating pixels with labeled ground-truth flow.
4. Experiments
Experiments evaluate SelFlow on public optical-flow benchmarks, compare unsupervised and supervised performance, and ablate its occlusion handling, self-supervision, temporal modeling, and pre-training.
- Evaluation: The evaluation uses MPI Sintel, KITTI 2012, and KITTI 2015, with EPE and Fl-all as the principal benchmark metrics.The code and models are publicly available for reproducibility.
- Unsupervised Results: 11.2% relative improvement reduces the previous best unsupervised Sintel-final EPE from 7.40 to 6.57.This result exceeds several fully supervised methods listed in the passage.
- Unsupervised Results: EPE=1.69 on KITTI 2012 training and EPE=4.84 on KITTI 2015 training improve on the previous best unsupervised DDFlow results by 28.1% and 15.3%, respectively.Testing results include Fl-all=7.68% on KITTI 2012 and Fl-all=14.19% on KITTI 2015.
- Supervised Fine-tuning: EPE=4.26 on Sintel final is the highest accuracy among submitted methods, while supervised fine-tuning also reaches Fl-all=6.19% on KITTI 2012 and Fl-all=8.42% on KITTI 2015.The paper reports state-of-the-art supervised results on all three datasets.
- Ablation Study: Self-supervised pre-training makes supervised fine-tuning converge quickly and achieve much better validation results than direct training with ground truth alone.Direct training without pre-training fails to converge well because the available training data are insufficient.
5. Conclusion
The paper presents self-supervised optical-flow learning that handles occlusions from unlabeled data and combines occlusion guidance with multi-frame temporal aggregation. Experiments report stronger unsupervised performance and state-of-the-art fine-tuned accuracy without synthetic labeled pre-training.
- Conclusion: SelFlow creates occlusions by perturbing superpixels, distills non-occluded flow to guide occluded-pixel learning, and aggregates multiple frames with a simple CNN.The method is designed to learn accurate optical flow from unlabeled data.
- Conclusion: The approach significantly outperforms existing unsupervised optical-flow methods and achieves state-of-the-art accuracy after supervised fine-tuning on leading benchmarks.The reported conclusion covers both unsupervised learning and fine-tuned evaluation.
- Conclusion: Self-supervised pre-training on unlabeled data removes reliance on synthetic labeled datasets within the paper’s reported setting.The conclusion states that superior performance is achieved through self-supervised pre-training.
1. Overview
The supplement reports occlusion-estimation results and documents submissions to the MPI Sintel, KITTI 2012, and KITTI 2015 public benchmarks.
- Overview: The supplement first evaluates SelFlow’s occlusion estimation performance, then presents screenshots of submissions to three public benchmarks.The screenshots are dated Nov. 23, 2018.
2. Occlusion Estimation
Occlusion estimation is evaluated with F-measure using forward-backward consistency, with sparse occlusion annotations noted for KITTI. SelFlow performs best on Sintel and comparably on KITTI.
- Occlusion Estimation: F-measure, the harmonic mean of precision and recall, evaluates occlusion estimation from a forward-backward consistency check without learned parameters.The method follows prior occlusion-estimation evaluations.
- Occlusion Estimation: SelFlow achieves the best occlusion-estimation performance on Sintel Clean and Sintel Final, and comparable performance on KITTI 2012 and KITTI 2015.KITTI occlusion maps are sparse, which qualifies those comparisons.
3. Screenshots on Benchmarks
The benchmark screenshots show that the proposed method performs strongly across MPI Sintel, KITTI 2012, and KITTI 2015, with unsupervised entries outperforming existing unsupervised methods. Its supervised fine-tuned entries also rank highly on these benchmarks.
- On MPI Sintel, the unsupervised entry outperforms existing unsupervised methods and several supervised methods, while the fine-tuned entry ranks No. 1 among submitted methods.
- On KITTI 2012, the unsupervised entry outperforms existing unsupervised methods and several recent fully supervised methods.
- On KITTI 2012, the supervised fine-tuned entry ranks second among published monocular optical flow methods, behind LiteFlowNet, while achieving better Out-All and Ave-All.