Source-linked AI summary
DDFlow: Learning Optical Flow with Unlabeled Data Distillation
Pengpeng Liu, Irwin King, Michael R. Lyu, Jia Xu
TL;DR
Training optical-flow CNNs on real-world sequences is limited by scarce dense labels, while existing unsupervised methods struggle with occluded pixels. DDFlow distills reliable teacher predictions from unlabeled data to guide a student network, achieving the highest reported accuracy among prior unsupervised methods across challenging benchmarks and running in real time.
Problem
High-performance optical-flow CNNs require densely labeled real-world data, while photometric and hand-crafted unsupervised methods lack the capability to learn flow for occluded pixels.
Method
DDFlow trains identical teacher and student CNNs, using teacher predictions on unlabeled data as annotations that guide the student to learn flow for non-occluded and occluded pixels.
Results
DDFlow achieved the highest accuracy among prior unsupervised methods on Flying Chairs, MPI Sintel, KITTI 2012, and KITTI 2015, including 4.57% Fl-noc error on KITTI 2012.
Takeaways & Limitations
The results show that CNNs can be self-trained from unlabeled data to estimate optical flow, including for occluded pixels, without human annotations.
Takeaways & Limitations
The distillation strategy works particularly well near image boundaries but is less effective for occluded pixels elsewhere.
Abstract
from arXiv · showhide
We present DDFlow, a data distillation approach to learning optical flow estimation from unlabeled data. The approach distills reliable predictions from a teacher network, and uses these predictions as annotations to guide a student network to learn optical flow. Unlike existing work relying on hand-crafted energy terms to handle occlusion, our approach is data-driven, and learns optical flow for occluded pixels. This enables us to train our model with a much simpler loss function, and achieve a much higher accuracy. We conduct a rigorous evaluation on the challenging Flying Chairs, MPI Sintel, KITTI 2012 and 2015 benchmarks, and show that our approach significantly outperforms all existing unsupervised learning methods, while running at real time.
Introduction
DDFlow addresses the difficulty of training optical-flow CNNs from real-world data without dense labels by distilling predictions from unlabeled images. Its teacher–student approach learns occluded-pixel flow and reports leading unsupervised benchmark accuracy.
- Motivation: Dense labels are difficult to obtain for real-world sequences, while synthetic-only training suffers from a domain gap with natural scenes.Unlabeled videos offer a readily available alternative at much larger scale.
- Motivation: Existing unsupervised methods use photometric or hand-crafted energy terms but lack the capability to learn optical flow for occluded pixels.Photometric losses provide misleading information when pixels are occluded.
- Approach: DDFlow trains identical teacher and student networks, using teacher predictions as annotations to guide the student on optical flow estimation.The teacher estimates non-occluded flow, while the student learns from the distilled guidance.
- Approach: The framework trains the teacher with photometric loss and the student with photometric plus occlusion losses, using only the student at test time.The occlusion loss is applied to pixels made occluded by cropping but non-occluded in the original images.
- Results: 4.57% Fl-noc error on KITTI 2012 and 14.29% Fl-all error on KITTI 2015 were reported, with performance exceeding published unsupervised methods.The method also outperformed several fully supervised methods fine-tuned for each dataset.
Related Work
Prior optical-flow methods range from computationally expensive classical energy formulations to supervised CNNs dependent on synthetic data. Unsupervised approaches use photometric or hand-crafted occlusion reasoning, motivating DDFlow’s unlabeled-data distillation strategy.
- Classical Methods: Classical optical-flow methods formulate estimation as energy minimization, while feature matching and coarse-to-fine refinement address large displacements.These methods are often time-consuming and require dataset-specific tuning.
- Learning-Based Methods: CNN-based methods improve optical-flow estimation but commonly rely on synthetic pre-training because densely labeled real-world data are difficult to obtain.Synthetic-only training can overfit and perform poorly on real sequences because of the domain gap.
- Unsupervised Methods: Unsupervised methods warp the target image and optimize photometric differences, but this loss provides misleading information for occluded pixels.Brightness constancy and spatial smoothness form the basis of these approaches.
- Unsupervised Methods: Later methods exclude occluded pixels or use multi-frame temporal consistency, yet still rely on hand-crafted energy terms and cannot learn occluded-pixel flow.Their performance remained behind state-of-the-art supervised methods.
- DDFlow: DDFlow distills annotations from a model trained with a classical optical-flow energy and retrains on those generated annotations without human labels.This data-driven procedure is designed to learn optical flow for occluded pixels with simple loss functions.
Method
DDFlow jointly trains teacher and student networks, using reliable teacher predictions from original frames to supervise student flow estimation in cropped patches, including hallucinated occlusions. The method combines photometric supervision for non-occluded pixels with data-driven distillation for occluded pixels.
- The teacher predicts flow for non-occluded pixels, while the student predicts flow for both non-occluded and occluded pixels; only the student is used at test time.
- DDFlow randomly crops patches from the original frames and associated teacher outputs, then constructs masks identifying pixels newly occluded by cropping.The valid mask selects pixels occluded in the cropped patch but non-occluded in the original image.
- PWC-Net is used as the compact backbone, with seven feature levels, coarse-to-fine warping and cost volumes, and quarter-resolution flow upsampled to full resolution.Identical architectures are used for teacher and student, but their input data and loss functions differ.
- Teacher predictions from original images become annotations for pixels that cropping makes occluded, enabling the student to learn their optical flow.The teacher prediction remains available even after the corresponding pixel becomes occluded in the cropped image pair.
- The teacher uses photometric loss with forward-backward occlusion masks, while the student combines photometric loss Lp with occluded-pixel loss Lo.Photometric loss is effective for non-occluded pixels; the student’s additional loss uses teacher predictions for cropped occlusions.
- Occlusion maps are estimated from forward-backward flow consistency, marking pixels occluded when consistency fails or the mapped location leaves the image boundary.The method uses threshold parameters α1 = 0.01 and α2 = 0.05 in all experiments.
Experiments
DDFlow is evaluated across standard optical-flow benchmarks, with ablations showing benefits from occlusion handling, census transforms, and data distillation, especially for occluded pixels.
- Comparison to State-of-the-art: DDFlow outperforms existing unsupervised methods across Flying Chairs, Sintel, KITTI 2012, and KITTI 2015 benchmarks.On Flying Chairs, EPE decreases from 3.30 to 2.97; pretrained-only DDFlow also achieves EPE=3.83 on Sintel Clean and EPE=4.85 on Sintel Final.
- Comparison to State-of-the-art: 14.0% relative improvement reduces Sintel Final EPE from 8.81 to 7.40 after fine-tuning.Sintel Clean EPE also improves from 7.23 to 6.18.
- Comparison to State-of-the-art: 37.7% relative improvement yields KITTI 2015 Fl-all=14.29%, while KITTI 2012 reaches EPE=3.0 and Fl-noc=4.57%.The KITTI 2012 EPE is 28.6% lower than the best existing counterpart, and KITTI 2015 Fl-all outperforms several supervised methods.
- Occlusion Estimation: Occlusion estimation is best on Sintel, while KITTI performance is comparable because its ground-truth occlusion masks are sparse.KITTI annotations contain only pixels moving out of the image boundary, whereas DDFlow also estimates occlusions within the image.
- Ablation Study: Occlusion handling improves flow estimation over ALL, NOC, and OCC pixels because brightness constancy fails for occluded pixels.The evaluation uses EPE, Fl, F-measure, and separate NOC and OCC EPE measurements.
- Ablation Study: Census transform consistently improves performance across all datasets by compensating for illumination changes.The preprocessing is reported as robust for optical-flow estimation.
- Ablation Study: Data distillation lowers EPE-OCC by 18.5% on Sintel Clean, 16.1% on Sintel Final, 58.2% on KITTI 2012, and 42.1% on KITTI 2015.Occluded prediction improves performance on all datasets, especially for occluded pixels.
- Ablation Study: DDFlow’s distillation works particularly well near image boundaries but is less effective for occluded pixels elsewhere.Random student cropping covers many occlusions; broader handling remains a stated future direction.
Conclusion
DDFlow learns optical flow from unlabeled data by using teacher predictions as annotations for a student network, including for occluded pixels.
- Conclusion: DDFlow uses teacher-network predictions as annotations to train a student network without human annotations.The approach is presented as data distillation for unlabeled optical-flow learning.