Source-linked AI summary
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss
Simon Meister, Junhwa Hur, Stefan Roth
TL;DR
Dense real-scene optical-flow ground truth is scarce, and synthetic supervision suffers from domain mismatch. UnFlow replaces it with an occlusion-aware bidirectional loss using the census transform, outperforming prior unsupervised methods and synthetic-only supervised counterparts on realistic benchmarks while supporting sparse-data fine-tuning.
Problem
Dense real-world optical-flow ground truth is difficult to obtain, while synthetic supervision differs from real imagery and limits generalization.
Method
UnFlow uses bidirectional flow estimation, explicit occlusion reasoning, and a robust census-transform photometric loss for unsupervised training.
Results
On KITTI, the unsupervised model outperforms previous unsupervised deep networks and similar supervised methods trained exclusively on synthetic data.
Takeaways & Limitations
The unsupervised loss enables optical-flow CNN pre-training and broader-domain training when only limited real-world ground truth is available.
Takeaways & Limitations
Choosing loss-term weights requires parameter search, increasing computation time when training on a new domain for the first time.
Abstract
from arXiv · showhide
In the era of end-to-end deep learning, many advances in computer vision are driven by large amounts of labeled data. In the optical flow setting, however, obtaining dense per-pixel ground truth for real scenes is difficult and thus such data is rare. Therefore, recent end-to-end convolutional networks for optical flow rely on synthetic datasets for supervision, but the domain mismatch between training and test scenarios continues to be a challenge. Inspired by classical energy-based optical flow methods, we design an unsupervised loss based on occlusion-aware bidirectional flow estimation and the robust census transform to circumvent the need for ground truth flow. On the KITTI benchmarks, our unsupervised approach outperforms previous unsupervised deep networks by a large margin, and is even more accurate than similar supervised methods trained on synthetic datasets alone. By optionally fine-tuning on the KITTI training data, our method achieves competitive optical flow accuracy on the KITTI 2012 and 2015 benchmarks, thus in addition enabling generic pre-training of supervised networks for datasets with limited amounts of ground truth.
Introduction
Optical-flow CNNs depend on synthetic ground truth because dense real-world labels are difficult to obtain, but synthetic-to-real differences limit generalization. UnFlow addresses this gap with bidirectional, occlusion-aware unsupervised training and reports strong benchmark performance.
- Dense real-world optical-flow ground truth is difficult to obtain, so most CNN methods train on synthetic images with abundant labels.
- Synthetic imagery differs intrinsically from real scenes, and limited synthetic variability makes generalization to real benchmarks challenging.
- UnFlow replaces synthetic supervision with photometric reconstruction, bidirectional flow, explicit occlusion reasoning, and a census transform for real-image robustness.
- On KITTI, the unsupervised model outperforms previous unsupervised deep networks by a very large margin and surpasses similar supervised methods trained only on synthetic data.
- Fine-tuning with sparse ground truth enables leading KITTI accuracy and pre-training for supervised methods on domains with limited ground truth.
Related Work
Early supervised FlowNet systems relied on synthetic data, while prior unsupervised FlowNet approaches used proxy losses but did not clearly overcome realistic-domain performance limitations. This motivates investigating stronger unsupervised alternatives.
- FlowNet introduced end-to-end supervised optical-flow CNNs trained with dense labels from synthetic scenes containing animated 3D chairs.
- FlowNet2 improved accuracy and complexity through stacked FlowNet networks, richer synthetic data, and KITTI fine-tuning on sparse ground truth.
- Prior unsupervised FlowNet methods replaced supervised losses with brightness-constancy and smoothness proxy losses trained on unlabeled KITTI image pairs.
- Prior unsupervised approaches did not clearly match or surpass supervised methods despite the synthetic-to-real domain mismatch.
- The paper investigates whether improved unsupervised learning can become a viable alternative or complement to supervised optical-flow learning.
Unsupervised Learning of Optical Flow
The method extends UnsupFlownet with bidirectional, occlusion-aware unsupervised loss, FlowNetC-based bidirectional estimation, and iterative refinement. Sparse supervised fine-tuning remains optional.
- The approach designs a symmetric occlusion-aware loss from forward and backward optical-flow estimates.
- It trains FlowNetC with the comprehensive unsupervised loss to estimate bidirectional flow.
- Iterative refinement is implemented by stacking multiple FlowNet networks.
- After unsupervised training, sparse ground truth can optionally provide supervised fine-tuning.
Unsupervised loss
The unsupervised loss estimates forward and backward flow, uses consistency to identify occlusions, and combines occlusion-aware photometric, consistency, and smoothness terms. It uses census-based photometric comparison, second-order flow regularization, and differentiable warping for training.
- Bidirectional estimation: The method jointly estimates forward and backward optical flow by applying symmetrical loss terms in both directions.Backward flow is obtained by exchanging the two input images in a second pass.
- Occlusion handling: Forward-backward consistency marks pixels as occluded when the two flows mismatch beyond a threshold, then masks those pixels from the data loss.For non-occluded pixels, the forward flow should invert the backward flow at the corresponding location.
- Data loss: The occlusion-aware data loss penalizes photometric differences for visible pixels and adds a constant penalty for occluded pixels to prevent all pixels becoming occluded.The photometric difference uses a robust generalized Charbonnier penalty and the ternary census transform, which tolerates common illumination changes.
- Regularization: The smoothness term imposes second-order regularization on both flow fields to encourage collinearity among neighboring flows.The neighborhood includes horizontal, vertical, and diagonal neighbors.
- Optimization: The final objective is a weighted sum of the individual loss terms and is optimized using bilinear backward warping with automatic differentiation.The same warping scheme is used to compare flow-displaced image samples during backpropagation.
Network architecture and computation
UnFlow-C predicts bidirectional dense flow with shared weights, while stacked refinement networks and multiscale losses guide increasingly detailed estimates. Fine-tuning uses sparse valid ground truth only at the final prediction.
- Network architecture: UnFlow-C applies FlowNetC to (I1, I2) and (I2, I1), sharing weights to estimate forward and backward flow.The network correlates separate image streams, encodes them to one-sixth resolution, and predicts dense flow through four upsampling stages.
- Stacked refinement: UnFlow-CS and UnFlow-CSS iteratively refine flow by stacking one or two additional FlowNetS networks with independent weights.Each refinement network receives the original images, warped imagery, brightness error, and the current flow estimate.
- Multiscale losses: Intermediate predictions receive weighted losses across refinement layers, with layer-specific settings and lower weights at higher resolutions.The census patch size is also reduced as resolution decreases.
- Supervised fine-tuning: Fine-tuning compares the bilinearly upsampled final estimate with ground truth wherever valid pixels exist, rather than downsampling sparse labels.Only the forward first pass and final prediction are used during fine-tuning.
Discussion
The discussion frames unsupervised optical flow as constrained by how faithfully its loss models video, while highlighting practical advantages and a tuning cost.
- Discussion: Unsupervised performance is limited by the loss function’s ability to faithfully model information available in video data.This is identified as the central challenge addressed by the method.
- Discussion: Parameter search over loss-term weights, including smoothness, increases training time when adapting to a new domain.The authors note that previous methods using unsupervised proxy losses share this limitation.
- Discussion: Unlike standard energy-based methods, the networks avoid expensive test-time optimization.Dataset-level stochastic minimization may also avoid pitfalls of optimizing a complex energy independently for each input.
Datasets for Training
Training uses synthetic SYNTHIA for pre-training and realistic unlabeled KITTI or Cityscapes sequences for unsupervised adaptation. Cityscapes additionally supplies skipped-frame pairs for larger displacements.
- SYNTHIA: The networks are first pre-trained on SYNTHIA, which provides synthetic imagery without optical-flow ground truth.The pre-training follows evidence that synthetic pre-training can help networks learn general flow concepts.
- KITTI: KITTI combines sparse ground truth benchmarks with a large raw dataset of 1392×512 image sequences without ground truth.Unsupervised training uses pairs of contiguous raw-KITTI images.
- Cityscapes: Cityscapes provides real driving sequences with semantic and instance annotations but no optical-flow ground truth.The model is trained on consecutive frames from its train, validation, and test sequences.
- Cityscapes: Cityscapes training also includes pairs skipping every second frame, producing about 230K image pairs at 8.5Hz for larger displacements.The original recording rate is 17Hz.
Experiments
Experiments train on SYNTHIA, raw KITTI, or Cityscapes with staged optimization, optional KITTI fine-tuning, augmentation, and benchmark-specific image resizing. Figure 3 compares baseline and best unsupervised losses visually.
- Training: SYNTHIA training runs for 300K iterations with batches of four image pairs and scheduled learning-rate reductions after 100K iterations.Stacked networks pre-train each network while freezing earlier networks.
- Training: Raw-KITTI training runs for 500K iterations with batches of four image pairs and learning-rate halvings after each 100K iterations.Stacked training uses the same staged procedure as SYNTHIA.
- Pre-processing and augmentation: Training applies random crops, noise, brightness, color, contrast, and gamma changes independently to both frames.Crop sizes differ across KITTI, SYNTHIA, and Cityscapes.
- Visual comparison: Figure 3 compares a baseline unsupervised loss with the best loss using ground truth, estimated flow, overlays, and error maps.Correct KITTI estimates are blue when within ≤3 px or ≤5% error; wrong estimates appear in red tones.
- Evaluation: KITTI benchmark images are upsampled to 1280×384 for evaluation and flow estimates are then downscaled to the original resolution.Flow components are scaled to account for changed pixel distances after resampling.
Unsupervised loss comparison
The study compares unsupervised optical-flow loss components on KITTI training data, showing that census data loss, higher-order smoothness, and occlusion-aware bidirectional consistency each improve the baseline.
- Unsupervised loss comparison: Approximately 35% improvement comes from replacing brightness constancy with the census loss.This comparison is made against the re-implemented bidirectional brightness-constancy baseline.
- Unsupervised loss comparison: Approximately 5% improvement and 17% outlier reduction result from second-order rather than first-order smoothness.
- Unsupervised loss comparison: Approximately 14% further improvement comes from occlusion masking combined with forward-backward consistency.
- Unsupervised loss comparison: Combining all three innovations reduces average endpoint error to less than 0.5× that of previous unsupervised training approaches.The AEE is not available to the network during training; training uses only the unsupervised loss.
- Unsupervised loss comparison: The ablation compares different unsupervised loss terms on the KITTI benchmark training sets using AEE and Fl-all metrics.AEE is reported over all and non-occluded pixels; Fl-all measures pixels wrong by both ≥3 pixels and ≥5%.
Results on benchmarks
Across KITTI, Middlebury, and Sintel, the unsupervised models outperform earlier unsupervised approaches and often compare favorably with supervised models, though the strongest supervised Sintel systems remain ahead.
- KITTI: On KITTI 2012, purely unsupervised UnFlow-C outperforms supervised FlowNetC and FlowNetS across all reported metrics.Compared with UnsupFlownet and DSTFlow, it more than halves both error metrics and strongly improves KITTI 2015 accuracy.
- KITTI: UnFlow-C-Cityscapes significantly outperforms UnsupFlownet and DSTFlow despite not being trained on KITTI images.
- KITTI: Fine-tuned UnFlow models perform similarly to the more complex FlowNet2-ft-kitti without a separate small-displacement network or custom training schedules.
- Middlebury: On Middlebury, UnFlow-C and UnFlow-C-Cityscapes outperform supervised FlowNetS, while stacked and fine-tuned variants fall between FlowNetS and FlowNet2.This indicates generalization beyond the driving domain used for training.
- Sintel: On Sintel, the unsupervised networks cannot compete with FlowNetS+ft and FlowNet2, but they outperform the similarly unsupervised DSTFlow.
Conclusion
The paper presents an end-to-end unsupervised approach that combines energy-based optical-flow components to train on unlabeled data and improve performance across realistic domains.
- Conclusion: The approach trains FlowNet networks on large datasets without optical-flow ground truth.
- Conclusion: Its loss combines census-transform data loss, higher-order smoothness, and occlusion reasoning from bidirectional flow estimation.
- Conclusion: The method more than halves KITTI error relative to previous unsupervised deep-learning approaches.
- Conclusion: Large real-world datasets paired with the unsupervised loss can outperform supervised training when only handcrafted synthetic datasets provide supervision.
- Conclusion: The unsupervised loss also provides a foundation for pre-training when only limited real-world ground truth is available.