Source-linked AI summary
What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Nikolaus Mayer, Eddy Ilg, Philipp Fischer, Caner Hazirbas, Daniel Cremers, Alexey Dosovitskiy, Thomas Brox
TL;DR
Dense optical-flow and disparity learning lacks abundant pixel-accurate real annotations, so the paper investigates synthetic training data and how its properties affect generalization. It finds that diversity, staged combinations of simple and complex data, and camera-effect simulation are more useful than realism alone, while some manually modeled scenes remain hard to scale.
Problem
Pixel-accurate ground truth for optical flow and disparity is difficult to obtain manually at large scale, limiting real training data.
Method
The paper compares procedural and manually modeled synthetic datasets, analyzes dataset properties and augmentation, and evaluates staged training schedules for FlowNet and DispNet.
Results
Diversity, multistage training across separate datasets, and simulated camera flaws improve generalization, whereas realistic lighting provides limited benefit.
Takeaways & Limitations
Simple synthetic data and augmentation can train useful low-level vision networks, while camera-specific effects and complementary training stages improve performance within the studied tasks.
Takeaways & Limitations
The conclusions may not hold for high-level tasks such as object recognition, and manually modeled scenes are difficult to scale.
Abstract
from arXiv · showhide
The finding that very large networks can be trained efficiently and reliably has led to a paradigm shift in computer vision from engineered solutions to learning formulations. As a result, the research challenge shifts from devising algorithms to creating suitable and abundant training data for supervised learning. How to efficiently create such training data? The dominant data acquisition method in visual recognition is based on web data and manual annotation. Yet, for many computer vision problems, such as stereo or optical flow estimation, this approach is not feasible because humans cannot manually enter a pixel-accurate flow field. In this paper, we promote the use of synthetically generated data for the purpose of training deep networks on such tasks.We suggest multiple ways to generate such data and evaluate the influence of dataset properties on the performance and generalization properties of the resulting networks. We also demonstrate the benefit of learning schedules that use different types of data at selected stages of the training process.
1 Introduction
Pixel-accurate optical-flow and disparity annotations are difficult to obtain at scale, motivating synthetic training data. The paper studies how dataset generation, properties, and training schedules affect generalization.
- Real optical-flow datasets are small and restricted because accurate ground truth is difficult to derive.
- Sintel demonstrated that rendered videos can provide optical-flow, disparity, and occlusion ground truth, but its training set contains only 1041 image pairs.
- The paper investigates procedural and manually modeled synthetic data for training FlowNet and DispNet, emphasizing dataset properties that support generalization to real data.
- Simplistic data can be effective: the 2D FlyingChairs dataset provides good starting data for optical-flow training despite lacking realism.
- Multistage training on separate datasets outperforms training on either dataset alone and a full mixture of both.
- Realistic lighting does not necessarily help, whereas simulating real-camera flaws improves performance on images from such cameras.
2 Related Work
Prior work used synthetic rendering and game engines for optical flow, depth, recognition, and specialized evaluation. This paper instead analyzes large-scale synthetic training data for general-purpose optical flow and disparity estimation.
- Earlier studies used virtual scenes, object-background compositions, and simple real scenes to benchmark optical-flow methods and analyze scene or motion complexity.
- Sintel and related Blender or Maya datasets provided dense flow and disparity ground truth, often for small or specialized settings.
- Game-engine approaches reused creator-made assets to generate datasets for difficult visual effects, driving, surveillance, and disparity evaluation.
- Recent large synthetic datasets commonly target narrow domains such as driving, indoor scenes, isolated objects, or human actions.
- This work focuses on datasets supporting general-purpose optical-flow and disparity estimation rather than narrow scenarios.
- Relatively little prior work analyzed which synthetic-data properties improve generalization to real data.
- The paper consolidates and extends earlier conference datasets and studies their training-data implications systematically.
3 Synthetic Data
The paper generates synthetic training pairs through procedural randomization or manual modeling, producing exact dense ground truth for optical flow and disparity. Procedural datasets scale better, while manually modeled scenes face rendering constraints.
- Synthetic data addresses the infeasibility of manually annotating dense, pixel-accurate optical-flow or depth ground truth.
- Procedural generation combines a simple general scene with randomly sampled dynamic elements, enabling abstract datasets of potentially unlimited size.
- Manual modeling can render existing or recombined scenes, but scene-specific optical tricks and stereo incompatibilities make it difficult to scale.
- FlyingChairs creates optical-flow pairs by pasting segmented objects onto backgrounds and applying randomized transformations between frames.
- FlyingThings3D extends procedural generation to rendered 3D scenes with ShapeNet objects, linear object and camera trajectories, and complex combined motions.
- Blender rendering produces dense ground truth for depth, disparity, optical flow, and object segmentation across both views of a virtual stereo camera.
- Procedural 3D generation allows arbitrary data amounts without manual effort and yields large, diverse datasets despite lacking naturalistic realism.
- Color and geometry augmentation provide complementary benefits, while disparity augmentation is more restricted by fixed stereo-camera geometry.
4 Learning Tasks
The optical-flow and disparity networks use contracting and expanding architectures with skip connections, while correlation layers fuse paired-image features. FlowNetC and DispNetCorr1D adapt this design for their respective tasks.
- FlowNet processes two full-resolution images and outputs the corresponding optical flow field.
- Contracting layers reduce spatial resolution while expanding layers restore it, with skip connections transferring high-resolution information around the bottleneck.After expansion steps, endpoint error loss provides additional training gradients through deep supervision.
- FlowNetC separately processes the two input images for three layers with shared weights, then fuses their features using correlation.The correlation layer computes scores between patches from the two feature streams.
- Optical-flow experiments use FlowNet2-c, a FlowNetC with each layer’s channels reduced to 3/8 of the original size.This reduction lowers training cost and enables extensive evaluations.
- DispNetCorr1D uses the same Siamese-and-correlation design, reducing correlation to one dimension for rectified stereo pairs.The 1D correlation can process a much wider receptive field at the same computational cost.
5.1 Testing environments
The study evaluates training-data properties for optical flow and disparity using established Sintel and KITTI 2015 benchmarks, rather than treating benchmark construction as its objective.
- Sintel clean and KITTI 2015 measure networks trained on different data, while benchmarking itself is not the paper’s aim.
5.2 What makes a good training dataset?
The study finds that synthetic training data quality depends on dataset size, diversity, motion statistics, augmentation, and imaging realism rather than visual realism alone. Performance can improve through staged training and camera-specific modeling, but no single dataset generalizes best across domains.
- Dataset scale and composition: 22k FlyingChairs samples outperform training on 908 Sintel samples, while FlyingThings3D performs worse despite greater diversity and realism.These comparisons motivate examining which dataset properties support generalization.
- Object shape and motion: Increasing shape and motion complexity generally improves performance, with nonrigid deformations producing the overall best result.Holes are counter-productive on benchmark datasets, while scaling and rotation help most when they match target-domain motion.
- Displacement statistics: Matching the test set’s displacement distribution is important, although learning schedules and mixtures of specialized networks can largely alleviate this dependence.Sintel-like displacement statistics give the best results, whereas large deviations cause large performance drops.
- Lighting: More complex lighting can help, but dynamic lighting provides only a small effect and can confuse networks when test objects lack corresponding lighting features.Surface features contribute more strongly than sparse highlights, and material diversity remains important.
- Data augmentation: Almost all augmentation types are complementary and important, except additional between-frame geometry augmentation when the raw data already varies displacements sufficiently.Color changes between frames add variation absent from the unaugmented dataset.
- Data quantity and augmentation: Randomized augmentation enables a ∼100-fold reduction in training data while still producing better results, although the best performance uses augmentation with as much raw data as possible.Augmentation both increases effective dataset size and can cover complementary input variations, but it cannot create every type of scene variation.
5.3 Learning schedules with multiple datasets
FlyingThings3D performs worse than FlyingChairs on Sintel despite greater diversity and realism, but sequential training—FlyingChairs followed by FlyingThings3D fine-tuning—produces the best results.
- FlyingThings3D performs worse than FlyingChairs on Sintel despite greater diversity and more realistic modeling.FlyingThings3D has fewer displacements below 5 pixels, but displacement statistics alone do not explain its inferior performance.
- Combining FlyingChairs and FlyingThings3D works best only when the datasets are used separately and in the order FlyingChairs → FlyingThings3D.A 1:1 mixture of the datasets does not produce the same improvement.
- The training schedule matters because presenting a sophisticated dataset too early is disadvantageous, whereas introducing it later helps.The simpler dataset may first teach general correspondence finding without confusing 3D-motion priors.
- The experiments use Slong and Sfine schedules, with Sfine providing fine-tuning on a second dataset.Most experiments use Sshort because it has lower computational cost, while this section uses the longer schedules.
5.4 Synthesizing the defects of real imaging
Synthetic training data becomes more useful for real-camera images when it reproduces camera-specific defects such as blur, color artifacts, and Bayer interpolation.
- Camera-artifact degradation can improve performance because the network can learn that imaging flaws convey no meaning, despite muddling the pixel-label relationship.The two effects are counteracting: degradation may complicate supervision but may also improve robustness to camera artifacts.
- The experiments compare clean and degraded synthetic training data on commercial stereo-camera images and on KITTI2015 disparity estimation.The degradation was tuned to simulate the actual lens and camera characteristics.
- Training on synthetic images degraded with radial blur, general blur, and oversaturated colors produces finer disparity details, especially near image boundaries.The network also makes more sensible guesses in regions occluded in the other view and appears to learn that object boundaries can be blurry.
- Finetuning on FlyingThings3D data with Bayer-interpolation artifacts improves the KITTI 2015 score by 5%.The artifacts are emulated by simulating a Bayer sensor and reconstructing an RGB image through interpolation.
- The measured Bayer-artifact improvement is limited because KITTI ground truth is sparse and has gaps along object contours.Many output changes occur at boundaries where KITTI provides no ground truth.
6 Conclusion
The paper analyzes synthetic training data for optical flow and disparity estimation and identifies dataset diversity, realism, training schedules, and camera modeling as important factors. These findings are supported for optical flow and disparity estimation but may not extend to high-level recognition tasks.
- Diversity improves generalization: networks trained on specialized data generalize worse to other datasets than networks trained on diverse data.
- Realism is overrated for basic optical flow learning, while simplistic data and data augmentation accomplish most of the task.Sophisticated lighting models provide only minor improvements.
- Learning schedules that combine simpler and more complex datasets can greatly improve generic network performance.
- Modeling camera distortions in training data largely improves performance when the network is run with that camera.
- The findings are evaluated for optical flow and disparity estimation but may not hold for high-level tasks such as object recognition.