Source-linked AI summary
A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, Thomas Brox
TL;DR
The paper addresses the lack of large, realistic, fully annotated data for learning disparity and scene flow with convolutional networks. It constructs three synthetic stereo-video datasets and trains networks for disparity and scene flow, obtaining competitive real-time disparity performance and initial full-scene-flow results. The dataset also supports evaluation, especially for scene flow where ground-truth resources were scarce.
Problem
Large, realistic, variable datasets with ground truth for disparity and scene flow were missing for training and evaluating convolutional networks.
Method
The paper creates three Blender-based synthetic stereo-video datasets and trains FlowNet-inspired convolutional networks, including a disparity network with horizontal feature correlation.
Results
The disparity network is on par with the state of the art while running 1000 times faster, and the scene-flow network provides promising initial results.
Takeaways & Limitations
The dataset can train large convolutional networks and serve as an evaluation resource for stereo, optical flow, and scene-flow estimation.
Takeaways & Limitations
Scene-flow training had not converged, was relatively slow at 0.28 seconds per forward pass, and was expected to improve with longer training.
Abstract
from arXiv · showhide
Recent work has shown that optical flow estimation can be formulated as a supervised learning task and can be successfully solved with convolutional networks. Training of the so-called FlowNet was enabled by a large synthetically generated dataset. The present paper extends the concept of optical flow estimation via convolutional networks to disparity and scene flow estimation. To this end, we propose three synthetic stereo video datasets with sufficient realism, variation, and size to successfully train large networks. Our datasets are the first large-scale datasets to enable training and evaluating scene flow methods. Besides the datasets, we present a convolutional network for real-time disparity estimation that provides state-of-the-art results. By combining a flow and disparity estimation network and training it jointly, we demonstrate the first scene flow estimation with a convolutional network.
1. Introduction
Scene flow estimation seeks depth and 3D motion for all visible points, but progress has been limited by scarce fully annotated data. The paper introduces large synthetic datasets and demonstrates convolutional-network applications for disparity and scene flow.
- Scene flow estimates the depth and 3D motion vectors of all visible points in stereo video.
- The datasets contain over 35 000 stereo frames with dense ground truth for optical flow, disparity, disparity change, and additional data such as object segmentation.
- A shortage of fully annotated ground truth data has constrained exploration of full scene flow beyond its disparity and optical-flow subtasks.
- Three Blender-based synthetic datasets provide stereo images and ground truth for disparities, optical flow, disparity change, motion boundaries, segmentation, calibration, and 3D positions.
- The paper trains a disparity network with competitive real-time benchmark performance and presents a convolutional scene-flow network with the first quantitative full-scene-flow results on a sufficiently sized test set.
2. Related Work
Prior datasets support disparity, optical flow, or restricted scene-flow evaluation, but differ in scale, realism, completeness, or annotation density. Existing learning approaches likewise remain limited in task coverage and availability.
- Middlebury provides real stereo scenes and optical-flow data but remains small, with small test sets encouraging manual overfitting.
- Sintel offers realistic synthetic scenes, dense optical-flow ground truth, and reliable evaluation, but its 1064 training frames are too few for training convolutional networks.
- KITTI supplies calibrated real road-scene stereo videos, but laser-based ground truth is restricted to static parts, sparse at distance and height, and incomplete for 3D motion.
- The Flying Chairs dataset is large for optical flow but limited to single-view 2D motion, lacks 3D motion, and was not publicly available.
- The related disparity method combines Siamese matching, cost aggregation, and semi-global matching rather than end-to-end convolutional-network training.
- Scene-flow research has comparatively few papers, with prior methods dominated by variational formulations and none using a learning approach.
3. Definition of Scene Flow
Scene flow represents visible-point depth and 3D motion through optical flow, disparity, and disparity change derived from stereo video. Camera parameters allow these components to reconstruct the underlying 3D quantities, while omitted occluded regions and noise sensitivity remain important boundaries.
- Scene flow provides each visible point’s 3D position and 3D motion vector from successive stereo frames.
- Optical flow is the projection of world 3D motion onto the image plane, whereas scene flow represents the underlying 3D motion field.
- A camera-independent scene-flow representation uses optical flow, disparity, and disparity change, from which visible 3D points and their motion can be computed when camera parameters are known.
- The figure distinguishes commonly used red disparity and flow relations from additional blue relations that the datasets also provide.
- KITTI evaluates only optical flow and disparities because disparity change is almost redundant, limiting reconstruction to points visible in both left and right frames.
- Reconstructing 3D motion from flow and disparities is more noise-sensitive because small optical-flow errors can produce large 3D-motion errors.
4. Three Rendered Datasets
The paper introduces three Blender-rendered stereo-video datasets designed to provide varied, large-scale, dense ground truth for disparity, optical flow, and complete scene flow. The datasets combine randomized objects and motion with distinct content styles, camera settings, and image-rendering variants.
- Dataset generation: The dataset suite derives complete forward and backward scene-flow ground truth from rendered current, future, and past 3D positions.Pixelwise differences produce 3D motion vectors, including in occluded regions because the renderer retains invisible scene points.
- Dataset generation: Depth, optical flow, disparity, and disparity change are computed from 3D positions, camera parameters, and the virtual stereo configuration.The datasets also provide object and material segmentation, motion boundaries, and full camera intrinsics and extrinsics.
- 4.1. FlyingThings3D: FlyingThings3D contains about 25 000 stereo frames generated from randomized 3D object trajectories and a large pool of models and textures.Its scenes use ShapeNet objects, procedural and photographic textures, animated cameras, and clean or postprocessed image passes.
- 4.2. Monkaa: Monkaa contributes nonrigid, softly articulated motion and challenging fur using assets from an animated short film and newly assembled scenes.Additional versions vary camera translation and rotation keyframes to increase the amount of data.
- 4.3. Driving: Driving provides a mostly naturalistic dynamic street scene from a driving-car viewpoint, with a wide-angle variant added to resemble KITTI.Its stereo baseline is set to 1 Blender unit, chosen to be comparable to KITTI’s camera and car dimensions.
5. Networks
The networks adapt FlowNet’s encoder-decoder design to disparity and combine pretrained flow and disparity networks for joint scene-flow estimation. DispNet adds expanding-part convolutions for smoother disparity, while correlation provides a cheaper option for larger horizontal displacements.
- Architecture: FlowNet-style networks use contracting and expanding parts with long-range links, enabling large-displacement estimation without an information bottleneck.The contracting path downsamples by 64, while the expanding path progressively upsamples features and integrates earlier representations.
- Disparity networks: DispNet adds convolutions between up-convolutional layers to regularize disparity maps and produce smoother results.The architecture specification alternates upconvolutions, convolutions, and loss layers, with earlier features concatenated into later layers.
- Disparity networks: DispNetCorr processes the two images separately through conv2 before horizontally correlating features over displacements up to 40 pixels.The one-dimensional correlation is computationally cheaper than FlowNet’s two-dimensional correlation and covers larger input-image displacements.
- Scene-flow network: SceneFlowNet combines FlowNet with two DispNets and fine-tunes the joint network to predict flow, disparity, and disparity change.The constituent networks are pretrained before being combined.
- Training: All networks are trained end-to-end with image inputs and optical-flow, disparity, or scene-flow ground truth outputs.Training uses progressive loss weighting and data augmentation through spatial and chromatic transformations.
6. Experiments
The experiments evaluate disparity and scene-flow networks on synthetic and real benchmarks, showing strong disparity performance while scene-flow training remains preliminary and computationally demanding.
- Evaluation setup: The evaluation compares existing disparity and optical-flow methods with the proposed networks using endpoint error, except KITTI 2015’s D1-all metric.D1-all counts pixels whose error exceeds both 3 pixels and 5% of the ground-truth disparity.
- DispNet results: DispNetCorr reaches KITTI 2015’s second-best result, runs roughly 1000 times faster than MC-CNN-acrt, and operates at 15 frames per second.It also achieves about 30% lower error than the best reported real-time method, Multi-Block-Matching.
- Generalization: KITTI fine-tuning improves KITTI results but increases errors elsewhere, apparently because the network loses accuracy on disparities larger than KITTI’s roughly 150-pixel range.Other datasets include disparities of 500 pixels and more.
- DispNet architecture: Adding convolutions between up-convolutions produces smoother disparity maps and roughly a 15% relative EPE decrease on KITTI-2015.The authors interpret the added layers as improving disparity-map regularization.
- DispNet architecture: 1D correlation layers systematically improve the disparity networks, plausibly because one-dimensional matching permits correlation at a finer grid.
- SceneFlowNet results: SceneFlowNet predicts disparity change well, including in regions that become occluded, but training takes 0.28 seconds per forward pass and has not converged.This forward pass is five times longer than for DispNet, and the authors expect longer training to improve results.
7. Conclusion
The paper introduces a large synthetic dataset for training and evaluating disparity, optical-flow, and scene-flow networks, then demonstrates strong disparity performance and promising scene-flow results.
- The dataset contains over 35 000 stereo image pairs with ground-truth disparity, optical flow, and scene flow.
- The dataset supports evaluation as well as training, addressing the shortage of ground-truth datasets for scene flow.
- The trained disparity network is on par with the state of the art and runs 1000 times faster, while the initial scene-flow network shows promising results.
1. Introduction
The supplied passage concerns supplemental material rather than the introduction’s research motivation or contribution.
- The supplemental material expands the dataset-generation description and provides additional DispNet details and qualitative results.
2. Dataset creation details
The dataset pipeline modifies Blender to render stereo imagery alongside 3D position data from current, previous, and subsequent frames, enabling scene-flow data conversion and segmentation outputs.
- Blender’s render pipeline produces stereo RGB images plus three additional data passes for each frame and stereo view.
- The base pass stores each visible scene point’s true 3D position in the camera coordinate system.
- The second pass stores previous-frame 3D positions projected using the current frame’s geometry and camera projection.
- The third pass analogously uses the subsequent frame instead of the previous frame.
- The Blender branch was modified from version 2.75b, whose multiview rendering support enabled the stereo pipeline.
- The three data structures encode the scene’s visible 3D structure and motion; 3D positions are converted into scene flow, while images and masks require no post-processing.
- Object and material indices provide consistent unique oversegmentation across frames and are included for applications beyond the reported experiments.
- The supplemental video demonstrates the created datasets and the pipeline’s final optical-flow and disparity outputs.
3. DispNetCorr
DispNetCorr explicitly correlates image features along horizontal scanlines, using the stereo geometry to estimate disparity. Its qualitative evaluations compare sharper detail, occlusion handling, and effects of KITTI finetuning across network variants.
- Architecture: DispNetCorr processes the two stereo images separately before correlating their features horizontally and further processing the result.This contrasts with DispNet, which uses the stacked RGB images as one six-channel input.
- Correlation layer: The correlation map assigns one channel to each possible disparity and computes each entry as a scalar product between horizontally shifted feature vectors.For location (x, y) and disparity d, the compared features are a(x,y) and b(x−d,y).
- Qualitative evaluation: On Sintel examples, DispNetCorr1D produces sharper estimates, improves smooth-region estimates, and handles occluded regions more reasonably than DispNet.The comparison is qualitative and is presented in Figures 5 and 6.
- Qualitative evaluation: KITTI finetuning produces smoother predictions but removes performance on large disparities and causes failures in sky regions absent from KITTI ground truth.Finetuned networks can nevertheless retain small structures and disparity discontinuities, such as delineator posts.