PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox

arXiv:1612.01925v1cs.CV

TL;DR

FlowNet struggled with small motions and real-world optical flow, limiting its competitiveness with traditional methods. FlowNet 2.0 combines improved training schedules, stacked warping networks, and small-motion specialization to achieve over 50% lower estimation error while remaining near interactive speed.

  • Problem

    Original FlowNet struggled to estimate small motions and compete with traditional methods on real-world optical flow.

  • Method

    FlowNet 2.0 combines scheduled multi-dataset training, stacked networks with image warping, and a specialized small-motion network.

  • Results

    FlowNet 2.0 performs on par with state-of-the-art methods, while its network family spans 8 to 140fps.

  • Takeaways & Limitations

    The FlowNet 2.0 family offers state-of-the-art optical flow across a broad accuracy–runtime range for varied scenes and applications.

  • Takeaways & Limitations

    The generic network is intended for varied scenarios, while scenario-specific performance may require optional fine-tuning.

Abstract

from arXiv · show

The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

1. Introduction · 2. Related Work

FlowNet 2.0 advances end-to-end optical-flow learning through curriculum-style data scheduling, stacked warping networks, and specialization for small motions and real-world data. The resulting system improves over FlowNet, reaches state-of-the-art performance, and offers accuracy–runtime variants from 8fps to 140fps, while related work spans CNN architectures, patch matching, refinement, warping, and curriculum learning.

  • 1. Introduction: FlowNet introduced direct optical-flow prediction from image pairs with a simple convolutional CNN, but initially struggled against highly fine-tuned established methods.The paper frames FlowNet 2.0 as successive consolidation of the original learning-based approach.
  • 1. Introduction: FlowNet 2.0 outperforms FlowNet by a large margin, matches state-of-the-art methods on Sintel and KITTI, and estimates small and large displacements with high detail at interactive frame rates.The system also resolves small-displacement problems and noisy artifacts in estimated flow fields, improving real-world applications such as action recognition and motion segmentation.
  • 1. Introduction: A multiple-dataset learning schedule improves results significantly, whereas using Mayer et al.’s more sophisticated training data in isolation produces inferior results.The introduction identifies dataset scheduling as the first decisive modification.
  • 1. Introduction: Stacking networks with a warping operation improves results and yields variants spanning 8fps to 140fps for controlling the accuracy–computational-resource trade-off.Warping compensates for preliminary motion estimates in the second image before subsequent processing.
  • 1. Introduction: A specialized training dataset and network address small, subpixel motion and real-world data, while a fusion network combines this component with the stacked network for arbitrary displacements.The fusion network is trained to combine the small-motion and stacked-network outputs optimally.
  • 2. Related Work: Related CNN-based optical-flow work includes 3D convolutions, unsupervised objectives, rotationally invariant architectures, and pyramidal coarse-to-fine approaches.These studies follow FlowNet’s end-to-end formulation, which takes an image pair as input and outputs a flow field.
  • 2. Related Work: Patch-based CNN methods learn image matching or descriptors but require exhaustive patch matching, making them restrictively slow for most practical applications.The related work discusses Deep Matching formulations and Siamese architectures as examples of this alternative.
  • 2. Related Work: Per-pixel CNN predictions can be noisy or blurry, motivating variational postprocessing and learned reaction-diffusion-style refinement.The paper also situates its approach within curriculum learning, where models train on gradually increasing tasks to improve complex real-world optical-flow estimation.

3. Dataset Schedules

Training-data order strongly affects optical-flow quality: despite its greater realism, Things3D performs worse alone than Chairs, while training on Chairs first and fine-tuning on Things3D gives the best results. Dataset and schedule changes improve reported FlowNetS and FlowNetC results by approximately 25% and 30%, respectively.

  • Datasets: Chairs contains about 22k image pairs with planar motions, whereas Things3D contains 22k renderings with true 3D motion, lighting effects, and greater object-model variety.Chairs uses chairs superimposed on Flickr backgrounds; Things3D uses moving ShapeNet models against static 3D backgrounds.
  • Experimental setup: The experiments train FlowNetS and FlowNetC on Chairs, Things3D, or an equal mixture using multiple learning-rate schedules.FlowNetS is an encoder-decoder architecture, while FlowNetC explicitly correlates feature maps; Sshort uses 600k iterations, with longer and fine-tuning schedules also investigated.
  • Dataset schedules: Training-data order matters, with Chairs-first then Things3D fine-tuning consistently outperforming Things3D alone and mixed-dataset training.Although Things3D is more realistic, Chairs-first training is conjectured to help learn color matching before greater complexity is introduced.
  • Architecture comparison: FlowNetC outperforms FlowNetS when both architectures are trained under the same conditions.The FlowNetS result with Sshort matches the earlier report, while FlowNetC achieves much better results under the tested conditions.

4. Stacking Networks

Stacking FlowNet architectures enables iterative flow refinement, especially when intermediate warping is used. The strongest stacks are trained sequentially with different network weights, while smaller variants improve the speed–accuracy trade-off.

  • Architecture: Stacked networks receive the images and previous flow estimate, allowing later networks to refine the preceding prediction.The first network takes I1 and I2; subsequent networks additionally receive wi = (ui, vi)⊤.
  • Warping: Warping I2 with the previous flow lets the next network focus on the remaining increment and optionally use the warped image and error.The warped image is computed by bilinear interpolation as ˜I2,i(x, y) = I2(x+ui, y+vi).
  • Training strategy: Warping makes stacking consistently beneficial, while an intermediate loss helps end-to-end training and freezing the first network produces the best result.Without warping, stacking improves Chairs but decreases Sintel performance, indicating over-fitting.
  • Training strategy: Sequentially training stacked networks avoids over-fitting while preserving the benefit of flow refinement after warping.This approach trains the networks one after another rather than jointly updating the full stack.
  • Stack design: Different-weight networks outperform recurrent reuse of identical weights, so the stack uses networks with distinct parameters and can vary network type and size.Repeated application and fine-tuning of a recurrent network did not improve results.
  • Results: 140 frames per second is achieved by FlowNet2-s with nearly the original FlowNet’s accuracy.A width factor of 3/8 is described as a good speed–accuracy trade-off for faster networks.
  • Results: ∼30% improvement over FlowNet2-C and ∼50% improvement over the original FlowNet are achieved by FlowNet2-CSS on Sintel train clean.Two small networks can also outperform one large network: FlowNet2-ss has 11M weights versus FlowNet2-S with 38M, and FlowNet2-cs has 11M versus FlowNet2-C with 38M.

5. Small Displacements

FlowNet struggled to reliably estimate small motions despite their relative ease for traditional methods. FlowNet2 addresses this with small-displacement fine-tuning, architectural modification, and a fusion network that performs well across displacement scales.

  • Motivation: FlowNet cannot reliably estimate small motions, although traditional methods handle them more easily.This limitation became apparent in real-world applications despite FlowNet’s strong Sintel performance.
  • Small-displacement fine-tuning: FlowNet2-CSS-ft-sd fine-tunes the whole network stack on a mixture of Things3D and ChairsSDHom for smaller displacements.A non-linearity downweights errors from large displacements during training.
  • Small-displacement fine-tuning: The fine-tuned network improves small-displacement performance without sacrificing performance on large displacements.Subpixel motion remains noisy, motivating a modification of the FlowNetS architecture.
  • Fusion network: A fusion network combines FlowNet2-CSS-ft-sd and FlowNet2-SD using flows, flow magnitudes, and brightness-warping errors.It contracts resolution twice and expands to full resolution, producing crisp motion boundaries across displacement scales.

6. Experiments

FlowNet2 is evaluated against state-of-the-art optical-flow methods on public benchmarks and application tasks, achieving strong accuracy across datasets while retaining a wide runtime advantage. It also demonstrates robustness on real-world data and competitive performance for motion segmentation and action recognition.

  • Public benchmarks: On Sintel, FlowNet2 consistently outperforms DeepFlow and EpicFlow, matches FlowFields, and reaches parity with Deep-DiscreteFlow on final data after fine-tuning.Fine-tuning slightly worsens clean-data performance but brings final-data performance to the level of the published state of the art.
  • Public benchmarks: On KITTI, FlowNet2-CSS is comparable to EpicFlow and FlowFields, while KITTI fine-tuning reduces error roughly threefold and achieves leading nonstereo results.Fine-tuning on combined KITTI2012 and KITTI2015 training data produces the error reduction; small-displacement fine-tuning instead degrades performance.
  • Public benchmarks: On Middlebury training data, FlowNet2 performs comparably to traditional methods, but test-set results remain substantially worse despite a large improvement over FlowNetS.The passage distinguishes the unexpectedly poor test-set performance from the stronger training-set comparison.
  • Efficiency: 8 to 140 frames per second are available across FlowNet2 variants, which outperform the best and fastest existing methods by large margins in Sintel runtime-error evaluations.The runtime range supports choosing a variant according to the application.
  • Real-world data: On real-world data, FlowNet2 is robust to homogeneous regions and image or compression artifacts while producing smooth flow fields with sharp motion boundaries.Its Sintel performance is similar to FlowFields, whereas the cited advantages are observed on real-world examples.
  • Application tasks: FlowNet2 is as reliable as other state-of-the-art methods for motion segmentation while being orders of magnitude faster, and its flow yields comparable action-recognition results.The original FlowNet was not useful for motion segmentation, and FlowNetS did not provide useful action-recognition results.

7. Conclusions · Supplementary Material for "FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks"

FlowNet 2.0 combines several improvements that achieve state-of-the-art optical-flow accuracy at orders-of-magnitude higher speed than FlowNet. Its estimated flow is reliable across diverse scenes and applications, including motion segmentation and action recognition.

  • 7. Conclusions: FlowNet 2.0 achieves accuracy fully on par with state-of-the-art methods while running orders of magnitude faster than FlowNet.The conclusion attributes both improvements to the combined FlowNet enhancements.
  • 7. Conclusions: Each proposed contribution was quantified experimentally, and all contributions were shown to play an important role.The authors explicitly evaluated the effect of every contribution.
  • 7. Conclusions: Experiments on motion segmentation demonstrate that FlowNet 2.0’s estimated optical flow is reliable across a large variety of scenes.Motion segmentation is one of the downstream applications used to assess reliability.
  • 7. Conclusions: Experiments on action recognition demonstrate that FlowNet 2.0’s estimated optical flow is reliable across a large variety of applications.Action recognition provides a second application-level reliability test.
  • 7. Conclusions: The reported reliability of FlowNet 2.0 extends beyond optical-flow evaluation to downstream scene and application experiments.The conclusion connects motion segmentation and action recognition results with reliability across varied settings.
  • 7. Conclusions: The conclusion presents FlowNet 2.0 as a family of improved networks built from multiple FlowNet advances.The passage introduces the FlowNet 2.0 family after summarizing its improvements, quantified contributions, speed, accuracy, and application reliability.

1. Video · 2. Dataset Schedules: KITTI2015 Results

The supplementary video presents FlowNet2 across diverse sequences, compares it with state-of-the-art methods, and illustrates speed–accuracy trade-offs. KITTI2015 results show that the Chairs→Things3D schedule performs best among the evaluated training schedules.

  • 1. Video: The supplementary video shows FlowNet2 results on diverse video sequences.It also includes comparisons with state-of-the-art methods and illustrates the FlowNet 2.0 family’s speed–accuracy trade-off.
  • 1. Video: The video compares FlowNet2 with state-of-the-art optical-flow methods.This comparison is presented alongside results on diverse video sequences.
  • 1. Video: The video illustrates the speed–accuracy trade-off across the FlowNet 2.0 model family.The supplementary material presents this trade-off as part of the FlowNet2 results.
  • 1. Video: Optical-flow visualizations encode displacement direction with hue and magnitude with color intensity.White denotes no motion, and flow fields are independently scaled for each displayed image pair because motion ranges differ across sequences.
  • 2. Dataset Schedules: KITTI2015 Results: Training with Slong + Sfine on the combined Chairs and Things3D datasets performs best on KITTI2015 and Sintel.The paper refers to this training procedure as the Chairs→Things3D schedule.

3. Recurrently Stacking Networks with the Same Weights

The bootstrap network predicts optical flow from scratch, whereas subsequent networks refine prior estimates and can therefore be recursively reapplied. The authors test recursive application of Net2 and continue training the full stack with multiple Net2 networks.

  • 3. Recurrently Stacking Networks with the Same Weights: The bootstrap network predicts a flow field from scratch without a previous flow estimate or warped image.Its task and inputs differ from those of the succeeding networks.
  • 3. Recurrently Stacking Networks with the Same Weights: Subsequent networks refine the previous flow estimate and can be applied recursively to their own output.This recursive use follows from their refinement-only task.
  • 3. Recurrently Stacking Networks with the Same Weights: The authors applied the best network from Table 2, Net2, recursively multiple times and then continued training the entire stack with multiple Net2 networks.The supplied passage identifies this as the experiment setup.

4. Small Displacements

FlowNet2 addresses small-displacement flow by introducing the ChairsSDHom dataset and fine-tuning a specialized network without sacrificing performance on large displacements. Its training emphasizes small motions, while a fusion network combines estimates and photometric consistency cues at full resolution.

  • Dataset: ChairsSDHom uses randomly placed and moved rendered 3D chairs over randomized backgrounds to model small-displacement real-world data.The dataset is rendered in raw and antialiased versions to obtain blend-free flow boundaries and antialiased images.
  • Training: FlowNet2-CSS-ft-sd fine-tunes FlowNet2-CSS on ChairsSDHom for smaller displacements while avoiding performance loss on large displacements.The section describes this fine-tuned network as the small-displacement variant.
  • Training: Mini-batches use 2 Things3D samples and 6 ChairsSDHom samples, while applying x0.4 to endpoint error emphasizes small-magnitude flows.The reported configuration was the best-performing one among the tested configurations.
  • Architecture: The small-displacement network concatenates both RGB images into 6 input channels and uses smaller initial strides and kernels than FlowNetS.It also adds convolutions between the upconvolutions.
  • Architecture: The fusion network merges two pretrained flow estimates using the first image, flow magnitudes, and squared Euclidean photoconsistency errors across 11 input channels.All inputs are provided at full image resolution, with previous flow estimates upsampled by nearest-neighbor interpolation.

5. Evaluation

FlowNet2 models provide a strong speed–accuracy trade-off across Sintel and KITTI2012, while stacked refinement improves flow estimates, especially for small displacements. The method also performs strongly on motion segmentation and KITTI2015 qualitative and quantitative evaluation.

  • Iterative refinement: Stacked networks progressively refine flow fields, with especially dramatic improvements for small displacements.FlowNet2-C’s initial prediction is noisy and is significantly refined by two succeeding networks.
  • Runtime and accuracy: FlowNet2 family models offer an excellent speed/accuracy trade-off on Sintel and KITTI2012.Figure 4 compares runtime with endpoint error across optical-flow methods; FlowNet2 outperforms other methods by a large margin.
  • Motion segmentation: FlowNet2 matches or exceeds the best optical-flow methods on all motion-segmentation metrics and is significantly better on VI.FlowNetS fails to fully converge on the training set, producing trajectories too short for test-set evaluation.
  • KITTI2015: FlowNet2-kitti significantly outperforms competing approaches quantitatively and qualitatively on KITTI2015.The model was not trained on the KITTI2015 images during fine-tuning, and sparse ground truth was bilinearly interpolated for visualization.

6. Warping Layer

The warping layer supports stacked networks by defining bilinear interpolation over images or feature maps, applying flow-guided warping in the forward pass, and differentiating the operation during backpropagation. It sets out-of-bounds warped pixels to zero and handles integer-coordinate nondifferentiability with directional derivatives.

  • Definitions: The warping layer applies to images or arbitrarily many-channel feature maps, using flow fields defined over image coordinates.Each feature-map channel is denoted I_c(x).
  • Definitions: Bilinear interpolation computes a continuous version of the input image before warping.The continuous image ˜I is obtained using the usual bilinear interpolation procedure.
  • Forward pass: The forward pass follows each flow vector and returns ˜I(x + w(x)) for in-bounds coordinates, otherwise assigning zero.This defines the warped image while explicitly handling flow vectors that leave the image domain.
  • Backward pass: The backward pass computes derivatives of the warped image with respect to input values and flow components at integer image locations.Derivatives are specified for I_c(x′) and the horizontal flow component u(x), with the derivative for v(x) analogous.
  • Backward pass: At integer coordinates, ceiling and floor functions are nondifferentiable, so directional derivatives are used.The treatment applies to these points in the derivative computation; the vertical-flow derivative is analogous to the horizontal case.
Loading 1612.01925v1…