PaperlayerView exact revision on arXiv

Source-linked AI summary

A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization

Tak-Wai Hui, Xiaoou Tang, Chen Change Loy

arXiv:1903.07414v3cs.CV

TL;DR

Optical flow CNNs seek accurate estimation without the large models and slow runtimes associated with FlowNet2. LiteFlowNet2 combines lightweight pyramidal feature processing, feature warping, cascaded inference, descriptor matching, and feature-driven regularization, outperforming FlowNet2 on Sintel and KITTI while using fewer parameters and running faster.

  • Problem

    FlowNet2 achieves accurate optical flow estimation but requires over 160M parameters and has a slow runtime.

  • Method

    LiteFlowNet2 uses a spatial-pyramid CNN with feature warping, cascaded flow inference, descriptor matching, and feature-driven local-convolution regularization.

  • Results

    LiteFlowNet2 outperforms FlowNet2 on Sintel and KITTI while being 25.3 times smaller in model size and 3.1 times faster in runtime.

  • Takeaways & Limitations

    LiteFlowNet2 provides lightweight, accurate, and fast flow computation suitable for real-time applications.

  • Takeaways & Limitations

    The f-warp layer's use for displacing channels differently when multiple flow fields are supplied is beyond the scope of this work.

Abstract

from arXiv · show

Over four decades, the majority addresses the problem of optical flow estimation using variational methods. With the advance of machine learning, some recent works have attempted to address the problem using convolutional neural network (CNN) and have showed promising results. FlowNet2, the state-of-the-art CNN, requires over 160M parameters to achieve accurate flow estimation. Our LiteFlowNet2 outperforms FlowNet2 on Sintel and KITTI benchmarks, while being 25.3 times smaller in the model size and 3.1 times faster in the running speed. LiteFlowNet2 is built on the foundation laid by conventional methods and resembles the corresponding roles as data fidelity and regularization in variational methods. We compute optical flow in a spatial-pyramid formulation as SPyNet but through a novel lightweight cascaded flow inference. It provides high flow estimation accuracy through early correction with seamless incorporation of descriptor matching. Flow regularization is used to ameliorate the issue of outliers and vague flow boundaries through feature-driven local convolutions. Our network also owns an effective structure for pyramidal feature extraction and embraces feature warping rather than image warping as practiced in FlowNet2 and SPyNet. Comparing to LiteFlowNet, LiteFlowNet2 improves the optical flow accuracy on Sintel Clean by 23.3%, Sintel Final by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%, while being 2.2 times faster. Our network protocol and trained models are made publicly available on https://github.com/twhui/LiteFlowNet2.

1 INTRODUCTION

The paper frames optical flow CNN design around classical data fidelity and regularization, seeking accurate estimation without FlowNet2’s large model and runtime costs. LiteFlowNet uses pyramidal features, feature warping, cascaded inference, and feature-driven regularization, while LiteFlowNet2 improves accuracy and speed.

  • Motivation: Optical flow is challenging because image correspondence is only partially observable, motivating coarse-to-fine estimation methods.Classical approaches iteratively refine flow from coarse to fine using image warping.
  • Motivation: FlowNet2 achieves accurate CNN-based flow estimation with over 160M parameters, whereas SPyNet is compact but less accurate.The paper targets the accuracy–efficiency gap between these approaches.
  • Design principles: LiteFlowNet maps image pairs into pyramidal feature descriptors and estimates flow through a coarse-to-fine decoder separated from feature extraction.This encoder–decoder separation is intended to improve efficiency and expose accuracy and model-size bottlenecks.
  • Design principles: Feature warping reduces feature-space distance directly, replacing image warping between inference stages and pyramid levels.The f-warp uses encoded features and a spatial transformer rather than regenerating features from warped images.
  • Design principles: Feature-driven local convolutions regularize flow using pyramidal features, flow estimates, and occlusion probabilities.The adaptive kernels make regularization both flow- and image-aware while addressing outliers and vague boundaries.
  • Design principles: Cascaded flow inference progressively corrects estimates and combines descriptor matching with later sub-pixel refinement.Short-range matching is enabled after feature warping reduces feature-space distance.
  • Results: LiteFlowNet2 outperforms FlowNet2 on Sintel and KITTI while being 25.3 times smaller and 3.1 times faster.The paper also reports up to 25 flow fields per second on Sintel-sized images using a NVIDIA GTX 1080 GPU.

2 RELATED WORK

Prior optical-flow research spans variational, machine-learning, and CNN-based methods. The paper positions LiteFlowNet among compact CNNs by combining feature warping, cost volumes, and cascaded residual-flow inference with learned regularization.

  • CNN-based methods: Variational methods historically dominated optical-flow estimation, while FlowNet and FlowNet2 established CNN-based alternatives.FlowNet2 uses a large cascade of FlowNet variants to improve flow adjustment.
  • CNN-based methods: FlowNet2 exceeds 160M parameters, while SPyNet uses 1.2M parameters but does not match FlowNet2’s accuracy.The contrast motivates lightweight architectures that preserve accuracy.
  • Concurrent work: LiteFlowNet differs from PWC-Net by using cascaded residual-flow inference that separates pixel-level matching from subsequent sub-pixel refinement.Both methods use coarse-to-fine inference, feature warping, and cost volumes, but their inference designs differ.
  • Feature matching: Dense pixel-wise matching can establish accurate correspondence but may require several seconds on a GPU, limiting practical use.The paper therefore emphasizes efficient sampled or short-range matching within a CNN pipeline.
  • Scope: The cited f-warp formulation is limited here to a single supplied flow field per feature map.Using different displacements for multiple supplied flow fields is explicitly outside the work’s scope.
  • Feature-based design: LiteFlowNet warps high-level CNN features rather than images and uses feature-driven convolution to regularize flow fields.This makes its warping mechanism more generic than image-only spatial transformation.

3 LITEFLOWNET

LiteFlowNet combines pyramidal feature extraction with cascaded flow inference and feature-driven regularization. It uses feature warping, cost-volume matching, sub-pixel refinement, and adaptive local convolutions to estimate and regularize flow efficiently.

  • Network Architecture: LiteFlowNet uses NetC for pyramidal feature extraction and NetE for coarse-to-fine optical-flow estimation.NetC produces multi-scale high-dimensional features, while NetE estimates flow from low to high spatial resolutions.
  • Feature Warping: Feature warping displaces second-image feature maps using the previous-level flow instead of warping images.This reduces feature-space distance before matching and supports residual-flow estimation.
  • Cascaded Flow Inference: At each pyramid level, cascaded inference first matches high-level features, then refines the resulting flow to sub-pixel accuracy.The two-step procedure uses descriptor matching followed by sub-pixel refinement, enabling early correction.
  • Cascaded Flow Inference: Multi-scale short-range matching, feature warping, and sparse cost volumes reduce the computational burden of cost-volume construction.The first two techniques reduce the search space, while sparse matching reduces matching frequency per pyramid level.
  • Flow Regularization: Feature-driven local convolution regularizes flow fields with position-specific filters adapted to features, flow variation, and occlusion probability.The regularizer is designed to smooth within flow regions while avoiding excessive smoothing across flow boundaries.

4 CORRESPONDENCES BETWEEN OPTICAL FLOW CNNS AND VARIATIONAL METHODS

The paper interprets optical-flow CNN components through the data-fidelity and regularization roles of variational methods. LiteFlowNet learns feature-based correspondence and adaptive flow smoothing rather than relying on explicitly engineered matching and smoothing terms.

  • Variational Formulation: Variational optical-flow estimation minimizes an energy combining data and regularization costs with a smoothness weight.The data term imposes correspondence constraints, while regularization enforces dependencies between neighboring flow vectors.
  • Data Fidelity: Classical data terms combine matching quantities such as brightness and gradient constancy, whose contributions require appropriate weighting.Maintaining differentiability is also necessary for solving the energy with the Euler–Lagrange equation.
  • Data Fidelity: LiteFlowNet trains pyramidal feature encoders and cascaded flow inference to minimize differences between high-level features across images.The learned feature correspondence plays the role of data fidelity without explicitly defining the classical matching quantities.
  • Regularization: Classical regularization can be viewed as convolution with an oriented Gaussian kernel applied to an intermediate flow field.This smoothing enforces spatial dependency among neighboring flow vectors.
  • Regularization: LiteFlowNet replaces engineered smoothing kernels with feature-driven local convolutions that adapt filters to features, intermediate flow, and occlusion probability.The resulting module regularizes each flow vector differently and corresponds to the variational regularization role.

5 RELATIONSHIP BETWEEN OPTICAL FLOW CNNS AND BASIS REPRESENTATION

The paper relates optical-flow CNN inference to conventional basis-field representations: learned filters act like flow bases, while feature maps provide corresponding coefficients. This analogy also applies to residual-flow estimation in LiteFlowNet2.

  • Parameterized image-motion models approximate flow within an image patch as a linear combination of basis vectors.The basis vectors lie in R2hw for an h × w image patch.
  • A vectorized flow field packs all x- and y-components into one vector, with coefficients ai estimated for each basis vector.The coefficients are the parameters estimated in the basis representation.
  • CNN convolution layers use filters, feature maps, biases, and activation functions to produce feature representations for flow inference.Wi denotes filters, Fi feature maps, bi trainable biases, and σ the nonlinear activation.
  • For residual flow, the vectorized convolution output Wi ∗ Fi + bi resembles the basis-vector term ai mi in conventional representation.The residual flow is estimated rather than the full flow in the comparison.
  • CNN feature-map channels correspond to basis-vector counts, while filters and feature maps correspond respectively to conventional bases and flow coefficients.Prior convolution layers compute the feature maps that play the coefficient role.
  • CNN filters are spatially small and feature maps scale with image dimensions, whereas basis fields match image-patch dimensions and coefficients are scalars.The comparison highlights dimensional differences between CNN inference and conventional basis representations.

6 EXPERIMENTS

Experiments show that LiteFlowNet2 improves accuracy and efficiency over prior lightweight and large CNN optical-flow methods across Sintel and KITTI, with gains shaped by architecture and training choices.

  • Training: Stage-wise training reduces training time and promotes lower overall-network training losses through initialization from previously trained stages.The later-added stages receive fewer iterations, while earlier stages provide their initialized filter weights.
  • Architecture: A pseudo flow-inference and regularization network reduces level-2 runtime from 52.51ms to 8.95ms and improves KITTI testing accuracy without significant Sintel testing improvement.The simplified network removes descriptor matching at level 2 and is more beneficial for challenging real-world variability than synthetic Sintel data.
  • Sintel: LiteFlowNet2 outperforms LiteFlowNet and PWC-Net on Sintel, with fine-tuned LiteFlowNet2 also outperforming FlowNet2-ft-sintel and EpicFlow on Sintel Final.LiteFlowNet2 retains sharper flow boundaries and fewer artifacts through flow regularization.
  • KITTI: LiteFlowNet2 outperforms FlowNet2, LiteFlowNet, and PWC-Net on KITTI, while fine-tuned LiteFlowNet2 exceeds LiteFlowNet, PWC-Net-ft, and PWC-Net+.KITTI training is more challenging because its datasets contain fewer than 400 image pairs and sparse flow labels.
  • Efficiency: 25.3× fewer parameters and 3.1× faster runtime than FlowNet2, while LiteFlowNet2 is 2.2× faster than LiteFlowNet.Its processing frequency reaches up to 25 flow fields per second, similar to PWC-Net+.

7 CONCLUSION

LiteFlowNet2 is a lightweight CNN for optical flow that incorporates data fidelity and regularization principles from variational methods. It combines accurate, fast computation with substantially reduced model size and supports real-time applications.

  • LiteFlowNet2 adopts data fidelity and regularization from variational methods in a lightweight CNN for optical flow estimation.
  • Its design uses pyramidal feature extraction, feature warping, multi-scale cascaded flow inference, and flow regularization.
  • 2.2 times faster and more accurate than LiteFlowNet, LiteFlowNet2 improves on its predecessor through its evolved architecture and training approach.
  • 25.3 times smaller and 3.1 times faster than FlowNet2, LiteFlowNet2 outperforms it on Sintel and KITTI benchmarks.
  • LiteFlowNet2 outperforms PWC-Net+ on KITTI 2012 and 2015 and matches it on Sintel Clean and Final while using 1.4 times less model size.

APPENDIX

LiteFlowNet2 uses two compact sub-networks, NetC and NetE, with NetC processing image pairs through shared filters and NetE supporting specialized modules.

  • LiteFlowNet2 consists of two compact sub-networks named NetC and NetE.
  • NetC uses two streams that share the same set of filters.
  • The input to NetC is an image pair, I1 and I2.
  • NetC and NetE architectures are specified across a six-level pyramid, with NetE detailed at pyramid level 5.
  • NetE marks layers used for descriptor matching, sub-pixel refinement, and flow regularization with the suffixes M, S, and R.
Loading 1903.07414v3…