Source-linked AI summary
LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
Tak-Wai Hui, Xiaoou Tang, Chen Change Loy
TL;DR
Optical-flow estimation needs accurate predictions, but FlowNet2’s model exceeds 160M parameters while SPyNet’s smaller model does not match its accuracy [21]. LiteFlowNet combines pyramidal feature extraction and feature warping with cascaded inference and feature-driven flow regularization. It matches or outperforms FlowNet2 on public benchmarks while using 30 times fewer parameters and running 1.36 times faster.
Problem
FlowNet2 exceeds 160M parameters, while SPyNet’s smaller model matches FlowNet rather than FlowNet2, motivating accurate flow estimation with greater efficiency.
Source: §1 · p. 1Method
Results
LiteFlowNet is on par with or outperforms FlowNet2 on public benchmarks, with 30 times fewer parameters and 1.36 times faster running speed.
Source: §1 · p. 2Takeaways & Limitations
LiteFlowNet shows that accurate flow estimation need not require a large model, and its lightweight computation is expected to suit applications such as SLAM and 3D reconstruction.
Source: §5 · p. 9A small amount of Things3D training data with extremely large flow displacement was excluded.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
FlowNet2, the state-of-the-art convolutional neural network (CNN) for optical flow estimation, requires over 160M parameters to achieve accurate flow estimation. In this paper we present an alternative network that outperforms FlowNet2 on the challenging Sintel final pass and KITTI benchmarks, while being 30 times smaller in the model size and 1.36 times faster in the running speed. This is made possible by drilling down to architectural details that might have been missed in the current frameworks: (1) We present a more effective flow inference approach at each pyramid level through a lightweight cascaded network. It not only improves flow estimation accuracy through early correction, but also permits seamless incorporation of descriptor matching in our network. (2) We present a novel flow regularization layer to ameliorate the issue of outliers and vague flow boundaries by using a feature-driven local convolution. (3) Our network owns an effective structure for pyramidal feature extraction and embraces feature warping rather than image warping as practiced in FlowNet2. Our code and trained models are available at https://github.com/twhui/LiteFlowNet .
1. Introduction
LiteFlowNet targets accurate, efficient optical-flow estimation by combining pyramidal features, feature warping, cascaded inference, and flow regularization. It is reported to match or outperform FlowNet2 on public benchmarks with a substantially smaller, faster model.
1. Introduction
- The encoder builds multi-scale feature pyramids, and the decoder estimates flow coarse-to-fine using same-resolution features at each level.Separating feature extraction from flow estimation makes it easier to pinpoint accuracy and model-size bottlenecks.Source: §1 · pp. 1–2
- Feature warping replaces image warping with a single step that reduces feature-space distance between the two images.The features of the second image are warped directly after encoder computation.Source: §1 · p. 2
- At each pyramid level, two lightweight inference networks progressively correct flow, enabling descriptor matching and pixel-level flow followed by sub-pixel refinement.Matching only sampled positions and using a short displacement reduces the computational burden of explicit matching.Source: §1 · p. 2
- A feature-driven local convolution regularizes flow using encoder features, the flow estimate, and an occlusion-probability map to address vague boundaries and artifacts.Its adaptive kernels make regularization both flow- and image-aware.Source: §1 · p. 2
- 30 times fewer parameters and 1.36 times faster than FlowNet2, while matching or outperforming it on public benchmarks.LiteFlowNet also outperforms FlowNet and SPyNet.Source: §1 · p. 2
2. Related Work
The related work spans variational, learning-based, and CNN approaches to optical flow, as well as feature matching and warping. LiteFlowNet draws on variational data fidelity and regularization while applying CNN features and in-network flow processing.
2. Related Work
- Variational methods use data fidelity and regularization, and LiteFlowNet draws inspiration from both when estimating and regularizing flow.Prior examples include brightness and gradient constancy, descriptor-based matching, and complementary regularization.Source: §2 · p. 2
- FlowNet2 improves accuracy through a cascade exceeding 160M parameters, whereas LiteFlowNet uses 30 times fewer parameters with performance on par with it.The passage contrasts FlowNet2’s large cascade with a more efficient LiteFlowNet architecture.Source: §2 · p. 3
- LiteFlowNet performs end-to-end in-network flow regularization, while FlowNet’s post-processing energy minimization is not end-to-end trainable.The proposed feature-driven local convolution plays a role similar to variational regularization.Source: §2 · p. 3
- LiteFlowNet reduces feature-matching cost through short-range matching of warped CNN features at sampled positions, followed by sub-pixel refinement.The passage notes that pixel-wise matching can take several seconds even on a GPU.Source: §2 · p. 3
- Unlike image-limited warping in FlowNet2 and SPyNet, LiteFlowNet warps high-level CNN features using a flow field that is not parameterized.The paper describes this as a relaxed version of Spatial Transformer.Source: §2 · p. 3
3. LiteFlowNet
LiteFlowNet separates pyramidal feature extraction from coarse-to-fine flow estimation, combining feature warping with cascaded inference and adaptive flow regularization at each level.
3. LiteFlowNet
- NetC extracts multi-scale features while NetE estimates coarse-to-fine flow, warping second-image features toward the first to reduce feature-space distance.The two-stream feature extractor shares filter weights; direct feature warping lets the decoder infer residual flow rather than the complete flow field.Source: §3 · p. 3
3.1. Cascaded Flow Inference
- At each pyramid level, descriptor matching estimates flow before sub-pixel refinement corrects it, reducing the risk of pixel-level errors being amplified during upsampling.Short-range matching follows feature warping, and matching is performed only at sampled positions at high spatial resolutions, with sparse costs interpolated for unsampled positions.Source: §3.1 · p. 4, Figure 3 · p. 5
3.2. Flow Regularization
- Feature-driven local convolution regularizes the inferred flow with filters adapted to local flow patches, aiming to smooth consistent regions without over-smoothing boundaries.The adaptive distance metric uses pyramidal features, the inferred flow, and an occlusion probability map; each filter is applied to a local feature patch.Source: §3.2 · p. 5
4. Experiments
LiteFlowNet variants outperform or match leading optical-flow methods across several benchmarks while using substantially fewer parameters and running faster than FlowNet2. The experiments also examine how feature warping, descriptor matching, sub-pixel refinement, and regularization affect flow quality.
4. Experiments
- Experiments compare variants across FlyingChairs, Sintel, KITTI, and Middlebury, using Chairs-only and Chairs-to-Things3D training schedules.Training is stage-wise, adding matching, sub-pixel refinement, and regularization units across pyramid levels.Source: §4.1 · p. 6, §4 · p. 6
4.1. Results
- Chairs-trained LiteFlowNet-pre outperforms the compared networks, and LiteFlowNetX-pre remains competitive despite omitting matching and shrinking the model.LiteFlowNetX-pre is reported as 43 times smaller than FlowNetC and 1.33 times smaller than SPyNet, while outperforming those networks and matching FlowNetC with explicit matching.Source: §4.1 · p. 6
- On Sintel final testing, LiteFlowNet-ft outperforms FlowNet2-ft-sintel and EpicFlow, while KITTI fine-tuning outperforms FlowNet2-ft-kitti.LiteFlowNet-ft and FlowNet2-ft-sintel perform best among the compared Sintel methods; LiteFlowNet-ft-kitti outperforms FlowNet2-ft-kitti.Source: §4.1 · p. 6, Table 2 · p. 7
- On Middlebury, LiteFlowNet has performance comparable to conventional methods and outperforms the listed CNN baselines.The listed baselines are FlowNetS/C, FlowNet2-S/C, SPyNet, and FlowNet2.Source: §4.1 · p. 7
4.2. Runtime and Parameters
- 30 times fewer parameters and 1.36 times faster runtime than FlowNet2 are reported for LiteFlowNet.Runtime was averaged over 100 Sintel image-pair runs; Table 3 notes that Torch timings in parentheses are not directly comparable to Caffe timings.Source: §4.2 · p. 7, Table 3 · p. 7
4.3. Ablation Study
- Without feature warping, flow fields become vaguer and AEE degrades by 33% on KITTI12 and 25% on KITTI15.Feature warping brings pyramidal features closer, facilitating residual-flow estimation at subsequent levels.Source: §4.3 · p. 8
- Without descriptor matching, AEE degrades on evaluations spanning wide flow displacements, indicating its usefulness for large-displacement flow.The example field had a maximum displacement of only 14.7 pixels and showed no noticeable visual difference without matching.Source: §4.3 · p. 8
- Sub-pixel refinement improves detail and AEE, while regularization reduces artifacts and flow bleeding and maintains crisp boundaries.Table 4 summarizes AEE for Chairs-trained variants with components disabled.Source: Table 4 · p. 7, §4.3 · p. 8
5. Conclusion
LiteFlowNet is a compact optical-flow network that performs on par with or better than FlowNet2 on public benchmarks, with faster runtime and a much smaller model. Its design combines pyramidal feature extraction, feature warping, cascaded inference, and feature-driven flow regularization.
5. Conclusion
- LiteFlowNet is on par with or outperforms FlowNet2 on public benchmarks while running faster and using 30 times less model size.The conclusion also reports that LiteFlowNet outperforms FlowNet.Source: §5 · pp. 8–9
- The network uses pyramidal feature extraction, feature warping, cascaded inference with short-range matching and sub-pixel refinement, and feature-driven local convolution for regularization.The paper describes these elements as addressing large-displacement, detail-preserving flow and crisp flow boundaries.Source: §5 · p. 9
- The authors expect LiteFlowNet's lightweight, accurate, and fast flow computation to support applications including motion segmentation, SLAM, and 3D reconstruction.Action recognition is also listed as a possible application.Source: §5 · p. 9
6. Appendix
The appendix passage describes LiteFlowNet's two compact sub-networks and points to architecture tables for their designs.
6. Appendix
- LiteFlowNet comprises two compact sub-networks, NetC and NetE; NetC takes an image pair as input and has two network streams that share the same set of filters.Source: §6 · p. 9