PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Faranak Shamsafar, Samuel Woerz, Rafia Rahim, Andreas Zell

arXiv:2108.09770v1cs.CV

TL;DR

Deep stereo networks can be too costly for moderate GPUs and resource-limited devices. MobileStereoNet proposes lightweight 2D and 3D end-to-end models using MobileNet blocks and a learnable cost volume, achieving competitive accuracy with substantially lower parameter and operation counts. The results support deployment-oriented accuracy-complexity trade-offs for edge devices.

  • Problem

    Accurate end-to-end stereo networks can require too much computation and memory for moderate GPUs and resource-limited embedded or mobile devices.

  • Method

    The paper develops 2D and 3D end-to-end stereo networks using MobileNet blocks, extends them to 3D, and introduces a learnable cost volume for the 2D model.

  • Results

    27%/95% fewer parameters/operations than GCNet are reported for 2D-MobileStereoNet, while 3D-MobileStereoNet achieves 72%/38% fewer parameters/operations than GwcNet-g with competitive or superior performance.

  • Takeaways & Limitations

    MobileStereoNets offer a favorable accuracy-complexity trade-off for deploying end-to-end stereo networks on edge devices.

Abstract

from arXiv · show

Recent methods in stereo matching have continuously improved the accuracy using deep models. This gain, however, is attained with a high increase in computation cost, such that the network may not fit even on a moderate GPU. This issue raises problems when the model needs to be deployed on resource-limited devices. For this, we propose two light models for stereo vision with reduced complexity and without sacrificing accuracy. Depending on the dimension of cost volume, we design a 2D and a 3D model with encoder-decoders built from 2D and 3D convolutions, respectively. To this end, we leverage 2D MobileNet blocks and extend them to 3D for stereo vision application. Besides, a new cost volume is proposed to boost the accuracy of the 2D model, making it performing close to 3D networks. Experiments show that the proposed 2D/3D networks effectively reduce the computational expense (27%/95% and 72%/38% fewer parameters/operations in 2D and 3D models, respectively) while upholding the accuracy. Our code is available at https://github.com/cogsys-tuebingen/mobilestereonet.

1. Introduction

Stereo matching estimates scene depth from binocular images, but accurate end-to-end networks can be too computationally and memory intensive for embedded deployment. MobileStereoNet addresses this with lightweight 2D and 3D architectures designed to remain competitive in accuracy.

  • Motivation: Stereo matching estimates depth from disparity between corresponding points in rectified binocular images.The technique supports applications including self-driving cars, robotics, and object detection.
  • Motivation: End-to-end stereo networks use 2D or 3D convolutions according to whether their cost volume is 3D or 4D.The 3D group generally provides higher disparity-estimation accuracy but incurs greater computational cost.
  • Motivation: 3D networks can exceed moderate-GPU memory limits and fail to fit or execute efficiently on memory-constrained embedded platforms.This resource constraint motivates lighter stereo architectures for devices such as mobile systems and FPGAs.
  • Proposed approach: The paper develops end-to-end 2D and 3D stereo networks using MobileNet-V1 and MobileNetV2 blocks for embedded platforms.The models are organized according to 3D or 4D cost-volume dimensions.
  • Proposed approach: A new learning-based cost-volume construction module is introduced for the 3D cost-volume pathway to support the 2D model’s accuracy.The construction interlaces features from two viewpoints, while the proposed architectures are reported as competitive with state-of-the-art counterparts.
  • Contributions: The main contributions include lightweight 2D/3D models, 3D extensions of MobileNet blocks, a learnable cost volume, and experiments analyzing accuracy-complexity trade-offs.The reported design findings are intended to apply to similar 2D/3D networks.

2. Related work

Prior stereo research increasingly used end-to-end deep networks, especially 3D convolutions, but their memory and computational demands limited mobile and real-time deployment. Related lightweight work reduced cost through pruning, simpler matching costs, and efficient 3D operations, while cost-volume design remained central to accuracy.

  • Deep stereo methods: Deep stereo methods include partial pipeline replacements and end-to-end formulations, with most end-to-end approaches relying on 3D convolutional layers.The related-work landscape spans methods that learn individual components and methods that learn the complete process.
  • Deep stereo methods: 3D-based architectures improve accuracy but require substantial memory, limiting their practicality for mobile and real-time applications.The cited applications include robotics and autonomous vehicles.
  • Lighter networks: Lightweight stereo methods reduce complexity through selective disparity evaluation, traditional local-descriptor costs, or alternatives to full convolutional feature extraction.DeepPruner uses PatchMatch, while another approach aggregates a cost volume built from traditional local descriptors.
  • Lighter networks: Other 3D-vision methods reduce convolutional cost using pseudo-convolutions or 3D depth-wise convolutions.These approaches decouple or simplify 3D processing for tasks such as video classification and 3D reconstruction.
  • Cost volume computation: Stereo cost volumes measure left/right feature similarity using operations such as absolute difference, Hamming distance, correlation, or feature concatenation.Cost-volume construction is treated as a central stereo-model component alongside feature extraction and regularization.

3. Methodology

The method builds lightweight 2D and 3D stereo networks by extending MobileNet blocks to cost-volume processing. A learned interlacing cost volume enables the 2D model to use 2D convolutions while retaining disparity information.

  • Light blocks replacing 2D/3D convolutions: The blocks replace standard convolutions with depth-wise and point-wise operations to reduce computational cost.The cost is measured using parameters and MACs; MACs depend on input size.
  • Light blocks replacing 2D/3D convolutions: MobileNet-V1 and MobileNet-V2 blocks are extended from 2D to 3D for processing stereo cost volumes.The 3D formulation operates on C × D × H × W data, where D is the disparity dimension, and uses corresponding 3D kernels.
  • Light blocks replacing 2D/3D convolutions: The reduction factor depends on kernel size, input and output channels, and the MobileNet-V2 expansion factor.The supplied computation-cost comparison evaluates standard, v1, and v2 blocks in both 2D and 3D settings.
  • Light blocks replacing 2D/3D convolutions: Increasing the MobileNet-V2 expansion factor decreases the reduction factor, with 2D blocks becoming heavier than standard convolutions beyond t = 5.The analysis uses Cin = Cout = {32, 64, 128} and k = 3.
  • Proposed models: The 2D baseline reduces extracted features to 32 channels, constructs a 3D cost volume, and processes it with a 2D encoder-decoder.The 3D baseline instead uses a 4D group-wise-correlation cost volume and 3D convolutions.
  • Proposed models: The interlacing cost-volume module interleaves left and traversed-right features, applies learned 3D kernels, and aggregates them across disparity levels.It is parameterized to learn feature aggregation while retaining an encoder-decoder composed only of 2D convolutions.

4. Experimental results and discussion

Experiments evaluate the proposed cost volume and MobileNet blocks, then compare the resulting 2D- and 3D-MobileStereoNets on SceneFlow and KITTI 2015. The models reduce computational and memory demands while retaining competitive stereo accuracy.

  • Cost-volume design: Interlacing cost volumes produce lower SceneFlow error rates than the evaluated alternatives.The study compares interlacing with correlation and direct concatenation of left/right unary features.
  • MobileNet-block analysis: Feature extraction accounts for much of the computational load in both 2D and 3D baselines.Replacing feature extraction with v1 blocks and the hourglass with v2 provides a favorable accuracy-complexity compromise.
  • MobileNet-block analysis: Replacing additional modules with MobileNet blocks can reduce both complexity and error in the 2D baseline, but some replacements deteriorate learning capability.The channel-reduction module and interlacing-cost-volume convolutions therefore remain standard convolutions.
  • KITTI 2015: On KITTI 2015 validation, 2D-MobileStereoNet matches PSMNet with 2.3x fewer parameters and 8x fewer operations, while 3D-MobileStereoNet outperforms PSMNet and GA-Net-11.Against GwcNet-g, the 3D model uses 3.6x fewer parameters and 1.6x fewer operations.
  • KITTI 2015: On the KITTI 2015 benchmark, 2D-MobileStereoNet uses 27%/95% fewer parameters/operations than GCNet, while 3D-MobileStereoNet uses 72%/38% fewer than GwcNet-g.The 3D model produces crisper edges, whereas the 2D model has visually similar outputs; both proposed methods also have smaller memory sizes.

5. Conclusion

The paper presents lightweight 2D and 3D stereo networks that use MobileNet blocks to reduce cost and a learned cost volume to improve the 2D model. The resulting models offer a favorable accuracy-complexity trade-off for edge-device deployment.

  • Conclusion: MobileStereoNets reduce parameters, operations, and model size using MobileNet blocks while preserving a favorable accuracy-complexity trade-off.The 2D model additionally uses a learned cost volume to improve accuracy.

(Appendix)

The appendix contains supplementary architectural, qualitative, implementation, and complexity details.

  • Appendix: The appendix covers baseline architectures, additional qualitative results, light blocks in other modules, implementation details, and complexity analysis.These sections provide supporting details for the main experiments.

A. Detailed baseline architectures

The 2D and 3D baseline architectures share feature extraction but differ in convolution dimensionality, cost-volume construction, and channel reduction.

  • Baseline architecture: The 2D and 3D baselines share the feature-extraction step and similar hourglass structures with intraconnections.Their architectural diagrams report each layer or module's output size.
  • Baseline architecture: The 2D baseline uses 2D convolutions throughout its hourglass, whereas the 3D baseline uses 3D convolutions there.The two baselines also differ in cost-volume construction and channel reduction.

B. More qualitative results

The paper compares qualitative disparity estimates on SceneFlow and KITTI 2015, including both proposed MobileStereoNet variants and established models on KITTI.

  • SceneFlow visualizations show disparity and error maps for 2D-MobileStereoNet and 3D-MobileStereoNet across test samples.Warmer error-map colors indicate higher errors.
  • KITTI 2015 visualizations compare the proposed models with PSMNet, GA-Net-11, GA-Net-deep, and GwcNet-g.The comparison uses models trained with a 160/40 split of the KITTI 2015 training data.
  • 2D-MobileStereoNet shows close performance to 3D models with the least number of operations.
  • 3D-MobileStereoNet achieves competitive or better accuracy with the least number of parameters among the compared methods.

C. Incorporating light blocks in other modules

The study replaces additional feature-extraction and pre-hourglass convolutions with MobileNet-V2 blocks, balancing EPE and computational cost differently in the 2D and 3D networks.

  • Replacing the two modules with MobileNet-V2 yields the least EPE in 2D-MobileStereoNet.The selected 2D baseline uses FE2D:v1 and HG2D:v2 with t = 2.
  • In 3D-MobileStereoNet, the same MobileNet-V2 combination produces slightly higher EPE but substantially reduces computation cost.The selected 3D baseline uses FE2D:v1 and HG3D:v2 with t = 2.
  • MobileNet-V1 was excluded for these modules because it deteriorated performance despite reducing computational cost more.

D. Implementation details

Experiments use PyTorch, four NVIDIA GeForce GTX 1080 Ti GPUs, Adam optimization, and staged learning-rate schedules on SceneFlow before KITTI fine-tuning.

  • Experiments were implemented in PyTorch and trained on 4 × NVIDIA GeForce GTX 1080 Ti GPUs.
  • SceneFlow training lasted 20 epochs with Adam, initial learning rate 0.001, and halving after epochs 10, 12, 14, and 16.The optimizer used β1 = 0.9 and β2 = 0.999.
  • The best model was selected by least EPE, then fine-tuned on the KITTI 2015 validation experiments.

E. Analyzing the complexity

MobileStereoNets reduce computation and parameter counts in feature extraction and encoder-decoder modules, with 3D encoder-decoders remaining the main bottleneck.

  • Feature extraction falls from 52.07 to 7.84 GigaMACs and from 7.84 to 0.39 million parameters.
  • Lighter 2D and 3D networks reduce encoder-decoder operations and parameters relative to standard-convolution baselines.The comparison covers feature extraction and encoder-decoder modules using MACs and parameter counts.
  • The encoder-decoder with 3D convolutions is the major computational bottleneck for 3D models.
Loading 2108.09770v1…