PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards real-time unsupervised monocular depth estimation on CPU

Matteo Poggi, Filippo Aleotti, Fabio Tosi, Stefano Mattoccia

arXiv:1806.11430v3cs.CVcs.RO

TL;DR

Real-time unsupervised monocular depth estimation is difficult on low-power CPUs because existing architectures are deep, complex, and resource-intensive. PyD-Net uses a compact pyramidal feature architecture trained with unsupervised image-reconstruction losses, achieving comparable accuracy with substantially lower complexity and enabling CPU and embedded deployment.

  • Problem

    Existing monocular depth-estimation architectures require substantial memory and run in real time mainly on power-hungry GPUs, limiting low-power CPU deployment.

  • Method

    PyD-Net is a compact CNN using multiple small decoders over a pyramid of features and unsupervised reconstruction-based training.

  • Results

    PyD-Net achieves comparable accuracy to the top-performing unsupervised method while using about 94% fewer parameters, with inference reaching about 2 Hz on a Raspberry Pi 3 and real-time performance on standard CPUs.

  • Takeaways & Limitations

    The method supports practical unsupervised monocular depth estimation on standard and embedded CPUs, including deployment on a Raspberry Pi 3.

Abstract

from arXiv · show

Unsupervised depth estimation from a single image is a very attractive technique with several implications in robotic, autonomous navigation, augmented reality and so on. This topic represents a very challenging task and the advent of deep learning enabled to tackle this problem with excellent results. However, these architectures are extremely deep and complex. Thus, real-time performance can be achieved only by leveraging power-hungry GPUs that do not allow to infer depth maps in application fields characterized by low-power constraints. To tackle this issue, in this paper we propose a novel architecture capable to quickly infer an accurate depth map on a CPU, even of an embedded system, using a pyramid of features extracted from a single input image. Similarly to state-of-the-art, we train our network in an unsupervised manner casting depth estimation as an image reconstruction problem. Extensive experimental results on the KITTI dataset show that compared to the top performing approach our network has similar accuracy but a much lower complexity (about 6% of parameters) enabling to infer a depth map for a KITTI image in about 1.7 s on the Raspberry Pi 3 and at more than 8 Hz on a standard CPU. Moreover, by trading accuracy for efficiency, our network allows to infer maps at about 2 Hz and 40 Hz respectively, still being more accurate than most state-of-the-art slower methods. To the best of our knowledge, it is the first method enabling such performance on CPUs paving the way for effective deployment of unsupervised monocular depth estimation even on embedded systems.

I. INTRODUCTION

Monocular depth estimation avoids multiple viewpoints and can be trained without depth labels, but existing CNNs are too complex for low-power real-time use. PyD-Net addresses this through a compact pyramidal design aimed at CPU and embedded deployment.

  • Monocular depth estimation infers depth from one image, avoiding multiple viewpoints during inference.
  • Unsupervised training replaces difficult depth labels with image-reconstruction signals from monocular sequences or stereo pairs.
  • PyD-Net reduces parameters by about 94% versus the top-performing comparison method, trading slightly lower depth accuracy for CPU speed.
  • PyD-Net infers depth at about 2 Hz on a Raspberry Pi 3 using less than 150 MB of its 1 GB memory.
  • The architecture is presented as enabling fast and accurate unsupervised monocular depth estimation on standard and embedded CPUs.

II. RELATED WORK

Related work progresses from supervised monocular depth estimation toward unsupervised reconstruction-based methods and more efficient pyramidal architectures. PyD-Net draws on these developments to reduce computational cost while retaining depth-estimation capability.

  • Supervised monocular depth estimation: Supervised monocular methods use labeled depth data and include CNN, semantic, nearest-neighbor, multi-scale, CRF, classification, and scene-prior approaches.
  • Unsupervised monocular estimation: Unsupervised methods train depth predictors through image reconstruction using multi-camera imagery, stereo pairs, or differentiable view synthesis.
  • Unsupervised monocular estimation: Godard et al. use bilinear sampling and left-right consistency while predicting depth from a reference image, establishing the cited state-of-the-art framework.
  • Pyramidal networks for optical flow estimation: Pyramidal optical-flow networks use small multi-resolution decoders over image or feature pyramids to reduce parameters and computation.

III. PROPOSED METHOD

The paper targets monocular depth models whose CPU execution is too slow for low-power deployment. It proposes a compact pyramidal CNN trained with unsupervised image reconstruction to improve efficiency while maintaining comparable accuracy.

  • Existing state-of-the-art models run in real time on high-end GPUs but require nearly a second on standard CPUs and more than 10 s on embedded CPUs.
  • The proposed compact CNN uses less than 150 MB at test time and reaches about 2 fps on Raspberry Pi 3 devices and tens of fps on standard CPUs.
  • PyD-Net adapts image-pyramid principles from optical flow through multiple small decoders operating at different resolutions.
  • Training casts depth prediction as image reconstruction and uses unlabeled stereo pairs, with the left frame processed to estimate inverse depth.

IV. PYD-NET ARCHITECTURE

PyD-Net is designed as a lightweight architecture that preserves state-of-the-art-comparable results while reducing parameters, memory footprint, and execution time. Its encoder builds a six-level feature pyramid for multi-resolution depth prediction.

  • PyD-Net is an architecture intended to achieve state-of-the-art-comparable results with fewer parameters, lower memory use, and shorter execution time.
  • The encoder contains 12 convolutional layers and progressively downsamples features to construct six pyramid levels.
  • The pyramid levels span resolutions from half the input image size down to the highest-level, lowest-resolution representation.

B. Depth decoders and upsampling

PyD-Net uses shallow depth decoders at multiple pyramid levels, progressively up-sampling processed features to refine depth estimates toward the highest resolution.

  • Depth decoders: At the highest pyramid level, a four-layer depth decoder processes extracted features before producing depth and forwarding features upward.The decoder has 96, 64, 32, and 8 feature maps across its layers.
  • Depth decoders: Each pyramid level infers depth at its own resolution using a shallow network.The architecture extracts a feature pyramid and estimates depth separately at each level.
  • Upsampling: A 2 × 2 deconvolution with stride 2 doubles spatial resolution before passing processed features to the next pyramid level.This progressively propagates information toward the highest-resolution estimate.
  • Upsampling: At test time, only a subset of decoders is required, reducing memory requirements and runtime for CPU deployment.The architecture is explicitly designed to support deployment on CPUs.

C. Training loss

PyD-Net is trained with a multi-scale loss combining reconstruction, edge-aware disparity smoothness, and left-right consistency terms across both stereo views.

  • Multi-scale loss: The multi-scale loss sums contributions computed at six pyramid scales during PyD-Net training.Each scale s ∈[1..6] contributes to the training objective.
  • Reconstruction: The reconstruction term measures differences between the original image and its depth-based warped reconstruction using SSIM and L1 difference.This term is computed for the left and right images.
  • Smoothness: The disparity smoothness term discourages depth discontinuities with an L1 penalty unless an image gradient occurs.Image gradients therefore modulate the smoothness constraint.
  • Consistency: The left-right consistency term enforces coherence between predicted left and right depth maps.The three loss contributions are also computed for right-image predictions.
  • Training versus testing: The right input and its prediction are used only during training, while testing operates as monocular depth estimation.Stereo information supplies training signals but is not required for inference.

V. IMPLEMENTATION DETAILS AND TRAINING PROTOCOL

The implementation uses a six-level pyramid with half-resolution maximum depth output, trains on KITTI under matched protocols, and omits stereo post-processing that would double computation.

  • Implementation: Six pyramid levels produce depth maps up to half the original input resolution, followed by bilinear interpolation to full resolution.Using full-resolution pyramid levels did not significantly improve accuracy but increased network complexity.
  • Training protocol: For fair comparison, PyD-Net is trained for 50 epochs on batches of 8 resized 512×256 images using 30 thousand KITTI raw images.A 200-epoch training schedule is also evaluated to measure the effect of longer training.
  • Evaluation scope: The evaluation excludes Godard et al.'s post-processing because forwarding the input image twice doubles processing time and memory.That procedure filters artifacts near depth discontinuities and image borders.

VI. EXPERIMENTAL RESULTS

On KITTI, PyD-Net is evaluated for both depth accuracy and hardware performance, including operation on low-powered CPUs while retaining satisfying results.

  • Evaluation: PyD-Net is evaluated against state-of-the-art methods on the KITTI dataset for accuracy and hardware performance.The hardware analysis covers different devices, including low-powered CPUs.
  • CPU performance: At about 2 Hz on a low-powered CPU, PyD-Net still produces satisfying results and is more accurate than most techniques in the literature.The reported evaluation emphasizes the trade-off between processing speed and depth accuracy.

A. Accuracy evaluation on Eigen split

On the KITTI Eigen split, PyD-Net substantially outperforms most compared methods while using far fewer parameters, and approaches Godard et al. in accuracy under longer training.

  • Evaluation protocol: 697 test images cover 29 KITTI scenes, while 22,600 frames from the remaining 32 scenes provide training data.Velodyne points reprojected onto the left image supply ground-truth labels.
  • KITTI-only training: PyD-Net with 50 or 200 training epochs significantly outperforms the considered methods on all metrics except δ < 0.1253.Liu et al. performs better on that metric than Godard et al..
  • KITTI-only training: After 200 epochs, PyD-Net achieves almost equivalent results to Godard et al., although it remains less accurate overall.The comparison uses supervised and unsupervised methods under the same evaluation protocol.
  • 50-meter evaluation: With predictions capped at 50 meters, PyD-Net uses 8+ times fewer parameters than Garg et al. and outperforms it in both training configurations.After 200 epochs, PyD-Net is close to or better than Godard et al. on the reported δ metrics.
  • Additional-data training: Additional CityScapes training benefits all networks; Godard et al. performs best overall, while 50-epoch PyD-Net still exceeds Zhou et al..PyD-Net trained for 200 epochs yields its best overall performance in this comparison.
  • Overall accuracy: PyD-Net outperforms more complex state-of-the-art methods and is comparable in most cases to the top-performing approach.The next evaluation examines the accuracy–runtime impact of this reduced complexity.

B. Runtime analysis on different architectures

Runtime experiments across GPU, desktop CPU, and embedded CPU show that PyD-Net trades a small accuracy loss for much faster inference, including practical Raspberry Pi deployment.

  • Hardware and resolutions: Table II compares PyD-Net and Godard et al. at Full, Half, Quarter, and Eight resolutions on a Titan X, i7-6700K, and Raspberry Pi 3.Full resolution is 256 × 512; reported times exclude Godard et al.'s post-processing.
  • Baseline runtime: Godard et al. falls below 2 Hz on the i7 CPU and requires more than 10 seconds per full-resolution map on Raspberry Pi 3.Its model is very fast on the high-end Titan X GPU but becomes unsuitable for embedded CPU deployment.
  • PyD-Net runtime: At Half resolution, PyD-Net takes 120 ms on the i7 and less than 2 s on Raspberry Pi 3, providing a 5× speed-up over.The speed-up comparison is at the same resolution.
  • PyD-Net runtime: At Quarter and Eight resolutions, PyD-Net runs at 20 and 40 Hz, delivering 11× and 18× speed-ups over, respectively.On Raspberry Pi 3, PyD-Net exceeds 1 Hz at the lower resolutions with 1.2 and 2.2 Hz.
  • Accuracy–runtime trade-off: At Eight resolution, PyD-Net has similar accuracy to while running at 40 fps on the i7 and about 2 fps on Raspberry Pi 3.Lower-resolution inference reduces accuracy for both methods, but the reduction is more gradual for PyD-Net.
  • Deployment implication: PyD-Net enables real-time processing on a standard CPU and practical deployment on embedded CPUs at a small accuracy cost.The evaluation includes a Raspberry Pi 3, USB camera, and standard smartphone power bank.

VII. CONCLUSIONS AND FUTURE WORK

PyD-Net is presented as an efficient unsupervised monocular depth architecture that maintains comparable accuracy while enabling CPU and embedded-system inference.

  • Conclusions: PyD-Net is trained unsupervised on rectified stereo pairs and achieves accuracy comparable to the state-of-the-art method.Its simplified configurations infer depth at about 2 Hz on Raspberry Pi 3 while exceeding most state-of-the-art methods in accuracy.
  • Future work: The authors identify mapping PyD-Net to embedded vision devices as future work for low-power applications such as UAVs and wearable systems.The proposed target includes the Intel Movidius NCS.
Loading 1806.11430v3…