Source-linked AI summary
FastDepth: Fast Monocular Depth Estimation on Embedded Systems
Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, Vivienne Sze
TL;DR
Embedded robotic platforms need depth sensing, but existing sensors are bulky and power-hungry, while accurate monocular depth networks are computationally complex. FastDepth combines an efficient encoder-decoder architecture, a low-latency decoder, pruning, and hardware-specific compilation, achieving order-of-magnitude faster inference with comparable accuracy and real-time performance on Jetson TX2.
Problem
Depth sensing is important for robotics, but existing sensors are bulky and power-hungry, while prior monocular depth-estimation networks have high computational complexity and underexplored decoder efficiency.
Method
FastDepth uses a MobileNet encoder, a low-latency decoder with nearest-neighbor interpolation and depthwise separable convolution, network pruning, and TVM compilation for embedded inference.
Results
178 fps on the Jetson TX2 GPU and 27 fps on its CPU, with active power consumption under 10 W, while maintaining comparable accuracy and achieving order-of-magnitude faster runtimes than prior work.
Takeaways & Limitations
FastDepth demonstrates real-time monocular depth estimation with high throughput and low latency on an embedded platform suitable for micro aerial vehicles.
Abstract
from arXiv · showhide
Depth sensing is a critical function for robotic tasks such as localization, mapping and obstacle detection. There has been a significant and growing interest in depth estimation from a single RGB image, due to the relatively low cost and size of monocular cameras. However, state-of-the-art single-view depth estimation algorithms are based on fairly complex deep neural networks that are too slow for real-time inference on an embedded platform, for instance, mounted on a micro aerial vehicle. In this paper, we address the problem of fast depth estimation on embedded systems. We propose an efficient and lightweight encoder-decoder network architecture and apply network pruning to further reduce computational complexity and latency. In particular, we focus on the design of a low-latency decoder. Our methodology demonstrates that it is possible to achieve similar accuracy as prior work on depth estimation, but at inference speeds that are an order of magnitude faster. Our proposed network, FastDepth, runs at 178 fps on an NVIDIA Jetson TX2 GPU and at 27 fps when using only the TX2 CPU, with active power consumption under 10 W. FastDepth achieves close to state-of-the-art accuracy on the NYU Depth v2 dataset. To the best of the authors' knowledge, this paper demonstrates real-time monocular depth estimation using a deep neural network with the lowest latency and highest throughput on an embedded platform that can be carried by a micro aerial vehicle.
I. INTRODUCTION
Monocular depth estimation offers a compact alternative to bulky depth sensors, but prior deep networks are too computationally demanding for embedded robotic systems. FastDepth addresses this gap with a low-latency encoder-decoder design, pruning, and embedded-platform compilation.
- Depth sensing supports mapping, localization, and obstacle avoidance, while monocular cameras offer lower cost, smaller size, and higher energy efficiency than typical depth sensors.
- Prior monocular depth methods emphasize accuracy, creating computation-intensive algorithms that struggle with embedded compute and latency constraints.
- Efficient auto-encoder design is particularly challenging because dense depth outputs preserve high dimensionality throughout decoding.
- FastDepth combines an efficient encoder-decoder architecture, a low-latency decoder, network pruning, and TVM compilation for embedded inference.
- The paper reports real-time monocular depth estimation with the lowest latency and highest throughput on an embedded platform carryable by a micro aerial vehicle.
II. RELATED WORK
Related work moved monocular depth estimation from hand-crafted and probabilistic methods toward deep neural networks, while increasingly prioritizing accuracy despite rising complexity and runtime costs.
- Early RGB depth estimation used hand-crafted features, probabilistic graphical models, and non-parametric retrieval from photometrically similar images.
- Modern methods train convolutional neural networks on large-scale datasets and add coarse-to-fine prediction, auxiliary tasks, or conditional random fields.
- Prior methods heavily pursue higher accuracy at increased complexity and runtime cost, with diminishing accuracy improvement.
- δ1 accuracy on NYU Depth V2 has saturated at around 82% in recent years.
B. Efficient Neural Networks
Efficient-network research has focused mainly on encoders that produce labels, leaving dense prediction architectures comparatively underexplored. FastDepth therefore targets decoder efficiency alongside pruning across the encoder-decoder network.
- MobileNet achieves similar accuracy to VGG-16 with 2.7 times fewer MACs and 32.9 times fewer weights for image classification.
- Efficient dense-prediction design is less explored than encoder optimization because decoders must reconstruct pixel-based outputs.
- Existing accurate depth-estimation designs often have decoders that dominate inference runtime.
- FastDepth uses depthwise separable convolution in the decoder so the decoder no longer dominates inference runtime.
- NetAdapt pruning is applied to both encoder and decoder networks to remove redundant parameters and computation.
1) Encoder Network:
FastDepth uses a fully convolutional encoder-decoder: an efficient MobileNet encoder extracts low-resolution features, while a lightweight decoder upsamples and refines them into a dense depth map.
- 1) Encoder Network:: High-complexity encoders such as VGG-16 and ResNet-50 provide strong accuracy but are unsuitable for real-time embedded applications.
- 1) Encoder Network:: Figure 2 reports intermediate feature-map dimensions as height × width × channels and depicts additive skip connections between encoder and decoder layers.
- 1) Encoder Network:: MobileNet is selected as the encoder because depthwise decomposition reduces convolutional complexity relative to standard convolution.
- 2) Decoder Network:: The decoder merges and upsamples encoder outputs to form a dense prediction, making its upsampling operation a key design choice.
- 2) Decoder Network:: NNConv5 uses five cascading upsample layers, nearest-neighbor interpolation after convolution, channel reduction, and depthwise decomposition for a slim decoder.
3) Skip Connections:
The approach combines an efficient encoder-decoder design, hardware-specific compilation, and posttraining pruning to reduce inference cost while evaluating accuracy on NYU Depth v2.
- 3) Skip Connections:: Feature maps are combined by addition rather than concatenation to avoid increasing the channels processed by decoder layers.
- 3) Skip Connections:: Depthwise decomposition is used in both encoder and decoder, with TVM compiling the design for embedded deployment.The paper motivates compilation because common frameworks do not fully optimize depthwise separable convolutions.
- 3) Skip Connections:: NetAdapt iteratively removes redundant feature-map channels from a trained network while selecting proposals with the best accuracy-complexity tradeoff.
- 3) Skip Connections:: The experiments compare encoder and decoder options using accuracy and latency metrics, including the effects of compilation and pruning.
B. Final Results and Comparison With Prior Work
FastDepth reduces inference runtime through decoder simplification, pruning, and target-specific compilation while retaining close-to-state-of-the-art accuracy. On the Jetson TX2, the approach achieves a 65 times reduction relative to the baseline and reaches 178 fps, with low-power operation also reported.
- B. Final Results and Comparison With Prior Work: 178 fps is achieved after pruning and compiling the network for the Jetson TX2, reducing runtime in both encoder and decoder.The baseline runtime is largely dominated by its decoder; decoder simplification is the first major runtime reduction.
- B. Final Results and Comparison With Prior Work: 65 times speedup relative to ResNet-50 with UpProj is achieved while maintaining accuracy.Figure 3 presents encoder-decoder runtime breakdowns, total runtimes, and δ1 accuracies for individual steps.
- B. Final Results and Comparison With Prior Work: An order-of-magnitude faster runtime on a Jetson TX2 is reported with accuracy close to state of the art.Table I compares δ1, RMSE, and runtime against prior work; MACs and runtimes were generated from re-implemented models.
- B. Final Results and Comparison With Prior Work: Under 10 W active power consumption is measured in TX2 max-N mode.
- B. Final Results and Comparison With Prior Work: Under 5 W active power consumption in max-Q mode still supports close-to-real-time CPU and above-real-time GPU speeds.
- B. Final Results and Comparison With Prior Work: Skip connections improve depth-map sharpness and visual clarity, while pruning preserves and can enhance clarity.The largest errors occur at boundaries and distant objects.
C. Ablation Study: Encoder Design Space
The encoder study compares ResNet-50, ResNet-18, and MobileNet for embedded depth estimation, selecting MobileNet for its lower GPU latency and accuracy–latency trade-off despite a CPU runtime limitation.
- Encoder selection: MobileNet is selected as the encoder for the final architecture.
- CPU limitation: MobileNet is an order of magnitude slower on the TX2 CPU than ResNet-18 despite its lower complexity.The passage attributes this to unoptimized depthwise-layer implementations in deep-learning frameworks.
- Evaluation setup: An encoder–decoder design space is evaluated using fixed-decoder depth metrics alongside isolated encoder metrics and TX2 runtimes.RMSE and δ1 apply to the complete encoder–decoder networks, while other metrics describe encoders alone.
D. Ablation Study: Decoder Design Space
The decoder design study examines upsampling, depthwise decomposition, and skip connections as the main dimensions of efficient decoder construction.
- Decoder design dimensions: Decoder efficiency is studied through upsample operation, depthwise decomposition, and skip connections.
- Decoder design dimensions: Efficient decoder design is especially challenging because dense depth outputs retain substantial information and remain high dimensional through decoding.
- Decoder design dimensions: The study evaluates decoder alternatives while targeting lower complexity and latency for dense depth estimation outputs.
1) Upsample Operation:
Among four upsampling designs, NNConv5 provides the preferred accuracy–runtime balance, while decoder simplification through depthwise layers substantially reduces GPU runtime but can reduce accuracy.
- Upsample Operation: NNConv5 achieves higher δ1 accuracy and lower RMSE than UpConv and DeConv5 with slightly lower GPU runtime.UpProj has the highest δ1 accuracy but is the slowest; UpConv remains too slow for real-time processing, while DeConv5 can introduce checkerboard artifacts.
- Depthwise Separable Convolution: The selected MobileNet–NNConv5 network is decoder-dominated in runtime, motivating replacement of decoder convolutions with depthwise separable convolutions.
- Depthwise Separable Convolution: Depthwise decomposition lowers GPU inference runtime by almost half.The reduction is accompanied by a slight accuracy loss attributed to depthwise layers.
- Upsample Operation: Nearest-neighbor interpolation is chosen because it is simpler and more consistently implemented across frameworks and compilers than bilinear interpolation.
3) Skip Connections:
Additive skip connections restore much of the accuracy lost from depthwise decoder layers without materially increasing runtime, while pruning further reduces computation and runtime.
- Skip Connections: Additive skip connections leave decoder channel counts unchanged and provide almost the same accuracy boost as concatenative skips with negligible runtime impact.Concatenative skips increase decoder channels, computational complexity, and CPU and GPU runtimes.
- Skip Connections: The final network uses additive skip connections between encoding and decoding layers.
- Network Pruning: Before pruning, the MobileNet–NNConv5 architecture exceeds real-time throughput on the TX2 GPU but remains below real-time speed on the TX2 CPU.Pruning targets a CPU framerate above 25 fps.
- Network Pruning: Pruning reduces MACs by 2 times and GPU runtime by 1.5 times, with almost the same accuracy, while preserving depth-map sharpness and visual clarity.The passage also reports a 1.8 times GPU-runtime reduction; its wording is retained as stated.
- Network Pruning: The pruned architecture contains bottlenecks in encoder layer mobilenet.9 and decoder layer decoder.2.
V. CONCLUSION
FastDepth enables high-speed monocular depth estimation on embedded systems by combining an efficient architecture, pruning, and hardware-specific compilation. On the Jetson TX2, it achieves runtimes an order of magnitude faster than prior work while maintaining comparable accuracy, with possible extension to other dense prediction tasks.
- V. CONCLUSION: An efficient network architecture uses a low-complexity, low-latency decoder that does not dominate inference runtime alongside a small MobileNet encoder.The decoder uses nearest-neighbor interpolation with depthwise separable convolution.
- V. CONCLUSION: Network pruning further reduces model size, while hardware-specific compilation converts complexity reductions into lower runtime on the target embedded platform.The approach applies a state-of-the-art pruning algorithm and uses compilation tailored to the target hardware.
- V. CONCLUSION: An order of magnitude faster runtimes than prior work are achieved on the Jetson TX2 while maintaining comparable accuracy.
- V. CONCLUSION: The approach may also support real-time deep-learning performance for other dense prediction tasks, such as image segmentation.This extension is presented as the authors’ belief rather than a demonstrated result in this work.