Source-linked AI summary
3D Packing for Self-Supervised Monocular Depth Estimation
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, Adrien Gaidon
TL;DR
Monocular depth estimation needs accurate scene structure without relying on additional depth sensors, while monocular geometry leaves metric scale ambiguous. The paper introduces PackNet, a self-supervised architecture trained on unlabeled videos, and reports strong benchmark performance, generalization, and real-time operation alongside the DDAD dataset.
Problem
Self-supervised monocular depth estimation avoids additional depth labels and calibration, but monocular structure-from-motion estimates depth and pose only up to an unknown scale factor.
Method
PackNet uses symmetrical, detail-preserving 3D packing and unpacking blocks for self-supervised monocular depth estimation, with optional velocity supervision for metrically accurate predictions.
Results
The approach outperforms existing self- and semi-supervised methods, competes with fully supervised methods, generalizes across datasets and unseen environments, and runs in real time without ImageNet pretraining.
Takeaways & Limitations
PackNet supports high-resolution, longer-range monocular depth estimation, including evaluation up to 200m on the DDAD dataset.
Takeaways & Limitations
Monocular structure-from-motion remains scale-ambiguous unless metric information such as velocity supervision is incorporated during training.
Abstract
from arXiv · showhide
Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset. Furthermore, it does not require large-scale supervised pretraining on ImageNet and can run in real-time. Finally, we release DDAD (Dense Depth for Automated Driving), a new urban driving dataset with more challenging and accurate depth evaluation, thanks to longer-range and denser ground-truth depth generated from high-density LiDARs mounted on a fleet of self-driving cars operating world-wide.
1. Introduction
The paper targets self-supervised monocular estimation of scene structure and camera motion without direct depth labels, introducing PackNet and DDAD to improve high-resolution, long-range depth evaluation.
- Self-supervised monocular depth learning uses geometric constraints from image sequences instead of ground-truth depth and cross-calibrated additional sensors.
- PackNet introduces packing and unpacking blocks with 3D convolutions to preserve dense appearance and geometric information while running in real time.
- DDAD uses diverse logs from calibrated self-driving cars equipped with cameras and high-accuracy long-range LiDARs for more accurate depth evaluation at range.
- Experiments report improvements over the state of the art, competitiveness with fully supervised methods, stronger generalization, and real-time high-resolution inference.
2. Related Work
Prior work progressed from supervised and stereo-based depth estimation toward self-supervised monocular depth and pose learning from unlabeled videos.
- Early convolutional depth networks regressed depth from single RGB images using multi-scale architectures and RGB-D supervision.
- Related dense prediction architectures explored pooling, learnable pooling, and sub-pixel convolutions to improve information flow and resolution.
- Stereo-based methods removed the need for ground-truth depth by synthesizing one view from the other using predicted depth and differentiable geometric losses.
- Monocular self-supervised methods jointly learn depth and camera pose from unlabeled videos using geometric constraints.
3. Self-Supervised Scale-Aware SfM
The method jointly learns scale-ambiguous monocular depth and ego-motion from image sequences using photometric and edge-aware depth losses, with optional velocity supervision for metric scale.
- Self-Supervised SfM: The model jointly predicts scale-ambiguous depth for each target-image pixel and 6-DoF ego-motion between the target and temporal source images.The practical source context uses the preceding and following frames.
- Self-Supervised SfM: The self-supervised objective combines appearance matching between synthesized and target images with edge-aware depth regularization.The objective masks invalid mappings and averages the loss across pixels, pyramid scales, and image batches.
- Appearance Matching: Photometric supervision combines SSIM with an L1 pixelwise term to compare target and synthesized images.The photometric loss is evaluated using the synthesized target image produced from the temporal context.
- Appearance Matching: Minimum photometric error across source images and an auto-mask reduce effects from occlusion, out-of-bounds mappings, static pixels, and dynamic objects without relative motion.The minimum-loss association is selected per pixel, while pixels whose warped loss exceeds the unwarped loss are removed.
- Scale Awareness: Monocular SfM cannot determine metric depth and pose scale from photometric supervision alone, so prior approaches use LiDAR measurements to scale predictions at test time.The paper introduces velocity supervision as an alternative route to metric scale.
- Scale Awareness: Velocity supervision compares predicted pose-translation magnitude with measured instantaneous velocity, enabling metrically accurate pose and depth estimates.The velocity term is weighted by λ2 in the final scale-aware objective.
4. PackNet: 3D Packing for Depth Estimation
PackNet replaces conventional downsampling and upsampling with symmetrical 3D packing and unpacking blocks designed to preserve spatial detail for high-resolution depth decoding.
- Architecture: PackNet addresses detail loss from aggressive striding, pooling, and traditional decoder upsampling with a new encoder-decoder architecture using 3D packing and unpacking blocks.The design targets fine-grained representations needed for accurate depth predictions.
- Packing Block: The packing block folds spatial dimensions into feature channels, then uses learned 3D feature processing and 2D contraction to compress the representation.Space2Depth reduces resolution while preserving information through an invertible rearrangement before learned compression.
- Unpacking Block: The unpacking block symmetrically decompresses packed channels into higher-resolution spatial features during decoding.It replaces nearest-neighbor or transposed-convolution upsampling and reverses the encoder’s 3D packing process.
- Detail Preservation: A packing-unpacking encoder-decoder nearly reconstructs input images, reaching a final L1 loss of 0.0079 while retaining sharp edges and fine details.The comparison uses one packing layer, one symmetrical unpacking layer, D = 2, K = 3, and C = 4.
- Network Design: The PackNet architecture combines packing and unpacking blocks with skip connections and four intermediate inverse-depth outputs used during training.The decoder upsamples intermediate inverse-depth maps before combining them with corresponding skip and unpacked feature maps.
5. Experiments
Experiments evaluate PackNet across KITTI, NuScenes, and DDAD, showing strong long-range accuracy, improved generalization, scalability, and competitive performance without supervised ImageNet pretraining.
- Depth Estimation Performance: PackNet-SfM significantly outperforms the state of the art on DDAD, with the performance gap increasing at longer distances up to 200m.DDAD provides dense LiDAR-based ground truth for cumulative and independently binned depth evaluation.
- Depth Estimation Performance: On KITTI, PackNet establishes a new state of the art in the self-supervised monocular setting through its architecture rather than modified training objectives.The evaluation uses both original and accumulated KITTI depth maps and includes qualitative comparisons.
- Depth Estimation Performance: Adding unlabeled CityScapes videos and increasing input resolution further improves performance, reaching results comparable to semi-supervised and fully supervised methods.The authors attribute the resolution benefit to preserving and processing spatial information end-to-end.
- Scale-Aware Depth Estimation Performance: PackNet produces scale-aware depth with similar performance to unscaled models while removing the need for ground-truth depth scaling at test time.Weak velocity supervision constrains pose translation magnitude during training and also supports metrically accurate trajectories from the pose output.
- Ablation Studies: PackNet achieves state-of-the-art results from randomly initialized weights and outperforms standard architectures across considered metrics without large-scale supervised ImageNet pretraining.Self-supervised pretraining on additional datasets can further improve the architecture.
6. Conclusion
PackNet is a self-supervised monocular depth architecture using symmetrical packing and unpacking blocks to preserve high-resolution visual information. It performs competitively across datasets and supervision settings while supporting real-time, scale-aware depth estimation.
- PackNet uses symmetrical, detail-preserving packing and unpacking blocks to compress and decompress high-resolution visual information for fine-grained predictions.
- The approach is trained solely on unlabeled monocular videos yet outperforms existing self- and semi-supervised methods and competes with fully supervised methods.
- PackNet generalizes better to different datasets and unseen environments, especially at longer depth ranges evaluated up to 200m on DDAD.
- The model runs in real time and does not require ImageNet pretraining to achieve state-of-the-art results.
- Weak velocity supervision makes the model scale-aware, producing metrically accurate depth maps from a single image.
A. Pose evaluation
PackNet-SfM is evaluated for monocular camera-pose estimation on KITTI odometry sequences 09 and 10 using training on sequences 00–08. With velocity supervision, it recovers metric scale without ground-truth test-time scaling.
- PackNet-SfM is trained from scratch on KITTI odometry sequences 00–08 using the same parameters and networks as the depth evaluation.
- Absolute Trajectory Error is averaged over overlapping 5-frame snippets on test sequences 09 and 10 for comparison with related methods.
- PackNet-SfM achieves competitive pose results relative to other related monocular methods on the KITTI odometry benchmark.
- With velocity supervision, PackNet-SfM does not require ground-truth scaling at test time because it recovers metrically accurate scale from monocular input.
B. Dense Depth for Automated Driving (DDAD)
DDAD is a diverse, calibrated urban-driving benchmark built from dense, long-range LiDAR measurements and synchronized camera data. Its denser ground truth and longer ranges make depth evaluation more challenging, with PackNet outperforming related methods under these conditions.
- DDAD uses high-resolution, long-range Luminar-H21 LiDAR with a maximum range of 250m and sub-1cm range precision.
- The dataset provides six calibrated cameras synchronized at 10 Hz, producing 360° vehicle coverage, although this work uses only the front-facing camera.
- Projected Luminar-H2 point clouds provide denser ground truth and longer-range evaluation, and PackNet outperforms related methods under these conditions.
- The performance gap increases when evaluation uses denser ground truth at longer ranges, across both the full interval and discretized distance bins.
- Table 6 reports average Absolute Trajectory Error in meters for methods trained on sequences 00–08 and evaluated on sequences 09–10, with M+v denoting velocity supervision.