PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Depth from Single Images with Deep Neural Network Embedding Focal Length

Lei He, Guanghui Wang, Zhanyi Hu

arXiv:1803.10039v1cs.CV

TL;DR

Single-image depth recovery is ambiguous because scene depth and focal length can produce indistinguishable projections. The paper proves and experimentally verifies this ambiguity, generates varying-focal-length training data, and embeds focal length in a depth network; embedded focal length significantly improves depth estimation across error measures.

  • Problem

    Monocular depth estimation lacks reliable recovery of true depth when camera focal length is unknown, while available datasets are generally fixed-focal-length.

  • Method

    The paper generates varying-focal-length datasets from fixed-focal-length data and trains a fine-grained network that fuses middle-level features while embedding focal length.

  • Results

    Depth inferred with embedded focal length significantly outperforms depth without focal length information across all error measures.

  • Takeaways & Limitations

    Focal length information is useful for extracting depth from a single image, and known focal length improves depth inference.

Abstract

from arXiv · show

Learning depth from a single image, as an important issue in scene understanding, has attracted a lot of attention in the past decade. The accuracy of the depth estimation has been improved from conditional Markov random fields, non-parametric methods, to deep convolutional neural networks most recently. However, there exist inherent ambiguities in recovering 3D from a single 2D image. In this paper, we first prove the ambiguity between the focal length and monocular depth learning, and verify the result using experiments, showing that the focal length has a great influence on accurate depth recovery. In order to learn monocular depth by embedding the focal length, we propose a method to generate synthetic varying-focal-length dataset from fixed-focal-length datasets, and a simple and effective method is implemented to fill the holes in the newly generated images. For the sake of accurate depth recovery, we propose a novel deep neural network to infer depth through effectively fusing the middle-level information on the fixed-focal-length dataset, which outperforms the state-of-the-art methods built on pre-trained VGG. Furthermore, the newly generated varying-focal-length dataset is taken as input to the proposed network in both learning and inference phases. Extensive experiments on the fixed- and varying-focal-length datasets demonstrate that the learned monocular depth with embedded focal length is significantly improved compared to that without embedding the focal length information.

I. INTRODUCTION

Monocular depth estimation is important for scene understanding but remains ambiguous because existing methods often overlook the camera’s focal length. The paper addresses this by generating varying-focal-length data and embedding focal length in a fine-grained depth network.

  • Single-image depth inference assigns a depth value to every pixel and supports semantic segmentation, pose estimation, and object detection.
  • Existing monocular depth methods improve accuracy with deep networks but commonly overlook ambiguity between scene depth and camera focal length.
  • The paper transforms fixed-focal-length datasets into varying-focal-length datasets and fills holes produced during image generation.
  • The proposed network fuses middle-level information and embeds focal length to learn fine-grained monocular depth.
  • Experiments on Make3D, NYU v2, and KITTI report better quantitative and qualitative performance without embedded focal length, with further gains after embedding it.
  • The contributions include proving focal-length ambiguity, synthesizing varying-focal-length images, and developing a middle-level feature-fusion network.

III. INHERENT AMBIGUITY

Monocular depth from a single image is intrinsically ambiguous because different focal-length and scene-depth configurations can produce the same image. The paper demonstrates this ambiguity experimentally and describes how depth inference uses receptive-field information and learned transformations.

  • Depth for each pixel is inferred by fusing global and contextual information from its receptive field, followed by affine transformations and nonlinear operations.The equation identifies the pixel depth, receptive field, activation function, and model parameters in this computation.
  • Deeper networks can improve depth estimation but may be difficult to train and may have actual receptive fields smaller than their theoretical receptive fields.The text attributes training difficulty to vanishing or exploding gradients and discusses receptive-field limitations.
  • Different focal lengths and scene depths can produce identical projected images, preventing recovery of true depth without the camera focal length.Under the pinhole model, a distant object viewed with a long focal length can match a nearby object viewed with a short focal length.
  • In a laboratory experiment using 250 image pairs captured at 50 mm and 105 mm, the inferred average depth difference was 0.07506 m versus an actual difference of 2 m.The image pairs had actual depth differences of at least 1 m, and the reported error rates were 89.76% for Liu et al. and 87.2% for Eigen et al.
  • The experiments demonstrate inherent ambiguity between focal length and scene depth in monocular depth estimation.The evaluation used depth predictions from two established methods on images captured at different focal lengths.

IV. DATASET TRANSFORMATION

The paper transforms fixed-focal-length RGB-D datasets into varying-focal-length datasets by geometrically reprojecting 3D points and resolving occlusions during image formation.

  • IV. DATASET TRANSFORMATION: Fixed-focal-length datasets are transformed by using camera intrinsics and corresponding RGB-D images to generate varying-focal-length data.The pipeline begins with the camera’s intrinsic parameters and RGB-D image, then transforms the scene before reprojection.
  • IV. DATASET TRANSFORMATION: Figure 4 compares RGB images and inferred depths at focal lengths of 105 mm and 50 mm for two existing methods.The rows show RGB inputs followed by depths inferred by Liu et al. and Eigen et al.
  • IV. DATASET TRANSFORMATION: The camera coordinate system is transformed to a new system through rotation and translation before projection with a specified new focal length.The transformation uses a rotation matrix and translation vector, followed by reprojection into new image coordinates.
  • IV. DATASET TRANSFORMATION: During reprojection, only the nearest 3D point is retained when multiple points map to one pixel, because farther points are occluded.The generated image is then quantized, with RGB values sampled from the corresponding original image.

B. Post-processing of the generated varying-focal-length datasets

The generated RGB-D images contain projection holes, which the paper fills using a binary-template interpolation procedure based on local 3 × 3 neighborhoods.

  • B. Post-processing of the generated varying-focal-length datasets: The reprojection process creates holes in generated RGB-D images, motivating a dedicated post-processing method.The paper describes the method as simple and effective for filling these holes.
  • B. Post-processing of the generated varying-focal-length datasets: The method locates empty pixels and applies 3 × 3 binary filters classified into three neighborhood-pattern classes.In the templates, 0 denotes a hole pixel and 1 denotes a pixel without a hole.
  • B. Post-processing of the generated varying-focal-length datasets: Class-a uses a 4-neighborhood template, class-b uses the corresponding 3 × 3 template, and class-c uses iterative interpolation.Class-c relies on intermediate interpolation results during a left-to-right, top-to-bottom scan.
  • B. Post-processing of the generated varying-focal-length datasets: The filtering process fills the projected holes in the generated RGB-D images.Examples of the resulting filled images are shown in Figure 6.

C. Implementation details

The implementation converts NYU and Make3D into varying-focal-length datasets using constrained camera transformations, geometric depth generation, and hole completion.

  • C. Implementation details: The transformation uses rotation angles within [−5°, 5°] and computes translation when the new image center coincides with the original.The implementation specifies separate rotations around the y and x axes.
  • C. Implementation details: The transformed depth maps are generated by geometric relationships, while contextual filtering completes the small hole regions introduced during quantization.The completed depth maps visually approach the ground truth at f = 580.
  • C. Implementation details: The transformed NYU and Make3D datasets use assigned focal lengths of 460, 500, 540, 620, 660, and 700 pixels around an initial value of 580.The generated image sets are illustrated for both datasets in Figure 8.
  • C. Implementation details: Visual verification indicates that the generated varying-focal-length datasets are geometrically reasonable.This assessment is reported for examples from both the Make3D and NYU datasets.

V. LEARNING MONOCULAR DEPTH WITH DEEP NEURAL NETWORK

The proposed network learns monocular depth with focal length information while fusing middle-level features to preserve fine-grained spatial structure. It combines a VGG-based encoder with global transfer, upsampling, and T-net modules.

  • V. LEARNING MONOCULAR DEPTH WITH DEEP NEURAL NETWORK: The model learns depth from single images by embedding focal length information using the newly generated varying-focal-length datasets.The focal length is incorporated during both learning and inference in the paper’s proposed approach.
  • V. LEARNING MONOCULAR DEPTH WITH DEEP NEURAL NETWORK: The network addresses resolution loss from convolution and pooling by combining upsampling with middle-level feature integration.Middle-layer fusion is motivated by the difficulty of restoring fine spatial structure from only the last convolutional layer.
  • V. LEARNING MONOCULAR DEPTH WITH DEEP NEURAL NETWORK: The architecture uses a pre-trained VGG-based first stage followed by global transfer and upsampling architectures to produce high-resolution depth.The network is trained end-to-end and transforms global category information into depth mapping.
  • V. LEARNING MONOCULAR DEPTH WITH DEEP NEURAL NETWORK: The VGG network is divided into five resolution-based blocks, while T-net modules propagate detailed structural information through shortcuts and feature concatenation.The T-nets reduce redundant channels, transform feature cues, and connect them to corresponding decoder features before unpooling and convolution.

B. Loss function

The network is trained with a loss defined between predicted and ground-truth depth, while the experiments compare datasets, metrics, and implementation settings across indoor and outdoor benchmarks.

  • Loss formulation: The standard MSE loss minimizes the squared Euclidean distance between predicted depth y and ground truth y∗.The loss is defined over valid pixels in the training batch.
  • Loss formulation: MSE can blur predictions by averaging plausible solutions, whereas L1 preserves better detail under uncertain high-frequency recovery.The paper therefore considers alternatives to the standard squared-error objective.
  • Loss formulation: The BerHu loss combines L2 and L1 behavior to address larger distant-depth errors while retaining detailed predictions.Large-error pixels receive stronger gradient propagation during training.
  • Experimental setup: Experiments cover NYU v2, Make3D, KITTI, synthetic varying-focal-length datasets, and SUNRGBD using adopted depth-error metrics.Make3D evaluation excludes ground-truth regions beyond 70m.
  • Experimental setup: NYU, Make3D, and KITTI use dataset-specific sampling, resizing, cropping, and augmentation procedures before training and evaluation.The datasets differ in scene type, resolution, and available depth measurements.

B. Analysis of the different architectures and loss functions

Ablations on NYU v2 compare architecture components and loss functions, showing benefits from BerHu loss and larger receptive fields through global-information fusion.

  • Experimental comparisons: The ablation compares T-net and skip connections, BerHu and L1 losses, and GIL-convolution and GIL-connected designs.These experiments isolate architectural and objective-function choices on NYU v2.
  • Loss functions: BerHu loss yields more accurate depth than L1 loss in the compared NYU v2 models.The comparison is part of an ablation over architectures and training losses.
  • Architectures: Depth performance improves as the receptive-field size increases from GIL-convolution to GIL-connected.The comparison evaluates alternative global information layers for bridging downsampling and upsampling paths.

C. Comparisons with the state-of-the-art

The proposed method is compared with reproduced VGG-based Laina and Eigen–Fergus baselines on NYU v2, Make3D, and KITTI, using reported depth-reconstruction results and qualitative visualizations.

  • Compared methods: The comparison uses VGG-Laina et al. and multi-scale Eigen–Fergus baselines, with reproduced variants denoted L. * [4] and E. and F. * [3].The baselines are evaluated alongside the proposed approach on three datasets.
  • Compared datasets: The proposed method is evaluated against these baselines on NYU v2, Make3D, and KITTI depth reconstruction tasks.The corresponding results are reported in dataset-specific comparison tables.
  • Qualitative evaluation: Qualitative comparisons additionally visualize predictions from the proposed method and competing approaches on the benchmark datasets.The figures present input images, ground truth, baseline predictions, and proposed predictions.

L. *-VGG [4]

Across Make3D, KITTI, and NYU v2 analyses, the proposed model is reported to improve depth quality and convergence while maintaining competitive execution time against VGG-based baselines.

  • NYU v2: The proposed method fuses middle-level information with T-nets, producing more detailed NYU v2 depth maps than Laina et al.The qualitative comparison is shown in Figure 10.
  • Make3D: VGG-ours and the modified Eigen–Fergus VGG model outperform other methods on Make3D, with stronger structural detail than Laina et al.Make3D errors are computed only where ground-truth depth is below 70m.
  • Varying focal length: Depth reconstruction errors are also compared between VFL-NYU and NYU test datasets.The comparison is summarized in Figure 14 and Table VII.
  • KITTI: The proposed approach yields lower error than the compared VGG baselines on KITTI and produces more fine-grained visual depth.The evaluation uses a fully convolutional replacement for the fully connected layer because of resolution differences.
  • Runtime: The method runs slightly slower than Laina et al. but faster than Eigen–Fergus, while recovering a 320×224 depth map in about 0.1 seconds.The runtime comparison covers NYU v2, Make3D, and KITTI resolutions.
  • Convergence: Training curves show lower error and faster convergence than the compared VGG baselines, especially on KITTI.The convergence analysis is presented for the benchmark training curves.

D. Evaluations of VFL dataset with focal length information

Embedding focal length consistently improves monocular depth estimation across varying-focal-length datasets and evaluation settings. The gains are reported across error measures, though they vary with dataset size and ground-truth characteristics.

  • VFL-NYU evaluation: Models with embedded focal length outperform corresponding models without it on VFL-NYU and NYU test data.For average relative error, several focal-length-aware models improve accuracy by about two percentage points on average.
  • VFL-Make3D evaluation: About two percentage points of average-relative-error improvement are reported for focal-length-aware models on VFL-Make3D and Make3D.All evaluated models with embedded focal length outperform their counterparts without focal-length information.
  • Evaluation conditions: VFL-NYU models are slightly weaker than models trained on the larger NYU dataset, while VFL-Make3D and Make3D have similar sample counts and smaller error differences.The authors attribute the VFL-NYU weakness mainly to fewer training samples and note that VFL-Make3D gains are less pronounced for root square error because of ground-truth range and training-set size.
  • SUNRGBD evaluation: On SUNRGBD, models with embedded focal length significantly outperform models without it in all error measures.The experiments reproduce competing methods for comparison on the same dataset.
  • Overall finding: Embedding focal length during both learning and inference boosts depth-inference accuracy across the evaluated varying-focal-length experiments.The reported conclusion concerns the benefit of including focal length in both phases of the network pipeline.

VII. CONCLUSION

The paper establishes focal-length ambiguity in monocular depth estimation and addresses it with varying-focal-length data and a fine-grained depth network. Experiments show that encoding focal length improves depth-learning accuracy.

  • Conclusion: The paper theoretically studies and experimentally verifies the ambiguity between scene depth and focal length in single-image estimation.The verification uses real images.
  • Conclusion: The authors generate varying-focal-length datasets from public fixed-focal-length datasets to address this ambiguity.The generated datasets support depth learning with focal length information.
  • Conclusion: A novel neural network infers fine-grained monocular depth from fixed- and varying-focal-length datasets.The model achieves competitive performance on public datasets without embedded focal length.
  • Conclusion: Embedding focal length yields significant performance increases in all error metrics relative to corresponding models without focal-length encoding.This result is reported for the proposed approach and state-of-the-art algorithms on varying-focal-length datasets.
Loading 1803.10039v1…