PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Optics for Monocular Depth Estimation and 3D Object Detection

Julie Chang, Gordon Wetzstein

arXiv:1904.08601v1cs.CVeess.IV

TL;DR

Monocular depth estimation and 3D object detection need depth information that single all-in-focus images do not directly provide, while specialized sensors can be costly and monocular models generalize poorly. The paper jointly designs depth-encoding optics and a CNN decoder, finding that optimized lenses improve depth estimation and 3D object detection, with chromatic aberrations providing a simpler near-best alternative. A physical prototype validates improved real-world depth estimation, although its field of view is limited.

  • Problem

    Single-image depth estimation is ambiguous and often generalizes poorly beyond training datasets, while specialized depth sensors can be costly or bulky for practical deployment.

  • Method

    The paper trains a differentiable optical image-formation model and CNN jointly, allowing backpropagation to optimize both network weights and physical lens parameters.

  • Results

    Optimized freeform lenses produce the most accurate depth estimation, chromatic aberrations perform nearly as well, and the depth-optimized lens also improves 3D object detection.

  • Takeaways & Limitations

    A simple camera with a single lens can encode useful depth information while preserving higher-level image understanding, including improved 3D object detection.

  • Takeaways & Limitations

    The physical prototype has a smaller field of view caused by camera vignetting and spatially varying real point-spread functions, preventing full indoor-room capture.

Abstract

from arXiv · show

Depth estimation and 3D object detection are critical for scene understanding but remain challenging to perform with a single image due to the loss of 3D information during image capture. Recent models using deep neural networks have improved monocular depth estimation performance, but there is still difficulty in predicting absolute depth and generalizing outside a standard dataset. Here we introduce the paradigm of deep optics, i.e. end-to-end design of optics and image processing, to the monocular depth estimation problem, using coded defocus blur as an additional depth cue to be decoded by a neural network. We evaluate several optical coding strategies along with an end-to-end optimization scheme for depth estimation on three datasets, including NYU Depth v2 and KITTI. We find an optimized freeform lens design yields the best results, but chromatic aberration from a singlet lens offers significantly improved performance as well. We build a physical prototype and validate that chromatic aberrations improve depth estimation on real-world results. In addition, we train object detection networks on the KITTI dataset and show that the lens optimized for depth estimation also results in improved 3D object detection performance.

1. Introduction

The paper applies deep optics—joint optical and image-processing design—to encode depth cues for monocular depth estimation and 3D object detection. Across experiments, optimized optics improve depth estimation and transfer benefits to higher-level 3D vision.

  • Motivation: Specialized depth sensors can be costly or difficult to deploy because they require custom hardware, illumination, electronics, or bulky form factors.
  • Motivation: Monocular methods based on pictorial cues and learned CNN features often generalize poorly beyond consistent training datasets, making absolute depth estimation ill-posed.
  • Approach: Deep optics jointly trains an optical encoder and electronic CNN decoder, optimizing both network weights and physical lens parameters through backpropagation.
  • Results: Optimized freeform lenses achieve the best depth-estimation results, while chromatic aberrations perform nearly as well and outperform standard fixed-lens alternatives.
  • Results: Images captured through optimized lenses improve 3D object detection over all-in-focus monocular images on KITTI.
  • Results: A simple U-Net with deep optics attains lower root-mean-square depth errors than more complex networks processing all-in-focus images.

2. Related Work

Prior work improves depth estimation by modifying camera optics or jointly learning optics and neural processing. This paper connects optical coding strategies, including chromatic aberration and optimized point-spread functions, with deep-learning reconstruction.

  • Deep Monocular Depth Estimation: Monocular depth networks can succeed on consistent datasets such as KITTI and NYU Depth but remain highly dependent on the training dataset.
  • Computational Photography for Depth Estimation: Computational photography encodes depth through amplitude or phase aperture coding, chromatic aberrations, and defocus blur.
  • Computational Photography for Depth Estimation: Chromatic aberrations make defocus blur depth cues unambiguous because their point-spread functions differ for distances on opposite sides of the focal plane.
  • Deep Optics: Deep optics jointly trains camera optics and CNN-based estimators for tasks including extended depth of field, superresolution, classification, and microscopy.
  • Deep Optics: The PSF simulation compares thin-lens defocus, chromatic aberrations, and an optimized mask initialized with astigmatism.

3. Differentiable Image Formation Model

The model differentiates wave-based image formation to generate depth- and wavelength-dependent PSFs, then uses them to synthesize sensor images for neural depth estimation. It supports conventional and freeform lens designs, layered depth compositing, and end-to-end optical optimization.

  • 3. Differentiable Image Formation Model: Wave-optics light transport models depth- and wavelength-dependent point spread functions while supporting refractive and diffractive elements.The simulated PSFs are used to represent the camera’s optical response at different depths and wavelengths.
  • 3.1. Modeling Conventional Cameras: A thin convex lens relates object depth and sensor distance through the thin-lens equation, with objects at the focus distance producing the sharpest PSF.PSFs spread as scene depth moves away from the focal plane.
  • 3.1. Modeling Conventional Cameras: For a point emitter, the model propagates the complex electric field through lens phase modulation and free-space transfer before converting it into an intensity PSF.The propagation uses an aperture function, Fourier-domain transfer function, and magnitude-squared intensity measurement.
  • 3.1. Modeling Conventional Cameras: The lens model includes finite aperture masking and Fourier-domain propagation, with the sensor intensity obtained by taking the propagated field’s magnitude-squared.This sequence produces a 2D PSF for each depth and wavelength of interest.
  • 3.2. Modeling Freeform Lenses: A freeform lens adds an arbitrary spatial thickness profile to the conventional lens model, increasing the degrees of freedom available for shaping depth-dependent PSFs.The freeform profile is parameterized with Zernike basis functions to encourage smoother surfaces.
  • 3.3. Depth-Dependent Image Formation: The simulated RGB image is formed by convolving each depth layer with its wavelength-specific PSF and combining the results using occlusion masks.The scene is represented as discrete planar depth layers, and blurred masks are renormalized so their sum equals one at each pixel.

4. Depth Estimation

The paper evaluates differentiable optical coding and jointly optimized lens–network systems for monocular depth estimation across synthetic and real-world settings. Depth-dependent blur, astigmatism, chromatic aberration, and freeform lens optimization improve depth prediction over all-in-focus inputs, while a prototype validates chromatic cues on captured images.

  • 4.1. Network and Training: A differentiable image-formation model converts lens parameters, all-in-focus images, and binned depth maps into depth-dependent sensor images for U-Net estimation.The model generates depth-specific PSFs and applies depth-dependent convolution before neural-network prediction.
  • 4.1. Network and Training: The experiments cover the Rectangles, NYU Depth v2, and KITTI datasets, using jointly adjusted lens parameters and U-Net weights for optimized lenses.The study compares fixed optical models with freeform and annular designs across three datasets.
  • 4.2. Analysis and Evaluation: All-in-focus inputs have the highest errors, while defocus, astigmatism, and chromatic aberration progressively provide stronger depth cues.Astigmatism helps resolve symmetric defocus ambiguity, and singlet-lens chromatic aberration further improves results.
  • 4.2. Analysis and Evaluation: The optimized freeform lens achieves the best depth-estimation results, whereas optimized annular lenses do not surpass fixed-lens chromatic aberration.For NYU Depth, initializing the freeform optimization with astigmatism produced better results.
  • 4.3. Experimental Results: On NYU Depth v2, adding the optical encoder produces results on par with state-of-the-art methods using more heavyweight networks.The comparison uses the default metrics reported for the NYU Depth v2 test set.
  • 4.3. Experimental Results: A chromatic-aberration prototype improves captured-image depth estimation relative to the corresponding all-in-focus approach, but its smaller field of view prevents full indoor-room capture.The field-of-view limitation arises from camera vignetting and the spatially varying real PSF.

5. 3D Object Detection

The paper tests whether an optical system optimized for depth estimation also benefits 3D object detection on KITTI. The optimized system preserves similar 2D detection performance while improving 3D detection relative to all-in-focus monocular images.

  • 5. 3D Object Detection: 3D object detection combines object recognition with regression of oriented 3D bounding boxes, making depth information critical to the task.The evaluation uses KITTI and the same optical system optimized for depth estimation.
  • 5. 3D Object Detection: The authors train FPointNet for the freeform lens optimized on KITTI depth estimation, using RGB-derived frustum proposals followed by 3D segmentation and box estimation.FPointNet supports both sparse LiDAR point clouds and dense depth images.
  • 5. 3D Object Detection: 3D object detection improves with the optimized optical system, while 2D AP remains similar to the all-in-focus system.The result indicates that optimized-lens sensor images retain comparable information for 2D detection and provide improved depth maps for 3D detection.

6. Discussion

The joint optical-encoder, electronic-decoder model outperforms all-in-focus optics-agnostic imaging while preserving 2D object detection and improving 3D object detection. The authors frame these conclusions as relative comparisons, constrained by simulated sensor images and an approximate, discretized image-formation model.

  • The joint optical-encoder, electronic-decoder model outperforms the corresponding optics-agnostic model using all-in-focus images.
  • The depth-optimized lens maintains 2D object detection performance while improving 3D object detection from a single image.
  • The conclusions are primarily based on relative performance because sensor images use ground-truth or pseudo-truth depth simulations and an approximate, discretized, layer-based image-formation model.
  • The paper considers the camera only as a single static optical layer, leaving more complex optical-computing and computer-vision designs for future research.
Loading 1904.08601v1…