PaperlayerView exact revision on arXiv

Source-linked AI summary

SharpNet: Fast and Accurate Recovery of Occluding Contours in Monocular Depth Estimation

Michaël Ramamonjisoa, Vincent Lepetit

arXiv:1905.08598v3cs.CV

TL;DR

Monocular depth estimation is ill-posed, and occluding contours remain difficult to reconstruct despite their importance for recognition and augmented reality. SharpNet trains with depth–contour and depth–normal constraints; on NYUv2, it outperforms prior methods on contour localization while placing second on standard depth metrics.

  • Problem

    Occluding contours are important for object recognition and augmented reality but remain difficult to reconstruct, and their depth annotations can be inaccurate.

    Source: §1 · p. 1

  • Method

    SharpNet jointly predicts depth, normals, and occluding contours during training, using loss constraints between depth and contours and between depth and normals.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    SharpNet outperforms previous methods on occluding-contour 2D localization and places second on all standard NYUv2-Depth metrics.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    SharpNet’s contour accuracy and 150 fps inference support realistic real-time integration of virtual objects in augmented reality.

    Source: §5 · p. 8

    The consensus terms assume orthographic projection, described as a good first-order assumption.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

We introduce SharpNet, a method that predicts an accurate depth map for an input color image, with a particular attention to the reconstruction of occluding contours: Occluding contours are an important cue for object recognition, and for realistic integration of virtual objects in Augmented Reality, but they are also notoriously difficult to reconstruct accurately. For example, they are a challenge for stereo-based reconstruction methods, as points around an occluding contour are visible in only one image. Inspired by recent methods that introduce normal estimation to improve depth prediction, we introduce a novel term that constrains depth and occluding contours predictions. Since ground truth depth is difficult to obtain with pixel-perfect accuracy along occluding contours, we use synthetic images for training, followed by fine-tuning on real data. We demonstrate our approach on the challenging NYUv2-Depth dataset, and show that our method outperforms the state-of-the-art along occluding contours, while performing on par with the best recent methods for the rest of the images. Its accuracy along the occluding contours is actually better than the `ground truth' acquired by a depth camera based on structured light. We show this by introducing a new benchmark based on NYUv2-Depth for evaluating occluding contours in monocular reconstruction, which is our second contribution.

1. Introduction

Monocular depth estimation is ill-posed but useful across several applications, while occluding contours remain difficult to reconstruct despite their importance for recognition and augmented reality. SharpNet adds depth–contour and depth–normal constraints to multi-task training, and the authors introduce NYUv2-OC to evaluate contour localization alongside standard depth metrics.

1. Introduction

  • Occluding contours remain difficult to reconstruct, though they matter for object recognition, augmented reality, and path planning.Depth annotations from stereo or structured-light cameras can be inaccurate near contours because nearby 3D points may be visible in only one image; contours also occupy little image area and may not influence training loss without special handling.Source: §1 · pp. 1–2
  • SharpNet jointly predicts depth, normals, and occluding contours, with loss terms constraining depth against both normals and contours.At runtime, it predicts only depth and runs at 150 fps on 640 × 480 images.Source: §1 · p. 2
  • The depth–contour constraint improves depth output beyond multi-task learning alone, while adding normal prediction helps training converge toward good depth predictions.The authors report that each aspect of their training procedure improves the depth output.Source: §1 · p. 2
  • Synthetic images provide pixel-perfect contour training annotations before the network is fine-tuned on NYUv2-Depth.Synthetic object-instance boundaries serve as contour annotations; NYUv2-Depth ground truth is used for training only during fine-tuning.Source: §1 · p. 2
  • NYUv2-Depth’s noisy contour ground truth makes it unsuitable for representative contour-accuracy evaluation, motivating manual contour annotations.Automatically identifying contours as depth discontinuities would also depend on the identification method’s parameters.Source: §1 · p. 2
  • SharpNet outperforms previous methods on occluding-contour 2D localization and places second on all standard NYUv2-Depth metrics.The authors manually annotated 100 randomly sampled validation images as NYUv2-OC and made the annotations and evaluation code publicly available.Source: §1 · p. 2

2. Related Work

Related work covers supervised and self-supervised monocular depth estimation, along with methods that target edges, occlusions, or reconstruction quality. The discussion contrasts their accuracy, contour sharpness, training requirements, and runtime with the needs addressed by SharpNet.

2.1. Supervised and Self-Supervised Monocular Depth Estimation

  • Earlier supervised methods used multi-scale depth estimation, while SharpNet uses a simpler mono-scale architecture and depth, normal, and contour constraints.Eigen et al. used multi-scale estimation for global and local information; the authors describe their own approach as efficient at runtime.Source: §2.1 · p. 2
  • Laina et al. produce smooth occluding contours, while a compared approach achieves the best performance on NYUv2-Depth but biases toward high-depth regions, reducing final error yet producing blurry depth maps.By contrast, the authors’ reconstructions are sharper along occluding boundaries, and their method is much faster.Source: §2.1 · pp. 2–3
  • Self-supervised methods use consistency across views, but the authors say they generally lag supervised accuracy and suit settings without annotated data.Source: §2.1 · p. 3

2.2. Edge- and Occlusion-Aware Depth Estimation

  • SURGE jointly predicts depth, normals, edges, and planar regions, then refines depth with a DCRF; it is not real-time and does not evaluate 2D contour localization.The passage says SURGE yields appealing planar-region reconstructions but underperforms state-of-the-art methods on global metrics.Source: §2.2 · p. 3
  • Edge- or occlusion-aware methods trained on stereo pairs or image sequences [36, 35, 37, 28, 10] require multiple calibrated training views, unlike monocular RGB-D datasets.Source: §2.2 · p. 3
  • Light-field methods [31, 14] report strong occlusion-aware results but require images that are more costly to obtain than RGB.Source: §2.2 · p. 3
  • Koch et al. introduced iBims-v1, a 100-image RGB benchmark with depth maps and annotations and metrics for occluding contours and planarity.The authors base their contour-reconstruction evaluation method on that work.Source: §2.2 · p. 3

3. Method

SharpNet predicts depth, surface normals, and occluding contours with a shared-encoder network, then uses supervision and cross-task constraints to improve geometric consistency. It pretrains on synthetic data and fine-tunes on NYUv2-Depth, where direct contour and normal supervision is unavailable.

3. Method

  • A shared ResNet50 encoder feeds task-specific decoders for depth, normals, and occluding-contour predictions, with skip connections between corresponding scales.At test time, the depth stream alone can be used for faster inference.Source: §3 · p. 3, Figure 2 · p. 3, §3.1 · p. 4

3.1. Training Overview

  • Synthetic PBRS training uses object-instance boundaries as proxy contour labels, then NYUv2-Depth fine-tuning relies on consensus constraints without direct contour or normal supervision.The authors argue that constraints between depth and contour predictions can also encourage prediction of intra-object occluding boundaries.Source: §3.1 · pp. 3–4

3.2. Loss Function

  • The loss combines depth, contour, and normal supervision with depth-contour and depth-normal consistency terms, while its supervision weights are adjusted during training.The weights are adjusted to prioritize learning local geometry before depth.Source: §3.2 · pp. 4–5

3.3. Supervision Terms Ld, Lc, and Ln

  • Depth supervision applies BerHu loss to log-distances; BerHu uses L2 loss for large deviations and L1 loss for small ones, with c = 1.The sum is over valid pixel locations.Source: §3.3 · p. 5
  • Contour supervision uses an attention loss for edge prediction to address the imbalance between contour and non-contour pixels; normal supervision minimizes angular error via dot products.The normal loss is disabled during NYUv2-Depth fine-tuning, as is the contour supervision loss.Source: §3.3 · p. 5

3.4. Consensus Terms Ldc and Ldn

  • The depth-contour consensus term encourages sharp depth edges at occluding contours by associating large depth gradients with high predicted contour probabilities.High-gradient regions incur a large loss unless contour probability is close to one.Source: §3.4 · p. 5
  • The depth-normal consensus term enforces geometric consistency between predicted depth and normals, assuming orthographic projection rather than requiring camera calibration.During fine-tuning, it can constrain depth and normals even when ground-truth normals are unavailable or insufficiently accurate.Source: §3.4 · pp. 5–6

4. Experiments

SharpNet is evaluated on NYUv2-Depth and the manually annotated NYUv2-OC contour benchmark, using standard depth metrics and boundary accuracy. It achieves the strongest reported occluding-contour accuracy while remaining competitive on standard depth metrics; ablations support the contribution of geometry-consensus terms.

4.1. Implementation Details

  • Training proceeds from synthetic PBRS data to NYUv2-Depth fine-tuning, with task losses monitored and decoders frozen during an initial real-data adaptation stage.PBRS instance boundaries serve as contour proxies; the consensus terms are initially disabled during fine-tuning, then restored after convergence.Source: §4.1 · p. 6

4.2. Evaluation Method

  • SharpNet outperforms state-of-the-art methods on occluding-contour accuracy and is a competitive second-best on standard depth metrics.Table 1 reports the final evaluation; the comparison uses the center crop and clips predictions to [0.7m, 10m].Source: §4 · p. 6, Table 1 · p. 7, §4.2 · p. 7
  • Occluding-contour accuracy is evaluated on NYUv2-OC, a manually annotated subset created because acquired NYUv2-Depth maps are especially noisy at boundaries.Detected depth edges are compared with annotated edges using truncated chamfer distance; distances above 10 pixels are ignored.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3. Ablation Study

  • Geometry-consensus terms improve both whole-dataset depth reconstruction and occluding-contour accuracy, while preserving their trade-off during real-data fine-tuning.The ablation compares models on NYUv2-OC; Table 2 reports RMSElog over the full NYUv2-Depth dataset.Source: §4.3 · p. 7, Table 2 · p. 8

5. Conclusion

SharpNet targets competitive monocular depth reconstruction with particular attention to occluding contours, using geometry-consensus terms during multi-task training. The paper reports real-time inference and introduces NYUv2-OC for evaluating contour reconstruction.

5. Conclusion

  • SharpNet achieves competitive single-image depth reconstruction with particular attention to occluding contours through geometry-consensus terms in multi-task training.Source: §5 · p. 8
  • At 150 fps, SharpNet’s depth and contour accuracy supports realistic integration of virtual objects in real-time augmented reality.Source: §5 · p. 8
  • The paper introduces NYUv2-OC as a first version of an occluding-contour dataset and notes that predicted normals and contours may aid other vision tasks.Source: §5 · p. 8

Related papers

Loading 1905.08598v3…