PaperlayerView exact revision on arXiv

Source-linked AI summary

The Edge of Depth: Explicit Constraints between Segmentation and Depth

Shengjie Zhu, Garrick Brazil, Xiaoming Liu

arXiv:2004.00171v1cs.CV

TL;DR

Self-supervised depth estimation remains ambiguous in challenging regions, and prior segmentation–depth approaches have mostly linked the tasks implicitly. This paper explicitly constrains depth borders using segmentation through iterative morphing and reports KITTI performance matching supervised state of the art on the absolute relative metric.

  • Problem

    Self-supervised depth inference is ambiguous in low-texture regions, reflective surfaces, and occlusions, motivating additional constraints.

    Source: §1 · p. 1

  • Method

    The method explicitly constrains depth borders using segmentation, iteratively morphing disparity toward a locally optimal map and penalizing its difference from the original prediction.

    Source: §1 · p. 2, §3.1.1 · p. 4

  • Results

    The method advances self-supervised monocular depth estimation on KITTI to state-of-the-art performance, matching supervised methods on the absolute relative metric.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates that segmentation borders can provide an explicit constraint for making estimated depth more consistent with segmentation.

    Source: §1 · p. 2

    The edge-consistency loss is asymmetric because segmentation edges are treated as more reliable than depth edges.

    Source: §3.1.1 · p. 4

Abstract

from arXiv · show

In this work we study the mutual benefits of two common computer vision tasks, self-supervised depth estimation and semantic segmentation from images. For example, to help unsupervised monocular depth estimation, constraints from semantic segmentation has been explored implicitly such as sharing and transforming features. In contrast, we propose to explicitly measure the border consistency between segmentation and depth and minimize it in a greedy manner by iteratively supervising the network towards a locally optimal solution. Partially this is motivated by our observation that semantic segmentation even trained with limited ground truth (200 images of KITTI) can offer more accurate border than that of any (monocular or stereo) image-based depth estimation. Through extensive experiments, our proposed approach advances the state of the art on unsupervised monocular depth estimation in the KITTI.

1. Introduction

Self-supervised monocular depth estimation remains ambiguous in difficult image regions, while prior uses of segmentation have mostly modeled its relationship with depth implicitly. This work introduces an explicit border constraint and reports KITTI performance matching supervised state of the art on the absolute relative metric.

1. Introduction

  • Self-supervised depth remains ambiguous in low-texture regions, reflective surfaces, and occlusions, while prior segmentation–depth methods used implicit relations that lacked a clear explanation.Source: §1 · p. 1
  • The method measures segmentation–depth edge distance and greedily morphs disparity toward a local optimum, penalizing the difference from the original prediction because the measurement is not differentiable.Source: §1 · p. 2
  • A stereo occlusion mask is proposed to alleviate bleeding artifacts near object boundaries and improve depth quality there.Source: §1 · p. 2
  • The paper explicitly defines a segmentation–depth border constraint to make estimated depth more consistent with segmentation.Source: §1 · p. 2
  • The method advances self-supervised monocular depth estimation on KITTI to state-of-the-art performance, matching supervised methods on the absolute relative metric.Source: §1 · p. 2

2. Related work

Related work uses photometric supervision and additional modalities to address limitations in self-supervised depth estimation. The paper highlights the underused segmentation–depth border relationship and the difficulty of making it explicit.

Self-supervised Depth Estimation

  • Self-supervised depth methods use predicted depth to build pixel correspondences across views and minimize photometric reconstruction loss, with video methods also estimating ego-motion.Source: §2 · p. 2
  • Photometric supervision can pair incorrect pixels under occlusion and provides weak signals in low-texture or color-saturated regions.Source: §2 · p. 2

Occlusion Problem

  • Stereo occlusions can produce bleeding depth artifacts; prior approaches used left-right consistency or depth-magnitude regularization, while the proposed mask uses one estimated disparity map.Source: §2 · p. 2

Using Additional Modalities

  • Prior work adds surface normals, segmentation, optical flow, or stereo matching proxies; segmentation–depth relations remain vague because segmentation has no definite mathematical link to depth.Source: §2 · p. 2

Using Semantic Segmentation for Depth

  • Segmentation–depth methods have shared weights, mixed features, or modeled statistical relationships, but shared borders pose partial-overlap and differentiability challenges.Source: §2 · pp. 2–3

3. The Proposed Method

The method aligns disparity edges to semantic boundaries by constructing a locally morphed target, then trains against it alongside photometric and stereo-proxy losses. It also masks stereo-occluded pixels to reduce bleeding at object borders.

3. The Proposed Method

  • Disparity is steered toward a segmentation-augmented map, making explicit edge consistency usable as a training loss despite its non-differentiability.The map is constructed by morphing disparity values, and the loss penalizes its difference from the original prediction.Source: §3 · p. 3, §3.3.1 · p. 5

3.1. Explicit Depth-Segmentation Consistency

  • The method extracts paired segmentation and disparity edges, then morphs depth values within their inner bounds to align depth edges with semantic edges.The training setup uses stereo image pairs and precomputed semantic labels, while inference requires only a monocular RGB image.Source: §3.1 · p. 3, §3 · p. 3

3.1.1 Edge-Edge Consistency

  • Depth edges are extracted from disparity, and segmentation edge points are associated with nearby depth edges within a preset distance because the correspondence is not one-to-one.The threshold k1 controls the gradient intensity needed for an edge; k2 limits the association distance.Source: §3.1.1 · pp. 3–4

3.1.2 Depth Morphing

  • The morph function shifts pixels near associated edge pairs, while distance-weighted morphing blends pairs and limits each pair’s influence to a local area.The local weighting makes pair effects beyond roughly 7 pixels negligible.Source: §3.1.2 · p. 4

3.2. Stereo Occlusion Mask

  • Stereo bleeding occurs when estimated foreground depth expands into background regions; the occlusion mask detects potentially occluded pixels and excludes them from reconstruction loss.The mask uses a one-directional neighborhood search based on disparity movement.Source: §3.2 · pp. 4–5

3.3. Network and Loss Functions

  • The encoder-decoder predicts disparity from a monocular RGB image, which is converted to depth using known camera parameters.Training consumes stereo pairs and precomputed semantic labels; inference uses only a monocular image.Source: §3 · p. 3, §3.3 · p. 5

3.3.1 Loss Functions

  • Training combines photometric reconstruction, morphing, and stereo-proxy losses on non-occluded pixels, applying morph loss when photometric loss improves.The stereo-proxy loss is used to help neutralize noise in predicted semantic labels.Source: §3.3.1 · p. 5

3.3.2 Implementation Details

  • Using Cityscapes semantics, object, vehicle, and human were foreground and all other classes background; they trained joint loss for 20 epochs at 1e−4, then applied lbn loss for 3 epochs.The lbn-loss phase used a learning rate of 1e−5.Source: §3.3.2 · p. 6

4. Experiments

Experiments show improved KITTI depth-estimation performance and border quality, with gains extending beyond object edges. Ablations examine the occlusion mask, training stabilization, finetuning, and morphing design.

KITTI Dataset

  • Evaluations use the KITTI Stereo 2015 dataset, eigen splits, seven standard metrics, the Garg crop, and an 80-meter distance cap.The method uses the same network structure as; its contributions are described as orthogonal to most monocular and stereo training methods.Source: §4 · p. 6

Depth Estimation Performance

  • δ < 1.25 improves from 0.890 to 0.898, while the method outperforms prior methods on all seven KITTI metrics and matches SOTA supervised performance in absolute relative error.The authors also report consistent gains with two ResNet variants, indicating robustness to backbone capacity.Source: §4 · p. 6

Depth Performance Analysis

  • Depth performance degrades near object edges by about 0.18 in δ relative to whole-image performance, while improvements occur near borders and inside objects.The reported gain peaks around 5 pixels into the background, but remains positive across the distance range.Source: §4 · p. 7, Figure 5 · p. 7

Depth Border Quality

  • Morphing aligns predicted depth borders more closely with object boundaries; applying it at inference also improves unsupervised, supervised, and multi-view systems using segmentation trained on 200 domain images.Using depth edges to morph segmentation instead is harmful to border quality.Source: §4 · p. 7, Figure 7 · p. 7, Table 3 · p. 8

Stereo Occlusion Mask

  • The stereo occlusion mask improves absolute relative error from 0.102 to 0.101 and δ < 1.25 from 0.884 to 0.887, while controlling bleeding artifacts.The resulting stronger borders support the consistency term and morphing operation.Source: §4 · p. 7, Table 4 · p. 8

Morph Stabilization

  • Training stabilization weights down low-image-variance regions and excludes regions that degrade photometric loss when noisy predicted segmentation labels guide morphing.The ablation also examines stereo proxy labels as a source of training stabilization.Source: §4 · pp. 7–8

Finetuning Strategy

  • Finetuning improves each of the ablated methods, including and the proposed method, suggesting the strategy is general.The comparison is reported in Tables 4 and 5.Source: §4 · p. 8, Table 5 · p. 8, Table 4 · p. 8

Morphing Strategy

  • Triangularization morphing is generally inferior to the proposed morphing strategy on RMSE and can introduce acute-angle boundary errors that add noise to supervision.The comparison is shown quantitatively in Table 6 and qualitatively in Figure 9.Source: §4 · p. 8, Table 6 · p. 8, Figure 9 · p. 8

5. Conclusions

The conclusion presents an explicit segmentation–depth border constraint and a stereo occlusion mask as components of a framework for self-supervised depth estimation. It reports state-of-the-art performance on KITTI and improved depth-border quality.

5. Conclusions

  • The framework morphs predicted segmentation and depth borders and uses the result as an additional supervision signal.This explicitly models their relationship, which prior work had primarily handled implicitly.Source: §5 · p. 8
  • A stereo occlusion mask is proposed to alleviate bleeding artifacts near object boundaries, alongside a finetuning strategy to boost generalization.Source: §5 · p. 8
  • The method advances KITTI self-supervised depth estimation to state-of-the-art performance, matching supervised methods in capacity and significantly improving estimated-depth border quality.Source: §5 · p. 8

1. Proof of Local Optimality

The proof argues that the proposed transformation can realign disparity edges with segmentation edges and attain a local optimum for the edge-consistency measure. The threshold condition is derived from the edge-detection threshold used on the original disparity map.

1. Proof of Local Optimality

  • Under the constructed transformation, semantic edge pixels can become disparity-edge pixels in the morphed map, making the consistency measure reach its local minimum of 0.The proof relies on the condition that the gradient at a semantic edge exceeds a threshold tied to k1.Source: §1 · p. 11
  • The construction realigns the depth edge set to the segmentation edge set, which is the basis for the claimed local optimality.Source: §1 · p. 11
  • The threshold t^(1+t) · k1 is derived as a condition satisfied under the morph and original-map edge threshold, not applied to the morphed map for edge detection.Source: §1 · p. 11

2. Network details

The experiments use ImageNet-pretrained ResNet encoders and a decoder structure shared with prior work. Training also uses common augmentation and loss practices.

2. Network details

  • The encoders are ImageNet -pretrained ResNet18 and ResNet50.Source: §2 · p. 12
  • The decoder structure follows Godard et al. and Watson et al..It is detailed in Table 1.Source: §2 · p. 12
  • Training incorporates color augmentation, random flips, edge-aware smoothness, and exclusion of stationary pixels.Source: §2 · p. 12

3. More Ablations

These KITTI Eigen-split ablations test morphing’s effect on edge-edge consistency and whether stereo occlusion masking sharpens borders.

Reducing edge-edge consistency via morphing

  • Morphing consistently lowers edge-edge consistency loss after one and two applications, and the resulting models are more consistent with segmentation than.Higher edge thresholds produce thinner edges and higher loss, but focusing on close-range objects yields a larger improvement margin over.Source: §3 · p. 12

Stereo Occlusion Mask

  • Without additional supervision, the stereo occlusion mask eliminates bleeding artifacts and reduces detected edge counts, indicating thinner, clearer borders.The ablations assess whether mask M boosts clear borders on the Eigen test splits of KITTI.Source: §3 · p. 12

More quality comparisons

  • Noisy predicted segmentation labels and weak photometric constraints cause blobbed borders in low-texture areas; texture weighting and proxy stereo labels suppress them.Source: Figure 3 · p. 13
  • Qualitative comparisons report improved border quality over [38] and more object detail with the clearest overall borders versus [13,29,30,38].Source: Figure 4 · p. 14, Figure 5 · p. 15

Related papers

Loading 2004.00171v1…