PaperlayerView exact revision on arXiv

Source-linked AI summary

LEGO: Learning Edge with Geometry all at Once by Watching Videos

Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia

arXiv:1803.05648v2cs.CV

TL;DR

Monocular 3D geometry learned from unlabeled videos can remain blurry and poorly aligned with scene structure. LEGO jointly learns depth, normals, and geometrical edges using a 3D-ASAP planar prior, and reports state-of-the-art performance across depth, normal, and edge estimation, including a 30% relative cross-dataset depth improvement.

  • Problem

    Unsupervised monocular geometry pipelines focus on per-pixel photometric errors, leaving estimated depths and normals blurry and insufficiently conformant to scene geometry.

  • Method

    LEGO jointly learns depth, surface normals, and geometrical edges using a 3D-ASAP prior that favors shared planar surfaces when no edge intervenes.

  • Results

    LEGO improves depth, normal, and edge estimation across the evaluated tasks, including 30% relative improvement in cross-dataset depth prediction over other SOTA methods.

  • Takeaways & Limitations

    Joint regularization of geometrical edges and 3D structure yields geometry with fine details, consistent scene layout, and cleaner edge alignment.

  • Takeaways & Limitations

    The 3D-ASAP prior is restrictive for large non-planar surfaces, although it fits the planar dominant surfaces of street scenes.

Abstract

from arXiv · show

Learning to estimate 3D geometry in a single image by watching unlabeled videos via deep convolutional network is attracting significant attention. In this paper, we introduce a "3D as-smooth-as-possible (3D-ASAP)" prior inside the pipeline, which enables joint estimation of edges and 3D scene, yielding results with significant improvement in accuracy for fine detailed structures. Specifically, we define the 3D-ASAP prior by requiring that any two points recovered in 3D from an image should lie on an existing planar surface if no other cues provided. We design an unsupervised framework that Learns Edges and Geometry (depth, normal) all at Once (LEGO). The predicted edges are embedded into depth and surface normal smoothness terms, where pixels without edges in-between are constrained to satisfy the prior. In our framework, the predicted depths, normals and edges are forced to be consistent all the time. We conduct experiments on KITTI to evaluate our estimated geometry and CityScapes to perform edge evaluation. We show that in all of the tasks, i.e.depth, normal and edge, our algorithm vastly outperforms other state-of-the-art (SOTA) algorithms, demonstrating the benefits of our approach.

1. Introduction

LEGO addresses blurry monocular geometry by jointly learning geometrical edges, depth, and normals under a 3D-ASAP prior. The framework reports improved fine-structure estimation across depth, normal, and edge tasks.

  • Motivation: Unlabeled-video methods improve monocular depth estimation but can produce blurry depths and normals that do not conform well to scene geometry.The pipeline uses view synthesis for supervision, while the cited results reveal geometric weaknesses.
  • Approach: LEGO jointly learns geometrical edges with depth and normals so the representations reinforce each other.Geometry filters internal image-gradient edges, while discovered edges provide long-range context and non-local regularization for fine details.
  • Approach: 3D-ASAP requires 3D-recovered points without intervening edges to lie on the same planar surface.The prior couples edge prediction with geometrical smoothness inside the learning pipeline.
  • Results: LEGO’s estimated depths and normals follow scene structure, while its geometrical edges align with layout and contain fewer noises than unsupervised edge detection.The discovered edges are geometrical rather than necessarily semantic.
  • Results: 30% relative improvement in cross-dataset depth prediction, 20% improvement in normal estimation, and 15% improvement in geometrical-edge detection are reported over prior methods.The experiments cover KITTI 2015, CityScapes, and Make3D; LEGO also runs without much extra computation compared with [64].

2. Related Work

Prior work spans geometric reconstruction, supervised and unsupervised CNN geometry estimation, non-local smoothness, and edge detection. LEGO combines unsupervised geometrical-edge discovery with unified 3D-geometry regularization rather than appended components.

  • Geometric reconstruction: Traditional SFM, SLAM, and DTAM can estimate 3D from video but may fail under low texture or drastic viewpoint changes and do not extend to single-view reconstruction.Single-view methods also use vanishing points, BRDF rules, or major-plane and box representations.
  • Supervised geometry: Supervised CNNs estimate dense single-view depth and normals, but these approaches require densely labeled ground truth that is expensive to obtain in natural environments.CRF-based depth-normal regularization further improves predictions over large planar surfaces.
  • Unsupervised geometry: Unsupervised CNN methods use videos or stereo images for single-view geometry estimation, motivated by richer 3D information and easier data acquisition.Deep3D learns view generation from stereo pairs using quantized depth values.
  • Non-local smoothness: Non-local graphical-model smoothness can improve detail recovery, but iterative CNN combinations are inefficient for learning and inference.The paper therefore adopts multi-scale strategies to learn non-local smoothness inside the unsupervised pipeline.
  • LEGO’s position: LEGO discovers geometrical edges unsupervised and jointly smooths 3D geometry without extra components, outperforming [37] in street-view cases.This unified regularization distinguishes the method from appended transfer or CRF-like modules.

3. Preliminaries

The preliminaries establish view synthesis as photometric supervision and add spatial and depth-normal regularization to resolve ambiguities in monocular geometry estimation.

  • View synthesis: Unsupervised geometry uses inverse warping from a target image to source views together with depth-normal consistency.The target frame is synthesized from source views using predicted depth and relative transformations.
  • View synthesis: A source image is warped into the target view using the target depth and an estimated transformation, then compared photometrically with the observed target image.Corresponding source pixels are obtained through perspective projection and bilinear interpolation.
  • Depth regularization: View-synthesis supervision is ambiguous because one pixel can match multiple candidates, so additional depth regularization is required.Prior work encourages locally similar depth where no significant image gradient exists.
  • Depth regularization: The spatial smoothness term penalizes the L1 norm of second-order depth gradients along the x and y directions in 2D space.The order value 2 denotes second-order gradients.
  • Depth-normal consistency: Depth-normal consistency strengthens scene structure by using 8-neighbor 3D relationships, with differentiable depth-to-normal and normal-to-depth layers.The consistency formulation and normal smoothness improve depth and normal predictions, especially along surface regions, and are incorporated into LEGO.

4. Learning edge with geometry from videos

LEGO introduces the 3D-ASAP prior to jointly learn geometrical edges with depth and surface normals from videos. The framework embeds learned edges into multi-scale geometry smoothness losses while addressing trivial edges, double edges, invalid gradients, and local gradients.

  • 3D-ASAP prior: 3D-ASAP assumes that, without visual cues such as edges, recovered points belong to a single planar 3D surface.For normals, pixels on the same surface are encouraged to share normals; for depth, intervening pixels are constrained to lie on the same 3D line.
  • Joint objective: The loss module combines visual synthesis, depth 3D-ASAP, normal 3D-ASAP, and edge losses, alongside depth-normal consistency.The resulting objective trains depth, normal, and edge maps together.
  • Learning geometrical edges: LEGO uses learned edge maps to determine affinities between pixels and jointly regularize depth, normal, and edge predictions.The edge decoder shares the depth network’s image encoder, and the learned affinity is based on the maximum edge response along the line between pixels.
  • Learning geometrical edges: An L2 penalty discourages the trivial solution of predicting every pixel as an edge when edge supervision is unavailable.Cross-entropy regularization performed poorly because the edge map contains sparse positives and lacks ground-truth balancing.
  • Training corrections: Clipping negative gradient values removes the double-edge artifact caused by smoothly estimated depth changes across object boundaries.The replacement g′ = max(g(pi, ∗), 0) retains one side of the second-order gradient response at each boundary.
  • Training corrections: A camera-motion-derived fly-out mask removes invalid gradients for target-view pixels that fall outside the source view.Multi-resolution losses are also used because bilinear spatial transformation depends only on four neighboring pixels.
  • Complementary geometry: Depth and normal smoothness are complementary: normal changes reveal intersecting-plane edges, while depth changes reveal edges where normals remain uniform.The learned edges are reported to remain consistent with both depth and normal information.

5. Evaluation

LEGO is evaluated for depth, normal, and edge estimation across KITTI 2015, Cityscapes, and Make3D, including cross-dataset generalization. It consistently improves geometry quality and edge alignment over competing methods while adding little computation.

  • Evaluation setup: Experiments cover depth, normal, and edge estimation on KITTI 2015, Cityscapes, and Make3D using established task-specific metrics.KITTI provides depth and normal evaluation, Cityscapes supports training and edge evaluation, and Make3D supports monocular depth evaluation.
  • Depth estimation: LEGO outperforms unsupervised depth methods on both KITTI test splits and performs comparably to the semi-supervised method.On KITTI split, LEGO also achieves the best reported Sq Rel performance, indicating fewer outlier depth values.
  • Normal estimation: LEGO achieves state-of-the-art normal performance on KITTI, outperforming baseline methods by a large margin despite not always leading on depth.The authors attribute this pattern partly to normal evaluation using neighboring points rather than only pixels with depth ground truth.
  • Qualitative geometry: LEGO preserves smooth surfaces while retaining clear geometrical edges, producing sharper depth boundaries, fewer normal artifacts, and clearer scene layout.Qualitative comparisons report smoother outputs within surfaces and better alignment with object boundaries than Yang et al.
  • Generalization: LEGO achieves state-of-the-art performance under both cross-dataset settings and a large margin when transferring from Cityscapes to KITTI.The authors suggest 3D-ASAP regularization jointly learns structural layout information, improving robustness to scene changes.
  • Edge detection: LEGO outperforms competing edge methods on all metrics, preserving object boundaries while ignoring trivial within-surface edges such as lane markings.Its edge predictions align better with ground-truth edges than edges generated from surface normals.

6. Conclusion

The paper concludes that LEGO jointly learns depth, surface normals, and geometrical edges without supervision using the 3D-ASAP prior. It reports state-of-the-art geometry results on KITTI and substantially better edge detection on Cityscapes.

  • Contribution: LEGO is an unsupervised framework for jointly learning depth, surface normals, and geometrical edges.Its 3D-ASAP prior regularizes scene layout by jointly considering these three 3D-scene descriptors.
  • Results: LEGO achieves state-of-the-art depth and normal performance on KITTI and outperforms other edge methods by a large margin on Cityscapes.The conclusion reports improvements across all three evaluated tasks.

1. Edge ground truth generation

Cityscapes geometrical edge ground truth is derived from semantic segmentation by combining categories that share a 3D surface, while preserving boundaries between instances.

  • Category combination: Four groups of semantic categories are combined before geometrical edges are extracted from the resulting segmentation.The combination removes boundaries between categories that are geometrically connected.
  • Edge extraction: Edges between different object instances remain preserved during geometrical edge generation.Thus, the target represents geometric discontinuities rather than every semantic-category boundary.
  • Generation pipeline: Figure 1 presents the generation sequence from RGB images and semantic ground truth to combined segmentation and geometrical edge ground truth.The visual pipeline proceeds from left to right through these four representations.

2. Inference between Eqn. 3 and Eqn. 4 / Eqn. 5

The derivation connects planar-surface normal consistency with a three-point collinearity condition, establishing equivalent constraints for the 3D-ASAP geometry prior.

  • Eqn. 3 to Eqn. 5: For three collinear 3D points, Eqn. 5 requires the gradients between each pair of consecutive points to be the same.The condition applies to points p_i, p_j, and p_k lying on a 3D line.
  • Equivalence: Because the two lines share p_j and have equal gradients, they are identical, making Eqns. 3 and 5 mutually necessary and sufficient.The argument uses the shared middle point to establish line identity.

3. Example outputs

LEGO jointly estimates depth, surface normals, and geometrical edges, with example outputs showing each predicted modality alongside the input image.

  • LEGO jointly estimates depth, surface normal, and geometrical edge.
  • The example outputs illustrate the three geometry-related predictions produced by LEGO.
  • Figure 2 presents the input image followed by predicted depth, predicted normal, and predicted edge.

4. Comparison with previous methods

Compared with prior methods, LEGO produces depth and normal results with better structure while preserving aligned object boundaries.

  • LEGO generates depth and normal results with better structure than methods [64] and.
  • LEGO preserves aligned object boundaries in its depth and normal predictions.

5. Qualitative results

Qualitative results include visual comparisons of depth and surface normal predictions across methods, plus Cityscapes video examples from diverse validation scenes.

  • The Cityscapes validation examples comprise 10 snippets from diverse scenes.
  • Each Cityscapes dataset snippet contains 30 frames around its key frames.
  • Figure 3 compares depth and surface normal results from LEGO, ground truth, and methods [60] and [64].
Loading 1803.05648v2…