PaperlayerView exact revision on arXiv

Source-linked AI summary

DeMoN: Depth and Motion Network for Learning Monocular Stereo

Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, Thomas Brox

arXiv:1612.02401v2cs.CV

TL;DR

Two-frame SfM must estimate scene structure despite correspondence, degeneracy, and scale challenges that constrain traditional pipelines and single-image depth methods. DeMoN addresses this by jointly learning dense correspondence, depth, camera motion, and related cues from unconstrained image pairs, with iterative refinement and a spatial-relative-difference loss. It outperforms traditional two-frame methods on most datasets but remains limited in handling cameras with different intrinsic parameters and in extending beyond two images.

  • Problem

    Traditional SfM is limited by sequential motion-depth estimation, sparse correspondence failures, small translations, and single-image methods' poor generalization to unseen structures.

  • Method

    DeMoN is an end-to-end convolutional network with iterative encoder-decoder refinement that jointly estimates optical flow, confidence, depth, surface normals, camera motion, and depth scale from image pairs.

  • Results

    DeMoN outperforms baseline methods on motion and depth accuracy by a factor of 1.5 to 2 on most datasets, except for motion on MVS, where it is on par with FlowFields.

  • Takeaways & Limitations

    Learning enables DeMoN to exploit motion parallax and integrate shape-from-X cues, supporting generalization to new scene types and improved two-frame SfM performance.

  • Takeaways & Limitations

    DeMoN has not yet reached the flexibility of classic approaches for cameras with different intrinsic parameters and is intended for the two-frame case.

Abstract

from arXiv · show

In this paper we formulate structure from motion as a learning problem. We train a convolutional network end-to-end to compute depth and camera motion from successive, unconstrained image pairs. The architecture is composed of multiple stacked encoder-decoder networks, the core part being an iterative network that is able to improve its own predictions. The network estimates not only depth and motion, but additionally surface normals, optical flow between the images and confidence of the matching. A crucial component of the approach is a training loss based on spatial relative differences. Compared to traditional two-frame structure from motion methods, results are more accurate and more robust. In contrast to the popular depth-from-single-image networks, DeMoN learns the concept of matching and, thus, better generalizes to structures not seen during training.

1. Introduction

DeMoN formulates two-frame structure from motion as end-to-end learning, jointly estimating depth and camera motion from unconstrained monocular image pairs. Its correspondence-aware iterative architecture addresses shortcuts and limitations of conventional pipelines and single-image depth methods.

  • Traditional SfM pipelines can propagate incorrect camera-motion estimates into depth and rely on sparse feature correspondences that fail in outliers, textureless regions, and small-translation cases.These limitations make it difficult to integrate priors for degenerate configurations.
  • DeMoN is the first reported convolutional network in this paper to jointly estimate depth and camera motion from an unconstrained image pair.The two-frame formulation is presented as a step toward learning-based large-scale SfM.
  • Single-image depth networks learn useful shape priors but can generalize poorly to unseen structures because they do not exploit stereopsis.DeMoN instead uses motion parallax to generalize to new scenarios.
  • A simple two-image encoder-decoder can ignore the second image, so DeMoN alternates optical-flow estimation with camera-motion and depth estimation to enforce correspondence.Optical flow requires using both images, helping prevent the single-image shortcut.
  • The method adds a shared-weight iterative component, previous-prediction inputs, and a spatial-relative-difference loss designed for structure-from-motion scale ambiguity.Training uses real Kinect images and rendered scenes.

2. Related Work

Prior SfM systems estimate geometry through multi-stage correspondence, motion, and dense-depth pipelines, while related learning methods typically address only selected components. DeMoN instead integrates dense correspondence, depth, and camera-motion estimation in one deep network.

  • Classic large-scale SfM pipelines match descriptors, estimate camera motion from an essential matrix, reject outliers, and then compute dense depth.Their multi-image reconstructions still depend on pairwise geometry quality for initialization.
  • LSD-SLAM jointly optimizes semi-dense correspondences and depth over short temporal windows, whereas DTAM tracks poses using dense depth but requires external depth initialization.DTAM's initialization ultimately relies on classic structure-and-motion methods.
  • DeMoN differs by training a single deep network that computes dense correspondences, depth, and inter-frame camera motion together.This integrates tasks that previous approaches generally separated across pipeline stages.
  • Single-image depth networks rely on learned priors, while DeMoN additionally uses motion parallax from image pairs to support generalization to arbitrary new scenes.The two-frame input supplies stereoscopic information absent from single-image prediction.
  • Prior ConvNet work includes learned matching, dense optical flow, stereo disparity, camera-motion estimation, and camera relocalization, but these approaches do not jointly provide DeMoN's outputs.Some camera-focused methods do not provide depth estimates.
  • DeMoN's architecture uses a chain of encoder-decoder networks that iterate over optical flow, depth, and egomotion, followed by refinement of the final depth resolution.The overview separates bootstrap, iterative, and refinement components.

3. Network Architecture

DeMoN uses stacked encoder-decoder components to estimate optical flow, depth, normals, camera motion, and confidence, then iteratively refines these predictions before producing full-resolution depth. Its architecture forces correspondence between the input images rather than relying on a single-image shortcut.

  • Architecture: DeMoN chains bootstrap, iterative, and refinement networks to estimate and progressively improve scene geometry and motion.The bootstrap and iterative components operate as encoder-decoder pairs, while the refinement net produces the final upsampled depth.
  • Bootstrap network: The bootstrap net first predicts optical flow and confidence, then uses them with the image pair and a warped image to estimate depth, normals, and camera motion.The second encoder-decoder also predicts a depth scaling factor tied to the depth-motion relationship.
  • Stereo correspondence: Feeding optical flow into the second encoder-decoder makes DeMoN use motion parallax, unlike naïve two-frame estimation, which performs no better than single-image depth.The architecture therefore yields a large performance improvement over the naïve image-pair approach.
  • Iterative refinement: Training simulates four iterations by appending earlier predictions to minibatches instead of backpropagating gradients through time.This saves memory and permits larger networks and more iterations.
  • Refinement network: The refinement net upsamples 64 × 48 predictions to the full 256 × 192 input resolution using the full-resolution first image and upsampled depth and normals.A three-iteration forward pass takes 110ms on an Nvidia GTX Titan X.

4. Depth and Motion Parameterization

DeMoN predicts depth and camera motion in a scale-normalized representation. Iterative refinement improves depth scale and optical-flow accuracy, while the final refinement stage preserves edges at higher resolution.

  • Pose representation: The network estimates depth in the first view and relative camera pose to the second view using rotation and translation vectors.Rotation uses angle-axis coordinates, while translation is represented in Cartesian coordinates.
  • Iterative refinement: After the first iteration, the L1-inverse error falls from 0.0137 to 0.0072 as iterative refinement improves depth scale.The same figure reports optical-flow endpoint error decreasing from 0.0176 to 0.0120, with motion edges recovered in the first iteration.
  • Resolution refinement: The refinement net generates a 256 × 192 depth map from a 64 × 48 estimate and the input image, preserving depth edges and repairing wrong measurements.The iterative net improves depth values, reducing building distortion in an MVS point cloud.
  • Scale normalization: Unknown-motion reconstruction is scale-ambiguous, so DeMoN normalizes translation and depth such that ∥t∥ = 1.The network consequently predicts a unit-norm translation vector.
  • Depth representation: DeMoN predicts inverse depth ξ = 1/z and multiplies it by a learned scale factor s to obtain final depth values sξ.Inverse depth represents points at infinity and reflects increasing localization uncertainty with distance.

5. Training Procedure

DeMoN combines point-wise and scale-invariant gradient losses for depth, normals, optical flow, confidence, and camera motion. Training proceeds in staged phases, including simulated iterative refinement and separate refinement-net training.

  • Loss design: The loss balances high-dimensional per-pixel outputs against low-dimensional camera-motion vectors while encouraging synergy between tasks.Loss weights are selected empirically and listed in the supplementary material.
  • Point-wise and motion losses: Point-wise losses supervise inverse depth, surface normals, optical flow, and flow confidence, while motion losses supervise rotation and translation.Depth uses L1 loss on scaled inverse depth; normals and flow use non-squared L2 norms, with flow corresponding to endpoint error.
  • Motion supervision: The translation target is normalized to unit norm, while the magnitude of the rotation target encodes the rotation angle.This matches the parameterization used by the motion losses.
  • Scale-invariant gradient loss: The scale-invariant gradient loss penalizes relative depth errors across five spacings, emphasizing discontinuities and smoothness within homogeneous regions.The same type of loss is applied to optical-flow components to sharpen motion discontinuities and smooth estimated flow fields.
  • Training stages: Training uses Caffe and Adam, with the bootstrap and iterative encoder-decoders trained sequentially before dedicated iterative-net training.The initial phase trains four encoder-decoder components for 250k iterations each, followed by 1.6 million iterative-net iterations.
  • Training stages: The refinement net is trained last for 600k iterations while all other weights remain fixed.This final phase targets the full-resolution output after the preceding network components have been trained.

6. Experiments

Experiments evaluate DeMoN across indoor, outdoor, synthetic, and single-image benchmarks using scale-aware metrics and comparisons with engineered baselines. DeMoN generally improves depth and motion accuracy, handles low-motion cases, and generalizes better to unexpected scenes.

  • Datasets and setup: The evaluation uses SUN3D, RGB-D SLAM, MVS, Scenes11, Blendswap, and NYUv2, with dataset-specific depth, pose, realism, and training constraints.Blendswap is training-only, while NYUv2 is test-only because it lacks camera pose information.
  • Metrics: Depth comparisons use a scale-invariant error because two-image methods estimate scale relative to camera translation, unlike single-image methods.Camera motion is evaluated by translation and rotation angles, while optical flow uses average endpoint error.
  • Baselines: DeMoN is compared with SIFT-, FlowFields-, Matlab-, and oracle-based baselines built from correspondence, essential-matrix, RANSAC, reprojection, and plane-sweep components.The baselines use full-resolution 640 × 480 images, whereas DeMoN receives 256 × 192 inputs.
  • Comparison to classic structure from motion: 1.5 to 2× better motion and depth accuracy is reported on most datasets versus baseline methods, with MVS motion accuracy matching the strong FlowFields baseline.DeMoN’s depth estimates also exceed traditional estimates given ground-truth motion on all datasets except MVS.
  • Comparison to classic structure from motion: DeMoN handles cases without or with very little camera motion, and concatenated pairwise poses produce locally consistent trajectories with mainly translational drift.The trajectory experiment applies no drift correction, although the authors state that the results support integration into SLAM systems.
  • Comparison to depth from single image: DeMoN outperforms single-frame methods on all but NYUv2, generalizes better to unexpected scenes, and produces more detailed and regular depth maps.The NYUv2 comparison is described as unfair because competing methods were trained on NYUv2 data.
  • Ablation studies: The full architecture combining scale-invariant loss, surface-normal estimation, and flow loss gives the best ablation results, while flow confidence improves egomotion estimation.The authors interpret confidence as helping identify the most accurate correspondences.

7. Conclusions and Future Work

DeMoN jointly estimates depth and camera motion from two unconstrained images, exploiting motion parallax and outperforming traditional two-frame structure-from-motion techniques. Its remaining limitations include camera-intrinsic flexibility and the restriction to two images.

  • DeMoN is the first deep network trained to estimate depth and camera motion from two unconstrained images.
  • Unlike single-image depth networks, DeMoN exploits motion parallax and generalizes to new scene types while estimating egomotion.
  • DeMoN outperforms traditional two-frame structure-from-motion techniques by training end-to-end and integrating multiple shape-from-X cues.
  • The method remains less flexible than classical approaches for cameras with different intrinsic parameters and is currently limited to two images.

– Supplementary Material –

The supplementary architecture chains encoder-decoder networks that exchange geometrically derived representations, with iterative refinement improving predictions. Training uses staged optimization, weighted losses, and bilinear image warping.

  • The network chains encoder-decoder modules whose later components receive predictions from earlier components.
  • Warped images are generated from optical flow using bilinear interpolation, with out-of-bound values filled by zeros.
  • The iterative network alternates optical-flow estimation with depth and camera-motion estimation to refine predictions.
  • Depth derived from optical flow and camera motion is scaled consistently with the estimated camera motion.
  • Training proceeds for 3200k iterations through sequential encoder-decoder, iterative-net, and refinement-net stages.
  • Loss weights emphasize flow, depth gradients, and camera rotation alongside depth, normals, and translation losses.

C. Datasets

The training data combines synthetic and real datasets to balance perfect ground truth, natural imagery, and scene diversity. Dataset limitations include scale ambiguity, restricted diversity, synthetic artifacts, and costly scene annotation.

  • Training uses five datasets spanning synthetic and real imagery because neither category alone provides perfect data properties.
  • SUN3D and RGB-D SLAM provide indoor sequences, while MVS contributes real outdoor imagery with reconstructed depth and poses.
  • The COLMAP test scenes comprise building exteriors and a sculpture scene reconstructed with computed depth and camera poses.
  • Several datasets retain scale or diversity limitations, including unknown absolute scale and small or non-diverse collections.
  • Synthetic Scenes11 offers perfect camera poses and depth but has an artificial appearance and simplistic camera-motion model.
  • Blendswap contains 150 varied scenes with realistic ground truth, but adding scenes requires time-consuming manual annotation.

D. Experiments with Ground Truth

Ground-truth experiments test the relationship among optical flow, depth, and motion and evaluate iterative refinement. Accurate ground-truth inputs substantially improve cross-modal estimation, while refinement improves depth and motion through iteration 3.

  • DeMoN alternates optical-flow estimation with depth and camera-motion estimation to progressively refine predictions.
  • Providing ground-truth optical flow or ground-truth depth and motion dramatically improves the corresponding estimates.
  • DeMoN remains far below the ground-truth-input upper bound, indicating that image-based estimation is the main difficulty.
  • Both depth and motion accuracy significantly improve through iteration 3, with no significant gains beyond three iterations.
  • On an MVS example, iterative refinement reduces building distortion in the reconstructed point cloud.

F. Error Distributions

DeMoN's error distributions are competitive with engineered baselines, with dataset-dependent trade-offs between accuracy and outlier resistance. Its motion estimates are especially strong on SUN3D, where baseline methods produce more failures.

  • The reported depth means average over pixels, whereas the main-table averages are computed over test samples.
  • Base-Oracle produces many very accurate depth estimates on highly textured MVS, whereas DeMoN's errors are more spread.The difference is attributed to Base-Oracle's higher-resolution input and use of ground-truth motion.
  • On SUN3D, Base-Oracle suffers significantly more depth outliers than DeMoN, although higher depth accuracy can matter more for some tasks.The distributions are more similar because homogeneous regions make matching difficult.
  • On MVS, Base-FF is slightly more accurate for rotation, while DeMoN is more accurate for translation; Base-Mat-F is less accurate than both.
  • On SUN3D, DeMoN estimates motion more accurately than both comparisons, while Base-FF produces rotation and translation outliers.DeMoN and Base-Mat-F have almost no rotation outliers on SUN3D.

G. Qualitative Depth Estimation Results

Qualitative comparisons show that DeMoN produces smooth depth maps while preserving sharp edges across several datasets. The method also handles thin objects, but moving people and dataset-specific generalization remain visible boundary cases.

  • DeMoN produces smooth depth estimates while preserving sharp edges across SUN3D, RGB-D SLAM, MVS, Scenes11, and NYUv2.The advantage is especially visible in qualitative point-cloud comparisons.
  • On NYUv2, DeMoN fails to predict upper bodies when people move between the two frames.
  • On SUN3D, Base-Oracle performs badly because of inaccurate motion ground truth, while Eigen-VGG works well for many images.
  • DeMoN handles very thin objects in RGB-D SLAM qualitative comparisons.
  • On MVS, single-image methods generalize poorly to the new dataset and produce coarse contour outliers, while Base-Oracle performs well.Most outliers occur in regions not visible in the second image.
  • DeMoN and Base-Oracle give the best qualitative depth results on Scenes11.
Loading 1612.02401v2…