PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding

Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia

arXiv:1806.10556v2cs.CV

TL;DR

Single-image geometry learning from unlabeled video commonly models camera motion but not moving objects. This paper adds dense 3D object motion to a self-supervised geometry framework, and reports state-of-the-art KITTI results across depth, scene flow, and moving-object segmentation evaluations.

  • Problem

    Prior geometry-learning approaches model rigid camera ego motion, while moving objects in videos also need to be represented.

    Source: §1 · pp. 1–2

  • Method

    The framework combines optical flow, camera motion, moving-object masks, and depth in a holistic 3D motion parser for joint geometry and dense object-motion learning.

    Source: §1 · p. 2, §5 · p. 13

  • Results

    On KITTI, the approach achieves state-of-the-art performance on all depth, scene flow, and moving object segmentation evaluations.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    Consistency between depth and 2D optical flow provides helpful supervision for depth learning.

    Source: §5 · p. 14

Abstract

from arXiv · show

Learning to estimate 3D geometry in a single image by watching unlabeled videos via deep convolutional network has made significant process recently. Current state-of-the-art (SOTA) methods, are based on the learning framework of rigid structure-from-motion, where only 3D camera ego motion is modeled for geometry estimation.However, moving objects also exist in many videos, e.g. moving cars in a street scene. In this paper, we tackle such motion by additionally incorporating per-pixel 3D object motion into the learning framework, which provides holistic 3D scene flow understanding and helps single image geometry estimation. Specifically, given two consecutive frames from a video, we adopt a motion network to predict their relative 3D camera pose and a segmentation mask distinguishing moving objects and rigid background. An optical flow network is used to estimate dense 2D per-pixel correspondence. A single image depth network predicts depth maps for both images. The four types of information, i.e. 2D flow, camera pose, segment mask and depth maps, are integrated into a differentiable holistic 3D motion parser (HMP), where per-pixel 3D motion for rigid background and moving objects are recovered. We design various losses w.r.t. the two types of 3D motions for training the depth and motion networks, yielding further error reduction for estimated geometry. Finally, in order to solve the 3D motion confusion from monocular videos, we combine stereo images into joint training. Experiments on KITTI 2015 dataset show that our estimated geometry, 3D motion and moving object masks, not only are constrained to be consistent, but also significantly outperforms other SOTA algorithms, demonstrating the benefits of our approach.

1 Introduction

Rigid-scene self-supervision can mis-handle moving objects, leaving reconstruction errors that destabilize depth learning. The paper models dense 3D motion alongside camera motion, depth, and object masks, and reports improved geometry and state-of-the-art results across evaluated tasks.

1 Introduction

  • With good depth and camera estimates, moving objects can still cause reconstruction error and unnecessary losses that destabilize learning.The framework’s joint 3D-flow and depth training is shown with less reconstruction error and consistent scene-flow and segmentation results.Source: §1 · p. 2, Figure 1 · p. 3
  • Dense per-pixel 3D scene flow, fused with camera pose, depth, optical flow, and a moving-object mask, models rigid background and moving-object motion together.The HMP recovers separate 3D motion for rigid background and dynamic objects.Source: §1 · p. 2
  • A differentiable HMP computes rigid and occlusion-aware 3D flows; their residual should be zero in visible rigid regions and spatially smooth within moving-object regions.These principles guide additional losses for end-to-end depth and motion learning.Source: §1 · p. 2
  • Monocular depth and motion can be confused under a projective camera model, so the framework adds stereo pairs when available and reports better joint-training results than either source alone.The paper describes a far moving object and a close relatively stationary object as potentially equivalent under the model.Source: §1 · pp. 2–3
  • The approach significantly outperforms other state-of-the-art methods on KITTI 2015 depth, 3D scene flow, and moving-object segmentation evaluations.The introduction also reports more accurate learned geometry and a more stable learning process.Source: §1 · p. 3

2 Related Work

Prior geometry methods include matching-based reconstruction, supervised depth networks, and self-supervised video or stereo approaches, with different assumptions and data needs. This paper’s related-work discussion places dense 3D motion estimation alongside scene-flow and moving-object segmentation research.

2 Related Work

  • Matching-based geometry methods can fail when image texture is low or perspective changes drastically, and several dynamic-scene methods require regularization assumptions such as rigidity or a fixed object count.The cited methods also do not extend to single-view reconstruction.Source: §2 · p. 3
  • Supervised CNN depth methods estimate dense geometry but require labeled ground truth that is expensive to obtain in natural environments.Source: §2 · p. 4
  • Self-supervised depth methods use stereo pairs or monocular videos, but most video-based models assume rigid scenes; this paper instead incorporates dense 3D motion into joint depth and motion learning.Source: §2 · p. 4
  • Scene-flow estimation evaluates both depth and optical-flow quality, and the paper reports results comparable to state-of-the-art algorithms using single-image depth and unsupervised optical flow.Source: §2 · p. 4
  • Unlike prior unsupervised moving-object segmentation approaches described here, the framework uses two consecutive images and 3D motion awareness rather than saliency or long trajectories.Source: §2 · p. 4
  • The framework pipeline combines optical flow, camera pose, moving-object masks, and independently estimated depths in an HMP that produces rigid and dynamic 3D motion maps.Source: Figure 2 · p. 5

3 Geometry Learning via Holistic 3D Motion Understanding

This section models rigid-background and moving-object 3D motion together to supervise geometry learning. It also combines stereo samples with monocular training to address depth confusion from moving objects.

3.1 Scene geometry with 3D motion understanding

  • The geometry model projects pixels between views using depth, camera pose, and object motion, with visibility masking occluded or out-of-frame pixels.The resulting correspondences support photometric supervision of the networks.Source: §3.1 · p. 5
  • The holistic 3D motion parser uses two-image depths, optical flow, camera pose, and a moving-object mask to separate rigid and dynamic motion.It computes rigid-background motion and an occlusion-aware full-image motion, then uses their difference in rigid regions and smoothness within moving regions as supervision principles.Source: §3.1 · p. 6

3.2 Training the networks.

  • Training compares rigid-background motion with learned optical flow, smooths object motion, and discourages masks that label every pixel as moving.Depth prediction also receives depth-normal consistency and edge-aware depth and normal smoothness regularization.Source: §3.2 · p. 7
  • Bidirectional view synthesis supplies intermediate supervision for both target and source images to address slower, harder convergence from scratch.Its similarity measure combines photometric distance and SSIM.Source: §3.2 · pp. 7–8
  • Stereo training resolves moving-object depth confusion that remains with monocular training.The paper reports that stereo-trained predictions correctly estimate moving-object regions.Source: Figure 3 · p. 8, §3.2 · p. 8

4 Experiments

The experiments cover depth, scene-flow, and moving-object segmentation, using KITTI 2015 for evaluation and Cityscapes for additional training data.

4.1 Implementation details

  • DepthNet uses a DispNet-like architecture, FlowNet uses PWC-Net, and MotionNet uses a U-net; HMP losses then finetune DepthNet and MotionNet.DepthNet and MotionNet are first trained separately from FlowNet.Source: §4.1 · p. 9

4.2 Datasets and metrics

  • Training uses 156 KITTI stereo videos (22,000 samples) and 18 Cityscapes videos (34,652 samples); Cityscapes has no depth ground truth.Cityscapes is used only for training.Source: §4.2 · pp. 9–10
  • Depth is evaluated on KITTI and Eigen splits, while scene flow and segmentation are evaluated on the 200-image KITTI test split.The scene-flow evaluation rescales estimated depth so its median matches the ground-truth depth median.Source: §4.2 · p. 10, §4.4 · pp. 11–12

4.3 Depth evaluation

  • Using stereo and sequential samples improves depth results over either sample type alone, and HMP finetuning further improves performance.The compared variants use the same amount of data.Source: §4.3 · p. 11
  • On KITTI-trained models, our method outperforms unsupervised SOTA methods on almost all depth metrics; with CS+K, it remains SOTA on Abs Rel.The comparison covers KITTI and Eigen depth splits.Source: §4.3 · p. 11, Table 2 · p. 12

4.4 Scene flow evaluation

  • Both stereo training and HMP finetuning improve scene-flow performance; applying the scene-flow solver further improves the Ours w/o HMP variant.Table 3 reports KITTI 2015 scene-flow results for supervised and unsupervised methods.Source: §4.4 · p. 12, Table 3 · p. 13

4.5 Moving object segmentation

  • The motion-mask segmentation outperforms masks used in depth reconstruction and masks computed from scene-flow residuals.The motion mask focuses on moving objects, unlike explainability masks optimized to filter reconstruction error.Source: §4.5 · p. 13

5 Conclusion

The paper concludes that its self-supervised framework jointly learns 3D geometry and dense object motion, with depth–optical-flow consistency helping supervise depth learning. On KITTI, it reports state-of-the-art performance across depth, scene flow, and moving-object segmentation evaluations.

  • The proposed self-supervised framework jointly learns 3D geometry and dense object motion, modeling consistency between depth and 2D optical-flow estimation to help supervise depth learning.Source: §5 · pp. 13–14
  • The approach achieves state-of-the-art performance on KITTI depth, scene flow, and moving-object segmentation evaluations.Source: §5 · p. 14, Table 4 · p. 14, Figure 6 · p. 14
  • The authors identify extension to datasets with deformable and articulated non-rigid objects as future work.Source: §5 · p. 14
Loading 1806.10556v2…