PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, Alan Yuille

arXiv:1810.06125v2cs.CV

TL;DR

Existing methods often separate depth and optical-flow learning and assume rigid scenes, despite moving objects and monocular scale confusion. EPC++ jointly learns depth, camera motion, optical flow, and moving-object segmentation with adaptive geometric consistency, using a holistic motion parser and stereo training. It reports state-of-the-art performance across five tasks on driving and non-driving datasets.

  • Problem

    Existing methods separate depth and optical-flow learning, while dynamic objects and monocular projective scale confusion complicate joint geometry and motion recovery.

  • Method

    EPC++ jointly estimates depth, camera motion, optical flow, and moving-object segmentation through adaptive consistency and a holistic motion parser, with stereo pairs used during training.

  • Results

    EPC++ achieves state-of-the-art performance across five tasks on KITTI and generalizes to Make3D and MPI-Sintel.

  • Takeaways & Limitations

    Joint geometric consistency supports a single framework for depth, motion, flow, segmentation, and scene understanding across driving and non-driving scenes.

  • Takeaways & Limitations

    The reported odometry metric can show only a small quantitative gap despite a large qualitative performance difference, and the authors identify future work on deformable and articulated objects.

Abstract

from arXiv · show

Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e. to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as "Every Pixel Counts++" or "EPC++". Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods. Code will be available at: https://github.com/chenxuluo/EPC.

1 INTRODUCTION

EPC++ jointly estimates depth, camera motion, optical flow, and moving-object segmentation from unlabeled videos, addressing dynamic scenes and enforcing geometric consistency across tasks. Its holistic parser and adaptive training strategy support strong performance across five tasks and multiple scene types.

  • Motivation: Existing depth methods assume static scenes, while moving objects violate rigid-scene assumptions and can degrade geometry learning.Prior approaches either ignore moving and occluded pixels or impose limits on the number of modeled objects.
  • Motivation: Optical-flow methods model rigid and non-rigid 2D motion but lack underlying 3D geometry, making prediction regularization difficult in occluded regions.Some joint methods still fail to model rigid/non-rigid motion, occlusions, or depth-flow consistency.
  • Approach: EPC++ jointly estimates per-pixel depth, camera pose, optical flow, and moving-object segmentation using adaptive consistency from unlabeled videos.On non-occluded pixels, flow guides depth and motion; in occluded regions, depth and motion help estimate flow.
  • Approach: The holistic motion parser recovers visibility, moving-object masks, and separate 3D motions for rigid background and moving objects using geometric rules and consistency.Its inputs are forward/backward flow, camera motion, and depth maps from consecutive frames.
  • Approach: Stereo image pairs are incorporated during training to address the ill-posed monocular depth-and-motion problem caused by projective scale confusion.A close object moving with the camera can appear equivalent to a far stationary object under monocular observation.
  • Results: EPC++ achieves state-of-the-art results across five tasks on KITTI and generalizes to Make3D and MPI-Sintel.The evaluated tasks include depth, optical flow, 3D scene flow, camera motion, and moving-object segmentation; adaptive learning is used to train coupled geometrical information.

2 RELATED WORK

Related work spans traditional and learned approaches to depth, optical flow, and 3D scene flow, with increasing efforts to combine these tasks. Existing methods remain limited by rigid-scene assumptions, incomplete motion or occlusion handling, and insufficient depth–flow consistency.

  • Depth and geometry: Traditional depth and motion methods use feature or patch matching, SfM, SLAM, and stereo, while early single-view methods rely on handcrafted scene assumptions and often produce sparse geometry.Examples include vanishing points, BRDF assumptions, and Manhattan-world layouts.
  • Depth and geometry: Unsupervised depth methods learn from unlabeled videos through view synthesis based on rigid structure from motion.A source image is warped to a target using predicted target depth and relative camera motion, with reconstruction error providing supervision.
  • Optical flow: Unsupervised optical-flow methods train with image synthesis and smoothness, but earlier systems lack explicit 3D scene-geometry understanding such as depth and camera motion.Occlusion masks can improve robustness by omitting photometric errors in occluded regions.
  • 3D scene flow: 3D scene flow estimates dense per-pixel 3D motion from image pairs and has been addressed using stereo, rigid constraints, piecewise moving planes, or supervised object-aware methods.Recent KITTI methods use semantic instance segmentation and supervised stereo disparity to handle large object displacements.
  • Joint depth and optical flow: Joint depth–flow approaches estimate residual or composite flow, yet prior work did not consistently benefit depth estimation or enforce 3D motion consistency.Some methods separately combine object and rigid-background flow rather than measuring 3D consistency.

3 LEARNING WITH HOLISTIC 3D MOTION UNDER-

EPC++ uses a holistic 3D motion parser to combine depth, camera motion, and optical flow, disentangle rigid and dynamic motion, and support joint learning with geometry-aware consistency.

  • 3.1 Geometrical understanding with 3D motion: For each target pixel, optical-flow correspondence and geometry-based projection should identify consistent source pixels across the two frames.The framework uses target/source depths, camera transformation, and optical flow to establish this relationship.
  • 3.1 Geometrical understanding with 3D motion: Monocular moving-object reconstruction is ill-posed because depth and object motion can compensate for one another, producing projective scale confusion.A nearby object moving with the camera can appear equivalent to a distant relatively static object.
  • 3.1 Geometrical understanding with 3D motion: EPC++ predicts depth, camera motion, and optical flow, then uses HMP to recover visibility, moving-object masks, and rigid and dynamic 3D motion.The three networks provide the inputs to the holistic motion parser.
  • 3.1 Geometrical understanding with 3D motion: Rigid background motion is computed from camera transformation and depth, while dynamic motion is recovered from flow-based 3D displacement after subtracting rigid motion.The back-projection function maps image pixels and depths into 3D camera coordinates.
  • 3.1 Geometrical understanding with 3D motion: HMP separates rigid and dynamic motion and estimates moving-object masks, enabling structural losses and regularization for jointly learning depth, motion, and flow.The parser also distinguishes visible from occluded regions for applying different supervision.
  • 3.2 Training losses: EPC++ uses view-synthesis, smoothness, rigid-aware 3D motion consistency, and occluded-region flow consistency losses, with masks selecting suitable training regions.The visibility and moving-object masks determine whether depth or flow supervision is applied.
  • 3.2.3 Adaptive stage-wise learning strategy: Joint training from scratch is unstable because randomly initialized coupled networks produce noisy masks and motion estimates, so EPC++ adopts adaptive stage-wise training.The strategy first trains DepthNet/MotionNet and OptFlowNet separately, then introduces consistency and alternates optimization with adaptive masks.
  • 3.3 Stereo to solve motion confusion: Stereo training addresses monocular projective confusion, correcting depth estimates for moving-object regions that monocular view synthesis can place artificially far away.The paper illustrates this correction by comparing monocular-trained and stereo-trained predictions.

4 EXPERIMENTS

EPC++ is evaluated across depth, optical flow, odometry, segmentation, and scene flow, with ablations and cross-dataset tests showing benefits from joint consistency and stereo training.

  • Depth evaluation: Adding stereo and sequential samples produces a large depth-performance boost over training with either sample type alone.The authors attribute the stereo benefit to removing monocular scale ambiguity during training.
  • Depth evaluation: EPC++ outperforms recent unsupervised methods on all reported single-view depth metrics, with depth-flow consistency reducing large prediction errors.The comparison uses KITTI Eigen test results; the squared-error metrics Sq Rel and RMSE show especially large gains after consistency is applied.
  • Depth evaluation: On MPI-Sintel, adding depth-flow consistency, visibility and motion masks, and adaptive training improves AbsRel from 0.866 to 0.524 and SqRel from 25.558 to 5.3206.The comparison is between EPC++ mono depth only and EPC++ mono on the training split after fine-tuning.
  • Optical Flow Evaluation: Explicitly modeling visibility and motion masks improves optical flow in both occluded and overall regions.Ablations comparing EPC++ vis-rigid and EPC++ non-vis show gains for both occ and all regions under stereo and monocular training.
  • Optical Flow Evaluation: EPC++ mono outperforms unsupervised monocular flow methods on KITTI 2015 F1-bg and F1-all, while stereo training further improves performance.The method uses two frames, whereas Multi-frame uses three; EPC++ still reports better full-region results.
  • Odometry estimation: The coupled pipeline yields strong odometry results, including a large margin over previous works on full-sequence translation and rotation errors.The authors link the improvement to depth and optical flow helping regularize jointly trained ego-motion.

5 CONCLUSION

EPC++ jointly estimates depth, camera motion, optical flow, and moving-object segmentation by enforcing geometric consistency across tasks. Experiments report state-of-the-art performance across driving and non-driving scenes.

  • EPC++ jointly estimates depth, camera motion, optical flow, and moving object segmentation masks in an end-to-end unsupervised framework.
  • The framework explicitly models motion rigidity and visibility so each pixel is explained as rigid, non-rigid/object motion, or occluded.
  • Adaptive training leverages depth and optical flow according to their differing strengths, outperforming uniform across-task consistency.
  • EPC++ achieves state-of-the-art performance across related tasks on driving scenes from KITTI and non-driving scenes from Make3D and MPI-Sintel.
  • The authors identify deformable and articulated nonrigid objects and multiple object segmentation as future extensions.
Loading 1810.06125v2…