PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Monocular Scene Flow Estimation

Junhwa Hur, Stefan Roth

arXiv:2004.04143v2cs.CVcs.LGcs.RO

TL;DR

Monocular scene flow must recover both 3D structure and motion from two images, an ill-posed task. The paper uses a self-supervised CNN that decomposes an optical-flow cost volume into depth and scene flow, and reports state-of-the-art accuracy among unsupervised/self-supervised monocular methods, with further gains from semi-supervised fine-tuning.

  • Problem

    Monocular scene flow is highly ill-posed because depth and per-pixel 3D motion must both be estimated from two consecutive monocular frames.

    Source: §1 · p. 1

  • Method

    A single CNN uses an inverse-problem approach to decompose a classical optical-flow cost volume into depth and 3D scene flow, trained self-supervised with 3D losses and occlusion reasoning.

    Source: §1 · pp. 1–2

  • Results

    34.0% accuracy increase over previous unsupervised/self-supervised monocular scene-flow approaches; the semi-supervised model approaches the best method's accuracy while being orders of magnitude faster.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A single joint decoder provides state-of-the-art scene-flow accuracy among un-/self-supervised monocular methods without the complex training schedules of multi-task approaches.

    Source: §5 · p. 8

    The 3D scene-flow regularizer is not ideal for optical-flow accuracy compared with a 2D optical-flow regularizer.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Scene flow estimation has been receiving increasing attention for 3D environment perception. Monocular scene flow estimation -- obtaining 3D structure and 3D motion from two temporally consecutive images -- is a highly ill-posed problem, and practical solutions are lacking to date. We propose a novel monocular scene flow method that yields competitive accuracy and real-time performance. By taking an inverse problem view, we design a single convolutional neural network (CNN) that successfully estimates depth and 3D motion simultaneously from a classical optical flow cost volume. We adopt self-supervised learning with 3D loss functions and occlusion reasoning to leverage unlabeled data. We validate our design choices, including the proxy loss and augmentation setup. Our model achieves state-of-the-art accuracy among unsupervised/self-supervised learning approaches to monocular scene flow, and yields competitive results for the optical flow and monocular depth estimation sub-tasks. Semi-supervised fine-tuning further improves the accuracy and yields promising results in real-time.

1. Introduction

Monocular scene flow seeks 3D structure and motion from two images, but jointly estimating depth and per-pixel 3D motion is highly ill-posed. The paper proposes a self-supervised CNN approach and reports a 34.0% accuracy increase over prior unsupervised/self-supervised monocular scene-flow methods.

1. Introduction

  • Monocular scene flow is highly ill-posed because two consecutive monocular frames must provide both depth and per-pixel 3D motion, while earlier approaches lacked both reasonable accuracy and real-time performance.Other sensor setups have their own constraints, including stereo calibration, expensive 3D sensing, or indoor-use limitations for RGB-D cameras.Source: §1 · p. 1
  • The method trains directly on the target domain with unlabeled data to address scarce 3D-motion ground truth and synthetic-dataset domain overfitting.Limited ground-truth data can optionally be used for semi-supervised fine-tuning.Source: §1 · p. 1
  • A single joint decoder decomposes a classical optical-flow cost volume into depth and 3D scene flow, simplifying joint estimation relative to multi-task CNN approaches.The network also outputs optical flow by projecting scene flow to 2D; the paper reports better depth and scene-flow accuracy with a simpler training setup.Source: §1 · pp. 1–2
  • The self-supervised method combines view-synthesis, 3D reconstruction, and occlusion-aware losses with an augmentation compromise validated by ablation.The geometric augmentations for the two tasks conflict, so the authors use an ablation study to select a suitable compromise.Source: §1 · p. 2
  • 34.0% accuracy increase over previous unsupervised/self-supervised monocular scene-flow approaches after training on unlabeled KITTI raw data and evaluating on KITTI Scene Flow.Semi-supervised fine-tuning moves accuracy closer to the semi-supervised method of while being orders of magnitude faster.Source: §1 · p. 2

2. Related Work

Prior scene-flow work spans stereo, RGB-D, point-cloud, and monocular inputs, with monocular estimation posing a more challenging joint structure-and-motion problem. Existing monocular methods include variational, hybrid energy-based, and multi-task CNN pipelines, while this paper contrasts its direct CNN-based scene-flow approach with those alternatives.

2. Related Work

  • Scene flow is commonly a dense 3D motion field; early variational methods had limited accuracy and long runtimes.The usual setup jointly estimates 3D structure and motion from stereo images.Source: §2 · p. 2
  • Piecewise-planar surfaces with rigid motion improved accuracy, especially in traffic, and semantic knowledge of rigidly moving objects brought further gains.These developments followed early variational approaches.Source: §2 · p. 2
  • Supervised CNN scene-flow methods use large synthetic datasets and limited in-domain data for state-of-the-art accuracy and real-time performance, while un-/self-supervised methods were developed to avoid ground-truth scarcity but lagged in accuracy.The paper situates its learning approach within this accuracy-versus-data context.Source: §2 · p. 2
  • Monocular scene flow jointly estimates 3D structure and motion from monocular images, a more challenging setup than using RGB-D or point-cloud structure cues.Scene-flow methods also use stereo image sequences.Source: §2 · p. 2
  • Earlier monocular methods included a variational approach requiring an initial depth cue and a CNN-plus-energy pipeline using pseudo-labeled depth and piecewise-rigid motion.The latter achieved state-of-the-art accuracy; the paper contrasts it with a purely CNN-based, real-time, end-to-end self-supervised method.Source: §2 · p. 2
  • Multi-task monocular CNNs estimate depth, optical flow, camera motion, and masks, but reconstructed scene flow is limited in occluded regions.A pipeline relating two depth maps and optical flow cannot recover 3D motion for occluded pixels because the second-frame depth is unknown; direct CNN scene-flow prediction bypasses this issue.Source: §2 · p. 2

3. Self-Supervised Monocular Scene Flow

The method jointly estimates 3D structure and motion from monocular image pairs by decomposing optical-flow correspondence evidence, while addressing the depth–motion ambiguity through learned depth and self-supervised losses. Its augmentation study finds different choices benefit monocular depth and scene flow.

3.1. Problem formulation

  • Monocular scene flow jointly estimates each pixel’s 3D point and motion from consecutive frames, but back-projecting optical flow leaves depth and motion ambiguous.The method treats optical flow as a 2D projection and uses its cost volume as an intermediate representation for recovering 3D structure and motion.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2. Network architecture

  • A single decoder jointly outputs scene flow and disparity, adapting PWC-Net’s pyramid to project updated scene flow into optical flow for the next-level feature warp.The decoder estimates residual scene flow but non-residual disparity; disparity is on an absolute scale using stereo guidance during training, while inference remains monocular.Source: §3.2 · p. 3, §3.3 · p. 3, Figure 3 · p. 4

3.3. Addressing the scale ambiguity

  • Stereo images and known camera geometry provide absolute-scale disparity supervision during training, enabling scene flow estimation on an absolute scale from monocular input at test time.Depth is recovered from disparity using the stereo baseline and focal length; stereo images are not used at test time.Source: §3.3 · p. 3

3.4. A proxy loss for self-supervised learning

  • The self-supervised objective combines disparity and scene-flow losses, using photometric reconstruction, 3D point reconstruction, smoothness, and occlusion reasoning.The model predicts forward and backward scene flow by reversing input order; stereo images guide the training loss but are not used at test time.Source: §3.4 · pp. 3–5

3.5. Data augmentation

  • Depth accuracy improves only with CAM-Convs, whereas scene-flow accuracy improves with geometric augmentation without CAM-Convs, as summarized in Table 1.The depth study reports that geometric augmentation deteriorates accuracy with or without CAM-Convs; the scene-flow study finds CAM-Convs hurt accuracy, so subsequent experiments use augmentation alone.Source: Table 1 · p. 5, §3.5 · p. 5

4. Experiments

The experiments evaluate a self-supervised monocular scene-flow model and its semi-supervised fine-tuning, with ablations of its proxy loss and decoder design. The model outperforms multi-task CNN methods on scene flow, while its fine-tuned version approaches a leading monocular method with substantially faster runtime; depth and flow results are also competitive.

4.1. Implementation details

  • The model is evaluated using KITTI raw data for training and KITTI Scene Flow Training for testing and ablation.The KITTI Scene Flow Training test set provides disparity and scene-flow ground truth for 200 images.Source: §4.1 · p. 6
  • Unlike approaches requiring pretraining or iterative training of multiple CNNs, this model can be trained from scratch all at once.The authors present this as a practical training advantage.Source: §4.1 · p. 6

4.2. Ablation study

  • Adding 3D point reconstruction improves scene-flow accuracy, especially when occluded pixels are excluded from the loss.The authors explain that the 3D loss penalizes Euclidean distance between corresponding points, while occlusion handling avoids misleading reconstruction targets in occluded areas.Source: Table 2 · p. 6, §4.2 · pp. 6–7
  • A single decoder gives stable training and comparable or better depth and flow accuracy than separate task-specific decoders.Separate decoders yield unstable training and trivial disparity solutions.Source: Table 3 · p. 7, §4.2 · p. 7

4.3. Monocular scene flow

  • Self-Mono-SF significantly outperforms multi-task CNN methods on the KITTI Scene Flow Training scene-flow metric, SF1-all.Table 4 compares the methods on this benchmark.Source: Table 4 · p. 7, §4.3 · p. 7
  • Self-Mono-SF-ft moves closer to Mono-SF accuracy on KITTI Scene Flow 2015 while running more than 400× faster.The paper also reports that the self-supervised model is more than two orders of magnitude faster than Mono-SF.Source: §4.3 · p. 7, Table 5 · p. 7

4.4. Monocular depth and optical flow

  • On KITTI, the method outperforms published multi-task methods for monocular depth and is competitive on the Eigen split.Some comparison methods use ground truth to scale predictions at test time; the authors report that their method still outperforms them.Source: Table 6 · p. 8, §4.4 · p. 8
  • For optical flow, the method has accuracy comparable to existing monocular and stereo methods, while its 3D regularizer is not ideal for optimizing 2D flow.The authors contrast this trade-off with stronger scene-flow accuracy than EPC++.Source: Table 7 · p. 8, §4.4 · p. 8

5. Conclusion

The conclusion presents a PWC-Net-based CNN that directly predicts monocular 3D scene flow using a single joint decoder and self-supervised training. The approach achieves state-of-the-art scene-flow accuracy among unsupervised/self-supervised monocular methods, while semi-supervised fine-tuning approaches the best monocular method with much faster runtime.

5. Conclusion

  • The proposed PWC-Net-based CNN directly predicts 3D scene flow with a single joint decoder for depth and scene flow.The conclusion says this design avoids complex training schedules and addresses limitations including lack of occlusion handling.Source: §5 · p. 8
  • The self-supervised approach uses 3D losses and occlusion reasoning, and the authors report that a suitable augmentation scheme is critical for competitive accuracy.Source: §5 · p. 8
  • The model achieves state-of-the-art scene-flow accuracy among unsupervised/self-supervised monocular methods; fine-tuning approaches the best monocular method with orders-of-magnitude faster runtime.The authors describe the resulting method as a foundation for real-time CNN-based monocular scene-flow estimation and follow-up work.Source: §5 · p. 8

A. Learning Rate Schedule

The schedule describes 400k self-supervised training iterations followed by 45k semi-supervised fine-tuning iterations. Both stages halve the learning rate at specified iteration steps.

A. Learning Rate Schedule

B. Details on Data Augmentation

The section specifies separate augmentation protocols for self-supervised training and semi-supervised fine-tuning, including their shared test-time resizing. It also notes that better protocols may be found with further investigation.

B. Details on Data Augmentation

  • Self-supervised training applies photometric changes with 50% probability: gamma [0.8, 1.2], brightness [0.5, 2.0], and per-channel color [0.8, 1.2].Source: B. Details on Data Augmentation · p. 11
  • Geometric training augmentation randomly crops at scales [93%, 100%], translates by [−3.5%, 3.5%], resizes to 256 × 832, and flips horizontally with 50% probability.The camera intrinsic matrix is adjusted for the geometric transformations.Source: B. Details on Data Augmentation · p. 11
  • Figure 6 distinguishes learning-rate schedules for self-supervised learning and semi-supervised fine-tuning.Source: Figure 6 · p. 11
  • At test time, inputs are resized to 256 × 832 without photometric augmentation; better augmentation protocols may be found through further investigation [65].Source: B. Details on Data Augmentation · pp. 11–12
  • Semi-supervised fine-tuning uses photometric augmentation with 50% probability and random cropping at [94%, 100%], without scaling to avoid changing sparse ground truth.Source: B. Details on Data Augmentation · p. 11

C. Hyper-Parameter Settings

The section narrows proxy-loss tuning to two scene-flow parameters after setting the task balance dynamically and adopting prior disparity settings. It reports a trade-off for the 3D point reconstruction weight and low sensitivity to the smoothness weight.

C. Hyper-Parameter Settings

  • The six-parameter proxy loss leaves λsf_sm and λsf_pt for grid search after adopting prior disparity settings and dynamically balancing scene-flow and disparity losses each iteration [21].Source: C. Hyper-Parameter Settings · p. 12
  • The grid search compares monocular depth, optical-flow, and scene-flow accuracy while varying λsf_sm and λsf_pt in turn.Source: C. Hyper-Parameter Settings · p. 12
  • Higher λsf_pt supports more accurate scene flow through better target-frame disparity, but trades off depth and optical-flow accuracy; λsf_sm has little effect overall.Source: C. Hyper-Parameter Settings · p. 12

D. In-Depth Analysis of the Decoder Design

The decoder analysis finds that splitting the proposed shared decoder harms scene-flow accuracy and, with further splitting, training stability. The experiments progressively separate decoder components while keeping network capacity similar.

D. In-Depth Analysis of the Decoder Design

E. Qualitative Analysis of Loss Ablation Study

The qualitative ablation examines how occlusion handling and the 3D point reconstruction loss affect disparity, optical flow, and scene flow. The examples show errors without the 3D loss and artifacts when that loss is used without occlusion handling.

E. Qualitative Analysis of Loss Ablation Study

F. Qualitative Comparison

The qualitative comparison with Mono-SF shows stronger disparity and optical-flow estimates in some examples, but weaker scene-flow results in others. The weaker cases are linked to disparity accuracy and the difference in depth-training data.

F. Qualitative Comparison

  • The comparison uses Mono-SF, an integrated pipeline of CNNs and an energy-based model, and presents both successful and failed examples.Source: F. Qualitative Comparison · p. 14
  • Some examples show more accurate disparity and optical-flow estimates than Mono-SF, without an explicit planar surface representation or rigid-motion assumption.The comparison includes successful and failed fine-tuned-model examples on the KITTI 2015 Scene Flow public benchmark.Source: F. Qualitative Comparison · p. 14
  • In failure cases, optical-flow accuracy is comparable to Mono-SF, but inaccurate disparity leads to less accurate scene flow.The authors attribute the disparity gap to Mono-SF using over 20,000 pseudo-ground-truth depth instances, while their method uses 200 images for fine-tuning.Source: F. Qualitative Comparison · p. 14

Related papers

Loading 2004.04143v2…