Deep Rigid Instance Scene Flow

1. Introduction

• Prior piece-wise rigid scene-flow methods used ego-car and rigid-object motion priors but required minutes per frame, precluding real-world robotics use.Deep low-level methods ran in real time but did not structure estimates to ensure consistency across pixels on an object.Source: §1 · p. 1
• DRISF models scene motion as the 3D rigid motion of each actor, with the static background represented by ego-car motion.It uses deep networks to estimate optical flow, disparity, and instance segmentation, then encodes geometric relationships as energy functions optimized with a Gaussian-Newton solver on GPU.Source: §1 · pp. 1–2
• DRISF reports 22% lower scene flow error and 800 times faster runtime than the existing best scene flow model.On KITTI, it also reduces D1 outliers by 43%, D2 outliers by 32%, and flow outliers by 24%.Source: §1 · p. 2

Optical flow

• Deep optical-flow methods improve flow prediction, but deep-feature matching may require post-processing, while PWC-Net is the flow module used in this work.SpyNet reduces model size at a performance cost; Lite-Flownet and PWC-Net are described as smaller and faster than earlier approaches.Source: §2 · p. 2

Stereo

• Deep stereo methods use learned matching or direct disparity regression, and this work uses PSM-Net as its stereo module.The related-work discussion describes 3D convolution, stacked hourglass, and pyramid spatial pooling as components of PSM-Net.Source: §2 · pp. 2–3

Scene flow

• The paper’s deep structured motion estimation model computes scene flow in less than a second, two to three orders of magnitude faster than most prior methods.The cited prior scene-flow approaches include piecewise-rigid methods and conditional-random-field refinement, which are described as computationally expensive.Source: §2 · p. 3

3. Deep Rigid Instance Scene Flow

• DRISF models each traffic participant and the static background as a rigidly moving instance, using optical flow, stereo, and instance segmentation as visual cues.The background motion represents ego-car motion.Source: §3 · p. 3, §3.1 · p. 3

3.1. Visual Cues

• Mask R-CNN provides instance masks, PSM-Net estimates stereo disparity, and a PWC-Net-based module estimates optical flow.The flow module pads out-of-image warping samples with the nearest boundary feature rather than zero; the authors report this improves performance.Source: §3.1 · p. 4

3.2. Energy Formulation

• For each instance, DRISF minimizes a weighted combination of photometric error, rigid fitting, and flow consistency to estimate 3D motion.The terms capture geometry and appearance agreement; the model uses a robust penalty to reduce outlier influence.Source: §3.2 · pp. 4–5

3.3. Inference

• Inference removes likely occlusions and RANSAC outliers, then initializes the non-convex energy minimization with the fitted rigid motion.The method unrolls Gaussian-Newton optimization as a recurrent neural network, with a differentiable computation graph.Source: §3.3 · pp. 5–6
• The estimated instance motions yield dense scene flow represented by first-frame stereo, warped second-frame stereo, and instance-wise rigid flow.The rigid-motion analysis reports over 80% of estimates have error below 1 m and 1.3°; larger errors often occur for distant, small vehicles with fewer observable points. Background motion yields average ego-car drift of 0.9 cm and 0.024° per 1 m of drive.Source: §3.3 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6

3.4. Learning

• The component networks are trained through back-propagation, with Mask R-CNN, PSM-Net, and PWC-Net each fine-tuned on KITTI after pretraining.Their pretraining sources and training losses differ by network.Source: §3.4 · p. 6

4.1. Dataset and Implementation Details

• KITTI provides 200 training and 200 test sets, split into train and validation at 4:1; foreground uses all energy terms, while background uses photometric error alone.RANSAC initializes the solver, which is unrolled for up to 50 Gauss–Newton steps and often reaches its best energy within 10.Source: §4.1 · p. 7

4.2. Scene Flow Estimation

• 43% fewer D1 outliers, 32% fewer D2 outliers, and 24% fewer flow outliers; versus ISF, scene-flow error is 22% lower and runtime is 800 times faster.The authors report state-of-the-art performance on almost every benchmark entry.Source: §4.2 · p. 7
• Qualitative examples show robust motion estimates under illumination changes and occlusion, but missed or inaccurate instance segmentation causes failures on the train and a middle-scene car.For an occluded textureless car, the method estimates motion from flow and disparity of its remaining visible part.Source: §4.2 · p. 7

4.3. 3D Rigid Motion Estimation

• Over 80% of vehicles have translation error below 1 m and angular error below 1.3°; ego-car odometry drifts 0.09 m and 0.24° every 10 m.Most vehicles with translation error above 1 m are at least 20 m away; the odometry result uses no pose filter or loop closure.Source: §4.3 · p. 8

4.4. Analysis

• All energy terms work best for foreground objects, while the background has lowest error with photometric error alone.Textureless vehicles with large displacements benefit from flow and rigid terms; adding terms may add noise in the more textured, lower-motion background.Source: §4.4 · pp. 8–9
• DRISF improves original flow/stereo outputs; replacing disparity and flow with ground truth reduces scene-flow error by 8% and 21%, while ground-truth segmentation does not improve results.The comparison with original flow/stereo is shown in Table 3.Source: Table 3 · p. 7, §4.4 · p. 9
• Stereo takes more than 50% of inference time, largely due to 3D CNN cost aggregation and stacked-hourglass refinement; runtime also depends on solver steps and points considered.The pipeline comprises visual-cue extraction, occlusion reasoning, and optimization; modules within a stage can run in parallel, with stage time determined by its slowest module.Source: Table 4 · p. 7, §4.4 · p. 9
• If segmentation misses a vehicle, DRISF treats it as background and assigns inverse ego-car motion; if optical flow fails completely, the flow-centric solver may not recover correct motion.The authors identify joint segmentation and scene-flow reasoning and additional flow-invariant energy terms as possible directions.Source: §4.4 · p. 9

• The model focuses on self-driving scenes, representing motion through the 3D rigid motion of each actor.Source: §5 · p. 9
• Deep learning extracts visual cues for each instance, while geometry-based energy functions encode their structural relationships to estimate scene flow.Source: §5 · p. 9
• On KITTI, the method achieves state-of-the-art performance, is reported as 22% better than prior art, and runs two to three orders of magnitude faster.Source: §5 · p. 9