Source-linked AI summary
SfM-Net: Learning of Structure and Motion from Video
Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, Katerina Fragkiadaki
TL;DR
SfM-Net addresses video motion estimation by learning geometrically interpretable scene, camera, and object motion from frame pairs. It achieves structure and motion prediction under limited supervision, while accounting for moving objects and enforcing forward-backward consistency.
Problem
Video-based structure and motion estimation must handle camera motion, object motion, depth, and moving objects while preserving image-formation geometry.
Method
SfM-Net predicts depth, camera and object motions, and pixel assignment masks, then uses geometric motion fields, differentiable warping, and forward-backward consistency for training.
Results
SfM-Net learns structure, object motion, and camera motion on KITTI, MoSeg, and RGB-D SLAM benchmarks using limited ground-truth annotations; stereo depth reaches a log RMSE of 0.31 on KITTI.
Takeaways & Limitations
Explicitly modeling moving objects improves video-based depth learning, especially on datasets containing more moving objects.
Takeaways & Limitations
Longer-range forward-backward consistency across multiple frame gaps produced no empirical gain in the reported experiments.
Abstract
from arXiv · showhide
We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.
1. Introduction
SfM-Net learns geometrically meaningful structure and motion from video by decomposing pixel motion into depth, camera motion, object motion, and masks. It supports self-supervised, camera-motion-supervised, and depth-supervised training and is evaluated across multiple benchmarks.
- Method: SfM-Net decomposes frame-to-frame pixel motion into scene depth, camera motion, object rotations and translations, and pixel assignment masks.These predictions generate an optical-flow field used to differentiably warp consecutive frames, while forward-backward consistency constrains structure.
- Training: SfM-Net supports completely unsupervised training, camera-motion supervision, or depth supervision from sources such as Kinect.Experiments examine these varying levels of supervision on KITTI, MoSeg, and RGB-D SLAM benchmarks.
- Training: Self-supervision trains the model through geometric scene-motion and image-formation constraints rather than requiring motion annotations.The model matches pixels across consecutive frames and uses forward-backward consistency on motion and 3D structure to provide training gradients.
- Predictions: The network predicts pixel-wise depth from a single frame together with camera motion, object motion, and object masks from a frame pair.This combines single-frame structure prediction with pairwise motion estimation in one deep network.
- Evaluation: The experiments evaluate structure, object motion, and camera motion prediction under different supervision levels using realistic video sequences with limited ground-truth annotations.The reported benchmarks are KITTI, MoSeg, and RGB-D SLAM.
2. Related work
SfM-Net builds on geometry-aware flow, differentiable warping, and learned motion estimation while extending prior approaches to dense structure and jointly estimated camera and object motion. Its explicit moving-object model addresses the static-world assumptions common in geometric methods.
- Back-propagating through warps and camera projection: Prior SE3-Nets predict rigid object motions and pixel-assignment masks under a static-camera assumption, whereas SfM-Net also estimates camera motion from video.SfM-Net retains a related 3D motion representation but removes the static-camera restriction described for SE3-Nets.
- Back-propagating through warps and camera projection: Earlier single-image landmark methods use synthetic pretraining, while SfM-Net estimates dense structure from videos to obtain self-supervision.The comparison distinguishes dense video-based structure estimation from sparse landmark prediction from static images.
- Back-propagating through warps and camera projection: Garg et al. predict depth from photometric error but require calibrated stereo pairs and do not infer camera or object motion.SfM-Net instead considers frame sequences in which camera and object motion must be estimated.
- Geometry-aware motion estimation: Geometric SLAM and related methods often assume a static world, making them susceptible to moving objects that SfM-Net represents with masks and rigid 3D motions.The paper contrasts static-world reconstruction with explicit moving-object modeling.
- Unsupervised learning in videos: Unlike unsupervised representation-learning methods aimed at pretraining recognition systems, SfM-Net extracts fine-grained 3D structure and motion from monocular videos with weak supervision.The distinction concerns the predicted representation: geometric structure and motion rather than semantic features.
3. Learning SfM
SfM-Net predicts depth, camera motion, rigid object motions, and pixel membership masks, then combines them geometrically to produce optical flow and differentiably warp frames. Training can use self-supervision, camera-motion supervision, or depth supervision, with forward-backward depth consistency supporting the self-supervised formulation.
- Architecture: SfM-Net predicts per-frame depth, camera motion, up to K rigid object motions, and pixel membership masks from frame pairs.Depth is predicted from a single frame, while camera and object motions are predicted from paired images.
- Optical flow: The model constructs optical flow by transforming depth-derived 3D points with object and camera motions, then projecting them through the camera intrinsics.Differentiable backward warping maps the next frame to the current frame using the resulting flow field.
- Architecture: Object masks weight rigid transformations per pixel, while pixels assigned to no object mask follow the static background motion induced by the camera.Sigmoid masks permit overlapping assignments, including compositions such as articulated kinematic chains; experiments fix K = 3.
- Supervision: Self-supervision minimizes photometric error between a frame and the next frame warped according to predicted motion, using differentiable image warping and brightness constancy.Robust spatial smoothness penalties are added for flow, depth, and inferred motion maps; depth uses second-order gradient penalties.
- Supervision: Forward-backward constraints require depths inferred in opposite temporal directions to remain consistent under the estimated scene motion.The scene-flow depth component W_t is used in the consistency relation, while composing cycles across longer frame gaps showed no empirical gain.
- Supervision: Camera trajectories or partial depth measurements can additionally supervise the corresponding predictions through rotation-and-translation losses or robust depth regression.When intrinsics are unavailable, default values yield depth estimates correct only up to a scalar multiplier.
4. Experimental results
SfM-Net is evaluated under unsupervised and supervised settings on KITTI, MoSeg, and RGB-D SLAM, testing depth, motion masks, optical flow, and camera motion. Results show benefits from explicit object-motion modeling, while performance remains sensitive to translation, occlusion, and difficult visual conditions.
- KITTI depth: 0.31 log RMSE is obtained with stereo pairs, while sequence-based depth estimation is less accurate because camera and object motion must be inferred.Stereo pairs provide a fixed camera pose; sequences require prediction of unknown motion between frames.
- KITTI depth: Explicitly modeling object motion improves unsupervised depth learning on videos containing independently moving objects.The gap between models with and without motion masks is wider on KITTI 2015, which contains more moving objects.
- KITTI depth: Depth estimation without camera-pose information matches stereo qualitatively when frames contain large translation, but fails when training examples contain little or no translation.Stereo pairs retain a constant offset, whereas sequence training may lack informative camera motion.
- Motion masks and flow: In KITTI 2015, predicted motion masks are often close to ground truth and help explain scene flow, but small distant objects and shadowed or occluded vehicles produce failures.Some semantically imperfect masks still yield reasonable flow, while heavy shadow makes the self-supervised photometric loss particularly difficult.
- MoSeg: On MoSeg, self-supervised motion masks achieve 0.29 IoU, comparable to prior unsupervised methods using four proposals, while higher IoU in prior work requires more than 800 proposals.MoSeg lacks annotations for quantitative depth, camera-trajectory, and optical-flow evaluation.
- RGB-D SLAM: With depth supervision on RGB-D SLAM, SfM-Net better estimates frame-to-frame translation and rotation than the benchmark baseline trajectories.The comparison uses relative camera-pose errors across Freiburg1 sequences.
5. Conclusion
SfM-Net incorporates learning into depth reconstruction, camera-motion prediction, and object segmentation while preserving image-formation constraints. The paper identifies training scenarios and longer-range consistency reasoning as open directions.
- SfM-Net combines learned depth reconstruction, camera-motion prediction, and object segmentation with image-formation constraints.
- The paper highlights extensive engineering, moving-object intolerance, and camera-calibration sensitivity as costs of current geometric SLAM methods.
- The exact training scenario needed for this more difficult inference problem remains an open question.
- Future work includes long-history and far-in-time forward-backward constraints with visibility reasoning.
- Annotated videos, self-supervised videos, synthetic datasets, and curriculum learning could help initialize network weights for learning.