Source-linked AI summary
RAFT-3D: Scene Flow using Rigid-Motion Embeddings
Zachary Teed, Jia Deng
TL;DR
Scene flow requires estimating pixelwise 3D motion from stereo or RGB-D video. RAFT-3D adapts RAFT to iteratively refine dense per-pixel SE3 motion using soft rigid-object embeddings and Dense-SE3 geometric consistency, achieving state-of-the-art results on FlyingThings3D and KITTI.
Problem
Scene flow estimates pixelwise 3D motion between video frames, but prior pipelines may require object instance segmentation and non-differentiable components.
Method
RAFT-3D is an end-to-end differentiable architecture that iteratively updates a dense pixelwise SE3 motion field using rigid-motion embeddings and Dense-SE3.
Results
RAFT-3D achieves state-of-the-art accuracy, improving FlyingThings3D two-view accuracy at δ < 0.05 from 34.3% to 83.7% and achieving KITTI error 5.77 versus 6.31.
Takeaways & Limitations
Rigid-motion embeddings enable dense and accurate 3D motion fields without object instance supervision.
Takeaways & Limitations
The method assumes known camera intrinsics and uses a pinhole projection model.
Abstract
from arXiv · showhide
We address the problem of scene flow: given a pair of stereo or RGB-D video frames, estimate pixelwise 3D motion. We introduce RAFT-3D, a new deep architecture for scene flow. RAFT-3D is based on the RAFT model developed for optical flow but iteratively updates a dense field of pixelwise SE3 motion instead of 2D motion. A key innovation of RAFT-3D is rigid-motion embeddings, which represent a soft grouping of pixels into rigid objects. Integral to rigid-motion embeddings is Dense-SE3, a differentiable layer that enforces geometric consistency of the embeddings. Experiments show that RAFT-3D achieves state-of-the-art performance. On FlyingThings3D, under the two-view evaluation, we improved the best published accuracy (d < 0.05) from 34.3% to 83.7%. On KITTI, we achieve an error of 5.77, outperforming the best published method (6.31), despite using no object instance supervision. Code is available at https://github.com/princeton-vl/RAFT-3D.
1. Introduction
RAFT-3D estimates pixelwise 3D motion from stereo or RGB-D video with an end-to-end differentiable architecture that iteratively refines dense SE3 motion. Its rigid-motion embeddings and Dense-SE3 layer exploit object rigidity without requiring object instance supervision, producing state-of-the-art benchmark results.
- Scene flow estimates pixelwise 3D motion between video frames for applications including path planning, collision avoidance, virtual reality, and motion modeling.
- Rigidly moving scenes exhibit piecewise constant 3D motion fields, providing a structural prior for scene-flow inference.
- Instance-detection and segmentation pipelines require instance supervision, introduce non-differentiable components, and cannot recover motion for new unknown objects.
- RAFT-3D retains RAFT’s iterative architecture while estimating pixelwise SE3 motion from stereo or RGB-D video.
- Rigid-motion embeddings softly group pixels into rigid objects, while Dense-SE3 enforces geometric consistency through differentiable least-squares updates.
- 83.7%: FlyingThings3D two-view accuracy at δ < 0.05, improving the best published 34.3%; KITTI error is 5.77 versus 6.31.
2. Related Work
Prior scene-flow methods use rigid-motion structure, optimization, object recognition, or point-cloud reasoning in different combinations. RAFT-3D builds on RAFT’s recurrent correlation-volume updates but replaces 2D flow with a per-pixel SE3 motion field and uses soft differentiable grouping rather than hard clustering.
- Optical Flow: RAFT constructs a 4D correlation volume and recurrently updates a high-resolution 2D flow field; RAFT-3D instead estimates a per-pixel SE3 motion field.
- Scene Flow: Scene flow recovers motion fields by aligning visually similar regions while imposing priors such as piecewise rigid motion and piecewise planar depth.
- Scene Flow: Traditional scene-flow optimization approaches include both variational and discrete inference, whereas RAFT-3D mimics iterative optimization by repeatedly refining the motion field.
- Scene Flow: Rigid-motion embeddings softly and differentiably group pixels into rigid objects instead of using hard clustering.
- Scene Flow: Object-detection and segmentation methods improve scene flow on driving datasets but generalize poorly to novel objects and introduce non-differentiable components.
- Scene Flow: Optical expansion methods avoid instance segmentation but cannot directly enforce rigid-motion priors.
- Scene Flow: Point-cloud methods suit sparse lidar measurements but do not directly exploit scene rigidity, which the paper identifies as critical for accuracy.
3. Approach
RAFT-3D estimates dense pixelwise SE3 motion from RGB-D or stereo image pairs by iteratively refining correspondences and geometrically consistent motion fields. Its architecture combines RAFT-style correlation and recurrent updates with rigid-motion embeddings and Dense-SE3 optimization.
- Input and output: RAFT-3D takes two image/depth pairs and outputs a dense transformation field assigning an SE3 motion to every pixel.For stereo inputs, depth estimates come from an off-the-shelf stereo network.
- Correlation and update: The network builds a 4D all-pairs correlation volume and repeatedly indexes it using current correspondence estimates.The correlation volume is organized as a multiscale pyramid, and local neighborhoods are sampled with bilinear interpolation.
- Correlation and update: A recurrent GRU update operator uses correlation and depth-derived features to predict correspondence revisions, rigid-motion embeddings, and confidence-weighted corrections.The predicted revisions correct optical flow and inverse depth before Dense-SE3 updates the motion field.
- Dense-SE3 optimization: Because SE3 transformations lie on a manifold, the method uses manifold-aware upsampling rather than directly taking linear combinations of transformations.The estimated field is produced at 1/8 resolution and then upsampled to the input resolution.
- Dense-SE3 optimization: Dense-SE3 maps revision predictions to SE3 updates through differentiable optimization, applying Gauss-Newton steps after recurrent updates.The layer maintains geometric consistency between the dense motion field, pixel correspondences, and embedding-based grouping.
- Rigid-motion embeddings: Rigid-motion embeddings softly group pixels into rigid objects, so similar embeddings make neighboring pixels contribute more strongly to the reprojection objective.This weighting prevents pixels from different rigid objects from being treated as if they shared one transformation.
4. Experiments
RAFT-3D is evaluated on synthetic FlyingThings3D and real KITTI data using scene-flow metrics, comparisons, and component ablations. It achieves strong benchmark results while showing benefits from iterative updates, local neighborhoods, inverse-depth revisions, and bi-Laplacian optimization.
- Evaluation setup: RAFT-3D is evaluated on real and synthetic datasets using AdamW, 12 unrolled update iterations, and an ImageNet-pretrained context encoder.The FlyingThings3D dataset combines camera and object motion spanning all 6 degrees of freedom.
- Evaluation setup: 2D EPE measures optical-flow error, 3D EPE measures scene-flow error, and threshold metrics measure the portion of pixels within a specified threshold.The 3D transformation field can produce predicted optical flow for 2D EPE evaluation.
- FlyingThings3D: 83.7% 3D δ < 0.05 accuracy improves over 34.3% on the FlowNet3D FlyingThings3D split using non-occluded points with depth <35 meters.This evaluation compares RAFT-3D with point-cloud-based scene-flow methods.
- KITTI: Ours ranks first on the KITTI leaderboard and outperforms DRISP despite using no instance supervision.DRISP combines PSMNet, PWC-Net, and Mask-RCNN with bounding-box and instance-mask supervision.
- Ablations: Performance improves with more update iterations up to about 16, while neighborhood radius 256 outperforms smaller radii and the full-image radius.Including inverse-depth revisions improves 3D metrics, and bi-Laplacian optimization raises 1px accuracy from 85.8 to 86.3 and 3D accuracy from 87.1 to 87.8.
5. Conclusion
RAFT-3D is an end-to-end network for scene flow that uses rigid-motion embeddings to represent soft groupings of pixels into rigidly moving objects. The authors report that these embeddings support dense and accurate 3D motion fields.
- RAFT-3D is an end-to-end network for scene flow.
- Rigid-motion embeddings softly group pixels into rigidly moving objects.
- The embeddings can be used to solve for dense and accurate 3D motion fields.
RAFT-3D: Appendix
The network combines feature and context encoders with a GRU-based update operator. Unlike RAFT, it adds context, correlation, and motion features elementwise before the GRU.
- The architecture includes a feature encoder, a ResNet50-based context encoder, and a GRU-based update operator.
- The GRU uses a dilated convolution pattern.
- RAFT-3D adds context, correlation, and motion features elementwise rather than concatenating them before the GRU.
A. Network Architecture
The appendix directs readers to Figure 4 for the network architecture, including its feature encoders and GRU-based update operator.
- Figure 4 shows the feature encoders and GRU-based update operator.
B. bi-Laplacian Optimization Layer Gradients
The optimization layer minimizes an objective involving finite-difference operators and flattened feature maps, then solves the resulting systems efficiently across channels and during backpropagation.
- The layer minimizes an objective function.
- Dx and Dy are linear finite-difference operators, while v is the flattened feature map.
- For a single channel, Wx and Wy are diagonal matrices formed from wx and wy, and the system is solved for u∗.
- Sparse Cholesky factorization and backsubstitution solve for u∗.
- Backward differentiation computes gradients with respect to the boundary weights by solving a linear system.
- Because H is positive definite, the forward factorization can be reused for the backward pass.
- Multiple Channels: The formulation extends to multiple channels by factoring H once and reusing it across the channel matrix.