Source-linked AI summary
Just Go with the Flow: Self-Supervised Scene Flow Estimation
Himangi Mittal, Brian Okorn, David Held
TL;DR
Scene-flow estimation is important for tracking independently moving entities, but existing methods require costly real-world annotations. This paper uses nearest-neighbor and cycle-consistency losses for self-supervised training, matching supervised performance without real-world annotations and exceeding state-of-the-art performance when combined with supervised learning.
Problem
Existing scene-flow methods require costly per-point annotations, and only a small amount of annotated real-world scene-flow data is available.
Method
The method combines nearest-neighbor and cycle-consistency losses, including an anchored reverse cycle, to train scene-flow networks on unlabeled sequential point clouds.
Results
The self-supervised method achieves performance comparable to supervised methods, while combining self-supervision with supervised training exceeds current state-of-the-art performance.
Takeaways & Limitations
Self-supervision enables scene-flow training and fine-tuning on large or arbitrary datasets that lack scene-flow annotations.
Takeaways & Limitations
Nearest-neighbor matching can fail on sparse point clouds and permits degenerate mappings, while cycle loss can become unstable without ground-truth annotations.
Abstract
from arXiv · showhide
When interacting with highly dynamic environments, scene flow allows autonomous systems to reason about the non-rigid motion of multiple independent objects. This is of particular interest in the field of autonomous driving, in which many cars, people, bicycles, and other objects need to be accurately tracked. Current state-of-the-art methods require annotated scene flow data from autonomous driving scenes to train scene flow networks with supervised learning. As an alternative, we present a method of training scene flow that uses two self-supervised losses, based on nearest neighbors and cycle consistency. These self-supervised losses allow us to train our method on large unlabeled autonomous driving datasets; the resulting method matches current state-of-the-art supervised performance using no real world annotations and exceeds state-of-the-art performance when combining our self-supervised approach with supervised learning on a smaller labeled dataset.
1. Introduction
Scene flow estimates the 3D velocity of each point in a scene, supporting motion understanding for autonomous driving. The paper addresses costly annotation requirements with two self-supervised losses that train scene-flow networks on large unlabeled datasets.
- Scene flow estimates the 3D velocity of each point, avoiding reliance on object detections that can introduce tracking errors.This provides point-level motion estimates for vehicles, pedestrians, cyclists, and other entities.
- Current state-of-the-art scene-flow methods are fully supervised and require costly per-point motion annotations, often relying on synthetic training data and real-world fine-tuning.
- The method combines cycle-consistency and nearest-neighbor losses to mimic human supervision without scene-flow annotations.The nearest-neighbor loss treats the closest point in the next cloud as a pseudo-ground-truth association, while cycle consistency enforces temporal consistency.
- These losses enable training a scene-flow network on large-scale, unannotated sequential point-cloud datasets.
- The method matches state-of-the-art performance without real-world annotations and exceeds state-of-the-art performance when combined with supervised learning on a smaller labeled dataset.
2. Related Work
Prior scene-flow work spans geometric, probabilistic, rigid-motion, regularization, and deep-learning approaches. Earlier self-supervised signals support tracking or representation learning, but the paper adapts self-supervision specifically to scene-flow estimation from point clouds.
- Early scene-flow methods used optical flow, stereo geometry, particle filters, local rigid-motion priors, and smoothness regularization.
- Modern methods use deep networks to estimate scene flow directly from paired 3D point clouds or structured representations.
- Prior self-supervised work tracked image patches through forward and backward cycles using cycle consistency and feature similarity.
- Other self-supervised signals target image ordering, temporal feature similarity, or point-cloud clustering and reconstruction, rather than directly estimating scene flow.
3. Method
The method estimates scene flow between consecutive 3D point clouds using self-supervised nearest-neighbor and cycle-consistency signals, avoiding dependence on dense ground-truth correspondences. Anchoring nearest neighbors stabilizes the reverse-flow computation, while the combined losses address degenerate solutions and structural distortions.
- Problem Definition: Scene flow describes each point’s 3D movement between consecutive point clouds, although corresponding points may be absent because of sparsity and differing cloud sizes.Each point combines Cartesian position with optional sensor features, and its future position need not coincide with an observed point in the next cloud.
- Nearest Neighbor Loss: The nearest-neighbor loss uses the closest point in the next cloud as a pseudo-correspondence for each predicted transformed point.It reduces the distance between the transformed source cloud and the target cloud, but can select incorrect correspondences when flow errors are large.
- Nearest Neighbor Loss: Sparse target clouds and collapsed mappings expose weaknesses of nearest-neighbor supervision alone, motivating an additional cycle-consistency loss.A nearest target point may not represent the true transformed point, and mapping all source points to one target point can achieve zero nearest-neighbor loss.
- Cycle Consistency Loss: Cycle consistency estimates forward flow, reverses the transformed points back to the original frame, and penalizes disagreement with their starting positions.Accurate forward and reverse flows should return each transformed point to its original point.
- Anchored Cycle Consistency: Using nearest neighbors as anchors stabilizes the transformed cloud before reverse-flow estimation, producing the anchored cycle-consistency loss.Anchoring removes some structural distortion and the experiments use a convex combination with λ = 0.5; it also helps eliminate the zero-flow degeneracy.
- Combined Objective: The sum L = LNN + Lcycle combines losses with conflicting local minima, acting as a stable surrogate for the unavailable supervised error.The nearest-neighbor loss remains non-zero for the zero-flow solution, complementing cycle consistency.
4. Experiments
Experiments evaluate self-supervised scene flow across KITTI and nuScenes, using unlabeled and labeled training configurations, qualitative comparisons, and ablations. The method matches supervised performance without real-world annotations and improves further when self-supervised pretraining is combined with supervised fine-tuning.
- Experimental setup: The experiments compare self-supervised training, supervised fine-tuning, and their combination across KITTI and nuScenes datasets.KITTI provides annotated scene flow, whereas nuScenes lacks scene flow annotations and therefore supports only self-supervised evaluation and training.
- Self-supervised training: The method can train on sequential point-cloud datasets without scene flow annotations, combining KITTI and nuScenes to provide 5x more real data than KITTI alone.The self-supervised loss enables unlabeled datasets to be integrated into training, addressing the annotation constraint of supervised learning.
- Quantitative results: 0.105 m EPE is achieved using no real-world annotations, with performance similar to supervised KITTI fine-tuning and better accuracy than that baseline.The self-supervised configuration uses synthetic pretraining followed by self-supervised fine-tuning on nuScenes and KITTI.
- Quantitative results: 0.091 EPE is obtained when self-supervised nuScenes training is followed by supervised KITTI fine-tuning, outperforming methods using only supervised training.This configuration achieves the best result reported on the benchmark and demonstrates benefits from unlabeled data even when annotations are available.
- Quantitative results: The method consistently outperforms the baseline at almost all ground-truth flow magnitudes and also exceeds HPLFlownet’s EPE of 0.1169.Figure 7 analyzes average EPE across flow-magnitude bins with 95% confidence intervals.
- Qualitative analysis: Qualitative results show greater overlap with the next point cloud than synthetic-only or purely supervised baselines on KITTI and nuScenes.The purely supervised baseline can underestimate motion, while the synthetic-only baseline may overestimate or fail to overlap with the next frame.
- Ablation studies: Removing the anchor from cycle consistency causes a large performance drop, while λ = 0.5 gives the best results by averaging predicted and nearest points.The ablation indicates that anchored point clouds stabilize cycle-consistency training.
5. Conclusion
The paper proposes self-supervised scene flow training using temporal cycle consistency and nearest-neighbor losses. It performs comparably to supervised methods without real-world annotations and exceeds state-of-the-art performance when combined with supervised training on unlabeled and labeled data.
- The method combines temporal cycle consistency and nearest-neighbor losses to train scene flow algorithms.
- Purely self-supervised training achieves performance comparable to supervised methods on the KITTI self-driving dataset.
- Augmenting supervised training with self-supervision on a large-scale, unannotated dataset exceeds current state-of-the-art performance.
- The method supports fine-tuning on arbitrary datasets that lack scene flow annotations.
Supplementary
The supplementary analyses describe the training setup, component ablations, point-density analysis, and endpoint-error distributions. Removing any component reduces performance, while the proposed method has a lower average endpoint error and fewer large outliers than the baseline.
- Implementation Details: All models use batch size 8 and Adam with learning rate 1e-4, β1 = 0.9, β2 = 0.999, and ϵ = 1e −8.
- Further Ablation Studies: Removing any self-supervised component causes a performance drop, especially on ACC 0.05 and ACC 0.1.When Cycle Consistency Loss is removed, anchoring is also removed because anchoring modifies that loss.
- Point Density Analysis: The point-density analysis bins flow estimates by local point density and reports mean EPE with 95% confidence intervals.
- Error Distribution: The proposed method has an average EPE of 0.091 and fewer large outliers than the baseline.Errors are computed for every point in every scan of the KITTI test set and displayed with log-binned x-axis values.