Source-linked AI summary
FLOT: Scene Flow on Point Clouds Guided by Optimal Transport
Gilles Puy, Alexandre Boulch, Renaud Marlet
TL;DR
Scene-flow estimation on point clouds can be viewed as recovering point correspondences, but real-world sampling and occlusions violate ideal transport assumptions. FLOT learns feature-based transport costs, relaxes transport constraints, and uses the resulting soft correspondences to estimate flow; it performs competitively with far fewer parameters, while FLOT0 nearly matches FLOT.
Problem
Point-cloud scene flow requires estimating correspondences despite imperfect sampling and objects appearing or disappearing through occlusions.
Method
FLOT learns pairwise transport costs from deep point-cloud features, relaxes transport constraints, and converts soft correspondences into refined flow estimates.
Results
FLOT performs similarly to the best methods on synthetic and real-world datasets with much fewer parameters, while FLOT0 performs nearly as well as FLOT.
Takeaways & Limitations
Most performance is explained by the learned transport cost, yielding the simpler FLOT0 variant.
Takeaways & Limitations
FLOT does not explicitly handle occlusions, although relaxed mass constraints limit their impact on performance.
Abstract
from arXiv · showhide
We propose and study a method called FLOT that estimates scene flow on point clouds. We start the design of FLOT by noticing that scene flow estimation on point clouds reduces to estimating a permutation matrix in a perfect world. Inspired by recent works on graph matching, we build a method to find these correspondences by borrowing tools from optimal transport. Then, we relax the transport constraints to take into account real-world imperfections. The transport cost between two points is given by the pairwise similarity between deep features extracted by a neural network trained under full supervision using synthetic datasets. Our main finding is that FLOT can perform as well as the best existing methods on synthetic and real-world datasets while requiring much less parameters and without using multiscale analysis. Our second finding is that, on the training datasets considered, most of the performance can be explained by the learned transport cost. This yields a simpler method, FLOT$_0$, which is obtained using a particular choice of optimal transport parameters and performs nearly as well as FLOT.
1 Introduction
FLOT estimates scene flow on point clouds by finding soft point correspondences with optimal transport, then converting them into and refining flow estimates. It is designed as a lightweight alternative that performs competitively without multiscale analysis.
- Scene flow estimates the 3D motion of surface points and supports low-level scene understanding, including autonomous driving.
- FLOT directly consumes point clouds using point-cloud convolutions, unlike related approaches based on images, occupancy grids, or cylindrical projections.
- FLOT separates estimation into soft-correspondence matching followed by scene-flow estimation.The first step uses optimal-transport tools inspired by graph matching; the second exploits correspondences to estimate flow.
- Deep point-cloud features define transport costs between points at consecutive times, with lower costs indicating likely correspondences.The initial flow estimate comes from linear interpolation and is refined by a residual network; parameters are learned under full supervision on synthetic datasets.
- FLOT is reported to match the best existing methods on synthetic and real-world datasets with much fewer parameters and no multiscale analysis.
- FLOT0 is a simpler variant derived from a particular optimal-transport parameter choice and achieves competing results with state-of-the-art methods.The simplification follows from the observation that most performance is explained by the learned transport cost.
2 Related Works
Prior scene-flow work uses varied point-cloud, image, depth, and self-supervised approaches, while FLOT relates most directly to graph matching by optimal transport and algorithm unrolling.
- Deep Scene Flow Estimation on Point Clouds: Deep scene-flow methods include rigid-motion estimation, continuous convolutions, PointNet++ flow embeddings, and coarse-to-fine PWC-Net adaptations.
- Deep Scene Flow Estimation on Point Clouds: FLOT’s optimal-transport module is independent of the point-cloud convolution type, although the paper uses PointNet++.Related work also includes self-supervised methods, whereas FLOT restricts training to full supervision.
- Graph Matching by Optimal Transport: Optimal-transport graph-matching methods compare graphs through node distributions, Wasserstein distances, Gromov-Wasserstein distances, or combinations of structure and node features.
- Graph Matching by Optimal Transport: FLOT uses the graph-matching approach that incorporates node features through Wasserstein distance.
- Algorithm Unrolling: Algorithm unrolling replaces a fixed number of iterations of an iterative algorithm with task-specific trained operations.The related technique has been applied to linear inverse problems and image denoising.
3 Method
FLOT estimates scene flow by first finding soft point correspondences with optimal transport, then converting them into flow and refining the estimate. Its relaxed transport formulation and learned feature costs address sampling imperfections while avoiding multiscale processing.
- Perfect World: FLOT models ideal scene-flow estimation as recovering a permutation matrix linking corresponding points across two consecutive point clouds.Each source point transports mass n^-1 to its matching target point in the perfect-world formulation.
- Perfect World: The optimal transport module produces a transport plan whose entries represent mass moved between points and thereby encode soft correspondences.The plan is computed from a nonnegative displacement-cost matrix under transport constraints.
- Real World: Real-world occlusions and unequal point sampling violate mass preservation, so FLOT relaxes the transport constraints using entropic and mass-variation regularization.The parameter ϵ controls plan sparsity, while λ controls deviation from uniform transported mass; strict preservation is recovered as λ approaches infinity.
- Real World: FLOT learns pairwise transport costs from cosine distances between deep point features rather than relying on Euclidean point distances.The feature network g is trained end-to-end through K unrolled Sinkhorn iterations, allowing gradients to reach the feature extractor.
- Architecture and Simplification: FLOT uses a lightweight single-resolution architecture, while FLOT0 removes mass-transport regularization and relies only on the learned cost matrix for correspondences.The architecture avoids downsampling, upsampling, and multiscale correspondence analysis; FLOT’s information mixing is explicit through ϵ and λ.
- Flow Estimation: The transport plan yields an interpolated initial flow, which a residual network refines because missing regions cannot always be reconstructed from corresponding target points.The refinement uses structural information from the source cloud and neighboring well-sampled regions.
4 Experiments
Experiments evaluate FLOT across synthetic and KITTI datasets, including ideal and occluded settings, and compare refinement, OT parameters, and competing methods. FLOT performs competitively with substantially fewer parameters, while results show that learned transport costs explain much of the performance and that occlusions remain a limitation.
- Datasets and evaluation: FLOT is trained on FlyingThings3D and evaluated on FlyingThings3D and KITTI Scene Flow using multiple point-cloud extraction variants.FT3Ds and KITTIs exclude occluded points, whereas FT3Do and KITTIo retain them without including them in loss or metrics.
- Datasets and evaluation: The evaluation reports EPE, two thresholded accuracy measures, and an outlier percentage.Validation results report means and standard deviations across repeated models and point samplings.
- OT parameter study: ϵ = 0.03 for all trained models, producing sparse transport plans whose correspondences concentrate around target points.The paper reports especially concentrated mass when target points lie near object corners.
- OT parameter study: On ideal FT3Dp, FLOT halves EPE relative to FLOT0, while increasing K from 1 to 3 further reduces error and stabilizes at K = 5.These results support coincidence between the OT optimum and the perfect-world optimum.
- OT parameter study: On realistic FT3Ds and FT3Do, FLOT generally outperforms FLOT0, but increasing K has less impact and can slightly reduce performance.The paper therefore treats K as a hyper-parameter and uses K = 1 or K = 3 in later experiments.
- Comparison with prior methods: FLOT outperforms FlowNet3D and HPLFlowNet on FT3Ds and KITTIs, has slightly better KITTIs EPE than PointPWC-Net, and uses 69 times fewer trainable parameters.PointPWC-Net retains better accuracy and fewer outliers, while FLOT remains competitive overall.
- Qualitative results: On KITTIs, the OT module already produces good flow estimates, and refinement further improves performance while correctly aligning objects in the illustrated scenes.The refined estimate is compared with the ground truth and input point clouds in Fig. 3.
- Occluded-data evaluation: FLOT and FLOT0 improve AS over FlowNet3D by 8.8 points on FT3Do and 17.7 on KITTIo, with corresponding reductions in outliers.FLOT at K = 1 performs best on FT3Do, while FLOT0 and FLOT at K = 3 perform best on KITTIo.
5 Conclusion
The paper proposes FLOT, an optimal-transport method for scene flow estimation, and derives FLOT₀ from the learned transport cost. FLOT achieves similar performance to the best-performing method with fewer parameters, while occlusions remain insufficiently handled.
- FLOT estimates scene flow using optimal transport tools.
- FLOT achieves similar performance to the best-performing method while requiring much fewer parameters.
- The learned transport cost explains most of FLOT’s performance and motivates the simpler FLOT₀ method.
- FLOT₀ performs nearly as well as FLOT.
- Occlusions negatively affect FLOT, and the method does not treat them explicitly.Relaxed mass constraints limit their impact, but explicit detection and treatment remain possible improvements.
A Networks architecture
The networks g and h use point-cloud convolutions based on local nearest-neighbor features and share an architecture without sharing weights.
- Each convolution processes point-cloud coordinates or incoming point features using the 32 nearest neighbors of each point.
- A shared MLP transforms each neighbor feature, followed by channel-wise max pooling to produce the new point feature.
- The networks g and h have the same architecture, but their weights are not shared.
B Datasets
The datasets are prepared from existing scene-flow data by different point-cloud extraction procedures, with distinct handling of occlusions and invalid flow points.
- FT3Ds and KITTIs contain no occluded points after processing, while random independent subsampling simulates different scene sampling.
- FT3Do uses masks for invalid flow caused by phenomena such as occlusions during training and evaluation.
- KITTIo provides no mask for invalid flow points, although such points remain in every network’s input.
C Performance metrics
Performance is evaluated by comparing estimated and ground-truth flow on randomly selected points, using EPE, accuracy rates, and outlier percentage.
- EPE is the average Euclidean end point error between estimated and ground-truth flow.
- AS and AR report the percentages of points meeting relative or absolute EPE thresholds of 0.05 and 0.1, respectively.
- Out. reports the percentage of points whose absolute or relative EPE exceeds the specified thresholds.
- Scores are computed on n randomly selected points from point clouds containing N provided points and averaged across the validation or test set.
D.1 Study of FLOT
Table 5 reports FLOT performance at the output of the optimal transport module on FT3Do, before refinement. Refinement improves EPE by around 2, confirming its utility in the presence of occlusions.
- Table 5 reports FLOT performance on FT3Do before refinement by the residual network h.The measurements are taken at the output of the optimal transport module.
- Refinement improves EPE by around 2 on FT3Do.
- The EPE improvement confirms refinement’s utility in the presence of occlusions.
D.2 Computation time in the OT module
The optimal transport module contributes a small share of FLOT’s computation time across the tested point-cloud sizes. Most computation is spent in the feature extractor, while restricting the cost-matrix computation offers a possible optimization.
- At n = 2048, OT-module times are 1.4, 2.0, and 2.2 ms for FLOT0, FLOT K = 1, and FLOT K = 3, respectively.
- At n = 8192, OT-module times are 13.1, 16.0, and 17.9 ms for FLOT0, FLOT K = 1, and FLOT K = 3, respectively.
- The OT module accounts for at most 8% of total computation time, while feature extraction uses at least 67% at n = 2048 and 86% at n = 8192.
- Restricting the dense cost-matrix computation to points within dmax meters could make the OT implementation faster.