Source-linked AI summary
HPLFlowNet: Hierarchical Permutohedral Lattice FlowNet for Scene Flow Estimation on Large-scale Point Clouds
Xiuye Gu, Yijie Wang, Chongruo wu, Yong-Jae lee, Panqu Wang
TL;DR
Large-scale scene-flow estimation from point clouds must preserve structure, fuse consecutive frames, and remain efficient under limited memory and uneven density. HPLFlowNet addresses this with hierarchical DownBCL, UpBCL, and CorrBCL operations on sparse permutohedral lattices, achieving strong benchmark results, cross-domain generalization, and processing of up to 86K points per frame.
Problem
Existing point-cloud networks have limited capacity for large scenes and can lose global or sparse-region information, while scene flow requires direct estimation and fusion of two consecutive point clouds.
Method
HPLFlowNet uses hierarchical DownBCL, UpBCL, and CorrBCL operations with sparse permutohedral-lattice convolutions to restore structure and fuse two point clouds.
Results
The method outperforms state-of-the-art approaches on FlyingThings3D and KITTI Scene Flow 2015, generalizes from synthetic to real data and across point densities, and processes up to 86K points per frame.
Takeaways & Limitations
Sparse lattice processing provides an efficient large-scale scene-flow architecture that retains performance while supporting real-world and varying-density point clouds without fine-tuning.
Takeaways & Limitations
Experiments use only point coordinates as input features, demonstrating the approach with minimum geometry information.
Abstract
from arXiv · showhide
We present a novel deep neural network architecture for end-to-end scene flow estimation that directly operates on large-scale 3D point clouds. Inspired by Bilateral Convolutional Layers (BCL), we propose novel DownBCL, UpBCL, and CorrBCL operations that restore structural information from unstructured point clouds, and fuse information from two consecutive point clouds. Operating on discrete and sparse permutohedral lattice points, our architectural design is parsimonious in computational cost. Our model can efficiently process a pair of point cloud frames at once with a maximum of 86K points per frame. Our approach achieves state-of-the-art performance on the FlyingThings3D and KITTI Scene Flow 2015 datasets. Moreover, trained on synthetic data, our approach shows great generalization ability on real-world data and on different point densities without fine-tuning.
1. Introduction
Scene flow estimates dense 3D motion directly from consecutive point clouds, but large-scale processing must preserve global structure, handle uneven density, and fuse both frames efficiently. HPLFlowNet addresses these challenges with hierarchical lattice operations and reports strong benchmark performance, generalization, and efficiency.
- Motivation: Direct point-cloud estimation avoids the indirect 3D reconstruction required by stereo optical flow and disparities.The paper focuses on efficient scene flow estimation directly from 3D point clouds.
- Motivation: Scene flow is a dense 3D motion field that represents ego-motion and dynamic-object motion for every point in the first point cloud.It is the 3D counterpart of optical flow and can support robotics, autonomous driving, human-computer interaction, visual odometry, and SLAM.
- Challenges: Existing point-cloud networks often split or subsample scenes, limiting point capacity while losing global context, boundary accuracy, or sparse-region performance.These limitations motivate processing entire scenes without chunking or subsampling.
- Challenges: Uneven sensor density and the need to combine two point clouds create additional requirements for density robustness and cross-frame information fusion.The introduction explicitly frames these as unresolved design questions.
- Approach: HPLFlowNet introduces DownBCL, UpBCL, and CorrBCL operations on permutohedral lattices for hierarchical processing, structural restoration, and two-frame fusion.Signals are processed through sparse lattice convolutions, downsampling, upsampling, and interpolation back to points in the first frame.
- Results: 86K points per frame can be processed as a complete KITTI frame pair, while experiments report state-of-the-art performance and generalization from synthetic to real data and across point densities.The reported generalization does not require fine-tuning.
2. Related work
Prior 3D learning methods use grids, point hierarchies, trees, graphs, or bilateral layers, while scene-flow methods remain relatively underexplored. HPLFlowNet focuses on efficient large-scale processing and two-cloud fusion through an end-to-end lattice architecture.
- 3D deep learning: Grid-based multi-view and volumetric CNNs suffer discretization issues, while point, tree, and graph methods provide alternative representations for unstructured 3D data.PointNet and PointNet++ use symmetric aggregation and hierarchical local neighborhoods, respectively.
- BCL-based methods: HPLFlowNet differs from prior BCL-based work by prioritizing efficient large-scale point-cloud processing rather than joint 2D–3D reasoning.The comparison also notes that HPLFlowNet must combine two point clouds, whereas the cited BCL work operates on one.
- Scene flow estimation: Earlier scene-flow systems use energy minimization, occupancy grids, filtering, or explicit geometric assumptions, whereas HPLFlowNet learns end to end from data.The cited prior methods include assumptions about local geometric constancy, smooth motion, or background filtering.
- Scene flow estimation: FlowNet3D mixes two point clouds with a flow-embedding layer but shares drawbacks associated with PointNet++, including its aggregation limitations.Another cited approach defines continuous convolution per point while using pooling as its aggregation mechanism.
- Architecture: Hierarchical DownBCLs and UpBCLs use Splat-Conv and Conv-Slice pipelines to move between finer and coarser permutohedral lattices.DownBCLs downsample onto coarser lattice points, while UpBCLs reverse the process.
3. BCL on permutohedral lattice
BCL maps continuous point features to a sparse discrete lattice, applies convolution only at occupied lattice locations, and interpolates results back. HPLFlowNet uses the permutohedral lattice to reduce high-dimensional interpolation cost while supporting multiscale processing.
- BCL: BCL extends fast high-dimensional Gaussian filtering with learnable weights and accepts general positioned inputs with associated signal values.In this application, the position space is 3D and the signal value is the point position.
- BCL: BCL gathers continuous input signals onto enclosing lattice points through interpolation, then performs sparse convolution on non-empty lattice locations.A hash table restricts convolution to lattice points that received signals.
- Permutohedral lattice: The integer lattice can require 2^d interpolated vertices per input point, making splatting and slicing exponential in dimension.This motivates using the permutohedral lattice instead.
- Permutohedral lattice: The permutohedral lattice is formed by projecting a scaled regular grid onto the zero-sum hyperplane, whose Delaunay cells are uniform d-simplices.Barycentric interpolation lets BCL use the same splat-and-convolve scheme on this lattice.
- Permutohedral lattice: Permutohedral-lattice simplex vertices and barycentric weights can be computed in O(d^2) time, improving efficiency over exponential-dimensional interpolation.The efficiency follows from the lattice’s special geometric structure.
- Multiscale processing: Scaling controls lattice resolution: larger scale factors produce finer lattices with fewer points per simplex, while DownBCL and UpBCL traverse these scales hierarchically.DownBCLs move toward coarser lattices and UpBCLs toward finer lattices.
4. Approach: HPLFlowNet
HPLFlowNet extends bilateral convolution with efficient downsampling, upsampling, correlation, and density normalization on sparse permutohedral lattices. These operations preserve structural information, fuse consecutive point clouds, and reduce computation for large-scale scene-flow estimation.
- DownBCL and UpBCL: DownBCL and UpBCL reduce BCL’s three-step pipeline to two steps, saving computation without introducing new operations.DownBCL uses Splat-Conv for iterative downsampling, while UpBCL uses Conv-Slice for upsampling.
- DownBCL and UpBCL: Fewer non-empty lattice points make computation depend on occupied volume rather than input-point count after the first DownBCL and before the last UpBCL.This reduces the data size handled across stacked DownBCL and UpBCL layers, especially at coarser resolutions.
- DownBCL and UpBCL: Removing DownBCL slicing and UpBCL splatting reduces interpolation errors that can alter signals and oversmooth per-point regression outputs.The issue arises because barycentric interpolation is asymmetric and makes outputs within each simplex smooth.
- CorrBCL: CorrBCL fuses two point clouds on a shared permutohedral lattice through patch correlation followed by displacement filtering.Patch correlation combines local neighborhoods, while displacement filtering matches locally shifted lattice positions and aggregates the resulting pairwise information.
- Density normalization: Density normalization is added during splatting to reduce computational cost and support generalization across point densities without fine-tuning.The normalization applies directly to CorrBCL and is reported to improve robustness under non-uniform sampling densities.
- Network architecture: Relative positions within enclosing simplices are concatenated with BCL inputs to compensate for positional information lost during continuous-to-discrete interpolation.The paper states that this improves translational invariance under quantization effects.
5. Experiments
Experiments evaluate accuracy, cross-domain and density generalization, efficiency, and component contributions on FlyingThings3D and KITTI. The method outperforms baselines, processes whole large-scale frames, and remains robust across densities.
- Experimental setup: Experiments cover synthetic FlyingThings3D training and evaluation, direct real-world KITTI testing, varying point densities, efficiency comparisons, and ablations.The study also reports evaluation metrics and compares against ICP, FlowNet3D, SPLATFlowNet, original BCL, and FlowNet3.
- Results on FlyingThings3D: Our method outperforms all baselines on every reported metric and is the only method with EPE3D below 10cm on FlyingThings3D.FlowNet3 is not directly comparable because it uses stereo inputs.
- Efficiency and qualitative behavior: The model handles complicated shapes, large motions, and neighboring objects with different motions, while its runtime does not linearly scale with input-point count.The authors attribute gains over SPLATFlowNet to the hourglass model and skip links.
- Generalization to KITTI: Without fine-tuning, the FlyingThings3D-trained model outperforms other methods on all KITTI metrics, with Ours/FlowNet3D EPE3D of 0.2366/0.3331 without ground removal.It also estimates scenes containing large ego-motion, multiple dynamic motions, trees, and bushes unseen during training.
- Efficiency and qualitative behavior: The network processes two complete point clouds in one pass, including KITTI frames with around 86K points, avoiding scene division into chunks.This design targets the global-information and boundary-point issues associated with chunking.
- Density robustness and ablations: Density normalization makes EPE3D remain unchanged across testing densities different from training density, although it introduces slight information loss.Ablations also find that skip links, multiple CorrBCLs, concatenation, and combined global-local positional information improve performance.
6. Conclusion
HPLFlowNet is a deep network for scene flow estimation on large-scale point clouds. Its DownBCL, UpBCL, CorrBCL, and density normalization scheme improve efficiency and robustness across lattice scales and point densities.
- HPLFlowNet introduces DownBCL, UpBCL, CorrBCL, and a density normalization scheme for scene flow estimation on large-scale point clouds.These components support robust processing on permutohedral lattices of different scales.
- The architecture reduces computational cost without sacrificing performance.The design operates across lattice scales rather than maintaining computation proportional to the full input point-cloud size.
- Extensive experiments demonstrate advantages over various comparison methods.