Source-linked AI summary
MESSENGER: Memory-Enhanced Sequential Scene Flow Estimation via Autoregressive Next-Frame Forecasting
Jiuming Liu, Jianing Li, Mengmeng Liu, Hongyang He, Hesheng Wang, Per Ola Kristensson
TL;DR
Pairwise scene-flow estimators lack long-term temporal correlation, while sequence-to-sequence methods can incur increasing computational costs and future prediction can degrade over longer horizons. MESSENGER forecasts sequentially from stored historical flows and latent states, reweighting retrievals by uncertainty. It reports state-of-the-art estimation and prediction, including EPE3D reductions over GenFlow3D in long-horizon extrapolation.
Problem
Pairwise scene-flow methods lack long-term temporal correlation, while multi-frame methods can incur increasing costs and prediction can degrade over longer horizons.
Source: §1 · p. 2Method
MESSENGER forecasts next-frame flow from a memory buffer of historical flows and latent states, using uncertainty-aware reweighting to filter unreliable retrievals.
Source: §1 · pp. 2–3Results
71.6% on nuScenes and 67.7% on Argoverse 2: MESSENGER reduces EPE3D versus GenFlow3D in long-horizon future extrapolation and reports state-of-the-art estimation and prediction.
Source: §1 · p. 3, §6 · p. 10Takeaways & Limitations
The authors attribute the long-horizon prediction advantage to explicit flow and hidden-state priors, next-frame forecasting, and uncertainty quantification that can mitigate long-term drift.
Source: §5 · p. 9The attention-based autoregressive forecasting has future prediction capability within a limited frame length.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Scene flow can capture low-level 3D motion displacements in dynamic scenarios. Early pairwise estimators relying on instantaneous two-frame motion lack long-term temporal correlation and also struggle with poor extrapolation ability in future prediction. Although some recent methods attempt to explore multi-frame scene flow estimation in a sequence-to-sequence manner, they typically suffer from heavy computational overhead with increasing input frames and long-horizon prediction degradation due to ineffective motion propagation. To address these problems, we propose a novel memory-enhanced sequential scene flow pipeline, called MESSENGER. To sufficiently mine long-term temporal dependencies naturally within consecutive sequences, a memory buffer is designed by explicitly storing multiple history flow estimates and latent states. For each input frame, the temporally stored flows and states are correlated and retrieved to predict the current initialized flow in a next-frame forecasting manner. Furthermore, we develop an uncertainty-aware reweighting module to filter unreliable retrievals and mitigate accumulated errors. Extensive experiments on nuScenes and Argoverse 2 demonstrate state-of-the-art performance of our MESSENGER, reducing EPE3D by 71.6% on nuScenes and 67.7% on Argoverse 2 in long-horizon future extrapolation. This superiority can be attributed to our designed autoregressive forecasting paradigm, which naturally forces the network to progressively learn the next-frame distribution based on history observations. Code will be released at https://github.com/liujiuming123/Messenger.
1 Introduction
MESSENGER addresses limits in pairwise and multi-frame scene flow by forecasting sequentially from stored flow and latent-state memories, with uncertainty-aware reweighting. It reports state-of-the-art estimation and prediction results, including substantial EPE3D reductions over GenFlow3D in long-horizon extrapolation.
1 Introduction
- Pairwise methods have limited temporal receptive fields, neglect long-term motion consistency, and struggle with occlusions, motion boundaries, and large displacements.Scene flow captures low-level 3D dynamics and supports tasks including dynamic reconstruction, motion segmentation, and SLAM.Source: §1 · pp. 1–2
- Multi-frame sequence-to-sequence methods incur increasing computational costs as frames are added, while GenFlow3D degrades over longer prediction horizons.The introduction also notes fixed-length input and ground-truth rigid-pose requirements for DeltaFlow, and GPU-memory constraints on longer sequences for GenFlow3D.Source: §1 · p. 2
- MESSENGER uses an autoregressive next-frame forecasting pipeline with a memory buffer that retrieves multiple historical flows and latent states to initialize current flow.The buffer stores explicit coarse flows and implicit states from multiple recent estimates; forecasting bases future motion on observations.Source: §1 · pp. 2–3
- Uncertainty-aware per-point reliability reweights memory fusion to filter unreliable historical retrievals and mitigate compounding errors.The method also uses multi-scale diffusion refinement decoders to recover fine-grained flow details iteratively.Source: §1 · p. 3
- 71.6% on nuScenes and 67.7% on Argoverse 2: MESSENGER reduces EPE3D versus GenFlow3D in long-horizon future extrapolation.The paper also reports state-of-the-art performance in both flow estimation and prediction tasks.Source: §1 · p. 3
2 Related Work
Prior work spans pairwise estimation, multi-frame estimation, and memory mechanisms, but the cited scene-flow methods leave future prediction or scene-flow memory comparatively underexplored. MESSENGER combines history-flow memory with progressive next-frame forecasting.
Pair-wise Scene Flow Estimation
- Pairwise methods improve matching and feature extraction, but limited temporal awareness constrains long-range motion consistency and challenging-scene matching.Examples include PointPWC-Net's PWC design, FLOT's optimal-transport matching, and transformer-based global matching in GMSF.Source: §2 · p. 3
Multi-Frame Scene Flow Estimation and Prediction
- Multi-frame methods use 4D processing, Mamba, residual flows, or temporal ensembling, yet are described as estimating only in-between flows rather than future motion.GenFlow3D supports prediction, but the passage says its default setting predicts one future frame and longer rollouts face accumulated errors.Source: §2 · p. 3
Memory Mechanism
- Memory mechanisms have been explored in optical flow and other vision tasks, while scene-flow memory remains underexplored.MESSENGER combines history flows as memory priors for progressive next-frame flow forecasting.Source: §2 · pp. 3–4
3 Methodology
MESSENGER estimates scene flow sequentially and extrapolates future flows from historical observations. Its memory retrieval, uncertainty reweighting, and multi-scale losses support this next-frame forecasting pipeline.
3.1 Preliminaries
- Given H historical point-cloud frames, the framework estimates flows between observed frames and can extrapolate K future flows.Future predictions are based on historical observations.Source: §3.1 · p. 4
3.2 Memory Buffers for Autoregressive Next-Frame Forecasting
- A fixed-length memory of historical coarse flows and hidden states supplies motion priors through cross-attention to initialize each next-frame flow.The latest state forms the query, while stacked hidden states provide keys and historical flows and states contribute values.Source: §3.2 · pp. 4–5
- Current-frame geometry refines the history-predicted coarse flow, then FIFO updates the buffer with the newest estimate and discards its oldest entry.The coarse prediction alone lacks current-frame geometry details and motion variations.Source: §3.2 · p. 5
3.3 Uncertainty-Guided Memory Reweighting
- A flow residual measures disagreement between historical predictions and current dynamics; per-point uncertainty then gates retrieved flows to limit unreliable memory propagation.The residual is mapped to an uncertainty value, and reliability-guided reweighting lets current correlations carry more weight when history is untrustworthy.Source: §3.3 · pp. 5–6
3.4 Loss Functions
- Training uses multi-scale supervised losses on estimated flows and flow residuals across observed and future predicted frames.At finer layers, estimated residuals are added to upsampled coarse flows to recover full flows.Source: §3.4 · p. 6
4 Experiments
Experiments evaluate MESSENGER on nuScenes and Argoverse 2 for scene-flow estimation and future prediction, alongside latency and architectural ablations. Results show strong estimation and long-horizon prediction performance, with memory and uncertainty designs affecting prediction quality.
4.1 Datasets and Implementation Details
- Evaluation uses nuScenes and Argoverse 2 LiDAR sequences; estimation uses H = 3, while prediction defaults to H = 6 and varies K from 1 to 4.The reported prediction comparison uses H = 6 and future intervals k = 1, 2, 3, 4.Source: §4.1 · pp. 6–7, Table 2 · p. 7
4.2 Comparison in Scene Flow Estimation
- On nuScenes and Argoverse 2, MESSENGER outperforms previous scene-flow estimators on EPE3D, Acc3DS, and Out3D.Against GenFlow3D, it has lower EPE3D and Out3D and higher Acc3DS.Source: §4.2 · p. 7
4.3 Comparison in Future Scene Flow Prediction
- With H = 6 observed frames, MESSENGER has half GenFlow3D’s EPE3D error at k = 1 and less than one-third its error at k = 4.The comparison covers nuScenes and Argoverse 2; the performance gap widens from k = 1 to 4.Source: Table 2 · p. 7, §4.3 · p. 8
4.4 Efficiency Analysis
- MESSENGER achieves the highest one-forward latency efficiency among the compared methods.The comparison includes DifFlow3D and GenFlow3D.Source: §4.4 · p. 8, Table 3 · p. 8
4.5 Ablation Studies
- Removing the memory buffer more than triples prediction errors, while increasing memory length from M = 3 to 6 improves prediction quality.Longer storage can slightly degrade performance when farther frames have scarce semantic similarity to the current frame.Source: §4.5 · p. 9, Table 4 · p. 9
- Without uncertainty modeling, EPE3D increases almost twofold, and storing prior-frame uncertainty estimates improves prediction.Normalization-based uncertainty construction outperforms the MLP-based alternative.Source: §4.5 · p. 9, Table 4 · p. 9
5 Discussions
The discussion attributes MESSENGER’s prediction performance to stored flow and state memories, autoregressive forecasting, and uncertainty-based filtering. It also examines rollout stability, uncertainty in dynamic regions, and class-specific estimation errors.
Why MESSENGER Excels Significantly in Prediction?
- Stored flow and hidden-state memories provide motion and semantic priors, while autoregressive forecasting retrieves temporal dependencies and uncertainty mitigates long-term drift.Both methods’ metrics drop with longer rollouts, but MESSENGER shows higher stability and lower metric fluctuations.Source: §5 · pp. 9–10
How Uncertainty Works?
- Learned uncertainty tends to concentrate in dynamic regions, where inconsistent car motion can make retrieved memories less reliable for forecasting.In such cases, the model adapts toward current geometric and correlation features rather than forecasting from memory.Source: §5 · p. 10
Fine-grained Per-class Error Analysis
- Pedestrians and vulnerable road users have higher estimation errors than regular cars for both methods, yet MESSENGER surpasses GenFlow3D across all dynamic classes.The passage attributes the greater difficulty partly to smaller size, non-rigid motion, and fewer LiDAR points for pedestrians and vulnerable road users.Source: §5 · p. 10, Table 5 · p. 10
- MESSENGER’s dynamic-region advantage is attributed to its memory buffer storing multi-frame historical flows and features for stronger motion consistency.The Three-way EPE averages foreground dynamic, foreground static, and background static categories without weighting.Source: §5 · p. 10
6 Conclusion
MESSENGER uses a dynamically maintained memory of coarse flows, latent states, and reliability scores to initialize current flow through autoregressive next-frame forecasting. The authors note that its future-prediction capability is limited to a finite frame length.
6 Conclusion
- MESSENGER progressively forecasts next-frame motion from retrieved historical flow, state, and reliability information, using uncertainty to weight memories and filter unreliable outliers.The conclusion reports improved scene-flow estimation and prediction, especially for long-horizon future predictions.Source: §6 · p. 10
- The authors identify limited frame length as a boundary of the current attention-based autoregressive forecasting approach.Future work could investigate large-scale pretrained backbones for long-horizon world modeling.Source: §6 · p. 10