Source-linked AI summary
FutureLift: Lifting Trajectory Proposals into Bidirectional Future Graphs for Motion Forecasting
Daniel Grimm, J. Marius Zöllner
TL;DR
Motion forecasters leave displacement and ranking errors in multimodal proposals, while existing refiners do not make the complete future explicit for correction. FutureLift lifts proposals into bidirectional future graphs and improves frozen proposals across Argoverse 2 and WOMD, outperforming native and specialized refiners in marginal and joint forecasting.
Problem
Existing refiners compress proposals or update segments sequentially, limiting the complete proposed future’s use as explicit relational context for correcting residual errors.
Source: §1 · p. 1Method
FutureLift lifts multimodal proposals into heterogeneous graphs of spatially grounded temporal nodes and uses one bidirectional graph pass to predict residual coordinate and confidence updates.
Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5Results
Takeaways & Limitations
Its proposal-only interface supports structurally different forecasting backbones without backbone-specific adaptation.
Source: §1 · p. 2, §5 · p. 8Open-loop forecasting metrics do not establish that geometric improvements translate directly to closed-loop planning or safety.
Source: §4.6 · p. 8
Abstract
from arXiv · showhide
Accurate motion prediction is critical for safe autonomous driving, yet even strong forecasters leave residual errors in their multimodal proposals. Existing refiners compress complete proposals or update their segments sequentially, preventing the full proposed future from serving as explicit relational context for correction. We introduce FutureLift, a backbone-agnostic refiner that represents each multimodal proposal as a heterogeneous future graph, with spatially grounded temporal nodes connected to agent histories, map elements, alternative modes, and, for joint prediction, other agents future trajectories. Bidirectional temporal edges allow later evidence to revise earlier segments in a single pass, and the updated nodes predict residual coordinate and confidence corrections. Requiring only proposals, scores, and scene context, FutureLift provides a common refinement interface for structurally different forecasting backbones. On Argoverse2 and the Waymo Open Motion Dataset, it consistently improves frozen proposals and outperforms native and specialized refiners across marginal and joint forecasting. On the WOMD motion-prediction leaderboard, FutureLift ranks first in minADE (0.5049m) and minFDE (0.9942m) among the compared standard-input methods, without LiDAR or ensembling, while achieving a competitive 0.4874 Soft mAP.
1. Introduction
Motion forecasters leave displacement and ranking errors in multimodal proposals, while existing refiners do not make the complete proposed future explicit for correction. FutureLift uses bidirectional future graphs to refine proposals and improves results across datasets, backbones, and forecasting settings.
1. Introduction
- Existing refiners can compress proposal geometry or use sparse anchors, while causal segment-wise sweeps cannot let later evidence revise earlier segments and iterative refiners require repeated computation.FutureLift instead exposes the complete proposed future so segments can interact with scene context and one another in both temporal directions.Source: §1 · p. 1
- FutureLift lifts proposal segments into spatially grounded nodes linked to histories, maps, alternative modes, and compatible other-agent futures for joint proposals, then predicts coordinate and confidence corrections in one bidirectional pass.It requires proposal coordinates, scores, and scene context rather than changes to the proposal mechanism.Source: §1 · p. 2
- Across Argoverse 2 and WOMD, FutureLift consistently improves frozen proposals; on WOMD it achieves 0.5049 minADE and 0.9942 minFDE, best among compared standard-input methods.A WOMD validation analysis also finds backward temporal information increasingly beneficial at longer horizons.Source: §1 · p. 2
- Experiments across multiple backbones, two datasets, and marginal and joint settings test FutureLift as a refinement interface, with each refiner trained for its target setting.The horizon-resolved analysis measures how backward temporal information contributes as prediction horizons lengthen.Source: §1 · p. 2
2. Related Work
Prior work structures observed scenes and embeds refinement within forecasting decoders, while post-hoc refiners use different interfaces and temporal strategies. FutureLift instead graphs generated proposal segments and conditions social future relations on coherent joint predictions.
2.1. Structured Representations for Motion Forecasting
- VectorNet [4], LaneGCN [13], and later scene graphs structure observed context; FutureLift instead graphs already generated proposal segments for backbone-independent refinement.Earlier scene representations organize the observed scene from which future trajectories are decoded.Source: §2.1 · p. 2
2.2. Trajectory Refinement
- MTR [19] and QCNet refine within decoders; SmartRefine and SRefiner [27] are post-hoc, but released SRefiner also consumes exported per-mode embeddings.FutureLift reconstructs its scene representation from proposal coordinates, confidence scores, and original scene inputs, without backbone embeddings or intermediate activations.Source: §2.2 · pp. 2–3
2.3. Marginal and Joint Motion Forecasting
- Equally indexed marginal modes need not represent the same joint world, so FutureLift adds social future edges only for scene-consistent joint hypotheses.This preserves marginal prediction without imposing joint semantics on its proposals.Source: §2.3 · p. 3
3. Method
FutureLift is a post-hoc refiner that lifts trajectory proposals into an explicit future-scene graph, then updates proposal coordinates and confidence scores using graph context. Its proposal-only interface and bidirectional temporal relations support refinement across different backbones and allow later evidence to revise earlier segments in one pass.
3.1. Problem Formulation and Proposal Interface
- The frozen backbone supplies multimodal trajectories and logits, while FutureLift uses the proposals, scores, and scene description to output coordinate and logit residuals.This interface lets the refiner operate on structurally different proposal generators after setting-specific training.Source: §3.1 · p. 3
3.2. Proposal-to-Future-Graph Lifting
- Each proposal is partitioned into temporal intervals and lifted into nodes anchored at predicted future locations, where neighborhoods are queried using the proposal geometry.Each node also retains its complete proposal segment in a local frame, preserving its motion pattern rather than only its endpoint.Source: §3.2 · p. 3
- The lifted graph connects future nodes to history, map context, alternative modes, and, for joint proposals, spatially neighboring agents; temporal edges are bidirectional.Social edges are disabled for marginal proposals because equal mode indices across agents do not identify corresponding hypotheses.Source: Figure 2 · p. 4, §3.2 · p. 4
3.3. One-Shot Bidirectional Future Refinement
- All future nodes are updated simultaneously on one graph built from the initial proposal, allowing later evidence to revise earlier segments without a causal sweep or graph reconstruction.The updated representation decodes coordinate corrections, uncertainty parameters, and confidence residuals.Source: Figure 2 · p. 4, §3.3 · p. 5
3.4. Training Objective
- FutureLift trains on fixed proposals with winner-takes-all trajectory regression and likelihood-based mode scoring, stopping gradients through proposal generation and backbone-specific latent features.The regression loss uses coordinate-wise Laplace negative log-likelihood; score refinement updates refined logits through the endpoint Laplace-mixture likelihood.Source: §3.4 · p. 5
4. Evaluation
Evaluation tests FutureLift across backbones, tasks, and datasets, then isolates the effects of graph relations, temporal direction, and resolution. Results show consistent refinement gains alongside limits inherited from post-hoc correction.
4. Evaluation
- Evaluation spans multiple backbones, marginal and joint prediction, AV2 and WOMD, and comparisons with native and specialized refiners.The evaluation also examines graph relations, temporal resolution, and representative failures.Source: §4 · p. 5
4.1. AV2 Marginal Prediction
- On AV2 marginal prediction, FutureLift improves every metric across QCNet, DenseTNT, and DONUT proposals, with lower errors than the compared SmartRefine results.The comparison uses shared frozen proposals for FutureLift and unrefined baselines; SmartRefine receives backbone trajectory features.Source: §4.1 · p. 6, Table 1 · p. 6
4.2. AV2 Joint Prediction
- On AV2 joint prediction, FutureLift reduces avgMinFDE and ActorMR versus shared unrefined proposals across FJMP, Forecast-MAE, and DONUT.On DONUT, adding social future relations further reduces avgMinFDE by 1.3% and ActorMR by 1.4%; SRefiner comparisons use different proposal exports and added embeddings.Source: §4.2 · p. 6, Table 2 · p. 6
4.3. WOMD Evaluation and Horizon Dependence
- On WOMD, FutureLift achieves the lowest valid horizon-averaged minADE and minFDE among listed standard-input methods, with competitive Soft mAP.At 8 s, minFDE is 1.6815 m versus 1.7513 m for the next-best standard-input result, NDPNet.Source: §4.3 · p. 6, Table 3 · p. 7
- On WOMD, bidirectional reasoning reduces minADE/minFDE versus causal reasoning by 0.5%/2.7% at 5 s and 2.2%/4.5% at 8 s, but increases them at 3 s.The WOMD test result uses one randomly selected seed; horizon-resolved validation results average three seeds.Source: §4.3 · p. 7, Table 3 · p. 7
4.4. What Makes Future-Graph Lifting Effective?
- On a frozen DONUT setup, one bidirectional sweep reduces avgMinFDE to 1.157 m, outperforming up to three causal sweeps; adding social future relations reduces it by another 0.015 m.The third causal sweep leaves avgMinFDE at 1.191 m; bidirectional refinement also lowers ActorMR from 0.151 to 0.148.Source: §4.4 · p. 7, Table 4 · p. 7
4.5. Temporal Resolution
- Increasing temporal resolution from R = 3 to R = 6 changes avgMinFDE by 0.003 m and avgMinADE by 0.001 m, while latency rises from 90 to 136 ms.The study selects R = 3 as the knee of the accuracy–latency trade-off.Source: §4.5 · p. 8, Table 5 · p. 8
4.6. Limitations and Failure Modes
- Post-hoc refinement cannot recover an absent maneuver, and the residual decoder can produce excessive curvature because it does not explicitly enforce vehicle dynamics.Open-loop metrics also do not establish that geometric improvements translate directly to closed-loop planning or safety.Source: §4.6 · p. 8, Figure 3 · p. 8
5. Conclusion
FutureLift refines multimodal trajectory proposals by lifting them into heterogeneous future graphs, where bidirectional relations let later evidence revise earlier segments. Across two datasets and marginal and joint forecasting, it improves frozen proposals; the evaluation remains limited to offline benchmarks with HD-map context.
5. Conclusion
- FutureLift lifts proposals into heterogeneous future graphs, using bidirectional relations to expose the complete proposed future for one-pass refinement.It improves frozen proposals across Argoverse 2 and WOMD in marginal and joint forecasting.Source: §5 · p. 8
- Within-backbone comparisons show FutureLift outperforming native and specialized refiners across marginal and joint forecasting.Models are trained separately for each setting.Source: §5 · p. 8
- Evaluation covers offline motion-forecasting benchmarks with HD-map context, not map-light settings or downstream closed-loop planning.The paper says gains in those settings have not yet been established.Source: §5 · p. 8
A. Experimental Setup
The experiments use Argoverse 2 and the Waymo Open Motion Dataset, with distinct history, horizon, and prediction-task settings. Both datasets evaluate multimodal forecasts, while the AV2 setup includes marginal and joint prediction.
A. Experimental Setup
- AV2 provides five seconds of history and requires six seconds of prediction at 10 Hz, with K = 6 modes in all experiments.Marginal forecasting ranks trajectories independently per actor; joint forecasting predicts coherent scene-level worlds.Source: §A.1 · p. 11
- WOMD provides one second of history and an eight-second horizon at 10 Hz, evaluating up to eight designated agents per scenario.The experiments use marginal prediction with K = 6 trajectories per designated agent.Source: §A.1 · p. 11
A.2. Metrics
The metrics quantify trajectory displacement, misses, confidence ranking, and overlap under dataset-specific evaluation rules. AV2 joint metrics select a predicted world based on its average final displacement error across scored actors.
A.2. Metrics
- AV2 marginal minFDE and minADE use the mode with the smallest FDE, while MR counts actors whose every mode exceeds 2 m FDE.Brier-minFDE adds a penalty based on the selected mode’s confidence.Source: §A.2 · p. 11
- AV2 joint avgMinADE and avgMinFDE are measured on the world with the lowest average FDE across scored actors; ActorMR counts selected-world actors exceeding 2 m final error.avgBrier-minFDE adds a confidence penalty to the selected world’s avgMinFDE.Source: §A.2 · p. 11
- For WOMD, minADE and minFDE take the minimum displacement error over K predictions at each cutoff, and MR counts agents with no matching mode.A match requires the endpoint to fall within speed-scaled lateral and longitudinal thresholds after rotating error into the ground-truth agent frame.Source: §A.2 · p. 11
- WOMD mAP and Soft mAP average confidence-ranking precision across motion buckets, with Soft mAP ignoring additional matching modes after the highest-confidence match.The main benchmark reports mAP v2 and Soft mAP v2 averaged over agent types and 3-, 5-, and 8-second cutoffs.Source: §A.2 · p. 11
A.3. Training and Evaluation Protocol
The protocol compares refiners on frozen proposals under controlled, setting-specific conditions and reports refinement-only efficiency. Supplemental AV2 test tables provide locally evaluated context distinct from the controlled validation comparisons.
A.3. Training and Evaluation Protocol
- FutureLift uses frozen proposal coordinates and logits without backbone queries, embeddings, or intermediate features, but is trained separately for each setting.Within-backbone comparisons generally use identical proposal files, splits, preprocessing, K, evaluator version, and random seeds; SmartRefine receives trajectory features required by its implementation, and daggered SRefiner results use a different export.Source: §A.3 · p. 11
- AV2 marginal and joint test tables are locally evaluated with released test annotations and the official evaluator, complementing rather than replacing controlled validation comparisons.The captions identify these as additional test-set context in Tables 6 and 7.Source: Table 6 · p. 12, Table 7 · p. 12
- Validation results average three seeds, with metric-wise standard deviations no greater than 3 × 10^-3; WOMD test uses a randomly selected trained seed.The models are trained for 40 epochs, and each run selects the checkpoint with the lowest validation loss.Source: §A.3 · p. 12
A.4. Efficiency Protocol
- Efficiency counts only refinement: parameter counts exclude the frozen backbone, while GFLOPs and latency include graph construction and the refiner forward pass.Measurements use an RTX 6000 Ada Generation GPU, PyTorch 2.5, FP32, batch size one, and average latency over 10 timed runs after 50 warm-ups.Source: §A.4 · p. 12
A.5. Complete AV2 Benchmark Context
The AV2 benchmark context reports local test evaluation of the selected frozen-DONUT configuration and representative geometric corrections. The refinement-cost table uses a common protocol that includes graph construction and averages latency over 10 timed runs.
A.5. Complete AV2 Benchmark Context
- The selected frozen-DONUT configuration is evaluated locally on AV2 test annotations because the leaderboard no longer accepts submissions; its values are not archived leaderboard results or identical-input comparisons.Published leaderboard values are included only for benchmark context.Source: §A.5 · pp. 12–13
- Figure 4 compares frozen proposals with refined hypotheses and ground truth, illustrating corrections to lateral drift, curved-lane misalignment, and a local trajectory discontinuity.These examples do not establish physical feasibility or closed-loop safety.Source: Figure 4 · p. 13
A.6. Detailed Refinement Cost
- Table 8 reports refinement-only efficiency under the common protocol, including graph construction, with latency averaged over 10 timed runs.The caption provides protocol details but no latency values.Source: Table 8 · p. 13
A.7. Number of Refinement Passes
The pass-count evaluation compares repeated application of the refinement stage with the default single-pass design. One pass gives the best reported spatial accuracy; further passes increase latency and degrade spatial metrics.
A.7. Number of Refinement Passes
- For J ≥ 1, the winning mode used for supervision is selected after refinement, consistent with the training objective.J is distinct from the number of attention layers within one pass.Source: §A.7 · p. 13
- 11.7% minFDE and 9.3% minADE reductions are reported for a single refinement pass relative to no refinement.The evaluation distinguishes outer refinement passes from attention layers within a pass.Source: §A.7 · p. 13
- Additional refinement passes increase latency and consistently degrade spatial metrics, supporting the single-pass design.Here, J denotes the number of outer passes, and J = 0 leaves predictions unrefined.Source: §A.7 · p. 13
A.8. Qualitative Analysis
Representative AV2 examples show FutureLift correcting geometric errors in frozen trajectory hypotheses. The detailed edge descriptions specify how temporal, map, social, and modal context is connected in the future graph.
A.8. Qualitative Analysis
- Figure 4 shows corrections to lateral drift, curved-lane alignment, and a localized discontinuity in the initial proposal, without establishing physical feasibility or closed-loop safety.Each scene compares the same hypothesis before and after refinement.Source: §A.8 · p. 13
B. Detailed Node and Edge Descriptions
- Lane nodes connect to neighboring lanes and nearby agents, while future nodes receive lane information within 150m and point information within 10m.Points belonging to a lane connect to that lane’s node.Source: Figure 7 · p. 14, Figure 9 · p. 14, Figure 6 · p. 14
- Agent nodes connect to up to 10 subsequent neighbors; future nodes receive information from up to 30 history nodes and other nodes in the same trajectory.The within-trajectory future-node relation depicts only k = 0 for visibility.Source: Figure 8 · p. 14, Figure 11 · p. 15, Figure 12 · p. 15
- Agent nodes connect to nearby traffic participants; future nodes connect to participants in the same joint mode and interval, and to the same participant’s alternative modes.The future-node social relation uses a 150m radius; the depicted social and mode relations show k = 0 for simplicity.Source: Figure 10 · p. 14, Figure 15 · p. 15, Figure 16 · p. 15