PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders

Jie Cheng, Xiaodong Mei, Ming Liu

arXiv:2308.09882v1cs.ROcs.CV

TL;DR

Self-supervised learning remains underexplored in motion forecasting, which also depends on annotated data and multimodal scene inputs. Forecast-MAE applies masked scene reconstruction to learn from trajectories and road networks, and it surpasses supervised methods and prior self-supervised work on Argoverse 2.

  • Problem

    Self-supervised learning is underexplored in motion forecasting, where annotated data and cross-modal inputs make pretext-task design challenging.

    Source: §1 · p. 1

  • Method

    Forecast-MAE pre-trains with a scene-reconstruction masked autoencoder, complementarily masking agent histories or futures and randomly masking lane segments.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    On Argoverse 2, Forecast-MAE surpasses supervised methods and previous self-supervised work, especially on minADE1 and minFDE1.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings suggest self-supervised learning can be a promising approach for motion forecasting.

    Source: §1 · p. 2

    Limited public dataset size leaves uncertain whether performance scales with increased training data and model capacity.

    Source: §5 · p. 8

Abstract

from arXiv · show

This study explores the application of self-supervised learning (SSL) to the task of motion forecasting, an area that has not yet been extensively investigated despite the widespread success of SSL in computer vision and natural language processing. To address this gap, we introduce Forecast-MAE, an extension of the mask autoencoders framework that is specifically designed for self-supervised learning of the motion forecasting task. Our approach includes a novel masking strategy that leverages the strong interconnections between agents' trajectories and road networks, involving complementary masking of agents' future or history trajectories and random masking of lane segments. Our experiments on the challenging Argoverse 2 motion forecasting benchmark show that Forecast-MAE, which utilizes standard Transformer blocks with minimal inductive bias, achieves competitive performance compared to state-of-the-art methods that rely on supervised learning and sophisticated designs. Moreover, it outperforms the previous self-supervised learning method by a significant margin. Code is available at https://github.com/jchengai/forecast-mae.

1. Introduction

Self-supervised learning remains underexplored in motion forecasting, where annotated data and multimodal inputs complicate pre-training. Forecast-MAE uses masked scene reconstruction to learn connections among agents’ trajectories and road networks, achieving competitive benchmark performance with a simple Transformer-based design.

1. Introduction

  • Motion forecasting is important for autonomous driving and challenging because agent behavior is multimodal and road networks vary.The task predicts future trajectories of vehicles and pedestrians while accounting for interactions and road layouts.Source: §1 · p. 1
  • Motion-forecasting pre-training relies on annotated tracking sequences and hand-crafted HD maps, making large-scale SSL pre-training challenging.These inputs typically require expensive onboard sensors and human annotation; the paper follows an existing-dataset approach.Source: §1 · pp. 1–2
  • Motion forecasting combines agent motion, map features, and scene context, while prior SSL approaches had not jointly modeled these modalities.PreTraM focused on history-trajectory/map connections, and SSL-Lanes used modality-specific pretexts.Source: §1 · p. 2
  • Complementary masking of agent history or future trajectories and random lane-segment masking trains a unified scene-reconstruction task.The design aims to learn bidirectional motion relationships and cross-modal connections between trajectories and road networks.Source: §1 · p. 2
  • Forecast-MAE achieves competitive performance against supervised state-of-the-art methods and significantly outperforms the previous self-supervised method on Argoverse 2.The approach uses standard Transformers with minimal inductive bias.Source: §1 · p. 2

2. Related Work

Related work describes progress through scene representations, model architectures, and prior knowledge, alongside a small but growing body of SSL research. Forecast-MAE takes a different direction by jointly learning multimodal representations through masked autoencoding.

Related Work

  • Scene representations have shifted from rasterized images toward vectorized and graph-based representations, which retain more structural information.VectorNet [13] helped popularize vectorized representations; other work explores graphs [26] [49] [9] [25].Source: §2 · p. 3
  • Motion-forecasting architectures have adopted Transformers and graph neural networks, with MTR drawing on DETR [2].Earlier rasterized approaches commonly used convolutional networks.Source: §2 · p. 3
  • Some forecasting methods use candidate trajectories, anchor points, drivable-area goals, or temporal constraints, often with added computation or model complexity.These approaches incorporate prior knowledge to address multimodal future prediction.Source: §2 · p. 3
  • Forecast-MAE uses self-supervised learning with a standard Transformer and minimal prior knowledge to compete with sophisticated supervised methods.This offers a different direction amid increasing architectural complexity and use of prior knowledge.Source: §2 · p. 3
  • Prior SSL work either used extra map patches and separate encoders or focused on modality-specific pretexts, whereas Forecast-MAE jointly learns cross-modal interconnections.The paper reports that Forecast-MAE outperforms SSL-Lanes by a large margin.Source: §2 · p. 3

3. Methodology

Forecast-MAE adapts masked autoencoding to motion forecasting by pre-training on visible agent trajectories and lane segments, then reconstructing masked elements. Complementary trajectory masking and random lane masking form its central pretext design.

3. Methodology

  • Forecast-MAE embeds visible history/future trajectories and lane segments for Transformer processing, then reconstructs masked elements.The framework is designed specifically for self-supervised pre-training of motion forecasting.Source: §3 · p. 3

3.1. Masking

  • Agents retain either history or future while lane segments are randomly masked; typed decoder mask tokens reconstruct the hidden trajectories and lanes.The complementary constraint avoids reconstructing a trajectory from only a single pose, which the authors describe as a meaningless pretext task.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Input Representation and Embedding

  • Agent trajectories and lane segments are represented as polylines; FPN embeds motion, mini-PointNet embeds lanes, and pose embeddings add global position.Agent motion features are fused across scales, while lane polylines are embedded with a lightweight network.Source: §3.2 · p. 4

3.3. AutoEncoder

  • A standard Transformer encoder processes only visible trajectory and lane tokens; the decoder receives their latents with learned type-specific mask tokens to reconstruct missing elements.Decoded mask tokens are passed through simple prediction heads.Source: §3.3 · pp. 4–5

3.4. Reconstruction Target

  • Prediction heads reconstruct normalized 2D coordinates for masked trajectories and lane polylines, using L1 trajectory losses and MSE lane loss.The loss weights correspond to history, future, and lane reconstruction terms.Source: §3.4 · p. 5

3.5. Motion Forecasting

  • Fine-tuning removes the MAE decoder, mask tokens, and future inputs, replacing pretext heads with a multimodal future decoder trained using Huber regression and confidence cross-entropy.A three-layer MLP predicts future modes, a separate three-layer MLP predicts their confidence, and winner-take-all optimizes the best prediction across scene agents.Source: §3.5 · p. 5

4. Experiments

Experiments on Argoverse 2 assess masking choices and compare Forecast-MAE with supervised methods, SSL-Lanes, and training from scratch. The final model performs strongly on the benchmark, while ablations examine masking ratios, strategy, and encoder depth.

4.1. Experimental Setup

  • The Argoverse 2 evaluation uses 250K non-overlapping scenarios, with 5 seconds of history and 6 seconds of future prediction, and official metrics including minADE, minFDE, MR, and brier-minFDE.The dataset is split into 199,908 training, 24,988 validation, and 24,984 test samples; metrics refer to six prediction modes unless specified.Source: §4.1 · p. 5

4.2. Ablation Study

  • History masking performs best at 40–50%, while 20% and 80% underperform; lane masking works across 30–60%, but >70% hurts and <20% raises ADE and FDE.Figure 3 compares fine-tuning metrics with a baseline trained from scratch for 30 epochs; lower is better.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
  • Complementary history/future masking achieves the best performance across all metrics; masking history while leaving future unmasked falls below scratch, whereas adding future masking improves minADE.The experiments compare different inputs and masking strategies.Source: Table 1 · p. 6, §4.2 · p. 6
  • Increasing encoder depth from 2 to 4 improves minADE by 4.6%; deeper encoders add no significant gains, making depth 4 the best performance-efficiency trade-off.Table 2 reports the encoder-depth comparison.Source: Table 2 · p. 6, §4.2 · p. 6

4.3. Results

  • The six-variant ensemble leads all methods across six metrics and beats GANet by 7.5% in minFDE1; without ensembling, Forecast-MAE leads minADE1 and minFDE1.The leaderboard comparison covers Argoverse 2 test results.Source: Table 3 · p. 7, §4.3 · p. 7
  • Forecast-MAE significantly outperforms all tested SSL-Lanes variants across metrics and achieves consistent validation and test performance, unlike SSL-Lanes’ reported test degradation.The comparison adapts SSL-Lanes to Argoverse 2 and uses its lane masking, distance-to-intersection, and success-failure classification pretext tasks.Source: §4.3 · p. 7
  • Fine-tuning improves all metrics over scratch, and extra scratch training does not help; across three held-out cities, fine-tuning also beats the baseline on all metrics.The cross-city evaluation trains on Miami, Pittsburgh, and Austin and evaluates on Dearborn, Palo-Alto, and Washington-DC.Source: §4.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 8

5. Conclusion

Forecast-MAE uses complementary trajectory masking and random lane-segment masking to learn bidirectional motion, road geometry, and cross-modal connections. On Argoverse 2, it surpasses supervised and prior self-supervised methods, while transfer and scaling remain unexplored.

  • Forecast-MAE jointly learns bidirectional agent-motion features, road geometry, and cross-modal connections through complementary trajectory masking and random lane-segment masking.Source: §5 · p. 8
  • Forecast-MAE surpasses supervised methods and previous self-supervised works on Argoverse 2, especially on minADE1 and minFDE1.Source: §5 · p. 8
  • Transfer and few-shot learning were not explored, and limited public dataset size prevents determining whether performance scales with more training data and model capacity.Source: §5 · p. 8

A. Implementation Details

The implementation uses a convolutional and neighborhood-attention history encoder, a PointNet-style lane embedder, and end-to-end fine-tuning. The paper reports experiment settings and describes training and adaptations for SSL-Lanes.

B. Additional Results

Additional results include qualitative comparisons on Argoverse 2, masked-scene reconstructions, and preliminary results on Argoverse 1 and WOMD. The visual comparisons report advantages in direction and velocity prediction and reconstruction under high lane masking.

Loading 2308.09882v1…