PaperlayerView exact revision on arXiv

Source-linked AI summary

Wayformer: Motion Forecasting via Simple & Efficient Attention Networks

Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, Benjamin Sapp

arXiv:2207.05844v1cs.CV

TL;DR

Motion forecasting must encode heterogeneous scene inputs while representing diverse possible futures, yet existing systems often rely on complex modality-specific designs. Wayformer uses a homogeneous attention-based scene encoder and decoder with alternative fusion and efficiency mechanisms, and reports state-of-the-art results on WOMD and Argoverse. The study also identifies scope limitations involving dense-scene computation, sparse state inputs, and independent per-agent future modeling.

  • Problem

    Motion forecasting requires encoding heterogeneous agent, road, traffic-light, and interaction inputs while representing multimodal futures, but prior systems use complex modality-specific designs.

  • Method

    Wayformer uses an attention-based scene encoder and cross-attention decoder, studying early, late, and hierarchical fusion with factorized and latent query attention.

  • Results

    Wayformer achieves state-of-the-art results on both WOMD and Argoverse challenges.

  • Takeaways & Limitations

    Simple, modality-agnostic design choices can provide strong motion-forecasting quality while exposing direct quality–latency trade-offs.

  • Takeaways & Limitations

    The study uses sparse abstract state inputs and independent per-agent future distributions, limiting representation of interactive-scene nuances and joint futures.

Abstract

from arXiv · show

Motion forecasting for autonomous driving is a challenging task because complex driving scenarios result in a heterogeneous mix of static and dynamic inputs. It is an open problem how best to represent and fuse information about road geometry, lane connectivity, time-varying traffic light state, and history of a dynamic set of agents and their interactions into an effective encoding. To model this diverse set of input features, many approaches proposed to design an equally complex system with a diverse set of modality specific modules. This results in systems that are difficult to scale, extend, or tune in rigorous ways to trade off quality and efficiency. In this paper, we present Wayformer, a family of attention based architectures for motion forecasting that are simple and homogeneous. Wayformer offers a compact model description consisting of an attention based scene encoder and a decoder. In the scene encoder we study the choice of early, late and hierarchical fusion of the input modalities. For each fusion type we explore strategies to tradeoff efficiency and quality via factorized attention or latent query attention. We show that early fusion, despite its simplicity of construction, is not only modality agnostic but also achieves state-of-the-art results on both Waymo Open MotionDataset (WOMD) and Argoverse leaderboards, demonstrating the effectiveness of our design philosophy

1 Introduction

Wayformer addresses the multimodal and heterogeneous nature of motion forecasting with a simple, modality-agnostic attention architecture. The paper studies fusion and efficiency choices and reports state-of-the-art results with minimal domain-specific assumptions.

  • Motivation: Future behavior prediction must represent diverse possible agent trajectories from heterogeneous temporal, spatial, and dynamic inputs.Inputs include agent histories, road geometry and connectivity, and time-varying traffic-light information.
  • Motivation: Previous approaches often use hand-engineered modality-specific architectures and preprocessing, making quality–latency trade-offs harder to control.The paper identifies excessive modality-specific tuning and heterogeneous preprocessing as obstacles to simpler model design.
  • Approach: Wayformer combines a self-attention scene encoder with a cross-attention decoder that generates diverse trajectories from multimodal scene data.The model family studies early, late, and hierarchical modality fusion.
  • Approach: The model family uses factorized attention and latent query attention to trade computational efficiency against forecasting quality.These techniques are introduced as ways to keep motion-forecasting models within practical real-time constraints.
  • Results: Wayformer achieves state-of-the-art results on both WOMD and Argoverse challenges.The experiments are presented as evidence that simple design choices and minimal domain-specific assumptions can be effective.

2 Multimodal Scene Understanding

Wayformer represents driving scenes as aligned multimodal tensors covering agent history, agent interactions, road geometry, and traffic-light state. The scene encoder uses these representations to support alternative fusion strategies.

  • Scene modalities: The scene representation includes road information, traffic-light state, agent history, and agent interactions across temporal and spatial dimensions.The notation supports modeled ego-agents, timesteps, feature dimensions, and contextual-object sets.
  • Agent modalities: Agent history stores past and current agent states, including position, velocity, acceleration, and bounding-box features.A context dimension is included for homogeneous tensor representations.
  • Agent modalities: Agent interactions represent nearby context agents in each modeled agent’s reference frame.The context set contains a fixed number of closest agents considered relevant to the modeled agent’s behavior.
  • Static and signal modalities: The roadgraph represents nearby road segments as endpoint-defined polylines annotated with type information.Road features use a singleton time dimension to remain homogeneous with other modalities.
  • Static and signal modalities: Traffic-light inputs contain nearby signal states and features describing each signal’s position and confidence.These features are represented per modeled agent and timestep.

3 Wayformer

Wayformer uses a compact scene encoder–decoder architecture to fuse heterogeneous driving modalities and generate diverse trajectories. It studies early, late, and hierarchical fusion alongside attention mechanisms that reduce computational cost for multidimensional inputs.

  • Architecture: Wayformer combines an attention-based scene encoder with a Transformer cross-attention decoder that generates k possible trajectories.Learned initial queries attend to the scene encoding to produce trajectory outputs.
  • Input representation: Input modalities are projected to a common dimension and combined into a scene representation Z across temporal and spatial dimensions.The scene representation is formed after modality-specific projections and positional embeddings.
  • Fusion: Early fusion uses one cross-modal self-attention encoder, while late and hierarchical fusion allocate self-attention capacity separately or between modality-specific and cross-modal encoders.Early fusion limits modality-specific parameters to projection layers; hierarchical fusion splits encoder depth across the two encoder types.
  • Attention: Multi-axis attention processes spatial and temporal dimensions together, whereas factorized attention applies self-attention over each dimension individually to reduce computational cost.The paper compares sequential and interleaved arrangements of temporal and spatial attention blocks.
  • Attention: Latent query attention maps the input sequence into a shorter latent space before subsequent encoder blocks process the latent representation.The reduction factor is defined as R = Lout/Lin and controls latent resolution.
  • Trajectory decoding: The decoder represents possible trajectories with a mixture of Gaussians, estimating component likelihoods and Gaussian parameters from decoder embeddings.Separate projections produce mixture likelihoods and four time series corresponding to Gaussian means and log-standard deviations.

4 Experimental Setup

The experiments evaluate Wayformer on WOMD and Argoverse using dataset-specific metrics and controlled model configurations. Training, capacity, mixture-output, and latent-query settings are specified to compare model variants and practical trade-offs.

  • Datasets: WOMD contains 1.1M examples from 103K driving scenarios, with 1 second of history and an 8-second future sampled at 5Hz.The dataset includes agent state attributes such as position, dimensions, velocity, acceleration, orientation, and turn signals.
  • Datasets: Argoverse contains 333K scenarios with 2 seconds of history and a 3-second future prediction horizon sampled at 10Hz.Inputs include trajectory histories, context agents, and lane centerlines.
  • Evaluation: Evaluation uses competition-specific metrics and considers only the top k = 6 most likely modes, using each mode’s mean trajectory.This output restriction is applied after trajectory aggregation.
  • Training: Models are trained with AdamW from an initial learning rate of 2e-4, linearly decayed to zero over 1M steps, using a total batch size of 256.Training uses 16 TPU v3 cores with a batch size of 16 per core.
  • Model configurations: Model capacity varies hidden size across {64, 128, 256} and depth across {1, 2, 4} Transformer layers.The feedforward intermediate size is fixed at either 2 or 4 times the hidden size.
  • Experimental variants: Architecture studies use m = 6 Gaussian components without aggregation, whereas benchmark experiments use m = 64 components followed by trajectory aggregation.Latent-query experiments reduce input resolution to 0.25, 0.5, 0.75, or 0.9 of the original sequence length.

5 Results

Wayformer experiments characterize quality–latency trade-offs across fusion strategies and attention mechanisms. Early fusion performs strongly on benchmarks, while latent queries substantially accelerate models with minimal quality regression.

  • Experimental setup: Models spanning 0.3M–20M parameters were evaluated on WOMD using inference latency, capacity, and minADE as quality measures.The ablations trained models for 1M steps and reported latency on a current-generation GPU.
  • Multi-Axis Attention: For models with latency x ≤16 ms, late fusion is optimal; hierarchical fusion gains quality at 16ms < x < 32ms, while early fusion matches it when x > 32ms.Late fusion avoids modality interaction during scene encoding, whereas hierarchical fusion adds a cross-modal encoder.
  • Multi-Axis Attention: Small models perform best with early fusion, but increasing capacity sharply reduces sensitivity to fusion choice.
  • Factorized Attention: Sequential and interleaved factorized attention perform similarly across fusion types, while factorized attention improves quality for early and late fusion.
  • Factorized Attention: Factorized attention reduces latency only for late fusion because road-graph tiling increases token counts in early and hierarchical cross-modal encoders.
  • Latent Queries: 2x-16x speedups are achieved with latent queries and minimal to no quality regression across all fusion models.Early and hierarchical fusion still provide the best quality, highlighting the role of cross-modal interaction.
  • Benchmark Results: Wayformer early-fusion models outperform other models across all Waymo Open Motion Dataset metrics, independent of multi-axis or factorized attention.The paper reports benchmark results on both Waymo Open Motion Dataset and Argoverse.

6 Related Work

Related motion-forecasting work uses rasterized representations, iterative cross-attention, or factorized attention to handle multimodal and long-sequence inputs. These approaches trade modeling flexibility against computational cost and modality-specific design.

  • Motion Forecasting Architectures: Motion-prediction models broadly fuse static, dynamic, social, and temporal scene aspects, reflecting the increasing availability of self-driving benchmarks.
  • Rasterized Representations: Rasterized top-down representations fuse multimodal scene information through spatio-temporal convolutional networks.The approach renders scene relationships in an orthographic plane, but convolutional structure is localized.
  • Iterative Cross-Attention: Iterative cross-attention processes modalities sequentially, including agent history, nearby agents, and maps, to reduce the cost of joint self-attention.
  • Factorized Attention: Factorized attention limits each operation to one axis to reduce the computational cost of self-attention on long, high-dimensional sequences.Prior applications include image generation and video classification.
  • Multimodal Encoding: Prior multimodal encoding work argues that independent temporal and spatial attention can lose information, while joint early self-attention increases computation for large inputs.

7 Limitations

The study’s scope is limited by repeated ego-centric computation, sparse abstract inputs, and independent per-agent future modeling. These choices improve efficiency but constrain scene detail and interactive multi-agent behavior.

  • Scope limitations: Ego-centric modeling repeats computations on dense scenes; encoding the scene once in a global reference frame could alleviate this limitation.
  • Scope limitations: Sparse abstract state inputs omit visual pedestrian cues and fine-grained vehicle contour or wheel-angle information.End-to-end perception and prediction could unlock improvements.
  • Scope limitations: Independent per-agent future distributions and conditional temporal independence simplify computation but fail to represent combinatorially many futures.The paper identifies multi-agent, temporally causal models as a possible route to further benefits in interactive situations.

A Factorized Latent Query Attention

Wayformer’s factorized latent-query architecture replaces selected initial encoder blocks with latent-query versions. Figure 7 summarizes the available blocks and their arrangement into complete encoders.

  • Encoder blocks: Factorized latent-query attention uses temporal and spatial latent-query encoder blocks in place of the first corresponding encoder blocks.
  • Architecture summary: Figure 7a overviews factorized latent-query attention encoder blocks, while Figure 7b shows how blocks are arranged to construct encoders.

B Hyperparameters

The ablation experiments use fixed model and input-feature hyperparameters, while trajectory aggregation selects and refines centroid modes to compress predictions.

  • Tables 2 and 3 specify model, training, and input-feature hyperparameters fixed or used across WOMD ablation experiments.
  • Trajectory aggregation first selects the fewest centroid modes that cover all output modes within distance threshold D.Centroids are chosen greedily by covered likelihood until every output mode is covered.
  • The refinement stage iteratively updates each centroid as the weighted average of its assigned output modes and reassigns modes to the closest centroid.

D SOTA Wayformer Details

Benchmark experiments use dataset-specific hyperparameters, with WOMD settings documented in Table 4 and Argoverse settings documented in Table 5.

  • Argoverse benchmark hyperparameters are documented in Table 5.The text identifies Tables 4 and 5 as the sources for WOMD and Argoverse benchmark settings, respectively.
  • WOMD benchmark experiments use model and training hyperparameters documented in Table 4.

E Metrics

The evaluation reports competition-specific motion-forecasting metrics using six aggregated, mean trajectories, alongside qualitative WOMD comparisons between Wayformer and MP++. The examples emphasize differences in collision avoidance, road following, interaction modeling, and yielding behavior.

  • Evaluation setup: Evaluation uses only the top k = 6 most likely modes after trajectory aggregation and the mean of each mode.
  • Metrics: The metric set includes minimum distance error, miss rate, minimum average distance error, mAP, overlap, minimum final displacement error, and brier −minFDE.The supplied passages define these metrics through distance, miss, precision-recall, overlap, endpoint, and probability terms.
  • Metrics: mAP averages behavior-bucket average precision values computed from precision-recall curves over possible true and false positives per agent.
  • Qualitative comparison: Qualitative WOMD figures compare Wayformer and MP++ predictions, using hue for the 0s–8s horizon and transparency for probability.Rectangles represent vehicles, while squares represent pedestrians or cyclists.
  • Qualitative comparison: Across examples, Wayformer predictions more often avoid collisions with parked or stopped entities and account for pedestrian and vehicle interactions.The figures describe MP++ trajectories passing through other agents or vehicles, whereas Wayformer predicts crossing, waiting, yielding, or nudging behavior around them.
  • Qualitative comparison: Wayformer also produces more road-following behavior in parking, lane, and intersection scenarios, including constrained turns, merges, and parking-lane entry.The examples contrast these predictions with MP++ trajectories that are off-road, collide, or assign broadly distributed futures.
Loading 2207.05844v1…