PaperlayerView exact revision on arXiv

Source-linked AI summary

HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding

Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, Junchi Yan

arXiv:2205.09753v2cs.AIcs.CVcs.LGcs.RO

TL;DR

HDGT addresses the challenge of encoding heterogeneous driving-scene elements, semantic relations, and relative spatial structure for trajectory prediction. It builds a heterogeneous graph, uses node-centric local coordinates and hierarchical Transformer aggregation, and achieves state-of-the-art results on two large-scale benchmarks.

  • Problem

    Driving scenes contain diverse object types, semantic relations, and relative spatial relationships that require comprehensive, ego-centric scene representations for trajectory prediction.

  • Method

    HDGT models agents and map elements as heterogeneous graph nodes, semantic relations as edges, localizes node and in-edge features, and updates them with hierarchical type-specific Transformers.

  • Results

    HDGT achieves state-of-the-art trajectory-prediction performance on the INTERACTION Prediction Challenge and Waymo Open Motion Challenge.

  • Takeaways & Limitations

    Symmetric local-coordinate encoding supports generalization and enables predicting multiple agents in one forward without performance drop.

Abstract

from arXiv · show

Encoding a driving scene into vector representations has been an essential task for autonomous driving that can benefit downstream tasks e.g. trajectory prediction. The driving scene often involves heterogeneous elements such as the different types of objects (agents, lanes, traffic signs) and the semantic relations between objects are rich and diverse. Meanwhile, there also exist relativity across elements, which means that the spatial relation is a relative concept and need be encoded in a ego-centric manner instead of in a global coordinate system. Based on these observations, we propose Heterogeneous Driving Graph Transformer (HDGT), a backbone modelling the driving scene as a heterogeneous graph with different types of nodes and edges. For heterogeneous graph construction, we connect different types of nodes according to diverse semantic relations. For spatial relation encoding, the coordinates of the node as well as its in-edges are in the local node-centric coordinate system. For the aggregation module in the graph neural network (GNN), we adopt the transformer structure in a hierarchical way to fit the heterogeneous nature of inputs. Experimental results show that HDGT achieves state-of-the-art performance for the task of trajectory prediction, on INTERACTION Prediction Challenge and Waymo Open Motion Challenge.

1 INTRODUCTION

HDGT addresses heterogeneous, relational, and ego-centric structure in driving scenes with a unified graph Transformer. It models diverse node and edge types, uses local coordinate systems, and reports strong benchmark performance.

  • Driving scenes contain heterogeneous objects and relations, including agent states, traffic lights, lane connections, and road-line types.
  • HDGT models agents and map elements as different node types and their semantic relations as edge types in one heterogeneous graph.
  • HDGT uses distinct Transformer parameters for different node and edge types while updating all nodes and edges simultaneously.The unified structure is intended to support stacking and scaling across different data types and sizes.
  • HDGT encodes each node and its in-edges in a local ego-centric coordinate system rather than using a single global reference.For an edge u → v, the source feature is combined with coordinate transformation information from u to v.
  • The symmetric spatial encoding enables multi-agent future prediction in one forward without performance drop, according to the authors.
  • HDGT achieves state-of-the-art performance on two large-scale benchmarks, with reported first- and second-place rankings by minADE/minFDE during the stated period.

2 RELATED WORK

Prior trajectory-prediction methods variously encode agents, maps, and interactions, but often rasterize maps or use shared processing across heterogeneous elements. HDGT instead unifies these elements in a heterogeneous graph with parallel, type-specific updates.

  • Transformer methods aggregate representations through dot-product attention and feed-forward networks, with attention producing normalized weighted combinations of elements.
  • VectorNet-based methods and SceneTransformer place agents and lanes in one fully connected graph with shared aggregation parameters, ignoring node types and semantic relations.
  • Some prior methods use different parameters for agent types but rasterize map elements, while vector-based approaches preserve map structure more directly.
  • HDGT’s pipeline combines graph construction, relative spatial encoding, repeated graph-Transformer updates, and regression-classification decoding for multimodal prediction.
  • HDGT models all node types and semantic relations in one heterogeneous graph, updating node and edge features in parallel with type-specific parameters.

3.1 Problem Formulation and Approach Overview

The task predicts future trajectories from agents’ history states and map elements, allowing multiple predictions with confidence scores. HDGT’s overview builds and encodes a heterogeneous scene graph before decoding target-agent features.

  • Inputs include agents’ past L time-steps with coordinates, velocities, headings, box dimensions, and agent types.
  • Map inputs include lanes, road lines, crosswalks, stop signs, traffic lights, speed bumps, and curbs represented by polylines or polygons.
  • The task predicts agents’ trajectories over the next T time-steps, with multiple predictions and confidence scores representing future uncertainty.
  • HDGT constructs a heterogeneous scene graph from agent and map types and relations, then transforms node and edge features into local reference systems.
  • Heterogeneous Driving Graph Transformer layers aggregate and update node and edge features repeatedly, followed by prediction heads using extracted agent-node features.

3.2 Construction of the Heterogeneous Graph

HDGT represents each scene as a directed heterogeneous graph whose nodes and edges encode object types and semantic relations. Connectivity uses flexible spatial thresholds and lane topology to organize relevant interactions.

  • HDGT represents each driving scene as one directed heterogeneous graph with node-type and edge-type mapping functions.
  • Edge types are determined by source-node type, destination-node type, and their relation, with all possible semantic-relation combinations considered.
  • Agent connectivity uses a distance threshold equal to speed times the prediction horizon plus a type-specific buffer value.
  • Lane and traffic-sign nodes are inversely connected to agents, while lane nodes also connect through left, right, entry, and exit relations.
  • Long lanes are divided into shorter segments with similar lengths so lane nodes receive more similar inputs.

3.3 Relative Spatial Relation Encoding

HDGT encodes each node and its spatial relations in local ego-centric coordinates, then converts heterogeneous geometric inputs into fixed-length representations for graph processing.

  • Symmetric scene encoding: Symmetric local encoding lets all agents process information in their own coordinate systems and supports predicting the entire scene in a single forward.This avoids forwarding multiple times with each agent as the reference.
  • Node feature encoding: Each node is initialized in its own ego-centric coordinate system, with coordinates and velocities translated and rotated relative to its reference.Agent headings are also expressed relative to the node reference orientation.
  • Node feature encoding: Agent histories are compressed with shared 1D CNNs, while polylines and polygons are encoded into single vectors with PointNet.The PointNet variant applies shared pointwise MLPs, max pooling, concatenation, and three stacked layers for map elements.
  • Edge feature encoding: For an edge u →v, HDGT transforms the source representation into the target view by combining it with the coordinate transformation ΔPoseu→v and applying an MLP.The transformation uses relative position and orientation information, and the MLP depends on the source-node type.
  • Design rationale: Relative spatial encoding separates single-object motion or geometry from pairwise spatial relations, reducing reliance on absolute coordinate values.The resulting inputs are represented in ego-centric coordinates and are less prone to overfitting absolute positions.

3.4 Heterogeneous Driving Graph Transformer Layer

HDGT uses Transformer-style aggregation and updates on heterogeneous nodes and edges, processing semantic relation types separately before integrating them.

  • Layer structure: HDGT alternates node and edge aggregation-update operations across K GNN layers to extract higher-order relations among agents and map elements.Node and edge features are updated simultaneously, with residual connections and layer normalization used for stability.
  • Design rationale: The hierarchical design addresses heterogeneous in-edges because agents, lanes, and traffic signs carry different kinds of information.The method separates semantic relations before combining them, rather than treating all in-edges as homogeneous.
  • Node aggregation and update: Node aggregation applies separate multi-head attention to each in-edge type, then concatenates the type-specific outputs and integrates them with an MLP.Node features provide queries, while in-edge features provide keys and values for each semantic relation type.
  • Heterogeneous updates: Different Transformer parameters are assigned to different node and edge types to preserve heterogeneity during information exchange.Node updates use type-specific feed-forward parameters, while edge updates use type-specific MLPs.
  • Edge aggregation and update: Edge aggregation retrieves the source-node feature and the pose transformation into the target view, then updates the edge with its previous feature through an MLP.The edge feature represents the source element from the target element’s local perspective.

3.5 Output Head and Training Objective

HDGT predicts multiple trajectory modes with regression and classification heads, training regression on the best-matching mode while encouraging diverse predictions.

  • Output head: The output head combines regression and classification MLPs within the MTP-loss formulation to produce trajectory modes and their confidences.Each mode contains predicted coordinates over time and a confidence score.
  • Training objective: For each agent, only the mode with the lowest regression loss receives regression gradients, while cross-entropy trains the classification head.The lowest-loss mode is positive and the remaining modes are negative.
  • Training objective: The regression loss uses the smooth L1 function, with the final objective combining regression and classification terms using λ = 0.1.The paper keeps λ fixed at 0.1 throughout training.

4 EXPERIMENTS

The experiments evaluate HDGT on large-scale driving datasets using trajectory-prediction metrics, multi-mode outputs, and specified implementation settings.

  • Metrics: The evaluation uses minADE, minFDE, and MR, with additional benchmark-specific metrics including minJointADE, minJointFDE, SMR, and mAP.The metrics are averaged over all target agents.
  • Datasets: The study evaluates HDGT on INTERACTION and Waymo Open Motion, two popular large-scale datasets.INTERACTION contains interactive scenarios with 1 second of history and 3 seconds of future vehicle trajectories.
  • Datasets: INTERACTION permits up to 6 prediction modes, while the model produces multi-mode trajectory predictions for evaluation.The INTERACTION input is sampled at 10 Hz, with 40K vehicle tracks in total.
  • Implementation Details: HDGT uses hidden dimension 128 and three graph-transformer layers on INTERACTION, while the Waymo model uses six layers and 12.1 million parameters.The implementation uses PyTorch, DGL, AdamW, and NVIDIA Tesla V100 GPUs.

4.2 Compared Baselines

The compared baselines represent diverse scene encoders and trajectory decoders, including rasterized, vector-based, graph, recurrent, heatmap, and Transformer approaches.

  • Compared Methods: DenseTNT uses a homogeneous fully connected VectorNet graph and selects goals from a dense candidate set.Its optimization-based decoding is intended to generate more diverse modes.
  • Compared Methods: MultiModalTransformer builds separate fully connected agent-agent and agent-map graphs and applies Transformers to both.Its decoder combines classification and regression MLPs.
  • Compared Methods: GOHOME and THOMAS use LaneGCN-like graph structures with heatmap-based prediction, with THOMAS adding hierarchical generation and attention-based decoding.Both methods use additional mechanisms to support diverse or consistent multi-agent predictions.
  • Compared Methods: ReCoG2 uses separate recurrent and convolutional encoders for vehicles and lanes, followed by GCN-based relation modeling and recurrent decoding.Its map representation is rasterized and its GCN does not model node heterogeneity.
  • Compared Methods: SimpleCNNOnRaster and ReCoAt encode map information with CNNs on rasterized images, while their agent encoders and decoders differ.SimpleCNNOnRaster also rasterizes agents, whereas ReCoAt uses an RNN for agents.
  • Compared Methods: SceneTransformer constructs a fully connected graph over agents and map elements and applies factorized spatial and temporal Transformers.This provides a Transformer-based comparison for scene encoding.
  • Compared Methods: Multipath++ uses relative-reference scene encoding and an ensemble decoder.It is described as a concurrent work.

4.3 Overall Performance Evaluation

HDGT achieves strong leaderboard performance and benefits from relative coordinates, HD-map information, semantic relations, and type-specific graph parameters.

  • Overall Results: HDGT ranks first in all four INTERACTION tracks and second on the Waymo Open Motion Leaderboard by minADE/minFDE.These rankings were reported between the final INTERACTION submission and the paper’s initial submission.
  • INTERACTION: HDGT outperforms other candidates by a large margin on INTERACTION minADE/minFDE metrics.It outperforms MultiModalTransformer and ReCoG2 across all reported metrics.
  • Waymo Open Motion: HDGT ranks first among single models on Waymo minADE, minFDE, and MR.The table notation distinguishes concurrent MultiPath++ ensemble results from single-model results.
  • Waymo Open Motion: Vector-based methods show an advantage over raster-based methods, while HDGT outperforms SceneTransformer with 12.1 million versus 15.3 million parameters.HDGT uses one sparsely connected heterogeneous graph, whereas SceneTransformer applies attention over two factorized fully connected graphs.
  • Ablation Study: Relative node-centric coordinates improve validation performance over a fixed global reference.Pose Change performs better than Relative Edge among the evaluated relative-relation strategies.
  • Ablation Study: Adding HD-map information improves performance, reflecting the role of lane and curb semantics in trajectory prediction.The ablation compares a homogeneous graph with HD-map information against an agents-only model.
  • Ablation Study: Semantic relations improve performance, and type-specific parameters for heterogeneous nodes and edges provide an additional significant boost.The full HDGT configuration outperforms the semantic graph with shared GNN parameters.

4.6 Analysis of Heterogeneous Graph Design

The analysis evaluates HDGT’s heterogeneous graph design, Transformer aggregation, and scalability across datasets of different sizes. Results show that Transformer aggregation and lane topology are especially important, while deeper models continue helping on the larger Waymo dataset.

  • Heterogeneous Graph Design: Replacing Transformer aggregation with a GCN-like function causes the largest performance drop.The comparison keeps different parameters for node and edge types while changing the aggregation and update functions.
  • Heterogeneous Graph Design: Ignoring lane topology causes the second-largest performance drop.The authors relate this result to vehicles tending to move along lanes and lane topology reducing prediction uncertainty.
  • Heterogeneous Graph Design: Merging lane and traffic-sign node types causes the least performance drop among the tested design changes.The experiment compares merged lane connectivity, homogeneous map nodes, and a GCN-like function against the proposed design.
  • Model Capacity: On INTERACTION’s 40K tracks, performance saturates at around 3 Heterogeneous Driving Graph Transformer layers.The scalability experiments also report inference time across validation sets.
  • Model Capacity: On Waymo Open Motion’s 7.64M tracks, performance continues improving with 6 layers.The authors interpret this as evidence of HDGT’s scalability when more data are available.
  • Qualitative Visualization: Visualized predictions use diverse scene elements and relations, including agents, lanes, stop signs, and traffic lights.The visualization is presented as evidence that explicitly modeling heterogeneous driving scenes benefits trajectory prediction.

5 CONCLUSION

HDGT extracts trajectory-prediction representations by modeling driving scenes as heterogeneous graphs and using Transformer-based graph aggregation and updates. It normalizes spatial features locally, achieves state-of-the-art results on two large-scale benchmarks, and is supported by ablation studies.

  • Conclusion: HDGT models the driving scene as a heterogeneous graph and uses Transformer aggregation and update functions.The graph contains heterogeneous scene information, while Transformer operations provide the graph processing mechanism.
  • Conclusion: HDGT normalizes spatial features into each node’s local coordinate system during aggregation.This design encodes spatial information in a local reference rather than a global one.
  • Conclusion: HDGT achieves state-of-the-art results on two recent large-scale and competitive benchmarks.The conclusion also reports thorough ablation studies validating the effectiveness of each module.
Loading 2205.09753v2…