Source-linked AI summary
Learning Lane Graph Representations for Motion Forecasting
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, Raquel Urtasun
TL;DR
Motion forecasting needs map representations that retain HD-map topology while accounting for interactions among traffic actors and lanes. The paper constructs a vectorized lane graph, applies LaneGCN, and fuses actor-map information through four interaction types; it significantly outperforms the state-of-the-art on Argoverse.
Problem
Rasterized maps lose information and make 2D convolutions inefficient for complex topology and long-range lane dependencies.
Method
The model constructs a lane graph from vectorized map data, uses LaneGCN for structured map features, and models four actor-map interaction types.
Results
The model significantly outperforms the state-of-the-art on the large-scale Argoverse motion forecasting benchmark.
Takeaways & Limitations
LaneGCN and complete actor-map fusion support prediction of realistic trajectories using structured map representations.
Takeaways & Limitations
The model currently uses lane centerlines and connectivity, excluding other map information such as traffic lights and traffic signs.
Abstract
from arXiv · showhide
We propose a motion forecasting model that exploits a novel structured map representation as well as actor-map interactions. Instead of encoding vectorized maps as raster images, we construct a lane graph from raw map data to explicitly preserve the map structure. To capture the complex topology and long range dependencies of the lane graph, we propose LaneGCN which extends graph convolutions with multiple adjacency matrices and along-lane dilation. To capture the complex interactions between actors and maps, we exploit a fusion network consisting of four types of interactions, actor-to-lane, lane-to-lane, lane-to-actor and actor-to-actor. Powered by LaneGCN and actor-map interactions, our model is able to predict accurate and realistic multi-modal trajectories. Our approach significantly outperforms the state-of-the-art on the large scale Argoverse motion forecasting benchmark.
1 Introduction
The paper addresses motion forecasting by preserving HD-map structure explicitly and modeling interactions between actors and maps. Its LaneGCN-based model significantly improves performance on Argoverse.
- Motivation: HD maps provide geometric and semantic information that helps forecasting models produce plausible and accurate trajectories.Actor behavior depends strongly on map topology, such as the availability of a left-turn lane.
- Limitations of Earlier Approaches: Heuristic map constraints can miss rare, non-compliant behaviors that may be safety critical.
- Limitations of Rasterization: Rasterized map representations lose information and use 2D convolutions inefficiently for complex topology and long-range lane dependencies.Spatially close lanes can nevertheless have different directional semantics and dependencies.
- Contributions: The model constructs a lane graph from vectorized map data and uses LaneGCN to capture topology and long-range dependencies.
- Contributions: The model represents actors and lanes as graph nodes and models actor-to-lane, lane-to-lane, lane-to-actor, and actor-to-actor interactions.
- Results: The approach significantly outperforms the state-of-the-art on the large-scale Argoverse motion forecasting benchmark.
2 Related Work
Prior work represents HD maps with lane geometry, connectivity, and semantic attributes, often using rasterized inputs for neural models. The paper situates structured lane graphs within this broader map-representation literature.
- Map Representations: HD maps have been represented with polylines, graphical models, and road-layout graphs to encode lane geometry and connectivity.
- Learning Map Representations: Rasterization-based methods encode roads, crosswalks, traffic controls, and entity histories as bird’s-eye-view image layers.
- Learning Map Representations: Prior work has fused rasterized maps with LiDAR for perception and prediction, and used them for end-to-end motion planning.
3 Lane Graph Representations for Motion Forecasting
The model separates actor and map encoding before fusing them through four interaction types to predict multi-modal trajectories. ActorNet processes observed trajectories, while MapNet builds structured lane features with LaneGCN.
- FusionNet: FusionNet uses actor-to-lane, lane-to-lane, lane-to-actor, and actor-to-actor blocks to exchange traffic and map information.Lane-to-lane propagation uses LaneGCN, while the other interaction blocks use spatial attention.
- Prediction: The prediction header converts after-fusion actor features into multi-modal future trajectories.
- ActorNet: ActorNet receives observed actor trajectories and uses 1D convolutions to extract actor features.The trajectory input contains displacement sequences and a padding mask.
- MapNet: MapNet builds a lane graph from vectorized map data and applies LaneGCN to produce map features.
- Map Representation: Each lane is represented by a centerline sequence with predecessor, successor, left-neighbor, and right-neighbor connectivity.
Map Data:
The lane graph uses fine-grained segment nodes and typed adjacency matrices to preserve directional topology. LaneConv extends graph convolution with connection-specific propagation and dilation for long-range along-lane dependencies.
- Lane Graph Construction: Lane graph nodes are straight segments between consecutive centerline points, enabling fine-resolution map queries near actors.
- Lane Graph Construction: Each lane node has predecessor, successor, left-neighbor, and right-neighbor relations represented by four adjacency matrices.
- LaneConv Motivation: Vanilla graph convolution cannot preserve connection types or straightforwardly capture long-range lane dependencies.
- LaneConv: LaneConv assigns separate adjacency and weight matrices to different connection types, allowing directional and cross-lane information flow.
- Dilated LaneConv: Dilated LaneConv expands the receptive field along predecessor and successor directions to capture long-range dependencies needed for multi-second forecasting.
- LaneGCN: LaneGCN combines LaneConv components across multiple dilation sizes to form a multi-scale lane-graph operator.
LaneGCN:
LaneGCN uses multi-scale LaneConv residual blocks to represent lane-graph structure across multiple dilation sizes. Its four-block architecture uses 128-dimensional features throughout.
- LaneGCN is composed of four LaneConv residual blocks using dilation sizes 1, 2, 4, 8, 16, and 32.Each block stacks a multi-scale LaneConv with a linear layer and shortcut connection.
- Each LaneGCN block combines a multi-scale LaneConv, a linear layer, and a residual shortcut.
- All LaneGCN layers have 128 feature channels and apply layer normalization and ReLU after LaneConv and linear layers.
3.3 FusionNet
FusionNet models interactions between actors and lane nodes through four information flows. Spatial attention handles actor-to-lane, lane-to-actor, and actor-to-actor interactions, while LaneGCN handles lane-to-lane propagation.
- FusionNet captures actor-to-lane, lane-to-lane, lane-to-actor, and actor-to-actor information flows between actors and lane nodes.
- Actor-to-lane attention introduces traffic information such as lane blockage or usage into lane nodes.
- Lane-to-lane propagation updates lane features, and lane-to-actor fusion returns the updated map information to actors.
- Spatial attention is used for actor-to-lane, lane-to-actor, and actor-to-actor interactions over context nodes within specified distance thresholds.The thresholds are 7, 6, and 100 meters respectively for these three interaction types.
3.4 Prediction Header
The prediction header produces multiple future trajectories and confidence scores for each actor. It separates trajectory regression from mode-confidence classification and derives confidence from distance embeddings and actor features.
- For each actor, the header predicts K future trajectories together with confidence scores for their modes.
- The regression branch uses a residual block and linear layer to predict K sequences of bird’s-eye-view coordinates.
- The classification branch applies an MLP to produce confidence scores for the predicted modes.Distance embeddings are concatenated with the actor feature before confidence prediction.
3.5 Learning
The model is trained end to end with classification and regression losses. A positive trajectory is selected by minimum final displacement error, while max-margin and smooth ℓ1 losses supervise confidence and coordinates.
- All differentiable modules are trained end to end using the sum of classification and regression losses.
- The positive trajectory is the predicted mode with minimum final displacement error relative to the ground-truth final location.
- Classification uses a max-margin loss, while regression applies smooth ℓ1 loss across all predicted time steps.
- The regression loss compares predicted coordinates with ground-truth bird’s-eye-view coordinates at each time step.
4 Experimental Evaluation
The model is evaluated on Argoverse using actor and vectorized lane-map data, with benchmark comparisons, module ablations, operator ablations, and qualitative hard-case analysis. Results report improvements over competing methods and support structured map representations, actor-map fusion, and LaneConv design choices.
- Dataset and Metrics: The evaluation uses Argoverse scenarios, actor trajectories, and lane centerlines with connectivity, while excluding rasterized drivable-area and ground-height maps.The benchmark contains over 30K scenarios from Pittsburgh and Miami, with a 3-second prediction horizon.
- Dataset and Metrics: The benchmark reports minADE, minFDE, and Miss Rate for K=1 and K=6, with leaderboard ranking based on minFDE for K=6.ADE and FDE measure average and final displacement, while Miss Rate counts best-mode predictions more than 2.0 meters from ground truth.
- Comparison with the State-of-the-Art: The model significantly outperforms four top entries and two official baselines across all reported metrics on the Argoverse test set.The comparison contrasts explicit structured map features and actor-map fusion with rasterized or independently encoded lane representations.
- Module Ablations: Adding all modules improves ActorNet, while actor-to-lane and lane-to-lane information flow significantly outperforms using lane-to-actor interaction alone.The ablation attributes useful traffic information to information flowing from actors into the map representation.
- Lane-Graph Operator Ablations: Residual blocks, multi-type connections, and dilation each improve the lane-graph baseline, with multi-type connections and dilation significantly boosting performance.The residual block adds about 7% parameters and facilitates training; the operator study compares these components with vanilla graph convolution.
- Qualitative Results: Qualitative cases show the model capturing right-turn modes, producing lane-following trajectories with limited history, and modeling stronger deceleration than baselines.None of the models captures the extreme-acceleration case well, possibly because the available information is insufficient.
- Limitations and Future Directions: The current lane graph uses centerlines and connectivity, while traffic lights and traffic signs remain potential additions for motion forecasting.The authors state that new map information could be incorporated by introducing additional nodes and connections.
5 Conclusion
The paper presents a motion forecasting model that learns lane-graph representations and complete actor-map interactions instead of using rasterized maps. On Argoverse, it significantly outperforms the state of the art, while future work will incorporate additional map data.
- Conclusion: The model constructs a lane graph from vectorized map data and uses LaneGCN to extract map-topology features.It replaces rasterized-map input with structured lane-graph representations.
- Conclusion: The model fuses actor and lane information through spatial attention and LaneGCN to perform a complete set of actor-map interactions.The conclusion frames actor-map fusion as part of the forecasting architecture.
- Conclusion: The model significantly outperforms the state of the art on the large-scale Argoverse motion forecasting benchmark.The reported conclusion is benchmark-level rather than tied to a single metric.
- Conclusion: Future work plans to incorporate other map data into the lane-graph representation.The paper identifies additional map information as an extension direction.
Appendix
The architecture combines temporal actor encoding, lane-topology encoding, recurrent actor-map fusion, and a prediction header for multi-modal trajectory output.
- Architecture: ActorNet extracts temporal actor features with a 1D CNN and merges multi-scale features using a Feature Pyramid Network.ActorNet is one of four modules in the architecture.
- Architecture: MapNet uses a Lane Graph Network composed of multi-scale LaneConv residual blocks to extract lane-topology features.The Lane Graph Network is stacked from four multi-scale LaneConv residual blocks.
- Architecture: The actor-map fusion cycle stacks actor-to-lane, lane-to-lane, lane-to-actor, and actor-to-actor interaction networks.The interaction modules use attention residual blocks, while lane-to-lane uses another Lane Graph Network.
- Architecture: The prediction header converts updated actor features into multi-modal trajectories and their confidence scores.This module follows the actor-map fusion cycle.