Source-linked AI summary
LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting
Wenyuan Zeng, Ming Liang, Renjie Liao, Raquel Urtasun
TL;DR
Forecasting dynamic actors is difficult because their intentions are latent and their motions depend on complex actor-to-actor and actor-to-map interactions. LaneRCNN represents each actor with a local lane graph, propagates information through a shared global lane graph, and predicts trajectories using lane-graph structure. It achieves state-of-the-art performance on the Argoverse motion forecasting benchmark and reaches first place on its leaderboard.
Problem
Future motion forecasting is challenging because actors have unknown intentions, may change plans, and perform complex maneuvers influenced by nearby actors and maps.
Method
LaneRCNN learns a distributed LaneRoI graph per actor, performs interaction message passing within a shared global lane graph, and parameterizes outputs using lane structures.
Results
LaneRCNN achieves state-of-the-art performance on the challenging Argoverse motion forecasting benchmark and first place on its leaderboard.
Takeaways & Limitations
LaneRCNN captures actor-to-actor and actor-to-map relations in a distributed, map-aware representation for motion forecasting.
Takeaways & Limitations
The formulation assumes that an HD map containing lanes and semantic attributes is accessible.
Abstract
from arXiv · showhide
Forecasting the future behaviors of dynamic actors is an important task in many robotics applications such as self-driving. It is extremely challenging as actors have latent intentions and their trajectories are governed by complex interactions between the other actors, themselves, and the maps. In this paper, we propose LaneRCNN, a graph-centric motion forecasting model. Importantly, relying on a specially designed graph encoder, we learn a local lane graph representation per actor (LaneRoI) to encode its past motions and the local map topology. We further develop an interaction module which permits efficient message passing among local graph representations within a shared global lane graph. Moreover, we parameterize the output trajectories based on lane graphs, a more amenable prediction parameterization. Our LaneRCNN captures the actor-to-actor and the actor-to-map relations in a distributed and map-aware manner. We demonstrate the effectiveness of our approach on the large-scale Argoverse Motion Forecasting Benchmark. We achieve the 1st place on the leaderboard and significantly outperform previous best results.
1. Introduction
Motion forecasting for autonomous vehicles is difficult because agents have hidden, changing intentions and interact through complex maneuvers and map structures. LaneRCNN addresses limitations of vector-based, fully connected approaches with distributed LaneRoI representations and map-aware interaction and prediction.
- Motivation: Hidden and changing agent intentions make future motion forecasting challenging even when complete trajectory histories are available.Possible maneuvers include yielding, nudging, acceleration, turning, and lane-changing.
- Limitations of prior approaches: Single feature vectors struggle to represent large spatial contexts for actors that may traverse about a hundred meters within five seconds.Existing models encode past motion and surrounding context into vectors using CNNs or RNNs.
- Limitations of prior approaches: Fully connected actor-interaction graphs ignore map structures, so spatial proximity may not reflect meaningful right-of-way interactions.An unprotected left-turn vehicle should yield to oncoming traffic, whereas nearby vehicles on opposite lanes may barely interact.
- LaneRCNN: LaneRCNN predicts final actor positions fully convolutionally and decodes full trajectories from those positions using the LaneRoI-based pipeline.Its overview consists of LaneRoI encoding, per-LaneRoI processing, graph-based interaction modeling, final-position prediction, and trajectory decoding.
- LaneRCNN: LaneRCNN constructs an actor-specific LaneRoI graph whose nodes encode local lane regions, past motion, and map semantics while preserving lane topology.The model embeds LaneRoIs into a shared global lane graph for message passing among interacting actors.
2. Related Work
Learning-based motion forecasting replaces handcrafted interaction rules with deep models that encode scene information, model actor interactions, and generate multiple possible futures. LaneRCNN builds on graph methods while explicitly incorporating lane topology into representations, interactions, and trajectory outputs.
- Motion Forecasting: Traditional motion-forecasting methods use handcrafted interaction and constraint rules, whereas learning-based approaches have significantly outperformed them.Handcrafted rules can be oversimplified and not scalable.
- Interaction Modeling: Social pooling aggregates nearby actors but may lose actor-specific information, motivating attention and GNN-based interaction graphs.These later methods update actor features through attention or message passing, often on fully connected or k-nearest-neighbor graphs.
- Interaction Modeling: Most prior interaction methods do not explicitly consider map structures, despite their influence on interactions among actors.LaneRCNN instead draws inspiration from ego-graphs to design LaneRoI for lane graphs.
- Future Prediction: Multi-modal futures are generated using CVAEs, multi-head or mode regressors, discrete trajectory samples, or occupancy maps.TNT independently uses lanes as forecasting priors, while LaneRCNN additionally contributes a graph representation and architecture.
- Graph Neural Networks: Graph neural networks generalize convolution and message passing from regular grids to irregular topologies, supporting graph representation learning across tasks.LaneRCNN adapts this general graph-learning concept specifically to lane graphs through LaneRoI.
3. LaneRCNN
LaneRCNN represents each actor and its context with an actor-specific LaneRoI graph, then uses topology-aware convolutions, pooling, and message passing to encode interactions and predict map-aware futures.
- 3.2. LaneRoI Representation: LaneRoI represents each actor’s relevant lane context as a graph whose nodes are lane segments and whose edges encode lane topology.Relevant lanes are retrieved over the prediction and observed-history horizons, with predecessor, successor, left-neighbor, and right-neighbor relations.
- 3.2. LaneRoI Representation: Each LaneRoI node embedding combines lane geometry and semantics with information relating the lane segment to the actor.The embedding is designed to capture center location, orientation, curvature, and semantic attributes of the lane segment.
- 3.3. LaneRCNN Backbone: Lane convolution aggregates multi-hop neighborhood features over lane relations, enlarging receptive fields while preserving graph topology.The implementation uses hops n ∈ {1, 2, 4, 8, 16, 32}, analogous to a dilated convolution mechanism.
- 3.3. LaneRCNN Backbone: Lane pooling learns topology-aware feature interpolation from nearby lane segments using relative position and orientation rather than a simple mean.The pooled feature is produced from neighboring nodes and their relative poses through a two-layer MLP.
- 3.3. LaneRCNN Backbone: LaneRCNN exchanges information among actor-specific LaneRoIs through a shared global lane graph, allowing global nodes to collect features from nearby interacting actors.The distribute step then transfers globally aggregated information back to nodes in each actor’s LaneRoI.
- 3.4. Map-Relative Outputs Decoding: The model predicts multimodal futures through actor goals near lane segments and refines trajectory proposals with a learnable header using lane-based representations.Goals correspond to final actor positions at the prediction horizon and are predicted in a fully convolutional manner from the LaneRoI.
4. Experiment
LaneRCNN is evaluated on Argoverse through benchmark comparisons, ablations of its LaneRoI encoder and interaction module, and qualitative validation examples. The experiments report state-of-the-art performance and show benefits from motion-specific shortcuts, map-aware interactions, and map-consistent multimodal predictions.
- Experimental Settings: LaneRCNN is evaluated on Argoverse, which forecasts 3 seconds of future motion from 2 seconds of observations across geographically separated train, validation, and test sequences.The dataset contains 205942 training, 39472 validation, and 78143 test sequences.
- Experimental Settings: The evaluation uses Miss-Rate, Average Displacement Error, and Final Displacement Error, with multimodal prediction evaluated using K=6 trajectories.Miss-Rate measures whether any prediction is within 2.0 meters at the final timestep; ADE averages future-timestep errors, while FDE uses the final timestep.
- Comparison with State-of-the-art: LaneRCNN achieves state-of-the-art performance on the Argoverse leaderboard and significantly improves the benchmark result despite closely clustered leading methods.The comparison includes official baselines and top leaderboard entries, with the leaderboard snapshot dated Nov. 12, 2020.
- Ablation Studies: Pooling features along an actor’s past trajectory significantly improves the LaneRoI encoder over global or center pooling by providing a larger, actor-motion-specific receptive field.Adding more shortcuts provides some benefit, but the gains diminish.
- Ablation Studies: Interaction-based models outperform the model without interactions, while fully connected interaction graphs achieve similar performance and naive average pooling ignores local structures.LaneRCNN’s map-aware interaction design significantly improves performance over the compared attention and GNN alternatives.
- Qualitative results: Qualitative predictions follow lane geometry and represent multimodal behaviors including curved-lane motion, lane changing, intersection turns, and overtaking.The examples also show slower turning and lane-following speeds consistent with the depicted scenarios.
5. Conclusion
LaneRCNN is presented as a graph-centric forecasting model that represents each actor with a distributed lane-graph structure, models interactions within a shared graph, and predicts trajectories using lane graphs. It achieves state-of-the-art performance on the Argoverse benchmark.
- Conclusion: LaneRCNN builds a distributed LaneRoI representation per actor to encode past motion and local map topology.The representation is constructed with learnable graph operators.
- Conclusion: Its interaction module captures interactions among actors within a shared global lane graph.This extends the actor-specific representations into shared map-topology-aware interaction modeling.
- Conclusion: LaneRCNN parameterizes output trajectories using lane graphs and achieves state-of-the-art performance on the challenging Argoverse motion forecasting benchmark.The conclusion identifies lane-graph parameterization as helping improve prediction.
A. Map-Relative Output Decoding
LaneRCNN decodes each actor’s future trajectory by first selecting a final goal, then constructing an initial trajectory, and finally refining that proposal with a learnable prediction header.
- Map-Relative Output Decoding: The decoder predicts an actor’s final goal from node embeddings before generating its trajectory.This is the first of the three decoding steps.
- Map-Relative Output Decoding: It proposes an initial trajectory from the predicted goal and the actor’s initial pose.The initial pose anchors the map-relative trajectory proposal.
- Map-Relative Output Decoding: A learnable header refines the initial trajectory proposal into the decoded prediction.This is the final step of the output-decoding pipeline.
A.1. Trajectory Proposal
LaneRCNN generates trajectory proposals by fitting a quadratic Bezier curve between an actor’s initial state and a predicted final pose, then evaluating positions along that curve over time.
- A.1. Trajectory Proposal: A predicted final pose, comprising location and tangent information, defines the boundary conditions for a quadratic Bezier curve.The curve satisfies zero-th and first-order derivative values at its endpoints.
- A.1. Trajectory Proposal: Each predicted goal uniquely determines a 2D curve parameterized by normalized distance along the trajectory.The normalized distance is denoted by s.
- A.1. Trajectory Proposal: The model unrolls a velocity profile along each curve to produce 2D waypoint proposals at every future timestamp.The velocity profile assumes constant acceleration over the prediction horizon and uses the initial velocity and traveled distance.
- A.1. Trajectory Proposal: Future actor positions are obtained by querying the curve at the distance traveled by the corresponding timestamp.The distance function is written as s(t).
A.2. Trajectory Refinement
LaneRCNN refines each initial trajectory proposal using LaneRoI features, allowing deviations from the proposal curve to be predicted at each future timestamp.
- A.2. Trajectory Refinement: Each trajectory proposal is represented using a shortcut layer over LaneRoI node embeddings before refinement.A two-layer MLP decodes proposal-specific values for every future timestamp.
- A.2. Trajectory Refinement: The refinement decoder predicts a longitudinal adjustment and a signed lateral deviation for each future timestamp.The sign of the lateral deviation indicates whether the trajectory shifts left or right.
B. LaneRoI Construction
LaneRoI construction retrieves map lanes relevant to each actor’s history and expands them along lane topology to capture future movement and interactions.
- B. LaneRoI Construction: LaneRoI retrieves lane segments nearest to an actor’s locations across its observed history.The method uniformly samples 1-meter centerline segments before performing nearest-segment association.
- B. LaneRoI Construction: The retrieved segments are expanded to predecessor and successor lanes within a velocity-dependent range D.D reflects expected future movement length plus a buffer, such as 20 meters.
- B. LaneRoI Construction: Left and right neighboring lanes are added to represent lane-changing behavior.The construction also uses predecessor and successor lanes to capture lane-following behavior.
C. Architecture and Learning Details
LaneRCNN encodes actor-specific lane graphs, pools them into a shared global graph for interaction, and trains node classification with regression and refinement objectives.
- C. Architecture and Learning Details: The architecture encodes each LaneRoI, pools node embeddings into a global graph, applies further lane convolutions, and distributes global information back to local representations.The lane-pooling neighborhood threshold is 2 meters.
- C. Architecture and Learning Details: Node classification identifies the lane segment closest to the actor’s final position, using that segment as the positive training example.Nodes more than 6 meters from the ground-truth location are negatives, while remaining nodes are excluded from classification loss.
- C. Architecture and Learning Details: Online hard-example mining subsamples negative nodes and selects the hardest 100 negatives per data sample for classification loss.The total loss combines classification, regression, and refinement terms.
- C. Architecture and Learning Details: Table 3 compares original-label upsampling and uniform lane-segment sampling at different densities, with and without a regression branch.The final model uses one of these sampling-and-regression configurations.
D. Ablation Studies
The ablations show that lane-segment sampling and trajectory parameterization materially affect LaneRCNN’s performance. Denser, uniform sampling and flexible, learnable trajectory representations improve the model, while lane-based output parameterization eases learning with limited data.
- LaneRoI Sampling: Uniform lane-segment sampling outperforms upsampling across all metrics, despite similar average segment lengths.Upsampling introduces segment-length variance that may harm representation learning.
- LaneRoI Sampling: Denser lane-segment sampling achieves better results than sparser sampling.
- Output Parameterization: Lane-based output parameterization explicitly uses map information as a prior, easing training compared with direct 2D-plane regression.Lane polylines naturally divide into segments used in the graph.
- Output Parameterization: 1% of training data yields a 20% miss-rate for LaneRCNN, whereas direct 2D regression requires substantially more data.The comparison uses the same backbone and a multi-modal Cartesian regression header for the regression-based variant.
- Trajectory Parameterization: Bezier curves significantly outperform straight-line parameterizations, while constant-acceleration rollout slightly improves over constant speed.A learnable refinement header further boosts trajectory-proposal performance.
E. Qualitative Results
LaneRCNN produces map-consistent and multimodal forecasts across diverse driving scenarios. Its qualitative examples include varied turns, curved roads, lane changes, braking, overtaking, and abnormal off-road behavior.
- Turning: Turning examples capture different directions and lane occupancies after the turn across varied map topologies.
- Interactions: The model predicts braking or overtaking when a leading vehicle blocks the ego-actor.
- Abnormal Behavior: LaneRCNN also predicts non-map-compliant behavior, such as driving out of roads, despite relying on lanes for prediction.
- Scenarios: The qualitative evaluation covers turning, curved roads, braking, overtaking, and abnormal behaviors on the Argoverse validation set.