Source-linked AI summary
VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, Cordelia Schmid
TL;DR
Behavior prediction for autonomous driving requires integrating complex agent dynamics with structured road context, but common approaches render these inputs as images for ConvNets. VectorNet instead uses a hierarchical graph network over vectorized maps and trajectories, augmented by masked-node reconstruction. It achieves on-par-or-better benchmark performance while reducing model size and computation, and reaches state-of-the-art performance on Argoverse.
Problem
Behavior prediction must integrate moving-agent dynamics with complex HD-map context to predict vehicle intent as trajectories.
Method
VectorNet represents maps and agent trajectories as vectors, aggregates vectors within polylines and then models polyline interactions with hierarchical graph networks and masked-node completion.
Results
70% model size saving and an order of magnitude reduction in FLOPs accompany on-par-or-better performance across two benchmarks, while VectorNet achieves state-of-the-art performance on Argoverse.
Takeaways & Limitations
Direct vectorized encoding provides a computationally lighter alternative to rendered ConvNet representations while retaining or improving reported behavior-prediction performance.
Takeaways & Limitations
The current coordinate normalization centers each target agent separately; sharing coordinate centers across interacting agents is left for future work.
Abstract
from arXiv · showhide
Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e.g. pedestrians and vehicles) and road context information (e.g. lanes, traffic lights). This paper introduces VectorNet, a hierarchical graph neural network that first exploits the spatial locality of individual road components represented by vectors and then models the high-order interactions among all components. In contrast to most recent approaches, which render trajectories of moving agents and road context information as bird-eye images and encode them with convolutional neural networks (ConvNets), our approach operates on a vector representation. By operating on the vectorized high definition (HD) maps and agent trajectories, we avoid lossy rendering and computationally intensive ConvNet encoding steps. To further boost VectorNet's capability in learning context features, we propose a novel auxiliary task to recover the randomly masked out map entities and agent trajectories based on their context. We evaluate VectorNet on our in-house behavior prediction benchmark and the recently released Argoverse forecasting dataset. Our method achieves on par or better performance than the competitive rendering approach on both benchmarks while saving over 70% of the model parameters with an order of magnitude reduction in FLOPs. It also outperforms the state of the art on the Argoverse dataset.
1. Introduction
VectorNet represents HD maps and agent dynamics directly as vectors, then uses hierarchical graph modeling to capture local polyline structure and higher-order interactions. An auxiliary node-completion task further trains the model to recover masked context and trajectory features.
- Behavior prediction must integrate agent dynamics with structured HD-map context to predict vehicle intent as trajectories.
- Rasterized approaches manually encode structured maps as color-coded images and use ConvNets with limited receptive fields.
- VectorNet represents lanes, crosswalks, signs, and agent trajectories as polylines composed of vectors with attributes.
- The model treats vectors as graph nodes, propagates map and agent context through GNNs, and decodes future trajectories from the target-agent node.
- Its hierarchical graph first aggregates vectors within semantically consistent polylines, then models interactions among all polylines.
- The auxiliary graph-completion objective randomly masks scene or trajectory node features and trains the model to reconstruct them from context.
- 70% model size saving and an order of magnitude reduction in FLOPs accompany on-par-or-better performance against a rendering baseline, with state-of-the-art Argoverse results.
2. Related work
Prior behavior-prediction methods use probabilistic hypotheses and ConvNet encoders for rendered maps, while VectorNet directly encodes vectorized scene context and agent dynamics.
- Learning-based behavior-prediction methods provide probabilistic interpretations of different behavior hypotheses but require representations for maps and trajectories.
- IntentNet jointly detects vehicles and predicts trajectories from LiDAR points and rendered HD maps, while related methods use ConvNets and trajectory anchors or generative models.
- VectorNet differs from these approaches by directly encoding vectorized scene context and agent dynamics, while assuming detections come from an existing perception algorithm.
3. VectorNet approach
VectorNet represents trajectories and HD-map annotations as vector sequences and encodes them with a hierarchical graph network. It first aggregates information within polylines, then models interactions among polylines and trains with an auxiliary masked-node completion objective.
- 3.1. Representing trajectories and maps: HD-map annotations and agent trajectories are approximated as sequences of vectors, forming an unordered vector set for graph encoding.Map splines, closed shapes, and points are represented by sampled vectors with attributes; sufficiently small intervals preserve close approximations.
- 3.1. Representing trajectories and maps: Each vector becomes a graph node with start and end coordinates, attributes, and a polyline-group identifier.Attributes can encode object type, timestamps, road-feature type, or speed limit, while coordinates are centered on the target agent's last observed location.
- 3.2. Constructing the polyline subgraphs: VectorNet connects vector nodes within each polyline, embeds them into polyline features, and then exchanges information across all polylines.The local subgraphs use shared MLP encoders, max pooling, and concatenation; the global interaction graph uses self-attention.
- 3.3. Global graph for high-order interactions: The global graph models high-order interactions among map polylines and agent trajectories before a trajectory decoder predicts future motion from moving-agent nodes.The implementation uses an MLP decoder, while deeper global graphs can model higher-order interactions when needed.
- 3.3. Global graph for high-order interactions: Randomly masking polyline-node features and reconstructing them adds a graph-completion objective that encourages context modeling during training.The node decoders are used only during training, and the total loss combines trajectory negative Gaussian log-likelihood with node-feature Huber loss using α = 1.0.
4. Experiments
The experiments cover datasets, metrics, a rasterized ConvNet baseline, ablations, computation cost, and comparisons with state-of-the-art methods.
- 4. Experiments: The experiments evaluate datasets and metrics, compare against a rasterized ConvNet baseline, conduct ablations, measure computation cost, and assess state-of-the-art performance.The planned cost analysis includes FLOPs and parameter counts.
4.1. Experimental setup
The experiments compare VectorNet with rasterized ConvNet inputs on two vehicle behavior prediction datasets, using matched scene information and standard displacement metrics.
- Datasets: 333K Argoverse sequences and 2.2M in-house training trajectories provide two vehicle behavior prediction benchmarks.Argoverse uses 5-second sequences, while the in-house trajectories span 4 seconds with separate observation and prediction intervals.
- Datasets: Both datasets derive input histories from automatic perception systems, making the observed trajectories noisy.Argoverse future trajectories are machine generated, whereas in-house future trajectories are manually labeled.
- Evaluation: ADE and DE@t at 1.0, 2.0, and 3.0 seconds measure trajectory displacement errors in meters.ADE is computed over entire trajectories.
- Baselines: The rasterized baseline stacks 10 in-house or 20 Argoverse 400×400 RGB frames and encodes them with a ResNet-18 ConvNet.The images contain road maps and detected object bounding boxes.
- Comparison protocol: The vectorized representation uses the same map information and visible road features as rasterization while retaining complex road features that are difficult to render.This setup is intended to ensure a fair comparison between representations.
- VectorNet configuration: VectorNet uses three polyline graph layers and one global interaction layer, with 64 hidden units in its MLPs.Coordinates are normalized around the target vehicle at the last observed timestep.
4.2. Ablation study for the ConvNet baseline
The ConvNet baseline is evaluated under different receptive-field and rendering-resolution settings. Larger crops and trajectory-aligned cropping help, while higher resolution generally improves performance with an Argoverse exception.
- Receptive field: Larger crop sizes and cropping along the observed vehicle trajectory improve ConvNet baseline performance.These strategies expand or reposition the receptive field used for vehicle-centric features.
- Rendering resolution: Performance generally increases with rasterized image resolution from 100×100 to 200×200 to 400×400.The tested resolutions correspond to 1, 0.5, and 0.25 meters per pixel, respectively.
- Rendering resolution: On Argoverse, increasing resolution from 200×200 to 400×400 slightly reduces performance.The paper attributes this observation to a decrease in effective receptive field.
- Trade-offs: Receptive-field design is important for rasterized inputs but can require carefully designed cropping strategies and increased computation.The baseline benefits from broader context, with added computational cost as a trade-off.
4.3. Ablation study for VectorNet
VectorNet benefits from combining map and agent trajectories with node completion, while its hierarchical representation outperforms or matches rasterized baselines across the evaluated datasets.
- Input node types: Adding map polylines significantly improves trajectory prediction, and adding other-agent trajectories improves it further.The ablation compares target trajectory alone, map context, and map plus agent trajectories.
- Training objective: The node completion auxiliary objective consistently improves performance, especially at longer prediction horizons.The objective is added alongside trajectory prediction.
- Graph architecture: Three polyline-subgraph layers perform best, while one global graph layer is sufficient.Wider MLPs do not improve performance and hurt on Argoverse.
- Dataset-dependent behavior: VectorNet's Argoverse advantage is attributed to capturing long-range context through its hierarchical graph network, whereas stationary cases favor local ConvNet patterns.The datasets differ in scenario composition: the in-house set contains many stationary vehicles, while Argoverse retains interesting cases.
4.4. Comparison of FLOPs and model size
VectorNet reduces computation and model size relative to ConvNet baselines while maintaining stronger prediction performance, with costs that depend on scene vectors and targets.
- ConvNet FLOPs increase quadratically with kernel and input image size, while VectorNet FLOPs depend on the numbers of vector nodes and polylines.
- VectorNet computation increases linearly with the number of predicting targets because coordinates are renormalized and features recomputed for each target.
- 200+ times fewer FLOPs and 29% of the parameters: VectorNet uses 0.041G FLOPs and 72K parameters versus 10.56G and 246K for ConvNets.The comparison is for a single agent; the prediction decoder is excluded from both FLOPs and parameter counts.
- VectorNet significantly outperforms the best ConvNet model while dramatically reducing computation cost.
4.5. Comparison with state-of-the-art methods
On the Argoverse test set with K=1, VectorNet improves the state-of-the-art DE@3s result from 4.17 to 4.01.
- 4.01 DE@3s at K=1: VectorNet improves the state-of-the-art result of 4.17 on the Argoverse test set.The reported results use the most likely prediction and compare against state-of-the-art methods from the Argoverse Forecasting Challenge.
5. Conclusion and future work
The paper represents HD maps and agent dynamics as vectors and uses a hierarchical graph network to model within-polyline and cross-polyline relationships. Experiments show lower computational cost than ConvNets and state-of-the-art Argoverse performance, while future work targets diverse trajectory generation.
- VectorNet represents HD maps and agent dynamics with vectors and aggregates information first within polylines, then across polylines.
- Experiments on the in-house and Argoverse datasets show that VectorNet outperforms its ConvNet counterpart while substantially reducing computational cost.
- VectorNet achieves state-of-the-art performance on the Argoverse test set for DE@3s with K=1.
- A proposed next step is combining the VectorNet encoder with a multimodal trajectory decoder to generate diverse future trajectories.