VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation

• The paper frames behavior prediction as learning vehicle-intent trajectories by integrating tracked agent dynamics with prior scene context such as HD maps.Source: §1 · p. 1
• The introduction identifies a limitation of learning-based approaches that rasterize structured HD maps: manual color-coded attribute specifications and ConvNets with limited receptive fields.Source: §1 · p. 1
• VectorNet represents map entities and agent motion as polylines composed of vectors, with vectors encoded by their endpoints and attributes as graph nodes.Map entities such as lane boundaries, crosswalks, and stop signs can be approximated as polylines.Source: §1 · p. 2
• Its hierarchical graph connects vectors within semantically labeled polylines, embeds them into polyline features, then fully connects polylines for information exchange using self-attention.The local graphs are implemented with multi-layer perceptrons.Source: §1 · p. 2
• An auxiliary graph-completion task randomly masks scene-context or trajectory node features and trains the model to reconstruct them, encouraging it to capture interactions between agent dynamics and scene context.Source: §1 · p. 2
• VectorNet reports on-par or better performance than a competitive rendering baseline on in-house and Argoverse datasets, with 70% model-size savings and an order-of-magnitude fewer FLOPs, and state-of-the-art Argoverse performance.Source: §1 · p. 2

Forecasting multi-agent interactions

• Unlike approaches to agent interactions such as Social LSTM, Social GAN [13], and graph-based methods [16] [18] [26], VectorNet jointly models agents and road-map entities in a hierarchical graph.Prior work models interactions through spatially aggregated LSTM states, adversarial discrimination, graph networks with variational RNNs, latent variables, or attention over predefined edges.Source: §2 · p. 3

Representation learning for sets of entities

• Unlike point-set methods PointNet [20] and PointNet++ [21], VectorNet preserves the closed-shape or directed structure of map entities and trajectories, encoding their attributes as graph node features.Point-set methods apply permutation-invariant operations, such as max pooling, to learned point embeddings.Source: §2 · p. 3

Self-supervised context modeling

• Inspired by self-supervised NLP methods [11], VectorNet predicts missing node features from neighboring nodes to encourage representations that capture interactions.This is introduced as an auxiliary loss for graph representations.Source: §2 · p. 3

3.1. Representing trajectories and maps

• VectorNet samples map geometry spatially and trajectories every 0.1 seconds into vectors, then centers vector coordinates on the target agent’s last observed position.Each vector node carries endpoint coordinates, attributes such as object type or timestamps, and its polyline ID; sufficiently dense sampling approximates the original geometry and trajectories.Source: §3.1 · p. 3

3.2. Constructing the polyline subgraphs

• Within each polyline, VectorNet connects vector nodes and repeatedly combines shared-MLP encodings with max-pooled neighbor information and concatenated relational features.Polyline-level features are then obtained by max pooling.Source: §3.2 · pp. 3–4

3.3. Global graph for high-order interactions

• A fully connected self-attention graph models interactions among polyline features; the implementation uses one GNN layer and decodes future trajectories from moving-agent nodes with an MLP.Additional GNN layers can model higher-order interactions, while alternative trajectory decoders are described as complementary to the input encoder.Source: §3.3 · p. 4
• During training, VectorNet randomly masks polyline features and learns to recover them from graph context, extending masked-context prediction to unordered graphs with end-to-end optimized node features.The node feature decoder is an MLP and is not used at inference; unlike methods that use precomputed visual features, VectorNet jointly optimizes its node features end to end.Source: §3.3 · pp. 4–5

3.4. Overall framework

• The overall objective combines trajectory negative Gaussian log-likelihood with masked-node Huber loss at α = 1.0, after L2-normalizing polyline features.Predicted trajectories are per-step coordinate offsets from the last observed position, in a coordinate system rotated to the target vehicle’s heading.Source: §3.4 · p. 5

• The section compares VectorNet’s performance with state-of-the-art methods.Source: §4 · p. 5
• It compares computational cost using FLOPs and parameter counts.Source: §4 · p. 5
• It reports ablation studies for both the rasterized baseline and VectorNet.Source: §4 · p. 5
• The experimental settings describe the datasets, metrics, and rasterized ConvNets baseline.Source: §4 · p. 5

4.1.1 Datasets

• Experiments use Argoverse [7] and an in-house vehicle-prediction dataset, with different sequence scales, prediction horizons, and future-label sources.Argoverse contains 333K five-second sequences; the in-house dataset has 2.2M training and 0.55M test trajectories, each four seconds long. Both use automatically derived history trajectories; Argoverse futures are machine generated, while in-house futures are manually labeled.Source: §4.1.1 · p. 5

4.1.2 Metrics

• Evaluation uses Average Displacement Error over entire trajectories and Displacement Error at 1.0, 2.0, and 3.0 seconds, measured in meters.Source: §4.1.2 · p. 5

4.1.3 Baseline with rasterized images

• The rasterized baseline stacks past-frame map and object images and encodes them with a ResNet-18 backbone, without LiDAR input.Vehicle-centered cropped features are average-pooled before a fully connected layer predicts future coordinates; ablations examine receptive field, cropping strategy, and image resolution. Training uses Adam for 25 epochs with learning-rate decay every five epochs.Source: §4.1.3 · pp. 5–6

4.1.4 VectorNet with vectorized representations

• For a fair comparison, VectorNet uses the same map features and visible road information as rasterization, while allowing more complex road features.Its setup uses three polyline-subgraph layers, one global-interaction layer, and 64 hidden units in each MLP; ablations vary context information and graph-layer counts. Training uses Adam for 25 epochs with learning-rate decay every five epochs.Source: §4.1.4 · p. 6

Impact of receptive fields

• At 400×400 resolution, increasing convolution kernel size from 3 to 5 or 7 brings slight performance improvements but quadratically increases computation cost.Source: §4.2 · p. 6
• Larger crops significantly improve performance, and cropping along the observed vehicle trajectory also helps, but these strategies require careful design and can increase computation.The study presents these findings as evidence that receptive fields matter when rasterized images are used as inputs.Source: §4.2 · p. 6

Impact of rendering resolution

• Performance generally increases with rasterization resolution, but on Argoverse, raising resolution from 200×200 to 400×400 slightly reduces performance as the fixed 3×3 kernel shrinks the effective receptive field.The tested resolutions were 400×400 (0.25 meter per pixel), 200×200 (0.5 meter per pixel), and 100×100 (1 meter per pixel).Source: §4.2 · p. 6

4.3. Ablation study for VectorNet

• Adding map polylines improves prediction over target-trajectory-only input, and adding non-target agent trajectory polylines improves it further.Table 2 compares target trajectory only (“none”), map polylines (“map”), and map plus agent polylines (“map + agents”).Source: §4.3 · p. 6, Table 2 · p. 7
• The node-completion auxiliary objective consistently improves performance, especially at longer prediction horizons.It is trained alongside trajectory prediction.Source: §4.3 · p. 6, Table 2 · p. 7
• Three polyline-subgraph layers perform best, one global-graph layer is sufficient, and wider MLPs do not improve performance and hurt on Argoverse.The authors suggest Argoverse’s smaller training dataset may explain why wider MLPs hurt there.Source: §4.3 · p. 6, Table 3 · p. 7
• On Argoverse, VectorNet significantly outperforms the best ConvNet baseline with a 12% reduction in DE@3; the authors attribute this advantage, presumably, to hierarchical-graph capture of long-range context in the dataset’s preserved “interesting” cases.Source: §4.3 · p. 7

4.4. Comparison of FLOPs and model size

• Excluding the decoder, VectorNet significantly outperforms R18-k3-t-r400 while using 0.041G vs 10.56G FLOPs and 72K vs 246K parameters.The FLOPs comparison is for a single agent; the in-house scenes average around 30 vehicles, and VectorNet's computation remains much smaller overall.Source: Table 4 · p. 7, §4.4 · p. 8
• ConvNet FLOPs grow quadratically with kernel and image size, while VectorNet FLOPs depend on scene vectors and polylines and grow linearly with prediction targets.Self-driving-vehicle-centered image features can be reused across targets, keeping backbone FLOPs constant; VectorNet re-normalizes coordinates and recomputes features per target. In-house scenes average 17 map polylines with 205 vectors and 59 agent polylines with 590 vectors.Source: §4.4 · pp. 7–8
• ConvNet parameter counts increase quadratically with kernel size.Source: §4.4 · p. 7

4.5. Comparison with state-of-the-art methods

• At K=1, VectorNet improves Argoverse Forecasting Challenge winners’ DE@3s from 4.17 to 4.01 on the Argoverse test set.The comparison reports the most likely predictions.Source: §4.5 · p. 8

Conclusion and future work

• VectorNet outperforms the ConvNet counterpart at substantially lower computational cost and achieves state-of-the-art DE@3s, K=1 performance on the Argoverse test set.The conclusion reports experiments on the large-scale in-house and publicly available Argoverse datasets.Source: §5 · p. 8
• VectorNet represents HD maps and agent dynamics with vectors, aggregating information within polylines before modeling higher-order relationships among polylines.Source: §5 · p. 8
• Future work could combine the VectorNet encoder with a multi-modal trajectory decoder to generate diverse future trajectories.The paper gives [6] [29] as examples of such decoders.Source: §5 · p. 8