PaperlayerView exact revision on arXiv

Source-linked AI summary

VectorMapNet: End-to-end Vectorized HD Map Learning

Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, Hang Zhao

arXiv:2206.08920v6cs.CVcs.RO

TL;DR

HD semantic mapping is constrained by manual annotation and rasterized predictions that omit instance information and need heuristic vectorization. VectorMapNet learns sparse polylines end-to-end from onboard sensors, achieving state-of-the-art results across nuScenes and Argoverse2, with reported gains of at least 14.2 mAP.

  • Problem

    Manual HD-map annotation has scalability issues, while rasterized predictions lack instance information and require heuristic post-processing to produce vectorized maps.

  • Method

    VectorMapNet converts sensor observations into vectorized HD maps by detecting and generating a sparse set of directional polylines.

  • Results

    VectorMapNet achieves state-of-the-art map-learning performance on nuScenes and Argoverse2, outperforming HDMapNet and another baseline by at least 14.2 mAP.

  • Takeaways & Limitations

    The predicted vectorized maps generate coherent, complex urban geometries and are compatible with downstream motion forecasting.

  • Takeaways & Limitations

    The model lacks guaranteed temporal consistency, has detector–generator feature mismatch from teacher forcing, and reduced interpretability when predicting occluded locations.

Abstract

from arXiv · show

Autonomous driving systems require High-Definition (HD) semantic maps to navigate around urban roads. Existing solutions approach the semantic mapping problem by offline manual annotation, which suffers from serious scalability issues. Recent learning-based methods produce dense rasterized segmentation predictions to construct maps. However, these predictions do not include instance information of individual map elements and require heuristic post-processing to obtain vectorized maps. To tackle these challenges, we introduce an end-to-end vectorized HD map learning pipeline, termed VectorMapNet. VectorMapNet takes onboard sensor observations and predicts a sparse set of polylines in the bird's-eye view. This pipeline can explicitly model the spatial relation between map elements and generate vectorized maps that are friendly to downstream autonomous driving tasks. Extensive experiments show that VectorMapNet achieve strong map learning performance on both nuScenes and Argoverse2 dataset, surpassing previous state-of-the-art methods by 14.2 mAP and 14.6mAP. Qualitatively, VectorMapNet is capable of generating comprehensive maps and capturing fine-grained details of road geometry. To the best of our knowledge, VectorMapNet is the first work designed towards end-to-end vectorized map learning from onboard observations. Our project website is available at \url{https://tsinghua-mars-lab.github.io/vectormapnet/}.

1. Introduction

VectorMapNet addresses scalability and representation limitations in HD semantic mapping by learning sparse vectorized polylines directly from onboard sensor observations. It replaces rasterization and heuristic post-processing with an end-to-end pipeline that models heterogeneous map elements and reports strong benchmark performance.

  • Offline manual annotation creates scalability issues, while deriving local maps from global maps requires precise ego-vehicle localization that can introduce meter-level errors.
  • Rasterized BEV maps lack instance information for distinguishing map elements with the same class label and require post-processing for vectorized outputs.
  • VectorMapNet predicts a sparse set of polylines directly from sensor observations, eliminating dense semantic pixels and sophisticated post-processing.
  • VectorMapNet achieves state-of-the-art performance on nuScenes and Argoverse2, outperforming HDMapNet and another baseline by at least 14.2 mAP.
  • Polylines provide a flexible representation for heterogeneous map elements, while encoded order accommodates their directional structure.
  • The method adapts DETR models to locate deformable map elements in 3D space, addressing limitations of centerpoint-based feature extraction for varying sizes and shapes.

2. Related Works

Related work covers semantic map learning, lane detection, geometric sequence generation, and vector representation learning from images. These approaches use rasterization, handcrafted geometric proposals, hierarchical graphs, autoregressive sequences, or vector graphics primitives.

  • Semantic map learning has used aerial images, LiDAR points, HD panoramas, and onboard camera images, with recent methods focusing on BEV semantics.
  • Lane detection commonly combines pixel-level segmentation with sophisticated post-processing or uses handcrafted proposals such as vanishing points, polynomial curves, line segments, and Bézier curves.
  • Geometric data generation often treats geometric elements as sequences produced by autoregressive models, while HDMapGen separately models global and local graphs.
  • LETR models line segments as a detection problem with a query-based detector rather than sequence generation.
  • Vector graphics methods learn vector representations from raster images, including CAD structures, canvas layouts, and stroke primitives.

3. VectorMapNet

VectorMapNet represents heterogeneous HD map elements as sparse polylines and predicts them end-to-end from sensor-derived BEV features. Its hierarchical detector estimates element locations and categories before a generator produces ordered, variable-length geometry.

  • Problem Formulation and Challenges: Map elements include irregular road boundaries, lane dividers, and polygonal pedestrian crossings, motivating representations that handle diverse geometric structures.The task uses onboard RGB cameras and/or LiDAR sensors and requires modeling geometric and topological relationships.
  • Polyline Representation: Polylines provide a unified primitive for points, lines, curves, and polygons while encoding direction through ordered vertices and matching downstream autonomous-driving representations.Polygons become closed polylines, while curves are simplified into polylines with the Ramer–Douglas–Peucker algorithm.
  • VectorMapNet: VectorMapNet formulates map construction as sparse set detection, converting sensor inputs into canonical BEV features before predicting a sparse set of map-element polylines.The BEV extractor lifts and aligns camera and LiDAR features into a shared representation.
  • Keypoint Representations: Three fixed-count keypoint representations—Bounding Box (k=2), SME (k=3), and Extreme Points (k=4)—are evaluated as intermediate abstractions of map-element outlines.These keypoints differ from the final polyline vertices; the polyline generator uses them for conditional prediction.
  • Map Element Detector: The map element detector uses hierarchical element and keypoint queries with a transformer set-prediction architecture to estimate element keypoints and class labels without extra post-processing.Element queries represent map elements, while fixed-count keypoints serve as intermediate representations passed to the polyline generator.
  • Polyline Generator: The polyline generator models each element as a discrete distribution over ordered vertex coordinates, flattening coordinates and appending an End of Sequence token for variable-length generation.Generation is conditioned on element keypoints, class labels, and BEV features, with the joint distribution decomposed into conditional vertex-coordinate distributions.

4. Experiments

Experiments evaluate VectorMapNet on nuScenes and Argoverse2 against representative baselines, with ablations, qualitative analyses, and downstream motion forecasting. Results show strong map performance, detailed polyline geometry, and useful predicted maps for forecasting.

  • Comparison with Baselines: Experiments use nuScenes and Argoverse2, comparing VectorMapNet with HDMapNet and STSU using Chamfer AP and Fréchet AP.Fréchet AP accounts for vertex order, unlike Chamfer AP.
  • Comparison with Baselines: 14.2 mAP separates VectorMapNet from HDMapNet in nuScenes Fusion, while camera-only and LiDAR-only gains are +17.9 and +9.9 mAP.Sensor fusion improves over camera-only and LiDAR-only by +4.3 mAP and +11.2 mAP, respectively.
  • Comparison with Baselines: VectorMapNet remains competent on Argoverse2 when overlapping boundaries and divider lines challenge rasterized representations.HDMapNet’s APdivider drops from 21.7 APdivider to 5.7 APdivider in the described case.
  • Qualitative Analysis: Polylines encode detailed geometry and direction, reducing ambiguous loopy curves compared with rasterized methods.The paper identifies these properties as benefits for representing real-world road layouts.
  • Qualitative Analysis: VectorMapNet models map topology and element locations before generating details, capturing small elements and detecting pedestrian crossings absent from annotations.The paper attributes this to information propagation between query embeddings through self-attention.
  • Ablation Studies: Bounding-box keypoints perform best in ablation, exceeding alternatives by 2.0 Fréchet mAP and 7.3 Chamfer mAP.The compared alternatives are Start-Middle-End and Extreme Points.
  • Motion Forecasting with Vectorized HD Maps from VectorMapNet: Predicted maps improve motion forecasting over trajectory-only inputs, with the gap from ground-truth maps reaching only -0.2% in miss rate.The forecasting task uses six possible three-second future trajectories from one second of past trajectories.

5. Discussions

The model has limitations involving temporal consistency, two-stage training mismatch, and interpretability when predicting occluded map elements.

  • Single-frame predictions can form coherent geometries but are not guaranteed to remain temporally consistent.
  • Teacher forcing creates a feature-space mismatch between the map detector and polyline generator, making fine-tuning and training schedules challenging.
  • Predictions in camera-occluded locations demonstrate scene understanding but reduce interpretability.
  • The paper points readers to Appendix § E for further discussion of potential societal impact.

6. Conclusions

VectorMapNet learns vectorized HD maps by representing map elements as polylines and decomposing prediction into detection and generation stages.

  • VectorMapNet uses polylines as primitives for representing vectorized HD map elements.
  • The model decomposes polyline prediction from sensor data into detection and generation steps.
  • Experiments show that polyline primitives support coherent and complex geometries for urban map elements.
  • The authors present this approach as a new perspective on learning HD maps.

A.1. Dataset

Experiments use nuScenes and Argoverse2 and evaluate predicted vectorized maps with distance-based polyline metrics rather than raster IoU.

  • Dataset: nuScenes contains 1000 autonomous-driving sequences with six camera images, LiDAR sweeps, and three annotated map-element categories.
  • Dataset: Argoverse2 contains 1000 logs split into 700 training, 150 validation, and 150 test logs, with camera images, LiDAR sweeps, and vectorized maps.
  • Chamfer distance: Chamfer distance measures similarity between unordered sets and is used after uniformly sampling each polyline to 100 vertices.
  • Fréchet distance: Fréchet distance additionally measures vertex order and evaluates geometric similarity between two polylines.
  • Fréchet distance: Discrete Fréchet distance is computed as the minimum norm over valid couplings between the sampled polyline sequences.

B.1. Visualization results of VectorMapNet(Fusion)

Fusion with LiDAR complements camera observations, producing more robust map predictions when visual information is blocked, ambiguous, or affected by weather.

  • LiDAR helps the fusion model recover map information hidden by nearby vehicles or camera blind zones.
  • Depth information helps detect missing lane boundaries near an occluded parking gate.
  • LiDAR distinguishes puddles from lane boundaries when light reflections create visual ambiguity.
  • The fusion model avoids falsely generating lane boundaries caused by misleading puddle reflections.

C.1. Overall Architectures.

The architecture uses BEV features and fixed-capacity element queries to predict vectorized map elements. Fusion with LiDAR depth improves results when cameras are occluded.

  • C.1. Overall Architectures.: The BEV feature extractor outputs a (200, 100, 128) feature map using a shared ResNet50 backbone and a PointNet LiDAR backbone.The LiDAR backbone aggregates points into pillars and produces 64-dimensional outputs.
  • C.1. Overall Architectures.: The map element detector uses Nmax = 100 element queries.
  • C.1. Overall Architectures.: With nearby-vehicle camera occlusion, VectorMapNet(Fusion) generates a more plausible map than VectorMapNet(Camera) using LiDAR depth cues.
  • C.1. Overall Architectures.: Both transformer decoders contain 6 decoder layers with hidden embeddings of size 256.
  • C.1. Overall Architectures.: The polyline generator represents its output space as a 200 × 100 grid with 0.3m × 0.3m cells.

C.2. Training settings.

Training combines detector and polyline-generation objectives, while camera features are projected into BEV through inverse perspective mapping. Experiments also compare sampling strategies and adapted baselines.

  • C.2. Training settings.: Camera features are transformed into BEV using inverse perspective mapping under a mostly planar-ground homography assumption.
  • C.2. Training settings.: The detector loss combines negative log-likelihood classification, smooth L1 regression, and IoU regression losses with coefficients 2, 0.1, and 1.
  • C.2. Training settings.: The polyline generator uses negative log-likelihood, and VectorMapNet trains by summing the detector and generator losses.
  • C.2. Training settings.: Ground-truth elements are padded with ∅ up to Nmax, then matched to predictions by the minimum-cost permutation σ* computed with the Hungarian algorithm.
  • C.2. Training settings.: For comparison, HDMapNet uses its official model while its NuScenes processing was adapted to create an Argoverse2 dataloader, and STSU was modified for lane-segment prediction.
  • C.2. Training settings.: Curvature-based vertex sampling outperforms fixed 1m sampling and achieves a leading 21.5 Fréchet mAP.

D.2. Vertex modeling methods.

The study compares continuous and discrete vertex modeling for polyline generation. Discrete coordinate embeddings substantially improve both Chamfer and Fréchet mAP.

  • D.2. Vertex modeling methods.: The comparison evaluates discrete modeling against a continuous mixture-of-Gaussians formulation using the same model structure.
  • D.2. Vertex modeling methods.: Discrete vertex coordinates increase Chamfer mAP from 18.2 to 32.5 and Fréchet mAP from 26.8 to 50.4.
  • D.2. Vertex modeling methods.: The authors attribute the improvements to the non-local characteristic of categorical distributions, which helps capture complex vertex-coordinate distributions.

D.3. Extrinsic Robustness

VectorMapNet maintains performance under substantial camera-extrinsic noise, with noisy-extrinsic training improving test-time performance. The paper also identifies uncertainty use in downstream tasks as an open question.

  • D.3. Extrinsic Robustness: Training with noisy camera extrinsics improves test-time performance, and the model maintains good performance at high noise levels.
  • D.3. Extrinsic Robustness: The stated societal-impact discussion focuses on privacy because the model relies on onboard observations rather than tracking global locations or individual movements.
  • D.3. Extrinsic Robustness: VectorMapNet produces instance-level confidence scores for detected map elements and point-level scores for generated polyline points.
  • D.3. Extrinsic Robustness: How to use prediction uncertainty effectively in downstream tasks remains an open question for future research.
Loading 2206.08920v6…