Source-linked AI summary
MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction
Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, Xinggang Wang
TL;DR
Online HD map construction seeks an efficient, precise alternative to costly and difficult-to-update offline mapping while preserving vectorized structure for autonomous driving. MapTRv2 introduces unified permutation-equivalent modeling with hierarchical queries, matching, efficient attention, and auxiliary supervision. It achieves state-of-the-art accuracy and efficiency on nuScenes and Argoverse2, including real-time inference and stable qualitative results.
Problem
Offline mapping is costly, hard to update, and vulnerable to ego-vehicle misalignment, while MapTR lacks directed-centerline modeling and scales poorly in memory and computation.
Method
MapTRv2 uses permutation-equivalent point-set modeling, hierarchical queries and bipartite matching, decoupled self-attention, one-to-many matching, dense segmentation, and depth supervision.
Results
MapTRv2 achieves state-of-the-art accuracy and efficiency on nuScenes and Argoverse2, including 33.7 FPS with 6.3 higher mAP on nuScenes using ResNet18.
Takeaways & Limitations
The framework provides end-to-end online vectorized HD map construction for map elements with arbitrary shapes and maintains stable, robust quality in complex driving scenes.
Takeaways & Limitations
The earlier MapTR framework does not account for the physical direction and connectivity of lane centerlines and incurs high memory and computation costs when scaling.
Abstract
from arXiv · showhide
High-definition (HD) map provides abundant and precise static environmental information of the driving scene, serving as a fundamental and indispensable component for planning in autonomous driving system. In this paper, we present \textbf{Map} \textbf{TR}ansformer, an end-to-end framework for online vectorized HD map construction. We propose a unified permutation-equivalent modeling approach, \ie, modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. To speed up convergence, we further introduce auxiliary one-to-many matching and dense supervision. The proposed method well copes with various map elements with arbitrary shapes. It runs at real-time inference speed and achieves state-of-the-art performance on both nuScenes and Argoverse2 datasets. Abundant qualitative results show stable and robust map construction quality in complex and various driving scenes. Code and more demos are available at \url{https://github.com/hustvl/MapTR} for facilitating further studies and applications.
1 Introduction
MapTRv2 addresses the efficiency, representation, and scalability limits of online vectorized HD map construction with a unified end-to-end framework. It supports directed and undirected elements, improves convergence, and achieves strong accuracy-efficiency results on nuScenes and Argoverse2.
- Motivation: Offline HD map construction is costly, difficult to update, and susceptible to ego-vehicle misalignment and localization error.Online construction instead builds maps around the ego vehicle at runtime using vehicle-mounted sensors.
- Motivation: Rasterized maps omit vectorized instance-level structure, while post-processing and autoregressive point prediction add complexity, latency, or error accumulation.These limitations motivate an end-to-end DETR-like paradigm for vectorized map construction.
- Method: MapTRv2 extends permutation-equivalent modeling to directed centerlines while retaining unified point-set representations for directed and undirected map elements.Centerlines use their given direction as the target permutation, unlike elements without fixed physical direction.
- Method: Decoupled self-attention reduces memory and computation complexity from O((N × Nv)^2) to O(N^2 + Nv^2).N denotes instance queries and Nv denotes point queries.
- Method: Hierarchical queries, bipartite matching, one-to-many matching, dense segmentation, and depth supervision improve structured learning and convergence.Dense supervision uses perspective-view and bird’s-eye-view foreground segmentation, while depth supervision guides 3D geometric representation learning.
- Results: 33.7 FPS and 6.3 higher mAP are achieved on nuScenes with ResNet18, while ResNet50 reaches 68.7 mAP at 14.1 FPS.The ResNet18 result is reported as 15× faster than VectorMapNet; the ResNet50 result is 22.7 mAP higher and 6× faster than VectorMapNet-ResNet50.
- Results: MapTRv2 reaches 73.4 mAP at 9.9 FPS with camera-only VoVNetV2-99 and exceeds VectorMapNet by 28.9 mAP on Argoverse2 3D map construction.The framework also maintains stable and robust qualitative construction quality in complex driving scenes.
2 Related Work
Related work spans segmentation-based, vectorized, lane-detection, contour-based, and Transformer approaches to map or shape prediction. MapTRv2 builds on these directions with permutation-equivalent modeling and efficient end-to-end construction.
- HD Map Construction: HDMapNet converts pixel-wise semantic segmentation into vectorized instances using heuristic and time-consuming post-processing.VectorMapNet instead predicts point sequences end to end with a cascaded coarse-to-fine autoregressive decoder.
- Lane Detection: Lane-detection methods commonly target single-view open-shape lane elements, limiting their coverage of broader HD map elements.LaneATT uses anchors for efficiency, while LSTR adopts a Transformer architecture.
- Contour-based 2D Instance Segmentation: Contour-based instance segmentation treats objects as closed polygons and predicts image coordinates for contour vertices.Representative methods use graph convolutions, polar coordinates, contour evolution, or Transformer-based refinement.
- Shape Modeling: MapTRv2 models polylines and polygons through point sets with equivalent permutations, enabling arbitrary-shape representation without relying on unique point ordering.This representation is illustrated for map elements such as lane dividers and pedestrian crossings.
- Detection Transformers: DETR established a fully end-to-end object-detection paradigm using query sets and a unified Transformer encoder-decoder architecture.MapTRv2 adapts this general paradigm to vectorized HD map construction.
3 Shape Modeling
MapTRv2 represents map elements as point sets while accounting for equivalent orderings that preserve their geometry. This resolves ambiguity for undirected polylines and polygons while retaining fixed ordering for directed centerlines.
- Shape abstraction: Map elements are abstracted as closed shapes, such as pedestrian crossings, or open shapes, such as lane dividers, then sampled into polygons or polylines.Points are sampled sequentially along each shape boundary.
- Permutation ambiguity: Undirected polylines can reverse direction, while polygons allow arbitrary starting points, two traversal directions, and cyclic shifts without changing geometry.These equivalent orderings make a single fixed permutation unsuitable as supervision.
- Permutation-equivalent modeling: MapTRv2 models each element as a point set V paired with a permutation group Γ covering all organization sequences representing the same shape.The point set defines geometry, while Γ captures equivalent orderings.
- Semantic permutation groups: For unspecific-direction polylines, Γ contains two permutations; for directed centerlines, it contains one; for polygons, it contains 2 × Nv permutations.The permutation set therefore depends on map-element semantics.
4 Architecture
MapTRv2 uses an encoder-decoder architecture that converts sensor inputs into BEV features and iteratively refines structured hierarchical map queries. Its decoder combines hierarchical queries, efficient attention variants, reference-point feature sampling, and hierarchical matching.
- Encoder-decoder architecture: The map encoder transforms sensor data into a unified BEV feature representation, while stacked Transformer decoder layers iteratively refine map-element predictions.The framework is fully end-to-end and supports vehicle-mounted sensors.
- Hierarchical queries: Hierarchical queries explicitly encode map elements using shared instance-level queries and point-level queries associated with each instance.This representation separates instance information from the points describing its shape.
- Self-attention variants: Decoupled self-attention separates inter-instance and intra-instance interactions, reducing complexity from O((N × Nv)^2) to O(N^2 + Nv^2).It is reported to reduce memory consumption and achieve higher performance than vanilla self-attention.
- Hierarchical matching: Hierarchical bipartite matching first assigns predicted instances to targets and then matches their points before defining the one-to-one set-prediction loss.The two-level assignments are denoted π̂ and {γ̂i}.
- Cross-attention variants: Dynamic reference points adapt to arbitrary map-element shapes and capture informative context by sampling BEV or projected perspective-view features.BEV-based cross-attention samples around BEV reference points, whereas PV-based attention projects them into camera images.
- Prediction head: The prediction head combines classification of instance classes with regression of normalized 2D or 3D point coordinates.Each map element produces a class score and a vector containing the coordinates of Nv points.
5 Training
MapTRv2 trains a fixed-size set of vectorized map elements through hierarchical instance- and point-level matching, then applies structured geometric losses and auxiliary supervision. One-to-many matching and dense prediction losses supplement the main one-to-one objective to accelerate convergence and exploit semantic and geometric information.
- Hierarchical Bipartite Matching: MapTRv2 predicts a fixed-size set of N map elements in one pass, padding ground truth with no-object entries to size N.Each prediction contains a class score and point set, while each ground-truth element contains a class label, point set, and permutation group.
- Hierarchical Bipartite Matching: Hierarchical bipartite matching first assigns predicted elements to ground truth using class and point-set position costs, then matches their points using valid permutations.Hungarian matching determines the instance-level assignment; point-level matching selects the lowest-cost permutation within the predefined permutation group.
- One-to-One Set Prediction Loss: Training uses classification, point-to-point, and edge-direction losses after the optimal instance- and point-level assignments are obtained.Point-to-point loss supervises point positions, while edge-direction loss additionally supervises the geometry of connecting edges.
- Auxiliary One-to-Many Set Prediction Loss: During training, an auxiliary branch repeats each ground-truth map element K times and assigns one ground-truth element to K predictions.The branch shares point queries and Transformer decoder layers with one-to-one matching, increasing positive samples and accelerating map-decoder convergence.
- Auxiliary Dense Prediction Loss: Three auxiliary dense prediction losses supervise depth, bird’s-eye-view segmentation, and perspective-view segmentation alongside the set-prediction objectives.Depth uses rendered LiDAR-based ground-truth maps, while BEV and PV segmentation use rasterized map masks in their respective views.
- Overall Loss: The overall training loss is a weighted sum of one-to-one matching, one-to-many matching, and dense prediction losses.The coefficients βo, βm, and βd balance these three objective groups.
6 Experiments
Experiments on nuScenes and Argoverse2 evaluate MapTRv2 for accuracy, efficiency, convergence, component contributions, and extension to directed centerline learning. MapTRv2 achieves strong map-construction results while reducing computational demands.
- Experimental setup: Experiments use nuScenes and Argoverse2, with AP based on Chamfer-distance thresholds evaluating selected map elements and centerlines.nuScenes provides six-camera 2D maps; Argoverse2 provides seven-camera logs with 3D vectorized maps.
- Main results: 64.7 mAP is achieved for 3D vectorized map construction on Argoverse2, 29.9 mAP higher than VectorMapNet with the same backbone.The method also achieves 67.4 mAP for 2D construction, 29.5 mAP higher than VectorMapNet.
- Ablation study: MapTRv2-ResNet50 reaches 61.5 mAP after component additions, versus 58.7 mAP for MapTR after 110 epochs, indicating faster convergence.The roadmap adds dense supervision, decoupled self-attention, and auxiliary one-to-many matching.
- Ablation study: Permutation-equivalent modeling improves performance by 4.1 mAP over fixed-order modeling and by 7.1 AP for pedestrian crossings.The approach addresses ambiguity in point-set ordering for map-element shapes.
- Ablation study: Decoupled self-attention reduces memory by 1985M under the reported ablation and provides a higher performance upper bound under a 24 GB RTX 3090 limit.Instance self-attention alone reduces training memory by 2265M with a 0.3 mAP drop; decoupling restores point-level interaction.
- Ablation study: Dense depth, perspective-view segmentation, and bird’s-eye-view segmentation provide complementary supervision, improving final performance by 4.9 mAP together.Auxiliary one-to-many matching improves performance by 3.9 mAP at K = 6, while increasing training memory.
- Centerline extension: Centerline learning extends MapTRv2 to directed map elements, achieving 54.0 mAP on nuScenes and 62.6 mAP for 2D maps and 61.4 mAP for 3D maps on Argoverse2.Centerlines provide directional information and indicate traffic flow for downstream planners.
7 Conclusion
MapTRv2 is a structured end-to-end framework for efficient online vectorized HD map construction. Experiments show precise perception of arbitrarily shaped map elements on nuScenes and Argoverse2.
- Conclusion: MapTRv2 combines permutation-equivalent modeling, hierarchical bipartite matching, and an encoder-decoder Transformer for vectorized HD map construction.The framework is designed for efficient online prediction of map elements with arbitrary shapes.