Source-linked AI summary
HDMapNet: An Online HD Map Construction and Evaluation Framework
Qi Li, Yue Wang, Yilun Wang, Hang Zhao
TL;DR
Traditional HD semantic-map construction depends on extensive annotation and maintenance, motivating a more scalable local-learning formulation. The paper introduces HDMapNet, which converts camera and/or LiDAR observations into vectorized bird’s-eye-view map elements and evaluates them with semantic- and instance-level metrics. On nuScenes, it reports improvements over baselines, including a fusion model that improves over baselines and the camera-only method by 50% relatively, while producing locally consistent maps applicable to real-time motion planning.
Problem
Traditional HD semantic-map construction requires extensive human effort to annotate and maintain globally consistent maps, limiting scalability.
Method
HDMapNet uses surrounding-camera images and/or LiDAR point clouds to predict vectorized map elements in bird’s-eye view, with semantic, instance, and direction outputs.
Results
50% relatively: the camera-LiDAR fusion model improves over baselines and the camera-only method, while reported nuScenes gains include 12.1 IoU and 13.1 mAP over existing methods.
Takeaways & Limitations
Local semantic map learning offers a potentially more scalable alternative for autonomous driving, with predicted maps immediately applicable to real-time motion planning.
Abstract
from arXiv · showhide
Constructing HD semantic maps is a central component of autonomous driving. However, traditional pipelines require a vast amount of human efforts and resources in annotating and maintaining the semantics in the map, which limits its scalability. In this paper, we introduce the problem of HD semantic map learning, which dynamically constructs the local semantics based on onboard sensor observations. Meanwhile, we introduce a semantic map learning method, dubbed HDMapNet. HDMapNet encodes image features from surrounding cameras and/or point clouds from LiDAR, and predicts vectorized map elements in the bird's-eye view. We benchmark HDMapNet on nuScenes dataset and show that in all settings, it performs better than baseline methods. Of note, our camera-LiDAR fusion-based HDMapNet outperforms existing methods by more than 50% in all metrics. In addition, we develop semantic-level and instance-level metrics to evaluate the map learning performance. Finally, we showcase our method is capable of predicting a locally consistent map. By introducing the method and metrics, we invite the community to study this novel map learning problem.
I. INTRODUCTION
The paper frames local HD semantic map learning as a more scalable alternative to labor-intensive global map construction, introducing HDMapNet and evaluation protocols for vectorized maps from onboard sensors.
- Traditional HD map pipelines build globally consistent maps from pre-captured point clouds and extensive semantic annotation, requiring substantial human effort.
- HDMapNet estimates vectorized local semantic maps from surrounding-camera images and/or LiDAR point clouds.
- Camera and LiDAR representations complement one another, and the fusion model outperforms single-modal counterparts across all categories.
- 12.1 IoU and 13.1 mAP improvements over existing methods are reported on nuScenes for semantic segmentation and instance detection, respectively.
- The paper proposes an online map-construction framework, comprehensive evaluation protocols, and metrics covering semantic- and instance-level performance.
- The framework includes a perspective-to-bird’s-eye-view feature projection module that implicitly models 3D environments and explicitly considers camera extrinsics.
II. RELATED WORK
Related work contrasts laborious, globally consistent HD map construction with local sensor-based map learning and reviews perspective-view lane detection approaches that produce increasingly data-driven and 3D outputs.
- Existing HD semantic maps are commonly built by manually or semi-automatically annotating LiDAR point clouds fused with SLAM and high-precision vehicle poses.
- The proposed local map learning framework uses onboard sensors to estimate local semantic maps instead of pre-annotating global semantic maps.
- Maintaining HD semantic maps remains laborious and costly because updates require high precision and timely revision.
- Perspective-view lane detection has progressed from hand-designed image processing and geometric projection toward deep models for segmentation, lane detection, and 3D outputs.
III. SEMANTIC MAP LEARNING
Semantic map learning predicts vectorized local map elements directly from onboard camera images and/or LiDAR point clouds. HDMapNet transforms image features into bird’s-eye view, optionally fuses modalities, and uses neural networks to produce the map.
- HDMapNet predicts vectorized map elements from single-frame surrounding images and/or point clouds using neural networks.The model variants use surrounding images, LiDAR, or both as inputs.
- The image branch encodes perspective-view features and transforms them into camera-coordinate features with a neural view transformer.The transformer models relations between camera-coordinate feature positions and perspective-view pixels using an MLP.
- The model overview supports image-only, point-cloud-only, or fused processing before producing a vectorized local semantic map.HDMapNet’s branches output semantic segmentation, instance embedding, and directions.
- Geometric projection with camera extrinsics converts camera-coordinate features into bird’s-eye-view features.The final image feature is formed by averaging features from the surrounding cameras.
- The point-cloud branch uses a PointPillar variant with dynamic voxelization to learn feature maps from pillar-wise point features.Each input point contains three-dimensional coordinates and additional features.
2) Point cloud encoder:
The point cloud encoder aggregates points within pillars and then encodes the resulting pillar features into a bird’s-eye-view feature map.
- Multiple points may occupy one pillar, so their features are aggregated before further encoding.The points corresponding to pillar j are represented as the set Pj.
- A convolutional network further encodes pillar-wise features into a bird’s-eye-view feature map.
3) Bird’s-eye view decoder:
The bird’s-eye-view decoder converts image and/or point-cloud features into map structure by jointly predicting semantics, instances, and lane directions. Post-processing clusters embeddings and vectorizes the predicted elements.
- Bird’s-eye view decoder: The decoder uses three branches for semantic segmentation, instance embedding, and direction prediction.Its input is the image feature map and, when available, the point-cloud feature map concatenated with it.
- Bird’s-eye view decoder: The decoder represents lane dividers and lane boundaries with instance-level and directional information rather than only pixel labels.A post-processing stage clusters instances and produces vector representations.
- Bird’s-eye view decoder: Image features from multiple cameras are fused and transformed into the ego-vehicle coordinate system using camera extrinsics.The feature transformation proceeds through perspective view, camera coordinates, and then bird’s-eye view.
- Bird’s-eye view decoder: Direction prediction discretizes lane directions into classes and uses the predicted direction to step from a current lane pixel to the next pixel.Forward and backward directions are treated as positive labels, while non-lane pixels receive zero direction vectors and are excluded from backpropagation.
- Bird’s-eye view decoder: Instance embeddings are clustered during inference with DBSCAN, followed by non-maximum suppression and vectorization.
B. Evaluation
The evaluation protocols assess semantic map learning at both semantic and instance levels.
- The evaluation includes semantic metrics and instance metrics for semantic map learning.
1) Semantic metrics:
The paper evaluates semantic map predictions with complementary Eulerian and Lagrangian metrics. IoU compares dense rasterized shapes, while Chamfer distance measures spatial distances between curve point sets.
- Eulerian metrics: IoU evaluates dense-grid semantic representations by comparing rasterized shape sets.The representations contain height, width, and category dimensions.
- Lagrangian metrics: Chamfer distance evaluates spatial discrepancies between predicted and ground-truth curves sampled as connected point sets.This metric targets structured curve outputs rather than only dense pixel values.
- Lagrangian metrics: Bi-directional Chamfer distance combines directional distances computed in both prediction-to-label and label-to-prediction directions.The directional and bi-directional forms are defined over two point sets sampled from the curves.
2) Instance metrics:
Instance detection is evaluated with average precision computed from ranked predictions and precision–recall pairs under Chamfer-distance matching.
- Instance detection: Average precision measures instance detection similarly to object detection.Predictions are ranked by semantic confidence before evaluation.
- Instance detection: Predictions are classified as true or false positives using a predefined Chamfer-distance threshold.A prediction below the threshold is a true positive; otherwise it is a false positive.
- Instance detection: Average precision is computed from the resulting precision–recall pairs.The procedure uses precision at specified recall values.
IV. EXPERIMENTS
The experiments evaluate HDMapNet on nuScenes through semantic map segmentation and instance detection, using several view-transformation baselines.
- Tasks & Metrics: The evaluation uses the nuScenes dataset and focuses on semantic map segmentation and instance detection.The considered static elements are lane boundaries, lane dividers, and pedestrian crossings.
- Architecture: The perspective-view image encoder uses EfficientNet-B0, followed by an MLP for feature conversion.EfficientNet-B0 is pretrained on ImageNet.
- Baselines: Baselines transform perspective-view predictions or features into bird’s-eye view representations using IPM variants.The baselines include IPM with a bird’s-eye-view decoder and an image-feature encoder with a bird’s-eye-view decoder.
C. Results
HDMapNet outperforms the evaluated baselines, with the camera–LiDAR fusion model achieving the strongest reported performance and supporting locally consistent map construction.
- Quantitative results: The camera-only HDMapNet(Surr) outperforms all evaluated baselines.The comparison concerns semantic map segmentation results in Table I.
- Temporal fusion: The model’s locally consistent map is generated by long-term temporal accumulation of occupancy probabilities over multiple frames.The resulting map is described as immediately applicable to real-time motion planning.
- Quantitative results: 50% relatively: the camera–LiDAR fusion model improves over baselines and the camera-only method.The fusion model achieves the best performance among the compared models.
- Metric analysis: Different models trade off directional and positional Chamfer components, so CD provides information complementary to IoU.VPN has the lowest CDP but the worst overall CD, whereas HDMapNet(Surr) achieves the best CD among camera-only methods.
- Modality comparison: HDMapNet(LiDAR) performs better on boundaries but worse on dividers and pedestrian crossings than HDMapNet(Surr).The result indicates that map-element recognizability differs across sensing modalities.
V. CONCLUSION
HDMapNet offers a scalable alternative for constructing local HD semantic maps directly from camera images and/or LiDAR point clouds, while exposing a trade-off between scalability and accuracy.
- HDMapNet predicts HD semantic maps directly from camera images and/or LiDAR point clouds.The local learning framework provides a more scalable alternative to global map construction and annotation, though the baseline sacrifices some accuracy.