Vector Map Quality Metrics for Contextual Autonomous Driving Systems

I. INTRODUCTION

• Map errors, omissions, or obsolete data can compromise autonomous vehicles’ safety, efficiency, and robustness.Source: §I · p. 1
• Map updates are shifting from high-precision Mobile Mapping Systems toward crowdsourced connected-vehicle data, whose lower-cost sensors bring variability, localization uncertainty, and partial visibility.Crowdsourcing can provide high-density data from many vehicles, supporting scalable and cost-effective maintenance.Source: §I · p. 1
• Existing update and matching strategies still leave a need for one interpretable metric covering geometric errors, missing detections, and semantic inconsistencies.Source: §I · p. 1
• The paper introduces GOSPAM as a new metric for assessing automotive map quality and monitoring map-maintenance uncertainty.Source: §I · p. 1

II. VECTOR MAPS

• Vector maps contain punctual, linear, and surface objects, including traffic signs, lane markings, and buildings.Source: §II · p. 2
• Map evaluation distinguishes true positives, false positives, and false negatives; true negatives cannot be evaluated because vector maps do not represent free space.Source: §II · p. 2
• A good navigation map should combine semantic and metric exactness with few false negatives and false positives.Source: §II · p. 2

A. Methodology

• Evaluation assumes a reference map whose quality and accuracy exceed those of the map being assessed, with map features represented as sets that may have different sizes.Source: A. Methodology · p. 2
• Features are associated within a vector tile using object-dependent map matching and a maximum-distance gating threshold before metrics are calculated.Source: A. Methodology · p. 2
• The methodology focuses on existence and location errors, while semantic exactness can be measured but is set aside as straightforward to evaluate.The considered errors include individual-feature coordinate errors, common-mode bias, missing features, and non-existing mapped features.Source: A. Methodology · p. 2

B. Usual metrics

• Precision, recall, and F1 summarize feature existence and completeness, while RMSE measures average location error among matched features.Precision is associated with false positives, recall with false negatives, and F1 combines precision and recall.Source: B. Usual metrics · pp. 2–3
• Location-error metrics do not account for false positives or false negatives, and conventional metrics do not yield one numerical value for overall map quality.Source: B. Usual metrics · p. 3

C. GOSPAM: A new metric

• GOSPA combines matched-feature localization errors with false-negative and false-positive counts in one real-valued measure.Source: C. GOSPAM: A new metric · p. 3
• GOSPAM normalizes GOSPA by the number of common features, addressing the size dependence that can make a sparse poor-quality map score better than a larger, more accurate map.The authors consider normalization by reference-map, evaluated-map, or common-feature cardinality and favor the common-feature count.Source: C. GOSPAM: A new metric · p. 3
• With no false negatives or false positives, GOSPAM reduces to RMSEM.Source: C. GOSPAM: A new metric · p. 4
• GOSPAM is expressed in meters, and lower values indicate more coherent maps.Source: C. GOSPAM: A new metric · p. 4

IV. CASE STUDY

• The case study focuses on two vector-map feature types: points and polylines.Source: §IV · p. 4

A. Point features

• Point features represent objects such as road signs, traffic lights, and georeferenced posts.Source: A. Point features · p. 4
• The study uses Euclidean distance for point matching because it is computationally more efficient and does not require covariance matrices.Source: A. Point features · p. 4
• The Munkres algorithm finds a minimum-cost one-to-one assignment, maximizing the number of matched pairs; matches are accepted below threshold c.Source: A. Point features · p. 4
• GOSPAM is calculated from RMSEM, TP, FN, and FP using Eq. (10).Source: A. Point features · p. 4

B. Polyline features

• Polylines represent road lanes, including lane markings, center lanes, and road borders.Source: B. Polyline features · p. 4
• The paper uses line-to-line matching to evaluate map polylines and populates the pairwise cost matrix with weighted Frenet distance.Source: B. Polyline features · p. 4
• Weighted Frenet distance measures average lateral deviation between a line's points and their projections onto reference lines, emphasizing overall alignment and shape consistency over isolated outliers.The paper considered Hausdorff distance but reports that it gives less smooth and less shape-consistent assignments.Source: B. Polyline features · p. 4
• After the cost matrix is populated, the Munkres algorithm determines the optimal polyline assignment, and GOSPAM is computed using matched and unmatched polylines.Source: B. Polyline features · p. 4

• The reference was a tile from an automotive supplier’s HD map, degraded with random metric errors, deleted elements, and added elements to create evaluation maps.The authors manipulated the map with QGIS.Source: §V · p. 4
• Point and polyline maps were considered separately, with one error parameter varied at a time to analyze GOSPAM sensitivity.Source: §V · p. 4
• In the point-feature example, circles mark reference features, diamonds mark evaluated features, dotted circles show association gating, and black lines show associations.Distances are in meters.Source: Figure 3 · p. 5
• The association-gating distance was set to 2 m in every experiment, corresponding to the map’s global accuracy.Source: §V · p. 5

A. Point features map

• In the point-map setup, the analyzed tile was reduced to 30 m × 30 m to limit the number of features and simplify the analysis.Source: A. Point features map · p. 5

1) Bias variation:

• GOSPAM increases linearly with bias while the bias remains below the 2 m gating distance; above 2 m, no true positives remain and the metric cannot be calculated.With no false positives or false negatives, GOSPAM is identical to RMSEM.Source: 1) Bias variation · p. 5

2) FN variation:

• False negatives produce a nonlinear GOSPAM response, and calculation requires at least one true positive to remain.The study therefore limited false negatives to four.Source: 2) FN variation · p. 5

3) FP variation:

• With true positives constant and no false negatives or MSEM, GOSPAM is proportional to false positives; the study reports no technical limit on the number added.Source: 3) FP variation · p. 5

4) Coordinates error variation:

• As random coordinate errors increase, median GOSPAM follows a nonlinear trend resembling a square-root function, while dispersion around the median widens.The study used 100 Monte Carlo tests per error value, with errors bounded by the 2 m gating distance to avoid false positives and false negatives.Source: 4) Coordinates error variation · p. 5

B. Polyline features map

• Polyline bias from 0 to 2 m produces a linear GOSPAM response similar to the point-feature case.Figure 5 depicts evaluated polylines with dotted lines and uses colors to distinguish three lane groups.Source: 2) FN variation · p. 5, Figure 5 · p. 5, B. Polyline features map · p. 5

2) FN variation:

• Polyline false negatives are applied randomly across lane groups, and the reported false-negative and false-positive behavior matches the point-map results.Source: 2) FN variation · p. 5, 3) FP variation · p. 5

4) Coordinates error variation:

• Polyline coordinate errors preserve the median trend seen for point maps but produce narrower box plots because more points must be affected to yield the same random variation.The authors describe this as a smoothing property of GOSPAM for polylines.Source: 4) Coordinates error variation · p. 5, Figure 6 · p. 6

C. Discussion

• GOSPAM responds linearly to bias, evolves explainably with false negatives and false positives, and shows RMSE-like dispersion under coordinate noise.Its sensitivity is of the same order across directions, so no input parameter dominates when error sources are combined.Source: C. Discussion · pp. 5–6
• Further studies are needed to determine which GOSPAM values indicate a map of sufficient quality.GOSPAM can be zero for a perfect map, but its value is not bounded.Source: C. Discussion · p. 6

VI. CONCLUSION AND PERSPECTIVES

• GOSPAM captures systematic, stochastic, and structural map perturbations in point and polyline simulations, with interpretable and balanced behavior across its inputs.The conclusion presents it as a single real value for assessing vector-map quality.Source: §VI · p. 7
• The paper links lower GOSPAM scores with better map fidelity and safer autonomous and assisted driving.The stated context is the use of accurate maps by perception and planning systems.Source: §VI · p. 7
• Planned work includes real-world validation and case studies to establish absolute quality ranges for a metric currently used relatively to compare maps or changes over time.The authors also plan to evaluate and compare crowdsourced map-updating strategies with GOSPAM.Source: §VI · p. 7