From Fragments to Global Maps: Learning Vectorized Map Aggregation with Large Language Models

1 INTRODUCTION

• Overlapping local predictions create redundant, fragmented, and geometrically inconsistent map elements that are difficult to associate and refine.The same road structures are repeatedly observed, while local prediction errors obscure their underlying geometry.Source: §1 · p. 1
• Fixed association thresholds and fitting intervals trade missed fragment merges against incorrect lane merges, and noise fitting against over-smoothing.The appropriate settings depend on local lane spacing, curvature, and detector noise, sometimes varying within one scene.Source: §1 · p. 2, Figure 1 · p. 2
• MapMergeLLM formulates aggregation as conditional sequence generation, directly generating global polylines from serialized local fragments.Synthetic corruptions of clean maps simulate duplication, fragmentation, and geometric perturbations to reduce dependence on any particular upstream detector.Source: §1 · p. 2
• Coordinate-token geometry-aware pretraining represents complete coordinate values as single tokens, while line-level association loss supervises correspondences between fragments of the same map element.The association loss helps reduce duplicated map elements.Source: §1 · pp. 2–3
• Road-boundary mean relative F1 improves by 23.1% on Argoverse2 and 21.9% on nuScenes over the strongest baseline.The evaluation uses local maps from recent vectorized map detectors.Source: §1 · p. 3
• Experiments on Argoverse2 and nuScenes show improvements over aggregation baselines across multiple recent upstream detectors.The framework is evaluated on local maps produced by recent vectorized map detectors.Source: §1 · p. 3

Vectorized HD Map Construction and Aggregation

• VMA uses predefined geometric rules, MonoLaM combines geometric association with graph optimization, and GNMap rasterizes vectorized tiles before producing a semantic map.MapMergeLLM instead aggregates fragmented observations directly in the vector domain and outputs global polylines.Source: §2 · p. 3
• Temporal map methods use historical observations or a continuously updated global map prior to improve prediction consistency.These methods build on local vectorized map prediction.Source: §2 · p. 3

Language Models for Map Generation

• Language models have been applied to map generation and refinement, but aggregating multiple vectorized map observations remains largely unexplored.Prior approaches include lane geometry generation, persistent mapping, and vision-language map refinement.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• MapMergeLLM generates global map polylines from serialized local maps, assuming local predictions are aligned using ego poses.The formulation focuses on element-level prediction errors rather than pose-induced misalignment.Source: §3 · pp. 3–4, §3.1 · p. 4

3.2 DETECTOR-AGNOSTIC CORRUPTION PROCESS

• Synthetic maps combine duplication, endpoint truncation or splitting, point-wise noise, and structured local geometric perturbations.Corruption operators are applied sequentially, and multiple failure modes may coexist in one map.Source: §3.2 · p. 5

3.3 LLM-BASED MAP AGGREGATION

• Coordinate tokenization represents each discretized coordinate as one token, while geometric and arithmetic pretraining teaches coordinate computation and comparison.Pretraining also includes next-token prediction on serialized local maps; only newly added coordinate-token rows are optimized while pretrained parameters stay frozen.Source: §3.3 · p. 6
• Stage 2 trains LoRA parameters on synthetic noisy local observations paired with clean global maps, using token-level cross-entropy.The pretrained model and coordinate-token parameters remain frozen during aggregation training.Source: §3.3 · p. 7
• A contrastive line-level loss complements token supervision by bringing same-element fragments together and separating unrelated ones.Line embeddings are mean-pooled from coordinate-token hidden states; positives are same-element observations from adjacent frames, while negatives are different elements.Source: §3.3 · p. 7
• At inference, serialized local predictions condition autoregressive generation of the global map sequence, which is parsed into polylines.Coordinates are converted back to values at the specified resolution.Source: §3.3 · p. 7

4.1 EXPERIMENTAL SETUP

• Training pairs use synthetic corruptions of ground-truth local maps cropped every 15 m over 60 m × 30 m regions, with the union of those regions as the target global map.At inference, MapTracker and StreamMapNet predictions are aligned in a common global frame before aggregation.Source: §4.1 · pp. 7–8
• VMA incrementally merges with predefined geometric rules, whereas MonoLaM associates spline elements geometrically and updates them through graph optimization.The evaluation uses line-level matching: over 75% of ground-truth points must fall within a distance threshold, with precision, recall, and F1 averaged across three thresholds.Source: §4.1 · p. 8

4.2 RESULTS

• MapMergeLLM consistently outperforms VMA and MonoLaM in F1 across datasets, detectors, and map categories; on AV2 with StreamMapNet, it exceeds MonoLaM by 5.8 points for lane dividers and 8.6 for road boundaries.The same setting yields gains over VMA of 35.0 and 29.7 percentage points, respectively.Source: Table 1 · p. 8, §4.2 · p. 8
• Across qualitative cases, MapMergeLLM reconstructs complete road boundaries while reducing fragmentation, geometric distortion, noise, and duplicate predictions.Examples include recovering a circular boundary and joining disconnected fragments into a continuous line.Source: §4.2 · p. 9, Figure 3 · p. 9
• Coordinate tokenization with pretraining and the line-level association loss each improve F1 across all four ablation settings.Removing coordinate-token pretraining lowers F1 in every setting; the association loss improves F1 over token-level generation alone.Source: §4.2 · p. 9, Table 2 · p. 9

• MapMergeLLM formulates map aggregation as conditional sequence generation from local vectorized map observations.Source: §5 · p. 10
• Synthetic noisy maps, geometry-aware coordinate-token pretraining, and line-level association loss address detector dependence and map-fragment association.The synthetic inputs are constructed from clean vector maps with simulated representative detection errors.Source: §5 · p. 10
• Experiments on Argoverse2 and nuScenes show consistent improvements over heuristic and optimization-based baselines across multiple upstream detectors.The conclusion presents data-driven aggregation as a direction for reducing reliance on heuristic merging rules.Source: §5 · p. 10

A.1.1 TRAINING SETUP

• Training uses Qwen3.5-4B with LoRA on attention query/value and feed-forward gate, up-projection, and down-projection layers.Source: §A.1.1 · p. 13
• Optimization uses batch size 1 with accumulation over 8 steps, AdamW weight decay 0.01, and learning rate 2 × 10^-4 with cosine scheduling and 200 warm-up steps.Source: §A.1.1 · p. 13
• The model is trained on a single NVIDIA B200 GPU, and evaluation uses the checkpoint with the lowest validation loss.Source: §A.1.1 · p. 13

A.1.2 IMPLEMENTATION DETAILS

• Coordinates in local observations and the target global map are translated so the first frame’s ego position becomes the origin.Coordinates use 0.1 m resolution over a range of −350 to 350 m.Source: §A.1.2 · p. 13
• Each map element is serialized as its category followed by ordered point coordinates, and input and target sequence lengths vary with the scene’s frames, polylines, and points.Source: §A.1.2 · p. 13
• Target polylines are direction-canonicalized and sorted by endpoint coordinates, while local polyline order is randomized during training to reduce sensitivity to input ordering.Source: §A.1.2 · pp. 13–14

A.2 PRETRAINING TASKS FOR COORDINATE TOKENIZATION

• Synthetic tasks cover translation, subtraction, midpoint prediction, point sorting, nearest-point selection, and bounding-box corner prediction.Coordinates use the new tokens, and loss is applied only to answer tokens.Source: §A.2 · p. 14, Table 3 · p. 14
• The synthetic tasks train spatial operations including coordinate-wise averaging, y-then-x ordering, distance comparison, and polyline extent prediction.These operations correspond respectively to midpoint prediction, point sorting, nearest-point selection, and bounding-box corner prediction.Source: §A.2 · p. 14
• Next-token prediction on serialized local maps teaches how coordinate tokens are used within polyline sequences.This complements the geometric and arithmetic relationships learned from the synthetic tasks.Source: §A.2 · p. 14

A.3 TRAINING ON DETECTOR OUTPUTS

• Synthetic training improves F1 particularly for road boundaries, while real-detector training is slightly higher for nuScenes lane dividers evaluated on StreamMapNet predictions.The comparison varies only the source of noisy local maps and evaluates on real MapTracker and StreamMapNet predictions.Source: §A.3 · p. 14, Table 4 · p. 15

A.5 FURTHER DISCUSSION

• Experiments show aggregation can be learned from synthetic local observations, reducing dependence on predefined association and merging rules across the two datasets and different upstream detectors.The discussion motivates further study on larger maps and related applications.Source: §A.5 · p. 15
• Experiments cover individual driving scenes; longer trajectories may exceed the LLM context window as local observations accumulate.Incremental aggregation over overlapping regions is proposed as a future direction for keeping inputs within the context window.Source: §A.5 · p. 15
• The 0.1 m coordinate resolution supports the reported datasets, while finer resolution or wider spatial range would increase vocabulary size.Hierarchical coordinate representations are suggested as a possible way to accommodate larger maps or finer resolutions with fewer additional tokens.Source: §A.5 · p. 15

A.4.1 COMPARISON WITH MAP AGGREGATION BASELINES

• MapMergeLLM better associates road-boundary fragments and reconstructs boundaries closer to ground truth than VMA and MonoLaM in the qualitative comparisons.VMA often leaves boundaries fragmented or incomplete, while MonoLaM can distort complex boundaries and add redundant predictions.Source: §A.4.1 · pp. 14–15, Figure 4 · p. 16

A.4.2 COORDINATE TOKENIZATION AND PRETRAINING

• The full model produces more complete road structures and better-preserved endpoints than vanilla tokenization or coordinate tokenization without pretraining.The ablated models can omit ground-truth boundaries or generate polylines extending beyond intended endpoints.Source: §A.4.2 · p. 15, Figure 5 · p. 17