Source-linked AI summary
MARS - A foundational map auto-regressor
Qi Zhang, Suvam Bag, Rupanjali Kukal, Lucas Thai, Jacob Kovarskiy, Mike Bianco, Xiaojie Guo, Raphael Tang, Mikael Figueroa, Rishi Madhok, Nikolaos Karianakis, Fuxun Yu
TL;DR
Map generation requires vectorized geometric outputs that do not fit common pixel-grid generative models. MARS represents roads and buildings as sequences and generates them end to end, while supporting user-guided completion. It reports stronger balanced F1 performance across datasets than prior state-of-the-art approaches.
Problem
Map features are variable, unstructured geometric shapes, posing a challenge for pixel-grid generative architectures.
Source: §1 · p. 1Method
MARS converts vectorized map primitives into a sequential language representation and uses an end-to-end transformer vision encoder and autoregressive decoder.
Source: §1 · p. 2Results
MARS achieves superior balanced F1 scores across datasets, outperforming previous state-of-the-art approaches on average.
Source: §5.2 · p. 8Takeaways & Limitations
“Chat with MARS” lets users guide completion of missing streets or buildings for interactive map generation and correction.
Source: §1 · p. 2Autoregressive decoding adds computational overhead relative to traditional one-shot segmentation methods because decoding is sequential.
Source: §5.5 · p. 10
Abstract
from arXiv · showhide
Map generation tasks feature extensive non-structural vectorized data (e.g., points, polylines, and polygons) and thus pose significant challenges to common pixel-wise generative models. Conventional approaches use multiple stages, first segmenting these features at the pixel level and then performing vectorized post-processing, with errors and complexity compounding at each stage. Motivated by the recent success of auto-regressive language modeling, we propose the first map foundation model, named Map Auto-Regressor (MARS), that is capable of generating both multi-polyline road networks and polygon buildings in a unified manner. For training MARS, we collected to our knowledge the largest multi-class map extraction dataset totaling 3.4M examples, which we call MAP-3M. Across four road and building datasets, MARS outperforms or matches the performance of multistage baselines. Additionally, we develop a "Chat with MARS" capability that enables interactive human-in-the-loop map generation and correction, supported by the auto-regressive nature of our end-to-end approach. We release our MAP-3M dataset and project demo page at (1) https://huggingface.co/datasets/bag-lab/MAP-3M and (2) https://huggingface.co/spaces/bag-lab/MARS, respectively.
1 INTRODUCTION
Map generation must represent points, polylines, and polygons, unlike pixel-grid outputs from common generative models. MARS treats these map primitives as a formal language and generates roads and buildings end to end, while MAP-3M supplies large-scale training data and “Chat with MARS” supports interactive correction.
1 INTRODUCTION
- Map features are variable, unstructured geometric shapes, creating a mismatch with pixel-grid generative architectures.Conventional approaches often segment pixels first and then apply vectorized post-processing; feature-specific heuristics limit generality and add hyperparameters.Source: §1 · pp. 1–2
- MARS converts roads and buildings into a sequential language representation and generates them end to end with a transformer vision encoder and autoregressive decoder.The framework applies no intermediate or post-processing.Source: §1 · p. 2
- “Chat with MARS” lets users prompt the model with a missing street or building starting point, which it completes for interactive map generation and correction.The capability is linked to MARS’s autoregressive nature.Source: §1 · p. 2
- MAP-3M provides three million images annotated for roads and buildings, addressing the limited quantity and single-class scope of existing map datasets.The authors describe it as 10× larger in image count and 100× broader in geographic coverage than common benchmarks.Source: §1 · p. 2
- MARS demonstrates superior performance to previous state-of-the-art models in benchmarks on MAP-3M and downstream applications.The introduction reports this overall comparison without specifying metric values.Source: §1 · p. 2
2 THE MARS FRAMEWORK
MARS turns vectorized map elements into sequences and generates them end to end from imagery. Its representation handles complex road networks, while a vision backbone and autoregressive transformer produce map tokens.
2 THE MARS FRAMEWORK
- MARS combines a map-to-sequence algorithm with an end-to-end autoregressor containing a vision backbone and a cross-attending autoregressive transformer.The transformer generates semantic class and coordinate tokens using visual context and previously generated tokens.Source: §2 · p. 3, §2.2 · p. 4
2.1 MAP-TO-SEQUENCE CONVERSION
- Points, polylines, and polygons are represented as vertex sequences, with polygons closing by repeating their starting vertex.Class tokens distinguish map-object types, and the resulting sequences support autoregressive learning without manual vectorized post-processing.Source: §2.1 · p. 3
- A stroke-based algorithm decomposes complex road graphs at intersections and merges consecutive segments within an angle tolerance, such as < 30°, into single-polyline roads.The resulting roads can be represented in the same sequence format, using the road token.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2 MAP AUTO-REGRESSOR ARCHITECTURE
- The autoregressive transformer predicts class and coordinate tokens with causal attention, teacher forcing, and a shared decoding vocabulary for semantic classes and pixel locations.Objects are ordered by distance from the image centroid and then clockwise angle when distances are similar.Source: §2.2 · p. 4
3 Chat with MARS: HUMAN-IN-THE-LOOP MAP GENERATION
Chat with MARS lets users intervene in autoregressive map generation using prompts. Start-, mid-, and end-sequence interactions respectively condition, redirect, or resume decoding to address missing or drifting objects.
3 Chat with MARS: HUMAN-IN-THE-LOOP MAP GENERATION
- Teacher-forced autoregressive training supports user interventions during decoding, forming an interactive human-in-the-loop map generation and correction capability.Users can indicate a missing object or correct drifting predictions; mixed interaction modes can form multi-round editing conversations.Source: §3 · pp. 4–5, §3.3 · p. 5
3.1 START OF SEQUENCE CHATTING
- Start-of-sequence chatting supplies a clicked starting point as a class token and x,y coordinates to guide subsequent generation.It is particularly useful for blurry or out-of-domain images, where an ill-conditioned first vertex may affect the sequence.Source: §3.1 · p. 5
3.2 MID OF SEQUENCE CHATTING
- Mid-sequence chatting replaces drifting coordinate tokens with a user-clicked x,y location to redirect road generation.New tokens are constrained to the current object, stopping at the next object-class token to avoid affecting other objects.Source: §3.2 · p. 5
3.3 END OF SEQUENCE CHATTING
- End-of-sequence chatting removes the end token and prompts a new object class and coordinates to resume generation for missed objects.The paper identifies this mode as a way to improve recall.Source: §3.3 · p. 5
4 MAP-3M: A LARGE-SCALE MAP DATASET
MAP-3M is a large aerial-image dataset with vectorized annotations for roads and buildings. Its image count and geographic coverage exceed those of prior literature datasets, while other map classes are omitted for now.
4 MAP-3M: A LARGE-SCALE MAP DATASET
- MAP-3M addresses the single-class, limited-quantity scope of widely used map benchmarks, which separately focus on roads or buildings.Cityscale and SpaceNet3 contain road annotations, while AI-Crowd focuses on buildings.Source: §4 · p. 5
- MAP-3M contains approximately 3M high-resolution images annotated for both buildings and roads, with 10× more images and 100× more spatial coverage than the previous largest dataset.The dataset comparison is shown in Table 1.Source: §4 · pp. 5–7, Table 1 · p. 6
- The released MAP-3M annotations cover buildings and roads; other classes, including waterbodies and parking lots, are omitted because they are much less dominant.The paper notes that other classes can be sequentialized with the same polygon/polyline representations.Source: §4 · p. 6
5 EXPERIMENTAL RESULTS
MARS is evaluated on unified map generation, downstream road and building benchmarks, pretraining, and interactive correction. Results show strong benchmark performance and benefits from pretraining and user input, alongside sequential-decoding overhead and difficult cases.
5.1 FROM SINGLE TO MULTI-CLASS: TOWARDS UNIFIED MAP GENERATION
- MARS handles single-polyline roads, complex multi-polylines, and mixed building-road tiles in one end-to-end model, with class diversity scaling without obvious performance degradation.The Table 2 ablation compares single-class and multi-class architectures; the unified model adds class tokens.Source: §5.1 · p. 7, Table 2 · p. 7
5.2 DOWNSTREAM FINETUNING COMPARISON
- 82.88% F1 on Cityscale improves on RNGDet++’s 78.44%, while MARS leads SpaceNet recall at 84.56% and is 0.46% below RNGDet++ in F1.On AICrowd-V1, MARS is reported closely on par with prior state-of-the-art building models despite being a generic map-generation model.Source: §5.2 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 8
5.3 IMPORTANCE OF MAP-3M PRE-TRAINING
- 70.45% to 82.05% on SpaceNet: MAP-3M pretraining improves MARS accuracy and speeds convergence compared with training without pretraining.The authors attribute the challenge to autoregressive next-token prediction being more sensitive to data scarcity and harder to optimize on limited data.Source: §5.3 · p. 9, Table 5 · p. 9
5.4 CHAT WITH MARS
- User input consistently improves precision, recall, F1, and IoU for roads and buildings across Cityscale, SpaceNet, and AICrowdV1-partial.The two protocols provide one or two vertices of a missing object; new predictions are combined with existing predictions for evaluation.Source: §5.4 · p. 9, Table 6 · p. 10
5.5 LIMITATIONS
- MARS’s autoregressive decoding has more computational overhead than one-shot segmentation because decoding is inherently sequential.The authors identify KV-cache acceleration and parallel decoding as techniques that can speed autoregressive models.Source: §5.5 · p. 10
- Complex intersections can lead MARS to overlook thinner road features, while tree canopy or shadows can obscure buildings and roads and cause missed vertices.The paper also shows successful examples in which the model infers plausible shapes under substantial occlusion.Source: §5.5 · p. 10
6 CONCLUSION
The conclusion presents MARS as an autoregressive, end-to-end framework for vectorized map generation, supported by MAP-3M and an interactive human-in-the-loop capability.
6 CONCLUSION
- MARS uses map-to-sequence conversion to frame map generation as language modeling and generates maps end to end without post-processing.The conclusion describes this as treating maps as a foreign language.Source: §6 · p. 10
- MAP-3M is presented as 10× larger than the then-current largest map dataset, supporting foundational model training.The conclusion lists dataset curation as one of MARS’s three major contributions.Source: §6 · p. 10
- The emerging Chat with MARS feature enables human-in-the-loop vectorized map generation.The conclusion names this as an interactive capability of the framework.Source: §6 · p. 10
A.1 MAP-3M DATA CURATION
MAP-3M combines imagery and vector annotations collected across U.S. cities, with a unified labeling format for buildings and roads.
A.1 MAP-3M DATA CURATION
- MAP-3M imagery comes from NAIP, usually at 0.6 meter per pixel, sampled across 5,000 cities in 50 states from imagery dated 2020–2024.Large image chips are tiled into non-overlapping 512×512 subtiles.Source: §A.1 · p. 12
- Building labels draw on Overture Map Foundation annotations, while road labels come from OpenStreetMap and cover road subtypes including highways, paths, and bridleways.The paper describes the sources as supporting broad geographic coverage and diverse structural representation.Source: §A.1 · p. 12
- Buildings are separate objects with dynamic-length vertex sequences, while roads are split into separate open-ended polylines after graph and stroke-based processing.The unified labeling format is derived from COCO and is intended to simplify multi-class annotation handling.Source: §A.1 · p. 13
A.2 AICROWDV2 VAL
The AICrowdV2 evaluation uses revised training and test splits; the authors report no prior published benchmarks on V2.
A.2 AICROWDV2 VAL
- AICrowdV2’s revised split addresses potential training-validation leakage reported for V1, whose validation set has about 60,000 images versus about 25,000 in V2’s test set.The paper presents its V2 results as the first public evaluation on the corrected version because published V2 benchmarks were unavailable.Source: §A.2 · p. 13
- AICrowdV1 remains included because it is widely adopted in prior research, while V2 provides a revised evaluation structure.The paper reports results on V1 and V2 in separate tables.Source: §A.2 · p. 13
- The paper evaluates building predictions with AP, AR, and IoU on the two AICrowd test sets.The authors describe these metrics as suited to polygonal predictions.Source: §A.2 · p. 13
A.3 ADDITIONAL RESULTS
The visualizations show MARS handling some occluded structures robustly, while identifying complex intersections and dense canopies as difficult cases where high accuracy is harder to preserve.
A.3 ADDITIONAL RESULTS
- MARS visualization shows robust handling of occluded structures.Source: Figure 8 · p. 14
- Complex intersections are difficult examples where MARS finds it difficult to preserve high accuracy.Source: Figure 9 · p. 14
- Dense canopies occluding roads and buildings are difficult examples where MARS finds it difficult to preserve high accuracy.Source: Figure 9 · p. 14