PaperlayerView exact revision on arXiv

Source-linked AI summary

MapAgent: An Industrial-Grade Agentic Framework for City-scale Lane-level Map Generation

Deguo Xia, Zihan Li, Haochen Zhao, Dong Xie, Yuyao Kong, Xiyan Liu, Jizhou Huang, Mengmeng Yang, Diange Yang

arXiv:2606.04513v2cs.AI

TL;DR

Lane-level mapping remains expensive and difficult to make specification-compliant because visual evidence and learned labels do not fully encode regulations or scene ambiguities. MapAgent adds selective, verification-driven refinement to a vectorization backbone, using structured judgments and deterministic edits. It improves lane-map correctness across backbones and supports production across over 360 cities with automation above 95%.

  • Problem

    Lane-level map production remains labor-intensive because end-to-end models do not explicitly represent cartographic and traffic constraints, especially in ambiguous long-tail scenes.

  • Method

    MapAgent selectively refines frozen backbone drafts with a bounded Judge–Planner–Worker loop that generates grounded diagnoses, deterministic edits, and re-validation.

  • Results

    MapAgent consistently improves lane-level correctness across GeMap and DuMapNet, while production integration supports over 360 cities and automation above 95%.

  • Takeaways & Limitations

    Agent-based refinement is presented as a practical and scalable approach for specification-compliant city-scale lane-level map generation and updating.

Abstract

from arXiv · show

Lane-level maps are critical infrastructure for autonomous driving and lane-level navigation, yet constructing and maintaining standardized lane networks for hundreds of cities remains highly labor-intensive. Recent end-to-end vectorized mapping methods can predict lane geometry and topology directly from sensor data, but they typically treat mapping specifications and traffic regulations as implicit, dataset-dependent supervision. Moreover, in complex scenes (e.g., worn or missing markings and occlusions), correct lane configurations are often under-determined by visual evidence alone, making specification violations a major source of human post-editing. We propose MapAgent, an industrial-grade agentic architecture that augments a vectorization backbone for specification-compliant lane-map production. Rather than merely adding an agent loop to map prediction, MapAgent couples backbone perception with explicit specification verification, constraint-aware reasoning, and deterministic map editing under a bounded, verification-driven Judge-Planner-Worker loop. A vision-language Judge diagnoses errors by jointly inspecting visual evidence and draft vectors, while a tool-calling Planner generates minimal corrective edits with post-edit re-validation. To remain scalable for city-scale production, MapAgent is selectively triggered only on tiles with low backbone confidence, adding modest overhead while preserving throughput. Experiments on real-world datasets show consistent gains over strong production baselines, especially in complex and long-tail scenarios. Additionally, MapAgent has been integrated into Baidu Maps, supporting lane-level map generation for over 360 cities nationwide and elevating the overall production automation to over 95%, demonstrating MapAgent's practicality and effectiveness for large-scale lane-level map generation.

1 Introduction

Lane-level maps support autonomous driving and navigation but remain costly to construct and maintain at nationwide scale. End-to-end vectorization has increased automation, yet industrial deployment still requires scalable quality and update performance.

  • Motivation: Lane-level maps provide centimeter-level priors for road geometry, lane topology, and traffic control beyond the sensing horizon.These priors support robust planning and decision-making.
  • Motivation: Constructing and maintaining lane-level maps for hundreds of cities remains extremely expensive and limits nationwide coverage and update cycles.Traditional pipelines rely heavily on trained experts for rule interpretation, annotation, and updating.
  • Prior Work: Recent end-to-end methods convert multi-sensor inputs into BEV features and directly decode vectorized polylines or topology, replacing much of the manual mapping pipeline.Examples include HDMapNet, VectorMapNet, MapTR, and MapTRv2.
  • Industrial Context: DuMapNet and LDMapNet-U extend industrial vectorization toward city-scale lane-level generation and historical-map-conditioned updating.DuMapNet was deployed in Baidu Maps, while LDMapNet-U targets map updating.

A. Baseline System

Existing one-pass systems learn visible lane structure but do not explicitly represent mapping specifications or reason about ambiguous scenes for geometric editing. MapAgent addresses this gap by refining backbone drafts through verification and deterministic edits.

  • Baseline Comparison: One-pass pipelines vectorize lanes from BEV images and rely on human post-editing to produce the final map.MapAgent instead adds automated verification and correction through a Judge–Planner–Worker loop.
  • Baseline Limitations: Existing systems learn visible lane boundaries, crosswalks, stop lines, and local topology but must also satisfy cartographic standards and traffic regulations.Industrial compliance requires consistent lane-group organization, geometry, attributes, and topology.
  • Baseline Limitations: Degraded markings, wide unstructured pavements, and adverse lighting create long-tail scenes where visual evidence alone can under-determine lane configurations.Commercial pipelines therefore continue to depend on expert editors for specification-based topology repair and compliance.
  • Proposed Shift: MapAgent treats frozen backbone outputs as mutable drafts and enforces specifications through verification and deterministic edits.The backbone remains responsible for scalable visual prediction, while the agentic layer handles controllable refinement.
  • Contributions: The framework is presented as an industrial-grade refinement layer for city-scale lane-level map generation and updating.Its contribution combines a frozen BEV backbone with a bounded, verification-driven Quality Agent and Judge–Planner–Worker loop.

2 MapAgent Framework

MapAgent treats backbone vector maps as mutable drafts and refines them through a specification-aware, verification-driven Judge–Planner–Worker workflow. Confidence-based routing, structured diagnosis, constrained planning, and deterministic editing provide scalable lane-map refinement while preserving the frozen backbone.

  • Overall architecture: MapAgent reframes lane-map generation as iterative refinement of a backbone-produced draft rather than a one-pass prediction.The system combines a frozen BEV vectorization backbone with an agentic verification-and-editing layer.
  • Overall architecture: The Quality Agent directly accepts tiles above confidence threshold δ = 0.7 and forwards only lower-confidence tiles to refinement.Selective routing focuses computation on difficult regions while preserving backbone throughput.
  • Refinement workflow: The bounded Judge–Planner–Worker loop re-validates each iteration and permits at most three refinement rounds to control latency and avoid cascading modifications.Validated edits are retained, while the system returns the refined map or best intermediate result after the budget is exhausted.
  • Judge Agent: The Judge produces lane-wise structured diagnoses containing an error category, confidence, and concise evidence summary for Planner consumption.Diagnoses are aggregated across selected low-confidence lanes into a step-level diagnosis.
  • Planner Agent: The Planner converts Judge outputs into executable, tool-grounded corrective plans under immutable geometric, topological, and mapping-specification constraints.Plans use a closed action schema and reject edits that fail the feasibility gate Ω.
  • Worker Agent: Workers apply sequential deterministic edits for category correction, deletion, local smoothing, or constrained local geometry regeneration.The fixed tool set supports auditable local modifications, while regeneration is the only worker using a learned model.

3 Experiments

Experiments evaluate MapAgent as a frozen post-hoc refinement module across datasets, backbones, Judges, runtime settings, and ablations. Results show improved lane-level correctness through specification-aware, conservative correction, with particular benefits in challenging scenes.

  • Experimental Setup: MapAgent is evaluated on GeMap and DuMapNet with frozen backbone predictions and without retraining.The protocol also compares Qwen3-VL-Instruct, Qwen3-VL-Thinking, and InternVL-3.5-8B under identical prompting and refinement budgets.
  • Runtime: 420 ms/tile mean latency and 1.6 s p99 latency are reported, with MapAgent triggered on about 30% of test tiles.The Judge averages 230 ms per tile, the Worker 140 ms per tile, and peak GPU memory is approximately 19 GB per A800.
  • Judge Results: 86.01% overall Judge accuracy follows GRPO alignment for Qwen3-VL-8B-Thinking, up from 83.55%, although Structure Error accuracy slightly decreases.SFT yields 70.16% for Qwen3-VL-8B and 58.23% for InternVL-3.5-8B; GRPO improves most class-wise precision and recall values.
  • Main Results: On GeMap, Qwen3-VL-Thinking raises Accuracy from 52.8 to 61.3 and F1 from 69.1 to 76.0.On DuMapNet, Accuracy rises from 52.2 to 63.9 and F1 from 68.6 to 78.0, with stable improvements across model variants.
  • Main Results: MapAgent gains concentrate in Precision, Recall, and classification correctness while geometric IoU metrics remain stable.This pattern reflects conservative correction of spurious lanes, category mismatches, and match failures rather than aggressive geometry alteration.
  • Ablation Study: With reasoning removed, Accuracy reaches 58.4 and F1 73.7, whereas the full Judge–Planner–Worker loop reaches F1 77.0 and Cls Acc 98.0.The comparison indicates that structured evidence helps the Planner produce actionable, localized, tool-grounded edits.
  • Ablation Study: Increasing the correction budget from T=1 to T=3 raises Accuracy from 58.3 to 62.6 and F1 from 73.6 to 77.0, with diminishing returns.BBox/Mask IoU changes only slightly, from 70.4/35.0 at baseline to 71.8/36.0 at T=3.
  • Case Study: In faint, missing, or occluded markings, both GeMap and DuMapNet often produce spurious lanes, fragmented segments, and inconsistent topology.MapAgent refines these noisy predictions into cleaner maps with fewer spurious segments and more consistent global topology while keeping geometry conservative.

4 Discussion

MapAgent targets production-scale, specification-compliant lane-map refinement while limiting intervention to safe, deterministic corrections. Its current scope excludes ambiguous lane additions and non-local topology changes, and it still depends on an existing backbone.

  • Discussion: MapAgent combines selective hard-tile triggering with bounded Judge–Planner–Worker refinement to reduce manual correction workload while preserving throughput.The loop diagnoses violations, applies deterministic edits, and re-validates results.
  • Discussion: The production system supports lane-level map generation for 360+ cities and reports over 95% automation.Automation is defined as the share of lane-level mileage completed fully automatically without human intervention.
  • Limitations: Extreme visual ambiguity can leave lane addition and topology modification under-determined, requiring stronger priors and principled uncertainty handling.The current production setting prioritizes manual post-editing categories addressable with safe, deterministic, local tools.
  • Limitations: MapAgent currently refines outputs from an existing backbone rather than autonomously orchestrating specialized perception backbones.A unified framework for scheduling specialized backbones and integrating complementary predictions remains future work.

5 Related Work

Related work spans end-to-end vectorized map generation and increasingly capable tool-using multimodal agents. MapAgent is positioned at their intersection, adapting agentic verification to the strict constraints of lane-map production.

  • Vectorized HD-map generation: End-to-end HD-map systems evolved toward direct vectorized regression, with methods such as VectorMapNet and MapTR decoding polylines using transformer architectures.Later systems such as GeMap and HiMap further address geometric consistency and structured modeling.
  • LLM/VLM-based agents: Agent research has developed reasoning, tool use, feedback-driven self-correction, modular routing, grounded execution, and multimodal visual grounding.Representative lines include ReAct, Toolformer, Reflexion, MRKL, Say-Can, ReWOO, Voyager, and multimodal foundation models.
  • Research gap: Prior agentic workflows had been applied to driving planning but had not been adapted to the strict safety constraints of map production.This leaves a gap between general agent capabilities and specification-compliant lane-map generation.

6 Conclusions

MapAgent reframes lane-map generation as bounded, verification-driven refinement over a frozen vectorization backbone. Experiments report consistent gains in challenging scenarios, alongside deployment across more than 360 cities with over 95% production automation.

  • Conclusion: MapAgent uses a lightweight Quality Agent and bounded Judge–Planner–Worker loop to refine hard tiles from frozen backbone predictions.The Judge diagnoses errors, the Planner generates tool-grounded edits, and deterministic Workers execute them safely.
  • Conclusion: The framework enforces geometric and topological validity together with traffic standards through constrained, deterministic edits.Post-edit validation keeps the refinement process verification-driven.
  • Conclusion: Experiments on large-scale real-world datasets show consistent gains over strong production baselines, especially in long-tail scenarios.The reported results support structured agentic refinement for challenging map-generation conditions.
  • Deployment: MapAgent supports lane-level map generation and updating for over 360 cities nationwide and elevates overall production automation to over 95%.The system has been integrated into the Baidu Maps production pipeline.

A Feasibility Check and Specification Library

MapAgent formalizes specification compliance as a boolean feasibility gate over geometric, topological, cartographic, and traffic constraints. Deterministic actions are accepted only when both the action and the resulting map state satisfy validity checks.

  • Feasibility formulation: The feasibility formulation uses a boolean QC function induced by a versioned library of hard specification predicates.These predicates derive from industry standards and internal cartographic specifications.
  • Specification checks: Geometric and topological checks cover conditions such as no self-intersection, bounded curvature or length, and lane-group consistency.Each predicate encodes a non-negotiable cartographic or traffic constraint.
  • Action validation: A deterministic tool transition updates the map state, but an action must satisfy schema, parameter, edit-magnitude, and lane-group-scope validity.The updated state is accepted only when it passes the QC gate.
  • SAM3 fine-tuning: SAM3 fine-tuning is described as a progressive strategy that gradually relaxes optimization constraints on pretrained components.The accompanying architecture figure presents the model design and progressive fine-tuning approach.

B.1 Training setup

Training uses a shared SAM3 initialization and consistent lane-dataset preprocessing, with staged optimization, matching, and heavily weighted segmentation supervision.

  • Data and preprocessing: All stages initialize from the same pretrained SAM3 checkpoint and use lane-dataset train.json and val.json files.Segmentation supervision and RLE mask loading remain enabled throughout training.
  • Data and preprocessing: Inputs are resized to 1008-square resolution with a minimum scale of 480, then padded and normalized using mean and standard deviation (0.5, 0.5, 0.5).Training also perturbs input boxes with Gaussian noise, filters empty targets, and caps images at 30 lane instances.
  • Optimization: Optimization uses AdamW, bfloat16 mixed precision, gradient clipping at max norm 1.0, and an inverse-square-root schedule with 2000 warmup steps.The base transformer learning rate is 2 × 10^-5.
  • Optimization: Backbone learning rates are smaller than the transformer rate, with staged unfreezing of transformer, vision, and language components across three training stages.Vision and language learning rates are 6.25 × 10^-6 and 1.25 × 10^-6 when unfrozen, respectively.
  • Matching and losses: Matching uses binary Hungarian assignment with focal loss and an auxiliary one-to-many branch, while lane losses emphasize classification, presence, masks, and Dice terms.The specified weights include 15.0 for classification and presence, 8.0 for mask loss, and 12.0 for Dice loss.

B.2 Progressive Fine-tuning Strategies

SAM3 fine-tuning progressively relaxes which components can adapt, balancing pretrained representation stability with lane-specific refinement. The section also illustrates priority-ordered Judge reasoning for diagnosing lane errors.

  • B.2 Progressive Fine-tuning Strategies: Three fine-tuning strategies progressively increase adaptation from constrained updates toward broader component updates to balance stability and task-specific adaptability.The strategies are framed as different optimization constraints rather than a single fine-tuning configuration.
  • Stage 1: Stage 1 freezes the vision and language backbones and optimizes only transformer layers, preserving pretrained visual representations while learning cross-modal interactions.This stage provides a stable reference for modeling thin and elongated lane structures.
  • Stage 2: Stage 2 adapts the vision backbone with a smaller learning rate and layer-wise decay while keeping the language backbone frozen.The design targets perspective effects and long-range continuity without disrupting deeper pretrained features.
  • Stage 3: Stage 3 updates all major components with carefully scaled learning rates, retaining the transformer as the primary optimization target.The asymmetric updates refine geometric perception and textual alignment while limiting drift from pretrained semantics.
  • Judge reasoning: The Judge’s priority order checks existence, category, geometry, and structure, stopping after a higher-priority condition is satisfied.The illustrated prediction is ultimately classified as structure_error after lower-priority checks are eliminated.

C Chain-of-Thought Data Generation with GPT-5.2

The paper constructs a controlled GPT-5.2-generated chain-of-thought dataset to supervise Judge diagnostics with fixed, visually grounded, priority-aware rationales.

  • Data generation: GPT 5.2 generates concise visual rationales for fixed oracle outcomes rather than inferring labels, preventing label ambiguity and decision leakage.The oracle annotation remains immutable, and GPT 5.2 is instructed only to explain the supplied decision.
  • Data generation: Each training instance contains a single frozen-backbone lane prediction, a Pred image, a GT image for supervision only, metadata, and an oracle error annotation.Metadata include the predicted category, a tight bounding box, and correctness or error type.
  • Visual grounding: Generated explanations must rely only on observable cues in the Pred image, despite the model receiving the GT image for supervision.The prompt forbids references to ground truth, labels, and cross-image comparison.
  • Reasoning structure: Reasoning follows a strict priority-based error taxonomy in which higher-priority errors short-circuit all lower-priority checks.This mirrors inference-time Judge behavior and prevents mixed or internally inconsistent explanations.
  • Reasoning structure: Every rationale contains exactly four sentences: elimination of higher-priority errors followed by an explicit oracle error conclusion.Fixed length simplifies downstream parsing and stabilizes supervised training.
  • Output constraints: The prompt requires concrete, image-verifiable descriptions such as early termination, curvature deviation, gaps, misalignment, and partial marking coverage.Outputs are constrained to a single JSON object containing the four-sentence rationale.

C.5 Resulting CoT Dataset

The resulting CoT dataset provides deterministic Judge supervision grounded in predicted-image evidence and aligned with oracle error labels. Its deployment value is paired with difficult cases where weak evidence or corrupted initial predictions limit conservative refinement.

  • Resulting CoT Dataset: The CoT rationales are grounded in Pred images, aligned with oracle error labels, and organized by the Judge Agent’s priority-structured decision logic.These properties make the rationales concise and deterministic supervision targets.
  • Bad cases: Weak lane evidence, shadows, occlusions, and ambiguous topology can leave missing, shifted, or structurally inconsistent segments unresolved.Refinement quality remains dependent on recoverable visual cues and the quality of the initial backbone prediction.
  • Judge supervision: Judge training uses generated rationales as reasoning references and oracle error types and evidence summaries as structured supervision targets.The model evaluates predictions against painted markings, curbs, asphalt boundaries, continuity, and local topology.
  • Judge supervision: Priority-constrained evaluation skips lower-priority checks after confirming a higher-priority error and permits topologically justified virtual lines without visible markings.A virtual line is labeled extra_lane_line only when such justification is absent.
Loading 2606.04513v2…