Source-linked AI summary
PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning
Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, Dongbin Zhao
TL;DR
Rule-based and learning-based planners have limitations in long-tailed and large-scale closed-loop settings, and recent LLM planning approaches face further evaluation, representation, and safety challenges. PlanAgent uses an MLLM with transformed scene inputs, staged reasoning, and reflection; the authors report state-of-the-art results on nuPlan closed-loop benchmarks.
Problem
Rule-based planners struggle with long-tailed situations, learning-based methods have not surpassed them in large-scale closed-loop planning, and recent LLM approaches face evaluation, representation, and safety limitations.
Source: §I · pp. 1–2Method
PlanAgent transforms scenes into BEV and lane-graph inputs, uses hierarchical MLLM reasoning to generate planner code, and applies simulation and scoring to filter unreasonable planners.
Source: §I · p. 2Results
PlanAgent achieves state-of-the-art performance on closed-loop reactive and non-reactive nuPlan Val14 and Test14-hard benchmarks.
Source: §I · p. 2Takeaways & Limitations
The authors conclude that scene extraction, staged reasoning, and reflection enhance planning safety.
Source: §VII · p. 10PlanAgent depends on accurate scene and planner-parameter understanding, but current MLLMs can misread these and generate unreasonable planner code.
Source: A. Limitations · p. 10
Abstract
from arXiv · showhide
Vehicle motion planning is an essential component of autonomous driving technology. Current rule-based vehicle motion planning methods perform satisfactorily in common scenarios but struggle to generalize to long-tailed situations. Meanwhile, learning-based methods have yet to achieve superior performance over rule-based approaches in large-scale closed-loop scenarios. To address these issues, we propose PlanAgent, the first mid-to-mid planning system based on a Multi-modal Large Language Model (MLLM). MLLM is used as a cognitive agent to introduce human-like knowledge, interpretability, and common-sense reasoning into the closed-loop planning. Specifically, PlanAgent leverages the power of MLLM through three core modules. First, an Environment Transformation module constructs a Bird's Eye View (BEV) map and a lane-graph-based textual description from the environment as inputs. Second, a Reasoning Engine module introduces a hierarchical chain-of-thought from scene understanding to lateral and longitudinal motion instructions, culminating in planner code generation. Last, a Reflection module is integrated to simulate and evaluate the generated planner for reducing MLLM's uncertainty. PlanAgent is endowed with the common-sense reasoning and generalization capability of MLLM, which empowers it to effectively tackle both common and complex long-tailed scenarios. Our proposed PlanAgent is evaluated on the large-scale and challenging nuPlan benchmarks. A comprehensive set of experiments convincingly demonstrates that PlanAgent outperforms the existing state-of-the-art in the closed-loop motion planning task. Codes will be soon released.
I. INTRODUCTION
PlanAgent is proposed to address weaknesses in closed-loop vehicle planning and recent LLM-based approaches. It uses an MLLM with environment transformation, hierarchical reasoning, and reflection, and the authors report state-of-the-art nuPlan results.
I. INTRODUCTION
- Existing rule-based planners struggle with long-tailed situations, while learning-based methods have difficulty generalizing from open-loop settings to large-scale closed-loop scenarios.The introduction also notes that learning-based approaches can underperform rule-based methods in large-scale closed-loop evaluation.Source: §I · p. 1
- PlanAgent combines a BEV map and lane-graph text inputs with hierarchical reasoning from scene understanding through motion instructions to planner-code generation, then simulates and scores plans to filter unreasonable outputs.The reflection step aims to reduce the safety impact of MLLM uncertainty.Source: §I · p. 2
- PlanAgent achieves state-of-the-art performance on closed-loop reactive and non-reactive nuPlan Val14 and Test14-hard benchmarks.The benchmarks cover common and long-tailed scenarios.Source: §I · p. 2
- PlanAgent is presented as the first closed-loop mid-to-mid autonomous-driving planning agent system based on an MLLM.The authors also report that its scene descriptions use one-third as many tokens as those of LLM-based state-of-the-art methods.Source: §I · p. 2
II. RELATED WORK
Related work spans mid-to-mid planning and LLM-based autonomous planning. The paper positions PlanAgent against methods whose settings, scene representations, or direct trajectory generation leave challenges for complex closed-loop planning.
A. Motion Planning for Autonomous Driving
- End-to-end approaches have advanced in architecture, representations, multimodal fusion, and generative models, but evaluations often remain open-loop or confined to simple simulator traffic.The authors identify a lack of realistic closed-loop simulators as limiting evaluation authenticity and diversity.Source: A. Motion Planning for Autonomous… · p. 2
- Mid-to-mid planning methods use recorded perception results and HD maps; PDM performs strongly on Val14 but struggles to generalize to the long-tail Test14-hard benchmark.Learning-based examples include lane-graph goal-oriented planning, DETR-based planning, and occupancy-heatmap forecasting, with some methods using post-optimizers.Source: A. Motion Planning for Autonomous… · pp. 2–3
B. Large Language Models for Autonomous Planning
- Prior LLM-based vision-centric approaches use images or LiDAR for commands or trajectories, often in open-loop settings; LMDrive is described as a closed-loop end-to-end system in CARLA.DriveLLM integrates LLMs with existing autonomous-driving modules for decision-making in real-world scenarios.Source: B. Large Language Models for… · p. 3
- Mid-to-mid LLM methods include DiLu’s reasoning, reflection, and memory; GPT-driver’s natural-language modeling; and LLM-ASSIST’s controller-parameter generation.PlanAgent instead uses BEV and lane-graph descriptions, hierarchical reasoning, and a more token-efficient environment transformation.Source: B. Large Language Models for… · p. 3
III. METHODOLOGY
PlanAgent uses an MLLM in a three-module planning pipeline: Environment Transformation supplies scenario prompts, Reasoning Engine generates a planner, and Reflection verifies it. Environment Transformation combines a BEV map with lane-graph-derived text to represent global scene semantics and local motion information.
A. Overall Architecture
- At each planning interval, Environment Transformation retrieves scene information, Reasoning Engine generates a planner, and Reflection verifies it.Reflection examines generated planners to mitigate MLLM uncertainty in planning safety.Source: A. Overall Architecture · p. 3
B. Environment Transformation
- The module represents global scene information in a BEV map and local motion information in lane-graph-based text, then uses both as multimodal scenario prompts.The BEV map visualizes map, agent, and obstacle information; agents are marked with direction arrows whose lengths indicate speed.Source: B. Environment Transformation · pp. 3–4
- Lane-graph text encodes positional and connection relationships between equal-length centerline segments, while nearby agents and obstacles are described by speed, distance, lane node, and traffic-light status.The described agents and obstacles are restricted to those on the eight lane nodes around the ego vehicle’s node.Source: B. Environment Transformation · p. 4
C. Reasoning Engine
The Reasoning Engine takes scenario and system prompts and guides an MLLM through hierarchical reasoning to generate IDM-based planner code. Its stages move from scene understanding to motion instructions and then planner-code generation.
C. Reasoning Engine
- The Reasoning Engine takes scenario prompts and predefined system prompts, then uses hierarchical chain-of-thought reasoning to generate planner code based on IDM.The MLLM is introduced to the driving planning task through in-context learning.Source: C. Reasoning Engine · pp. 4–5
- System prompts define the task and code format, provide driving common sense and planner examples, and guide stepwise reasoning.The task prompt explains the physical meanings of IDM parameters, including centerline, leading agent, speed limit, acceleration, and deceleration.Source: C. Reasoning Engine · p. 5
- Scene understanding questions cover global scenario type and traffic-light status as well as local ego and same-lane vehicle status.The authors motivate this stage by noting that scenario types can require different IDM parameters and planners.Source: C. Reasoning Engine · p. 5
- The MLLM next gives longitudinal and lateral motion instructions with rationale, then generates Python planner code.Longitudinal instructions include accelerating, decelerating, or maintaining speed; lateral instructions include changing lanes or keeping the current lane.Source: C. Reasoning Engine · p. 5
D. Reflection
Reflection simulates each generated planner and uses its score to decide whether to execute it or ask the Reasoning Engine to rethink. The module uses short-term simulation and limits repeated executions.
D. Reflection
- Every generated planner is simulated and scored using measures such as collision potential and comfort; a score below threshold λ prompts the Reasoning Engine to reprocess its reasoning and generate another planner.The assessed score determines whether to execute or rethink the planner.Source: D. Reflection · p. 6
- Short-term simulation is intended to limit uncertainty and cumulative error in predicting other agents; after maxexec is exceeded, the last generated controller code is selected.The execution limit reflects decision-making time constraints.Source: D. Reflection · p. 6
A. Implementation Details
The implementation uses a pretrained MLLM and evaluates PlanAgent on nuPlan Val14 and Test14-hard with closed-loop scores. The section identifies the benchmark metrics and comparison methods, but the supplied table and figure captions do not report result values.
A. Implementation Details
- PlanAgent uses a pretrained MLLM without training for planning, with GPT-4V selected in this paper; the score threshold λ is 0.75 and maxexec is 3.The authors state that experiments with other MLLMs will be shown in Sec. V-E.Source: A. Implementation Details · p. 6
B. Datasets
- Evaluation uses nuPlan Val14 and Test14-hard; Test14-hard represents the benchmark’s long-tailed scenarios.nuPlan contains 1500 hours of human driving data from Boston, Pittsburgh, Las Vegas, and Singapore.Source: B. Datasets · p. 6
C. Metrics
- The evaluation uses NR-CLS and R-CLS, both scored from 0 to 100 with higher scores better, measuring driving performance in simulation.R-CLS uses IDM-controlled background traffic, whereas NR-CLS uses log-replay of other agents.Source: C. Metrics · p. 6
D. Baseline Methods
- The comparison set includes learned planners, rule-based PDM, and LLM-ASSIST; the table caption covers Val14 and Test14-hard, while Fig. 5 compares NR-CLS with PDM-Closed across 14 Test14-hard scenarios.Because LLM-ASSIST is not open-source, only its publicly available Val14 results are reported.Source: D. Baseline Methods · pp. 6–7, Table II · p. 7, Figure 5 · p. 7
V. EXPERIMENTS
Across nuPlan Val14 and Test14-hard, PlanAgent performs strongly against rule-based, learning-based, and LLM-based methods. Its lane-graph scene descriptions also use fewer tokens than the compared LLM-based methods.
A. Comparison with SOTA Methods
- On Val14, PlanAgent outperforms rule-based, learning-based, and LLM-based methods on both NR-CLS and R-CLS.On Test14-hard, it has the highest R-CLS among those method categories, with only a slightly lower NR-CLS than planTF.Source: A. Comparison with SOTA Methods · p. 7
- Against PDM-Closed, PlanAgent performs better in Test14-hard lane changes, turns, high-lateral-acceleration, and pickup/dropoff scenarios.These are scenarios requiring complex driving maneuvers.Source: A. Comparison with SOTA Methods · p. 7
- PlanAgent performs well on both Val14 and long-tailed Test14-hard, while PDM-Closed generalizes poorly to Test14-hard and planTF drops on regular scenarios.PlanAgent has good performance across both benchmark settings, unlike these contrasting patterns.Source: A. Comparison with SOTA Methods · p. 7
- PlanAgent has the lowest average scene-description token usage among the compared LLM-based methods, with its lane-graph-based text description contributing to more efficient scene descriptions.The comparison adapts DiLu and GPT-driver prompts to nuPlan and counts LLM-ASSIST tokens from sample prompts.Source: A. Comparison with SOTA Methods · pp. 7–8, Table III · p. 8
B. Ablation Study
Ablations on Test14-hard report improvements from both Environment Transformation inputs and the Reasoning Engine and Reflection components.
Ablation Study of Environment Transformation Module
- Adding BEV map input improves NR-CLS by 1.5 and R-CLS by 1.2, while lane-graph representation improves NR-CLS by 1.3 and R-CLS by 1.6.These are the respective comparisons against the ablated configurations in Tab. IV.Source: B. Ablation Study · p. 8
Ablation Study of Reasoning Engine Module and Reflection Module
- Removing scene-understanding and mid-level motion-instruction in-context learning lowers NR-CLS by 2.42 and 1.91, respectively; removing Reflection lowers it by 2.67.The Reflection module performs safety checks on planner code generated by the MLLM.Source: B. Ablation Study · p. 8
C. Qualitative Results.
Qualitative cases show PlanAgent translating traffic context into maneuvers that avoid stopping in the road. Its MLLM-call interval also affects planning performance, with a moderate interval performing best.
C. Qualitative Results.
- At a merging roundabout, PlanAgent infers that the vehicle ahead is waiting and changes left, avoiding the middle-of-road standstill seen with PDM-Closed.The figure presents this as an example of PlanAgent’s hierarchical reasoning and comparison with PDM-Closed.Source: C. Qualitative Results · p. 8, Figure 6 · p. 9
- At a red-light crosswalk, PlanAgent infers that traffic is waiting and decelerates for a safe, comfortable stop at the stop line if pedestrians enter.This avoids stopping in the middle of the road as PDM-Closed does.Source: C. Qualitative Results · p. 8
D. Interval of MLLM Calls.
- A 2-second MLLM-call interval gives PlanAgent its best performance; both higher and lower call frequencies hurt performance.At the nuPlan simulation frequency of 10 Hz, ten planning iterations equal one second in the real world.Source: D. Interval of MLLM Calls · p. 8, Table VI · p. 8
E. Performance of Different MLLMs.
Open-source MLLMs used within PlanAgent outperform PDM-Closed on Test14-hard NR-CLS. The paper also identifies prompt sensitivity, fixed-interval calls, and inference time as limitations, and outlines future work on these issues.
E. Performance of Different MLLMs.
- LLaVa-13B outperforms the other tested open-source MLLMs; CogVLM-chat, LLaVa-7B, and LLaVa-13B improve NR-CLS over PDM-Closed by 5.1%, 4.1%, and 6.7%, respectively.The results are on nuPlan Test14-hard; the table also reports average inference time per MLLM call.Source: E. Performance of Different MLLMs · p. 10
A. Limitations.
- PlanAgent is sensitive to prompt quality, fixed-interval MLLM calls add computational burden, and GPT-4V takes about 5 seconds per call, complicating deployment.The authors also report that inaccurate scene understanding or planner-parameter interpretation can produce unreasonable planner code.Source: A. Limitations · p. 10
B. Future Work
- Future work targets fine-tuning with autonomous-driving data, deciding when MLLM involvement is needed, and using closed-loop simulator feedback for human-like alignment.The authors aim for reduced MLLM calls and lower time and computational-resource requirements.Source: B. Future Work · p. 10
VII. CONCLUSION
- The conclusion describes PlanAgent’s BEV and lane-graph inputs, hierarchical reasoning through planner-code generation, and simulation-based Reflection, alongside competitive, generalizable benchmark results and improved token efficiency.The reported benchmarks are nuPlan Val14 and Test14-hard.Source: §VII · p. 10