PaperlayerView exact revision on arXiv

Source-linked AI summary

PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving

Yuan Gao, Sebastian Müller, Mattia Piccinini, Marc Kaufeld, Yuchen Zhang, Finn Rasmus Schäfer, Qunying Song, Johannes Betz

arXiv:2609.08965v1cs.AIcs.CLcs.RO

TL;DR

Autonomous-driving scenario testing is fragmented across generation, retrieval, modification, execution, and analysis, motivating a unified framework. PlannerForge integrates these stages with LLM-based routing, enhancement, and benchmarking, and evaluates the resulting pipeline across models and tasks. Across 80,000 off-the-shelf calls, modules score from 0.88 to 1.00, while end-to-end chaining retains 83% of commercial and 78% of open-model seed queries.

  • Problem

    Scenario-based ADS testing remains fragmented across separate tools for generation, selection, modification, execution, and analysis, while prior LLM work largely focuses on scenario generation.

  • Method

    PlannerForge is a full-lifecycle LLM-agent framework that unifies scenario testing with module routing, planner enhancement, and cross-planner benchmarking.

  • Results

    Across 80,000 off-the-shelf calls, module scores range from 0.88 for Selection to 1.00 for Planner Testing, with end-to-end retention of 83% commercial and 78% open-model seed queries.

  • Takeaways & Limitations

    PlannerForge provides a benchmark-ready, unified platform for scenario-based motion-planner testing without domain-specific fine-tuning.

  • Takeaways & Limitations

    PlannerForge runs open-loop, uses Frenetix for quantitative planner results, and evaluates a modification corpus dominated by Germany.

Abstract

from arXiv · show

Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used to validate Autonomous Driving Systems (ADSs), but it remains a fragmented modular pipeline in which scenario generation, retrieval, modification, ADS execution, and results analysis are performed by separate tools with little interaction. Large Language Model (LLM) agents have shown promise across ADS sub-systems such as perception, planning, and control. However, no prior work covers the whole scenario-based testing pipeline for ADSs with a unified LLM-agent framework. We present PlannerForge, an LLM-agent framework that extends all scenario-based testing stages (from Scenario Generation to ADS Assessment) and adds two further LLM-enhanced stages: ADS Enhancement and ADS Benchmarking. We evaluate PlannerForge with 10 off-the-shelf LLMs across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing, and Enhancement) under 5 prompt conditions. Best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends match commercial APIs on most tasks. Open-source models such as Qwen3.6:35B match commercial APIs on three of the five tasks. Chaining the modules end-to-end retains 83% / 78% of seed queries (commercial / open). It outperforms Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-language generation (193 vs. 144 executable of 200) and realises 92-96% of requested city, road and vehicle attributes. It outperforms BM25 (Robertson and Zaragoza, 2009) at rank 1 selection (92.0% vs. 67.5%) and From-Words-to-Collisions (Gao et al., 2025) on physically valid edits (>=94% vs. 31%). At N=400, cost-tuning lifts planner success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4%, without domain-specific fine-tuning.

1 Introduction

PlannerForge addresses the fragmented, manual nature of motion-planner testing with a unified LLM-powered framework spanning the scenario-based testing lifecycle. It integrates generation, retrieval, modification, execution, analysis, and comparative planner evaluation through a chatbot-driven system.

  • Scenario-based ADS testing remains fragmented across generation, selection, modification, execution, and analysis, with separate tools and manual workflows.
  • The framework generates scenarios from OSM maps, retrieves cases from an open-source database, modifies traffic objects and behaviors, executes planners, and analyzes performance.
  • PlannerForge unifies the full scenario-based testing lifecycle in a single LLM-powered framework.
  • The evaluation covers ten off-the-shelf LLM backends across five core tasks and five prompt conditions without domain-specific fine-tuning.
  • PlannerForge adds a unified multi-planner interface for comparative evaluation on generated and modified scenarios.

2 Related Work

Prior work develops individual scenario-based testing components and increasingly applies LLMs to generation, enhancement, and analysis. PlannerForge targets the resulting gap by unifying the classical testing pipeline and adding LLM-guided enhancement and benchmarking.

  • Classical scenario-based testing uses a six-component taxonomy spanning scenario source, generation, database, selection, execution, and ADS assessment.
  • LLM-powered systems have expanded across scenario generation, traffic simulation, crash reconstruction, traffic rules, and adversarial generation.
  • Recent LLM work also supports ADS enhancement through planning, control, cost tuning, speed decisions, and supplemental reasoning modules.
  • Most reviewed LLM-based ADS testing papers focus on scenario generation rather than the full testing process.
  • PlannerForge addresses this gap by unifying the six classical stages and adding ADS Enhancement and ADS Benchmarking for planner tuning and shared-scenario comparison.

3 Problem Formulation

PlannerForge formalizes scenario-based testing as language-to-structured-output decisions over scenarios, databases, planner configurations, histories, execution outcomes, and analyses. Generation or selection initializes a session, after which a router dispatches user intents to constrained downstream operations.

  • The formalization represents utterances, scenarios, databases, planner configurations, conversation histories, outcomes, and natural-language analyses as distinct spaces.
  • A session begins by generating a scenario from an utterance or selecting one from the scenario database.
  • The Module Router classifies each post-initialization utterance into modification, tuning, testing, analysis, or question answering.
  • The routed operators map scenarios, planner configurations, and test inputs to modified scenarios, tuned configurations, and execution outcomes.
  • Generation, modification, and tuning are constrained generators whose outputs must satisfy scenario or planner-configuration schemas.

4 Methodology

PlannerForge implements the unified pipeline as a chatbot-oriented architecture with six modules, structured prompting, stateful interaction, and router-based post-selection navigation. Its modules cover scenario creation and retrieval, SUMO-based editing, planner testing and tuning, and LLM-mediated result analysis.

  • Framework architecture: The framework contains Generation, Selection, Module Router, Modification, Testing, and Analysis modules, with the router dispatching user intent after scenario selection.
  • Prompting: Structured prompts combine output schemas, syntactic constraints, operators, chain-of-thought steps, and in-context examples for module-specific tasks.
  • Framework setup: A Gradio frontend and LangChain backend maintain conversational memory, visual chat history, session state, and routing across modules.
  • Generation: Generation converts natural-language intents into OSM and SUMO-based CommonRoad scenarios, including road networks, traffic, ego vehicles, and planning problems.
  • Selection: Selection extracts dialogue slots across five metadata steps, progressively prunes database candidates, and falls back to semantic similarity when exact matches fail.
  • Modification: Modification converts CommonRoad scenarios through SUMO files so the LLM can edit trajectories and behaviors before round-tripping to kinematically feasible trajectories.
  • Testing and analysis: Testing provides unified execution and parameter tuning, while Analysis aggregates batch statistics, logs, configurations, and recommendations into natural-language feedback.

5 Results & Discussion

PlannerForge evaluates ten off-the-shelf LLM backends across five scenario-testing tasks and eight modification slices, showing strong module-level performance and meaningful end-to-end retention. Against established tools, it improves natural-language generation, rank-1 selection, physically valid modification, and planner enhancement, while prompt strategies must match task failure modes.

  • Quantitative Evaluation: Best task scores span 0.880 to 1.000, while open-source 20–35B models are fully capable of driving the entire pipeline.Qwen3.6-plus reaches 0.880 on Selection, Gemma4:31b reaches 0.997 on Module Routing, and Gpt-5.4-mini reaches 1.000 on Planner Testing and Enhancement.
  • End-to-End Composition: 83% versus 78% of seed queries survived the chained commercial and open-source pipelines, with Selection and Modification identified as leak points.The chain used actual outputs between Generation, Database, Selection, Modification, Test and Enhancement on N=200 queries.
  • Comparison with Prior Scenario-Testing Tools: 193 versus 144 of 200 scenarios were executable for PlannerForge and Scenario Factory 2.0, respectively.PlannerForge also realises 92–96% of requested city, road and vehicle attributes, but induces 20.0% versus 6.1% planner collisions.
  • Comparison with Prior Scenario-Testing Tools: 92.0% versus 67.5% of queries achieved valid rank-1 selection with PlannerForge and BM25, respectively.The Any@5 gap is smaller, but LLM slot extraction resolves paraphrase, location ambiguity and implicit range constraints that keyword matching cannot.
  • Comparison with Prior Scenario-Testing Tools: At least 94% of PlannerForge’s four modification types were physically valid, compared with roughly 31% for From-Words-to-Collisions.PlannerForge routes every edit through SUMO, whereas From-Words-to-Collisions writes raw coordinates without vehicle dynamics.
  • Comparison with Prior Scenario-Testing Tools: 70.2% planner success versus 50.4% at N=400 followed LLM cost-tuning, while collisions fell from 19.0% to 8.4%.Across batch sizes, success rose by 17.6–21.3 percentage points and collisions fell by 10.5–13.3 points.
  • Transferable Insights: Prompting must match failure modes: CP saturates schema-constrained tasks, ICL handles refusals, and CoT helps joint constraints but can conflict with native reasoning.On Selection, adding CoT to CP hurt reasoning-enabled models while lifting nonthinking Qwen3.6:35b; an executable harness validates both form and downstream execution.

6 Conclusion and Future Work

PlannerForge presents a full-lifecycle LLM-agent framework for motion-planner testing and reports strong module, end-to-end, and benchmark performance. The framework also improves planner outcomes through cost tuning, while future work includes extending it to additional simulators.

  • PlannerForge is a full-lifecycle framework with a Module Router, schema-checked modules, and a unified motion-planner interface.
  • 83% / 78% of seed queries survive end-to-end chaining from Generation through Enhancement for commercial / open models.The evaluation used 80,000 off-the-shelf calls without fine-tuning.
  • 0.88 to 1.00 are the module scores, spanning Selection to Planner Testing, with Generation at 0.957 and the Router at 0.997.Modification reaches near 100% on headline checks.
  • 193 vs. 144 executable scenarios are produced against Scenario Factory 2.0, while rank-1 selection reaches 92.0% vs. 67.5% against BM25.Physically valid modification reaches ≥94% vs. 31% against From-Words-to-Collisions.
  • At N=400, cost-tuning raises planner success from 50.4% to 70.2% and reduces collisions from 19.0% to 8.4%.
  • Future work will extend the framework to simulators such as CARLA.

Limitations

The evaluation exposes losses during simulation round-trips and selection, while the framework remains limited in scope by open-loop testing, planner coverage, simulator format, and a Germany-dominated modification corpus.

  • 13–17% of surviving queries are lost for Trajectory edits and 12–16% for Population edits during simulation round-trip.These losses affect simulation round-trip rather than headline semantics.
  • 6%/11% of commercial/open queries are lost in Selection because of tag over-prediction, with best sat_all at 0.880.
  • PlannerForge runs open-loop, so other agents follow recorded or SUMO-exported trajectories and do not react to the ego vehicle.Closed-loop falsification is left to future work.
  • Quantitative planner, collision, and cost-tuning results use Frenetix, while the MP-RBFN comparison is qualitative.
  • Evaluation uses CommonRoad, and the modification corpus is Germany-dominated, with DEU contributing 80–92 queries per task among 15 country codes.

Ethical Considerations

PlannerForge addresses risks from hallucinated structured outputs through validation before acceptance, but residual risk remains where checking is incomplete, especially in Analysis.

  • Off-the-shelf LLM agents can hallucinate structured outputs, including scenario tags, map or vehicle identifiers, module calls, and unsupported Analysis claims.
  • Schema validation and execution are applied before scored module outputs are accepted to reduce invalid tests and misleading diagnostics.
  • Residual risk remains where checks are incomplete, particularly because the Analysis module is unscored against ground truth.

A.1 Implementation Details

The implementation combines cloud and local LLM backends with a Gradio interface, conversational memory, CommonRoad-SUMO simulation, motion planners, and a semantic scenario database using OSM data sources.

  • Ten LLM backends are grouped by provider and reasoning mode, with cloud models served through APIs and local open-weight models evaluated with configured reasoning settings.Table 6 reports local VRAM for Ollama backends and no local VRAM for cloud models.
  • PlannerForge is exposed as a Gradio web application with chatbot and OSM-based generation tabs that render scenarios as animated GIFs.
  • ConversationSummaryBufferMemory preserves multi-turn context by summarising older messages to fit token limits.
  • CommonRoad scenarios use SUMO traffic dynamics and integrate the Frenetix and MP-RBFN motion planners.
  • The scenario database indexes 500+ curated CommonRoad scenarios in ChromaDB using SentenceTransformer embeddings for semantic retrieval.Scenario metadata covers locations, roadside infrastructure, participants, and ego-vehicle properties.
  • OSM generation fetches road networks through Overpass and geocodes city names through Nominatim, using mirror endpoints and polite request delays.

A.2.1 LLM Scope and Hallucination Isolation

PlannerForge confines LLMs to structured intent parsing and ego selection, while deterministic tools handle map retrieval, conversion, and simulation. This separation isolates geometric and dynamic hallucinations, with schema validation catching remaining LLM-side errors.

  • Hallucination isolation: The generation design prevents LLMs from emitting map geometry, road topology, or vehicle trajectories directly, ruling out three downstream hallucination classes.This structurally excludes invalid lanelet IDs or topologies, kinematically infeasible trajectories, and references to nonexistent map fragments.
  • LLM responsibilities: The LLM parses free-form requests into fixed-schema intents containing location, road classes, traffic density, vehicle mix, and simulation duration.Schema-defined defaults apply when values are omitted or invalid.
  • LLM responsibilities: The LLM selects an ego vehicle using first-car, type-based, or index-based strategies, while the goal region is attached deterministically.The goal is assigned to a chosen lanelet or forward offset along the ego trajectory.
  • Deterministic-tool responsibilities: OSM fetching, CommonRoad–SUMO conversion, and microscopic SUMO simulation execute without LLM involvement.Simulation populates traffic according to the parsed vehicle mix and density while enforcing car-following and lane-changing dynamics.
  • Validation: Schema validation catches misparsed locations, incorrect traffic-density levels, and missing ego specifiers, applying defaults or surfacing errors.Per-key pass rates are reported in Table 7.

A.2.2 Query Corpus

The evaluation uses a 200-query generation corpus and broader multi-task retrieval, modification, routing, and enhancement experiments. Results show schema prompting improves structured generation, strict funnel retrieval preserves precision, and prompting substantially improves routing and cost-configuration outputs.

  • A.2.2 Query Corpus: N = 200 natural-language generation queries are split into CLEAN (n = 158) car-ego requests and ADVERSARIAL (n = 42) non-car-ego requests.The adversarial bucket tests the rule that the planned ego must be a car while preserving the requested non-car type in surrounding traffic.
  • A.2.2 Query Corpus: 55 queries mention traffic density and 21 request explicit duration, testing defaults of low density and 20 s simulation duration.These queries probe defaults-compliance behavior.
  • A.2.2 Query Corpus: Each generation cell aggregates 200 end-to-end runs across geocoding, OSM fetching, CR conversion, SUMO simulation, and Frenetix execution.Reported pipeline metrics include JSON validity, defaults compliance, loadability, traffic, runnability, and safe goal-reaching.
  • A.2.4 Evaluation Results: Defaults compliance rises from ≈0.50 to 1.00 for nine of ten models when the JSON schema and defaults are injected.Additional chain-of-thought or in-context examples provide only marginal gains, with overall scores converging to 0.95–0.96.
  • A.3.3 Evaluation Results: The strict-AND funnel reaches 63.5–88.0% precision, while semantic retrieval returns 0%, hybrid prefilter reaches 5.5–6.0%, and RRF remains below 1.0%.Under recall-oriented sat_any@5, RRF is approximately tied with funnel, but this parity is attributed to the top-five hit-rate metric.
  • A.5.4 Evaluation Results: Prompting raises strict-match routing accuracy, with gpt-5.4-mini increasing from 43.5% to 98.5% and Gemini-3-flash-preview from 62.0% to 99.0%.The largest gain comes from the categorical action schema and ordering rule supplied by contextual prompting.
  • A.6.3 Evaluation Results: Prompting lifts full-YAML matching from 36% at baseline to at least 98% for almost every model, while 81.2% of gpt-5.4-mini outputs produce runnable planner configurations.The remaining failures include collisions caused by modified cost weights, which YAML matching alone does not detect.
Loading 2609.08965v1…