LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

1. Introduction

• Modular and end-to-end driving systems struggle to handle arbitrary user commands such as overtaking a car ahead.The paper also identifies a lack of established LLM integration paradigms and driving benchmarks.Source: §1 · p. 1
• LaMPilot uses language model programs as the action space, with APIs that connect natural-language instructions to executable driving plans through code generation.This approach is inspired by Code as Policy.Source: §1 · p. 1
• LaMPilot-Bench is introduced as the first benchmark for evaluating LLM programs in autonomous driving, with simulated tasks and programmatic policy scoring.The benchmark tasks are intended to be completed safely and efficiently.Source: §1 · p. 1
• Experiments on LaMPilot-Bench demonstrate the potential of off-the-shelf LLMs to handle diverse driving scenarios and follow user instructions.The experiments assess off-the-shelf LLM performance using the LaMPilot framework.Source: §1 · pp. 1–2

2.1. Language-Guided Driving

• Prior LLM applications in autonomous driving address generalization, interpretability, and human-vehicle interaction; examples include DiLu and LMDrive.LMDrive processes camera-LiDAR data and natural-language instructions to generate vehicle control signals.Source: §2.1 · p. 2
• Prior language-guided driving approaches use high-level goals, future trajectories, or detailed vehicle controls, whereas LaMPilot generates policy code executed by classical planners or controllers.The authors describe this combination as using the strengths of LLMs and traditional autonomous-driving components.Source: §2.1 · p. 2

2.2. Large Language Models for Code Generation

• LLMs have generated code for robotics, object manipulation, visual processing, information extraction, and logical planning, alongside broader code-generation uses.Examples include Code as Policy, ProgPrompt [41], VoxPoser, and ViperGPT.Source: §2.2 · p. 2

3. LaMPilot Methodology

• LaMPilot uses language model programs as a temporally generalizable, hierarchically structured action space, with complex maneuvers decomposable into conditional sequences of actions.It is designed as an add-on to existing systems: LLM code snippets guide strategic navigation while classical algorithms handle driving execution.Source: §3 · p. 2

3.1. Prompts

• The LLM receives API documentation, human instructions, and driving context, then uses Chain-of-Thought reasoning to produce policy code.The prompt process is formulated as [R, P] = ℓ([A, I, C]), where ℓ denotes the LLM backbone.Source: §3.1 · p. 2

3.2. Functional Primitives

• LaMPilot policies call functional primitives implemented with classical planning and control algorithms, while LLMs are reserved for strategic planning rather than low-level control.The framework overview traces prompts to generated policies, simulator execution, and evaluator assessment.Source: §3.2 · pp. 2–3, Figure 1 · p. 3

3.3. Completions

• The generated code can use if-else statements and loops to form dynamic feedback policies, including a right-lane overtaking example that waits for safe lane changes and monitors distance until passing.The example takes API documentation, an instruction, and driving context as inputs.Source: §3.3 · pp. 3–4
• Yield statements let the autonomous-driving system execute low-level actions for the LLM’s high-level policy, combining strategic decisions with real-time classical algorithms.The paper presents this combination as supporting flexible instruction following.Source: §3.3 · p. 4

4. LaMPilot Benchmark

• LaMPilot-Bench combines a simulator, dataset, and evaluator to assess LLM-based agents’ instruction-following capabilities in autonomous driving.The benchmark’s stated objective is for agents to accomplish assigned tasks safely and efficiently.Source: §4 · p. 4

4.1. Simulator

• The simulator extends HighwayEnv with interfaces for LLM-based agents and custom intersections to diversify driving scenarios.HighwayEnv provides driving models and simulated multi-vehicle interactions.Source: §4.1 · p. 4

4.2. Dataset

• The dataset contains 4,900 semi-human-annotated traffic scenes, with 500 samples reserved as the test set.Samples include a natural-language instruction, an initial simulator state, and goal-state criteria aligned with the instruction.Source: §4.2 · p. 4

4.3. Evaluator

• The evaluator scores safety using time-to-collision and speed variance, with TTC scores based on a 2-second safety margin.TTC values above 2 seconds receive a score of 100; the minimum positive TTC across vehicles and task time represents the nearest potential collision.Source: §4.3 · p. 5
• Task completion requires meeting the instructed objective without collisions and within the time limit; the overall score weights TTC, SV, and TE.The time-efficiency score evaluates completion within a predefined limit, and higher TE scores indicate greater efficiency.Source: §4.3 · pp. 5–6

5.1. Experiment Setup

• Experiments benchmark five named proprietary and open-source LLMs; scenarios have a 60-second limit, and incomplete cases receive a driving score of zero.The score combines successful-case scores with success-rate and collision-rate terms; Table 3 compares baselines under different learning settings.Source: §5.1 · p. 6, §5 · p. 6, Table 3 · p. 7

5.2.1 Heuristic Baselines

• The baselines include IDM, MOBIL, human drivers, and zero- and few-shot chain-of-thought prompting, with three human-written code exemplars in the few-shot setting.The exemplars are created by a Python-proficient programmer using API documentation and non-test scenarios.Source: §5.2.1 · p. 6, §5.2.2 · pp. 6–7

5.2.3 Human Feedback Baselines

• The human-feedback baseline lets LLMs revise generated policies using context-specific feedback and retrieve approved code snippets without fine-tuning.The repository uses function descriptions as retrieval keys, and each method's feedback process uses 100 random non-test samples.Source: §5.2.3 · p. 7

5.3. Quantitative Results

• IDM and MOBIL have zero collision rates, but their instruction-independent policies achieve success rates of 20.4% and 15.3%, respectively.They provide reference points for assessing whether LLM-based agents follow human instructions.Source: §5.2.1 · p. 6, §5.3 · pp. 7–8
• Zero-shot GPT models and PaLM 2 outperform rule-based methods in task completion, but incur collision rates of 1%-4%.The authors attribute the task-completion advantage to understanding natural-language instructions and generating code; they note that the policies do not fully capture driving-task complexities.Source: §5.3 · p. 8
• All evaluated LLMs improve completion rates with three-shot code examples; Llama 2 rises from 20.4% zero-shot to 63.3% three-shot.The same three programmer-written code examples and API documentation are used across test cases.Source: §5.2.2 · pp. 6–7, §5.3 · p. 8
• GPT-4 achieves the highest driving score, 64.0, when combined with human-in-the-loop learning.The authors attribute the improvement to iterative refinement of generated policies with human knowledge.Source: §5.3 · p. 8

5.4. Discussion

• The framework adds LLM-side safety reasoning and actuator-API hard constraints, while its textual input/output limits visual understanding and instruction tuning may risk generalization when training data contain spurious correlations.The authors present multimodal perception and generalizable instruction-tuning solutions as future directions.Source: §5.4 · p. 8

• LaMPilot integrates LLMs into autonomous driving systems, and LaMPilot-Bench evaluates instruction-following by LLM-based agents.Source: §6 · p. 8
• Experiments show that off-the-shelf LLMs can generate driving-task code policies based on human instructions.Source: §6 · p. 8
• The observed collision rate indicates a need for further research on the complexities and safety requirements of real-world driving.Source: §6 · p. 8