PaperlayerView exact revision on arXiv

Source-linked AI summary

A Language Agent for Autonomous Driving

Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, Yue Wang

arXiv:2311.10813v4cs.CVcs.AIcs.CLcs.RO

TL;DR

Autonomous-driving pipelines do not fully capture human reasoning and experiential knowledge, partly because language-based priors do not align naturally with numerical neural representations. Agent-Driver uses an LLM as a cognitive agent coordinating tools, memory, and staged reasoning, and reports large benchmark gains, interpretability, and few-shot learning; real-time feasibility remains uncertain.

  • Problem

    Conventional perception-prediction-planning systems do not fully model human decision-making, while language-based human knowledge is difficult to integrate with numerical neural representations.

  • Method

    Agent-Driver uses an LLM as an interactive scheduler with a dynamic tool library, cognitive memory, and a reasoning engine for chain-of-thought reasoning, planning, and self-reflection.

  • Results

    Agent-Driver significantly outperforms state-of-the-art systems, including over 30% collision improvements in motion planning, while demonstrating interpretability and few-shot learning on nuScenes.

  • Takeaways & Limitations

    The findings support LLMs as agents for integrating human-like intelligence into autonomous-driving systems.

  • Takeaways & Limitations

    Accurate inference time could not be obtained, so whether Agent-Driver meets real-time demands for commercial driving remains uncertain.

Abstract

from arXiv · show

Human-level driving is an ultimate goal of autonomous driving. Conventional approaches formulate autonomous driving as a perception-prediction-planning framework, yet their systems do not capitalize on the inherent reasoning ability and experiential knowledge of humans. In this paper, we propose a fundamental paradigm shift from current pipelines, exploiting Large Language Models (LLMs) as a cognitive agent to integrate human-like intelligence into autonomous driving systems. Our approach, termed Agent-Driver, transforms the traditional autonomous driving pipeline by introducing a versatile tool library accessible via function calls, a cognitive memory of common sense and experiential knowledge for decision-making, and a reasoning engine capable of chain-of-thought reasoning, task planning, motion planning, and self-reflection. Powered by LLMs, our Agent-Driver is endowed with intuitive common sense and robust reasoning capabilities, thus enabling a more nuanced, human-like approach to autonomous driving. We evaluate our approach on the large-scale nuScenes benchmark, and extensive experiments substantiate that our Agent-Driver significantly outperforms the state-of-the-art driving methods by a large margin. Our approach also demonstrates superior interpretability and few-shot learning ability to these methods.

1 Introduction

Conventional autonomous driving decomposes human driving into perception, prediction, and planning, but this simplifies human decision-making and poorly integrates language-based knowledge with neural systems. Agent-Driver uses an LLM-centered architecture with tools, memory, and reasoning to pursue more human-like driving.

  • Motivation: The perception-prediction-planning framework represents perception, foresight, and decision-making through object detection or occupancy, future-motion prediction, and collision-free trajectory planning.These stages may use hand-crafted rules or learned data-driven methods.
  • Motivation: Conventional pipelines overly simplify human decision-making and cannot fully model driving complexity.
  • Motivation: Language-based human knowledge and reasoning are difficult to incorporate because conventional systems process numerical sensory signals, boxes, and trajectories.
  • Agent-Driver: Agent-Driver introduces LLMs as an interactive scheduler and uses natural language as a unified interface between human knowledge and neural modules.
  • Agent-Driver: The system combines dynamic function-call tools, configurable cognitive memory, and a reasoning engine for chain-of-thought reasoning, task planning, motion planning, and self-reflection.
  • Reported contributions: Over 30% collision improvements in motion planning, strong few-shot learning, and interpretability are reported on the nuScenes benchmark.

2 Related Works

Related work models driving through conventional modular pipelines and has begun applying LLMs to autonomous-driving planning and system control. Agent-Driver extends this direction by using an LLM to schedule the whole driving system.

  • Conventional systems: Modern systems use perception-prediction-planning modules to recognize scenes, estimate future object motion, and plan driving trajectories from sensory inputs.
  • LLMs in driving: LLMs offer commonsense reasoning and language understanding, but their use for autonomous driving remains an open challenge.
  • LLMs in driving: GPT-Driver reformulates motion planning as language modeling, while DriveGPT4 proposes an end-to-end LLM-based driving approach.
  • LLMs in driving: Using an LLM as an agent to schedule the whole driving system has produced strong performance on a real-world driving benchmark.

3 Agent-Driver

Agent-Driver connects neural driving modules with human knowledge through text, then uses retrieved memory and staged reasoning to generate and safety-check trajectories. Its architecture includes a tool library, cognitive memory, and reasoning engine.

  • 3 Agent-Driver: Agent-Driver uses text representations as a unified interface connecting neural networks with human knowledge.
  • 3.2 Tool Library: The tool library dynamically extracts decision-relevant information from neural outputs through functions for detection, prediction, occupancy, and mapping.The library contains more than 20 functions, including textual object descriptions, predicted trajectories, occupancy probabilities, and lane information.
  • 3.3 Cognitive Memory: The cognitive memory stores text-based common sense and driving experiences, including traffic regulations, risky behaviors, prior scenarios, and decisions.
  • 3.3 Cognitive Memory: A two-stage search retrieves similar driving experiences using embedding-based K-nearest-neighbor search followed by LLM-based ranking.
  • 3.3 Cognitive Memory: Retrieved experiences, commonsense knowledge, and environmental information jointly form the reasoning engine’s inputs through the shared text interface.
  • 3.4 Reasoning Engine: The reasoning engine models driving as step-by-step reasoning, hierarchical planning, and self-reflection.
  • 3.4 Reasoning Engine: Chain-of-thought reasoning identifies key objects and their effects, while task planning converts context into a high-level behavior-and-velocity plan guiding motion planning.
  • 3.4 Reasoning Engine: Self-reflection invokes collision checking and trajectory optimization, improving the safety of the planned trajectory.

4 Experiments

Experiments evaluate Agent-Driver across open-loop and closed-loop planning, few-shot learning, interpretability, LLM compatibility, stability, and instruction strategies. The system achieves strong benchmark performance and shows interpretable execution, few-shot capability, compatibility across LLMs, and stable outputs under limited instruction data.

  • Open-Loop Results: Agent-Driver exceeds prior open-loop methods on L2 and collision rate, including 0.74 L2 error and 0.21% collision rate under UniAD metrics.These values are respectively 11.9% and 32.3% better than the second-best methods GPT-Driver and UniAD.
  • Closed-Loop Results: Agent-Driver achieves the highest route completion in Town05-Short, surpassing VAD by 4.1%, while its 57.33% driving score is on par with prior methods.The closed-loop evaluation uses the CARLA simulator.
  • Few-Shot Learning: With 1% data, Agent-Driver’s motion planner exceeds the state-of-the-art motion planner trained on full data, demonstrating few-shot learning ability.The experiment fine-tunes the motion-planning LLM with varying fractions of training data while keeping other components unchanged.
  • Interpretability: Agent-Driver records tool-library, cognitive-memory, and reasoning-engine messages, making its driving decision process transparent and interpretable.Figure 7 also distinguishes detected, meaningful, and notable objects and compares planned trajectories with human driving trajectories.
  • Compatibility: Agent-Driver maintains satisfactory motion-planning performance across Llama-2-7B and two gpt-3.5-turbo models, indicating compatibility with diverse LLM architectures.The tested foundation models are Llama-2-7B, gpt-3.5-turbo-1106, and gpt-3.5-turbo-0613.
  • Stability: After exposure to 1% of training data, Agent-Driver produces zero invalid waypoint outputs across 6,019 validation scenarios.Invalid outputs are defined as waypoint outputs containing non-numerical values.
  • In-Context Learning vs. Fine-Tuning: In-context learning performs slightly better for reasoning and task planning, whereas fine-tuning significantly outperforms it for motion planning.The comparison evaluates both instruction strategies across the three modules using downstream motion-planning performance.

5 Conclusion

The paper concludes that Agent-Driver uses LLMs to schedule autonomous-driving modules through a tool library, cognitive memory, and reasoning engine. Experiments support its effectiveness, few-shot learning ability, and interpretability, while real-time inference remains future work.

  • 5 Conclusion: Agent-Driver uses LLMs as agents to schedule autonomous-driving modules and adds a tool library, cognitive memory, and reasoning engine.These components are intended to bring human-like intelligence into driving systems.
  • 5 Conclusion: Experiments substantiate Agent-Driver’s effectiveness, few-shot learning ability, and interpretability on a real-world driving dataset.The conclusion presents these findings as evidence for the potential of LLM agents in human-level intelligent driving systems.
  • 5 Conclusion: Optimizing the LLMs for real-time inference is identified as future work.

Ethics Statement

The ethics statement argues that Agent-Driver’s use of language generation targets autonomous-driving representations and decision-making rather than text generation. It also states that the driving data excludes personal and location information.

  • Ethics Statement: Agent-Driver’s language generation is used for autonomous-driving reasoning and planning, not for text generation itself.The statement therefore characterizes its potential negative impact on misinformation or deepfakes as minimal.
  • Ethics Statement: The driving data used by Agent-Driver does not include personal or location information, according to the ethics statement.The authors conclude that they believe Agent-Driver does not lead to ethics concerns.

Reproduciblity Statement

The paper and supplementary material provide implementation details, function definitions, tool-use examples, and code intended to support faithful reproduction and extension of Agent-Driver.

  • Reproducibility Statement: The authors provide implementation details in the paper and appendix, alongside code covering the LLM, neural modules, tool library, cognitive memory, and reasoning engine.This material is intended to let others reproduce results and build on the approach.
  • Reproducibility Statement: The paper states that all tool-library function definitions are included in Tables 6 and 7.
  • Reproducibility Statement: The tool library functions cover detection, prediction, occupancy, and mapping, enabling flexible environmental information collection.
  • Reproducibility Statement: Figures 4 and 5 provide a detailed example of LLM tool use, with prompts, responses, and collected data shown in distinct colors.System prompts guide dynamic function invocation to collect necessary environmental information.
  • Reproducibility Statement: The appendix describes the data format and retrieval process for commonsense and experience memory.

B.1 Memory Data

Agent-Driver retrieves relevant driving experience through vector similarity followed by LLM-based fuzzy search, while its reasoning engine combines environmental information with memory data to guide decisions. The memory stores configurable commonsense knowledge and editable scenario–trajectory experiences.

  • Memory Contents: The commonsense memory stores essential safe-driving knowledge in a configurable text-based format.It can encode traffic regulations and risky-behavior knowledge, and users can customize it for different driving conditions.
  • Memory Contents: The experience memory caches environmental information and corresponding driving trajectories from training scenarios.Expert human-driver demonstrations can be inserted online to benefit subsequent decision-making.
  • Memory Search: A first search stage vectorizes past scenarios using ego-states, mission goals, and historical trajectories, then computes weighted similarity scores.The highest-scoring candidates are passed to a second-stage search.
  • Memory Search: An LLM-based fuzzy search selects the most similar past scenario from the top-K candidates using their text descriptions.The corresponding driving trajectory is retrieved as a reference for the current decision.
  • Decision Support: Retrieved experience and driving decisions can support the current decision-making process.The reasoning engine takes environmental information and memory data as inputs, alongside chain-of-thought reasoning, planning, and self-reflection.

C.2 Task Planning

Agent-Driver converts reasoning and environmental context into configurable high-level driving plans, then generates and safety-checks executable trajectories. Its ablations indicate that the tool library, memories, reasoning, task planning, and self-reflection each contribute to collision-rate performance, while the system remains effective without ego-states and across replaceable neural modules.

  • C.2 Task Planning: Task planning generates high-level driving plans from reasoning results, environmental information, and memory data.The plan combines discrete driving behaviors with speed estimations.
  • C.2 Task Planning: Agent-Driver defines 6 driving behaviors and 6 speed estimations, producing 31 configurable driving plans.The combinations cover most driving scenarios, while additional behavior and speed types can be added for long-tailed cases.
  • C.2 Task Planning: In-context learning is more appropriate than fine-tuning for instructing the LLM in task planning.The same comparison reports that in-context learning produces more diverse reasoning outputs and better motion-planning performance.
  • Motion Planning: Motion planning generates a safe and comfortable trajectory from the driving plan, reasoning results, environmental information, and memory data.The LLM outputs text-based waypoint coordinates that are transformed into numerical trajectories for execution.
  • C.4 Self-Reflection: Self-reflection checks planned trajectories against occupancy maps and optimizes colliding trajectories toward collision-free alternatives.The optimization balances similarity to the original trajectory with separation from nearby obstacle points.
  • D.3 Ablation Study: Removing the tool library increases input tokens by approximately 2 times and worsens collision rate, while removing memory, reasoning, task planning, or self-reflection also worsens collision rate.The ablation study used 10% training data for instructing the LLMs.
  • D.4 Impact of Ego-States on Open-Loop Motion Planning: Without ego-states, Agent-Driver remains on-par with UniAD, while the ego-only baseline performs worse than Agent-Driver.These settings were trained with 10% data.
  • Module Flexibility: Agent-Driver maintains favorable performance with variable neural-module combinations, supporting plug-and-play replacement without retraining upon every module change.The authors attribute this flexibility to the system design.

D.6 Language Justification on the BDD-X Dataset

Agent-Driver extends its tool library to language justification, generating explanations for driving actions from visual inputs. Results on BDD-X, including qualitative examples, demonstrate this capability.

  • Agent-Driver provides language justification for actions based on visual inputs using a vision-language model as a tool.The paper evaluates this capability on the BDD-X language benchmark.
  • The BDD-X evaluation reports quantitative and qualitative evidence for Agent-Driver’s language justification ability.The quantitative results appear in Table 5, while qualitative examples appear in Figure 1.
  • The examples justify forward driving because the path is clear and justify slower driving because traffic ahead is moving slowly.
  • Other examples explain slowing because a vehicle ahead stopped, continuing because no other cars are present, and stopping because a traffic light turned red.

E.1 Qualitative Ablation

Qualitative ablations indicate that memory and reasoning-related components affect planned trajectories, while visualizations show progressive object identification and safe trajectory planning. Failure cases highlight the importance of accurate heading prediction for large objects.

  • Qualitative Ablation: Removing the memory module produces a planned trajectory with a larger discrepancy from the ground-truth trajectory.
  • Qualitative Ablation: Disabling chain-of-thought reasoning and task-level planning causes greater deviation from the ground-truth trajectory.The paper describes reasoning as a critical component for accurate planning.
  • Interpretability: Agent-Driver progressively identifies critical objects through tool use and reasoning before planning a safe driving trajectory.Figures 13–15 visualize the identified objects and planned trajectories.
  • Failure Cases: Heading errors in large objects such as buses critically affect motion planning, underscoring accurate heading prediction in detection networks.

F Limitations

The paper reports that accurate inference time could not be obtained because of OpenAI API limitations, leaving real-time suitability for commercial driving applications uncertain. It points to faster LLM inference and evolving onboard hardware as possible paths forward.

  • Accurate Agent-Driver inference time could not be obtained because of OpenAI API limitations.
  • Whether Agent-Driver meets the real-time demands of commercial driving applications remains uncertain.
  • Recent advances in accelerating LLM inference and rapidly evolving onboard computational power are described as promising directions for resolving the inference restriction.
Loading 2311.10813v4…