Source-linked AI summary
Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, Yu Qiao
TL;DR
The paper examines why conventional autonomous-driving systems struggle with long-tail corner cases and asks whether an LLM can understand driving scenes more like a human. It proposes a human-learning-inspired framework centered on reasoning, interpretation, memorization, and expert interaction, and demonstrates GPT-3.5 in a closed-loop driving setting. Qualitative experiments report comprehension and long-tail problem-solving abilities that provide insights for human-like autonomous driving.
Problem
Conventional optimization-based and modular autonomous-driving systems have limitations on long-tail corner cases because they overlook infrequent patterns and lack common-sense experience.
Method
The paper proposes a human-learning-inspired driving framework with reasoning, interpretation, memorization, environment interaction, memory, and expert feedback, then implements a GPT-3.5 closed-loop system.
Results
Qualitative experiments report that the LLM demonstrates comprehension and the ability to solve long-tailed driving cases.
Takeaways & Limitations
The findings provide insights for developing autonomous-driving systems that drive more like humans by using common sense and accumulated experience.
Abstract
from arXiv · showhide
In this paper, we explore the potential of using a large language model (LLM) to understand the driving environment in a human-like manner and analyze its ability to reason, interpret, and memorize when facing complex scenarios. We argue that traditional optimization-based and modular autonomous driving (AD) systems face inherent performance limitations when dealing with long-tail corner cases. To address this problem, we propose that an ideal AD system should drive like a human, accumulating experience through continuous driving and using common sense to solve problems. To achieve this goal, we identify three key abilities necessary for an AD system: reasoning, interpretation, and memorization. We demonstrate the feasibility of employing an LLM in driving scenarios by building a closed-loop system to showcase its comprehension and environment-interaction abilities. Our extensive experiments show that the LLM exhibits the impressive ability to reason and solve long-tailed cases, providing valuable insights for the development of human-like autonomous driving. The related code are available at https://github.com/PJLab-ADG/DriveLikeAHuman .
1 Introduction
The paper argues that optimization-based and modular autonomous-driving systems struggle with long-tail cases because they focus on dominant patterns and lack common-sense experience. It proposes human-like driving built around reasoning, interpretation, memorization, and interaction with expert feedback, then explores GPT-3.5 in a closed-loop environment.
- Motivation: Optimization-based autonomous-driving systems can become trapped in local optima and overlook infrequent long-tail cases.Adding more data may reduce the gap to current performance but does not address the capacity limit of optimization-based methods described by the paper.
- Motivation: Humans handle inexhaustible long-tail situations through accumulated experience and common sense, motivating systems that learn through continuous driving.The paper contrasts this with fitting a limited training corpus.
- Human-like driving abilities: The paper identifies reasoning, interpretation, and memorization as the three abilities required for autonomous driving that resembles human driving.Reasoning uses common sense and experience; interpretation supports introspection and declarative memory; memorization retains prior experiences for similar situations.
- Human-like driving system: Its canonical system contains an environment, agent, memory, and expert, connected through interaction, reflection, and supervision flows.The agent perceives and decides using memory, while expert feedback supports training and correction.
- LLM exploration: The study builds a GPT-3.5 closed-loop driving system and uses qualitative experiments to examine comprehension, environment interaction, reasoning, and memorization.The system is evaluated in driving scenarios, including long-tailed cases that are difficult for modular systems but easy for human drivers.
- Results: The experiments report impressive comprehension and an ability to solve long-tailed cases, offering insights for human-like autonomous-driving development.The paper presents this as an initial feasibility demonstration rather than a complete autonomous-driving solution.
2 Closed-loop interaction ability in driving scenarios
The closed-loop system equips GPT-3.5 with perception tools and prompts to reason over driving actions, interact with HighwayEnv, and maintain decision consistency. Its behavior is contrasted with RL- and search-based approaches through safety-aware lane changes, common-sense evaluation, and zero-shot performance.
- Closed-loop system: GPT-3.5 uses perception tools, agent prompts, and ReAct cycles of thought, action, and observation to control vehicles in HighwayEnv.The prompts provide current actions, driving rules, and cautions, enabling a text-only model to interact with the environment.
- Human-like decision-making: GPT-3.5 evaluates action consequences and uses common sense to make more human-like decisions than RL- and search-based approaches.The paper contrasts this with RL agents’ reward-driven unconventional behavior and search agents’ objective-driven aggressive or meaningless lane changes.
- Performance comparison: Over 60% zero-shot pass rate in HighwayEnv was achieved by GPT-3.5 without fine-tuning, while the RL-based approach required numerous iterations for competitive performance.The RL agent also learned an unconventional collision-avoidance policy that slows initially before accelerating.
- Scenario reasoning: GPT-3.5 checks the safety of available actions and selects a lane change after reasoning that it is safe and provides greater flexibility.Acceleration may conflict with the front vehicle, whereas changing to lane_2 is safe with veh1; the model explains its choice using the vehicles’ relative speeds.
- Decision consistency: Including the previous decision and explanation in later prompts supports consistent acceleration decisions and reduces tool invocations and reasoning cost.In the example, GPT-3.5 continues accelerating because that action remains safe and consistent with its prior decision, shortening the gap to the leading vehicle.
3 Reasoning ability with common sense
The paper evaluates whether LLMs can use common sense to distinguish harmless cargo from genuinely hazardous traffic-cone situations. GPT-3.5 reached opposite, situation-appropriate recommendations for two visually similar cases.
- Evaluation setup: LLMs are evaluated on long-tail driving cases involving pickup trucks carrying traffic cones.The setup compares two similar photographs, one with cones contained in the truck bed and one with additional cones scattered on the ground.
- Case comparisons: In the first case, GPT-3.5 inferred that the cones were cargo being transported and judged the scenario non-hazardous.It advised that slowing down was unnecessary and could potentially endanger traffic flow.
- Case comparisons: In the second case, GPT-3.5 treated scattered cones around the truck as potentially dangerous and advised deceleration and increased distance.The recommendation was intended to avoid collisions with the cones.
- Implication: The examples are presented as evidence of zero-shot understanding and reasoning in driving scenarios using common-sense knowledge.The paper states that this knowledge supports more rational decisions compatible with human driving behavior.
4 Performance enhancement through memorization ability
The paper uses expert feedback and self-reflection to build a memory mechanism for accumulating driving experience. In an initially overcautious narrow-lane case, the human driver’s different decision supplies experience that can improve later similar decisions.
- Motivation: Continuous learning is presented as a defining part of how human drivers develop skills through repeated exposure to new traffic situations.The paper contrasts this accumulation of experience with optimization-based approaches that seek to imitate continuous learning by acquiring more data.
- Initial decision: GPT-3.5 advised the ego car to stop and wait because two vehicles moving toward each other on the same narrow lane risked collision.The advice was described as the safest action based on the available information.
- Expert comparison: The human driver instead continued without slowing and nudged slightly left, allowing both vehicles to pass without collision.This decision provides an expert outcome that differs from the model’s initial cautious recommendation.
- Memory mechanism: The memory module stores decision scenarios that deviate from expert evaluations or real-world human-driver decisions.After feedback, the LLM self-reflects on why its decision differed from the expert’s and subsequently uses the stored experience.
- Memory mechanism: Retrieving accumulated cases is intended to support continuous learning, reduce decision cost in similar scenarios, and improve practical performance.Figure 7 is presented as an example of the LLM’s self-reflection and memorization process.
5 Related work
The related work situates autonomous driving across modular and end-to-end paradigms and reviews LLM progress in zero-shot reasoning, embodied agents, and transportation applications.
- Self-driving autonomy: Modular autonomous driving uses interconnected perception, planning, and control components, offering modularity and versatility but creating tuning and error-propagation challenges.The paper contrasts this architecture with end-to-end autonomy, which directly maps inputs to outputs.
- Large language models: Recent LLM research reports progress in zero-shot prompting, complex reasoning, embodied-agent research, and transportation problems.The paper also notes efforts to adapt pretrained LLMs for multimodal prompts through fine-tuning.
6 Conclusion
The paper proposes human-like autonomous driving centered on reasoning, interpretation, and memorization to address long-tail corner cases. GPT-3.5 provides preliminary evidence of traffic-scene understanding, while the work frames closed-loop LLM driving as an opportunity for further research rather than a finished driving agent.
- Conclusion: The proposed human-like driving paradigm is built around three abilities: reasoning, interpretation, and memorization.These abilities are presented as responses to imperfections in previous optimization-based autonomous driving systems.
- Conclusion: GPT-3.5 showed an impressive ability to understand traffic scenarios in the paper’s preliminary experiments.The authors describe the work as only scratching the surface of LLM potential in closed-loop driving.
- Conclusion: The paper emphasizes adopting LLM technology to highlight benefits and opportunities rather than using LLMs as the driving agent.Its stated aspiration is to motivate AGI-based autonomous driving systems that drive like humans.