Source-linked AI summary
On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving
Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, Botian Shi, Yu Qiao
TL;DR
Traditional driving approaches have difficulty with open-vocabulary perception, road-user intentions, and rare corner cases, while earlier LLMs could not process visual data. This paper evaluates GPT-4V across driving-scene understanding, reasoning, and driver-like decisions; it finds common-sense understanding and generalization alongside limitations in perception and spatial reasoning.
Problem
Traditional approaches struggle with open-vocabulary perception, road-user intentions, and rare corner cases, while earlier LLM applications lacked visual-data processing.
Source: §1.1 · p. 4Method
The authors qualitatively test GPT-4V from scenario understanding through reasoning to frame-by-frame driving decisions using images, time sequences, navigation information, and vehicle data.
Source: §1.1 · p. 5Results
GPT-4V demonstrates traffic common-sense understanding and out-of-distribution generalization, but has deficiencies in directional and traffic-light recognition, localization, and 3D spatial reasoning.
Source: §5.2 · p. 40Takeaways & Limitations
The authors identify integrating VLM common-sense knowledge with conventional autonomous-driving perception techniques as a future development direction.
Source: §5.2 · p. 40VLMs are not yet suitable replacements for existing autonomous-driving perception methods because of deficiencies in directional recognition, traffic-light identification, 3D spatial reasoning, and localization.
Source: §5.2 · p. 40
Abstract
from arXiv · showhide
The pursuit of autonomous driving technology hinges on the sophisticated integration of perception, decision-making, and control systems. Traditional approaches, both data-driven and rule-based, have been hindered by their inability to grasp the nuance of complex driving environments and the intentions of other road users. This has been a significant bottleneck, particularly in the development of common sense reasoning and nuanced scene understanding necessary for safe and reliable autonomous driving. The advent of Visual Language Models (VLM) represents a novel frontier in realizing fully autonomous vehicle driving. This report provides an exhaustive evaluation of the latest state-of-the-art VLM, GPT-4V(ision), and its application in autonomous driving scenarios. We explore the model's abilities to understand and reason about driving scenes, make decisions, and ultimately act in the capacity of a driver. Our comprehensive tests span from basic scene recognition to complex causal reasoning and real-time decision-making under varying conditions. Our findings reveal that GPT-4V demonstrates superior performance in scene understanding and causal reasoning compared to existing autonomous systems. It showcases the potential to handle out-of-distribution scenarios, recognize intentions, and make informed decisions in real driving contexts. However, challenges remain, particularly in direction discernment, traffic light recognition, vision grounding, and spatial reasoning tasks. These limitations underscore the need for further research and development. Project is now available on GitHub for interested parties to access and utilize: \url{https://github.com/PJLab-ADG/GPT4V-AD-Exploration}
1 Introduction
The paper evaluates whether GPT-4V can address visual understanding and reasoning gaps in autonomous driving, then examines its abilities and limits through qualitative tests. Its curated image-text evaluation spans scenario understanding, reasoning, and driver-like decisions, using diverse sources and a specific model version.
1.1 Motivation and Overview
- Traditional driving approaches struggle with open-vocabulary perception, traffic participants’ intentions, and rare corner cases because they lack common-sense reasoning.The authors frame this as a challenge for nuanced scenario understanding and causal reasoning.Source: §1.1 · p. 4
- GPT-4V’s image-understanding capabilities motivate testing whether it can improve scene understanding and causal reasoning in autonomous driving.Earlier LLM applications were largely limited to decision-making and planning because they could not process visual data.Source: §1.1 · p. 4
- The evaluation progresses from scenario understanding to reasoning and continuous judgment and decision-making as a driver in real-world scenarios.Tests cover recognition, causal reasoning, concatenated time-series images, navigation-image association, and frame-by-frame driving decisions with vehicle information.Source: §1.1 · p. 5
- The authors present qualitative image-text results as initial insights, noting that this approach is less stringent but permits comprehensive analysis.The stated aim is to support future research on GPT-4V in autonomous driving.Source: §1.1 · pp. 4–5
1.2 Guidance
- Test images and videos come from public driving datasets, simulation, and internet sources, and may overlap with GPT-4V training data through April 2023.The article’s text queries were newly generated.Source: §1.2 · p. 5
- Experiments used the web-hosted GPT-4V version from September 25, 2023, and later model updates may produce different responses.All experiments were conducted before November 5, 2023.Source: §1.2 · p. 5
2 Basic Capability of Scenario Understanding
The paper evaluates GPT-4V’s understanding of driving environments and traffic participants across varied visual inputs. Results show strengths in recognizing weather, signs, and road users, alongside notable traffic-light and description errors.
2.1 Understanding of Environment
- Daytime and nighttime scenes reveal temporal and traffic-detail recognition, while four same-intersection images are correctly classified as cloudy, sunny, overcast, or rainy.Nighttime descriptions include a distant vehicle’s tail lights and an inference that it is stationary or moving away.Source: §2.1 · pp. 6–7, Figure 2 · p. 6, Figure 3 · p. 8
- GPT-4V misreads small, distant traffic lights and countdowns, responding correctly only when the signal is zoomed in.It distinguishes yellow street lights from red traffic lights in some nighttime cases, but subsequent tests also show misrecognitions.Source: §2.1 · p. 7, Figure 4 · p. 9, Figure 5 · p. 9
- GPT-4V recognizes most tested traffic signs, including distant height limits, but misidentifies a “Speed Bump” sign in one sample.All three signs in the other sample are correctly recognized.Source: §2.1 · p. 7, Figure 6 · p. 10
2.2 Understanding of Traffic Participants
- Front-view and fisheye tests show recognition of road users and vehicles, but nighttime vehicle descriptions and some scene details can be inaccurate or hallucinated.The model identifies pedestrians and nearby parked vehicles in fisheye views, but invents a nonexistent charging station.Source: §2.2 · pp. 11–12, Figure 7 · p. 11, Figure 8 · p. 13
- In 2D LiDAR point-cloud visualizations, GPT-4V identifies some road and building patterns and estimates vehicle counts, but misinterprets a circular pattern and sometimes miscounts incomplete vehicles.The paper notes that projecting point clouds into 2D loses 3D geometric information.Source: §2.2 · p. 12, Figure 10 · p. 15
- V2X images elicit recognition of freeway layouts, mixed traffic, and signals; CARLA images are identified as simulations, though simulated traffic lights remain difficult.The model also recognizes virtual pedestrians and, in rare cases, notes when they run red lights.Source: §2.2 · p. 12, Figure 11 · p. 16, Figure 12 · p. 17
3 Advanced Capability of Reasoning
The reasoning tests examine GPT-4V on unusual driving situations, relationships across camera views, temporal sequences, and combined camera-map inputs. The examples show useful scene interpretation and decision-making, alongside failures in spatial reasoning and some driving judgments.
3 Advanced Capability of Reasoning
- The reasoning assessment tests responses to unexpected events and dynamic environments, using curated perceptual corner cases that include out-of-distribution objects.The authors note that driving data can only approximate the boundaries of dynamic, continuous, and random environments.Source: §3 · p. 18, §3.1 · p. 18
3.1 Corner Cases
- GPT-4V recognized unusual road scenes and proposed cautious responses, including safe clearance around work areas, waiting for an emergency-landed aircraft to be cleared, and yielding to pedestrians.In another example, it identified a cement mixer and waited for the red light to turn green before proceeding.Source: §3.1 · p. 18
3.2 Multi-view Images
- With correctly ordered surrounding views, GPT-4V inferred overlapping vehicle positions, but other combined-view examples included vehicle-count and shape errors and a false left-turn sign.When asked to sort out-of-order views using the correct front view, it produced all wrong answers and struggled to connect adjacent images.Source: §3.2 · p. 22, Figure 16 · p. 23
3.3 Temporal Sequences
- Across temporal examples, GPT-4V explained a stop for a crossing pedestrian and an SUV yielding to a pedestrian, but also misread a lane-change scene and a green light.It also inferred a front car’s start after the signal turned green despite camera flare caused by rain.Source: §3.3 · p. 26
3.4 Visual-Map Navigation
- With front-view images and map navigation, GPT-4V localized itself and made a correct left turn in one example, but chose an incorrect left turn in a more complex one.It recognized roadside details and correctly obtained speed and distance information from the map in the latter example.Source: §3.4 · p. 31, Figure 23 · p. 31, Figure 24 · p. 32
4 Act as A Driver
Across five real-world driving scenarios, GPT-4V combines traffic-element observations to recommend generally cautious, safety-aware actions, but perception and spatiotemporal reasoning errors still affect decisions.
4.1 Driving in Parking Lot
- At the parking exit, GPT-4V recognizes security infrastructure and advises waiting for checks and pedestrians, but sometimes misidentifies zebra crossings.It also ambiguously assesses pedestrians and distant vehicles, prompting low-speed, cautious decisions.Source: §4.1 · p. 33, Figure 25 · p. 34
4.2 Turning at Traffic Intersection
- At a busy intersection, GPT-4V uses vehicle taillights but repeatedly misreads a distant traffic light before recommending a cautious left turn.The authors note that poor recognition of small, distant objects may affect its intersection decisions.Source: §4.2 · p. 33, Figure 26 · p. 35
4.3 Turning at Highway Ramp
- On a nighttime highway ramp turnaround, GPT-4V follows signs and lane markings and adjusts its driving, but object recognition and positioning remain limited at night.Its sequence includes braking, signaling left, and adjusting speed after entering the highway.Source: §4.3 · pp. 33, 36, Figure 27 · p. 37
4.4 Road Merging
- During nighttime merging, GPT-4V tracks merge progress and suggests lane changes, yet may misjudge road signs and lanes and tends toward conservative actions.It also mistakenly identifies a solid line and a motorcycle in the ego lane.Source: §4.4 · p. 36, Figure 28 · p. 38
4.5 U-Turning at Traffic Intersection
- At a nighttime U-turn, GPT-4V confuses side-road lights with signals for its lane and remains stopped, reflecting difficulty with temporal and spatial context.The authors characterize its overall strategies as conservative.Source: §4.5 · p. 36, Figure 29 · p. 39
4 Act as A Driver
- Across five tests, GPT-4V combines traffic-element states to make reasonable decisions in varied scenarios, with strong rule adherence and safety awareness but limited distant-object and nighttime perception and spatiotemporal inference.The authors describe its driving strategies as relatively conservative.Source: §4.5 · p. 36
5 Conclusions
The evaluation reports potential for GPT-4V to exceed existing autonomous systems in scenario understanding, intention recognition, and driving decisions. However, limitations in perception and spatial reasoning mean VLMs are not yet suitable replacements for existing perception methods.
5.1 Capabilities of GPT-4V in Autonomous Driving
- GPT-4V shows potential to surpass existing autonomous systems in scenario understanding, intention recognition, and driving decision-making.The authors also report handling out-of-distribution cases, interpreting traffic interactions, and guiding decisions using navigation-app interfaces.Source: §5.1 · p. 40
5.2 Limitations of GPT-4V in Autonomous Driving
- GPT-4V sometimes confuses left and right, including at complex junctions and during lane changes.The paper identifies direction discernment as a critical navigation task.Source: §5.2 · p. 40
- Traffic-light recognition fails in some scenes, while cropping the light region separately enables successful recognition.The authors suspect full-image semantic information may weaken the traffic-light embedding.Source: §5.2 · p. 40
- GPT-4V can indicate approximate image regions but struggles to specify pixel-level coordinates or bounding boxes.Source: §5.2 · p. 40
- It struggles with precise spatial judgments in multiview stitching and relative-position estimation, possibly because 3D space must be interpreted from 2D images.Source: §5.2 · p. 40
- The model is less reliable at interpreting non-English signs and counting overlapping traffic participants in congestion.Source: §5.2 · p. 40
- Given deficiencies in direction recognition, traffic-light identification, 3D reasoning, and entity localization, the authors say VLMs are not yet suitable replacements for existing perception methods.They identify integrating VLM common sense with conventional perception and ensuring output safety and reliability as ongoing challenges.Source: §5.2 · p. 40