PaperlayerView exact revision on arXiv

Source-linked AI summary

DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?

Aditya Ramabadran, Simon Mahns, Tobias Gessler

arXiv:2609.38948v1cs.ROcs.AI

TL;DR

Frontier models have advanced on digital tasks, but whether general-purpose vision-language models can drive a real car remains unmeasured. DrivingBench evaluates them as direct, closed-loop drivers of a Toyota Corolla, with commands executed while models think. GPT-6 Astra alone completed the course, on its second attempt; the evaluation used one independent trial per model, with up to three dependent attempts.

  • Problem

    Whether unmodified, general-purpose vision-language models can drive a real car remains an unmeasured physical-world ability, despite progress on digital tasks.

    Source: §1 · p. 1

  • Method

    DrivingBench connects agent applications to a real car so models observe camera images and directly command steering and speed in closed loop while the car may continue moving.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    GPT-6 Astra alone completed the course, on its second attempt; Claude Fable 5.1 reached 45%, and GPT-5.6 Sol and Grok 4.6 failed to pass the first corner.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark shows that a general-purpose model can complete a real-world driving task end-to-end at low speed under supervision, while evaluated models remain far from human-level.

    Source: §7 · p. 12, §8 · p. 12

    Each model was evaluated in one independent trial, with up to three dependent in-context-learning attempts.

    Source: §7 · p. 12

Abstract

from arXiv · show

Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present DrivingBench, to our knowledge the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and velocity around a parking lot cone course at low speeds. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We also detail the design principles behind our action interface, and show how the tool output format and the framing of the task combined to determine whether models would drive at all or refuse. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.

1 INTRODUCTION

DrivingBench asks whether general-purpose vision-language models can drive a real car, testing a physical skill that remains difficult despite rapid progress on digital tasks. It evaluates models as direct, closed-loop drivers, with the car continuing to move while they think; GPT-6 Astra alone completed the course.

1 INTRODUCTION

  • DrivingBench tests whether an unmodified, general-purpose vision-language model can drive a real car, a physical-world ability not yet measured by the benchmark question posed here.The motivation contrasts rapid progress on digital benchmarks with the difficulty of physical perception and action.Source: §1 · p. 1
  • The model directly observes camera images and chooses steering and speed in a closed loop, with no intermediate driving planner beyond openpilot’s low-level actuation.Unlike robotic pick-and-place tasks, driving mistakes cannot simply be undone or retried in the same attempt.Source: §1 · pp. 1–2
  • Because the car continues executing an active command while the model thinks, inference latency is part of the driving task.Each new command replaces the currently running command.Source: §1 · p. 2
  • GPT-6 Astra completed the course on its second attempt; Claude Fable 5.1 reached 45%, while GPT-5.6 Sol and Grok 4.6 failed to pass the first corner.Two of the four models improved across attempts, while the other two diagnosed mistakes without materially changing their driving.Source: §1 · p. 2
  • DrivingBench provides an open-source evaluation harness and releases traces, commands, transcripts, and appropriately blurred videos for reproducibility.The paper also examines how task framing and tool-output format affected whether models drove or refused.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates language models in recorded-scene question answering, simulation, real vehicles, and robotics, often with driving-specific training or an autonomy stack. DrivingBench instead evaluates general-purpose models directly controlling a real car.

2 RELATED WORK

  • Driving evaluations have mainly used recorded-scene question answering or simulated closed-loop driving with driving-trained models or small general-purpose VLMs.Simulator work has also tested LLM driving through code and APIs, including memory and reflection across episodes, as in a car protocol similar to DrivingBench (Ma et al., 2024; Wen et al., 2024).Source: §2 · p. 3
  • Robotics work has used language models to write policy code, select pre-learned skills, or control robots with trained models; some untuned VLM systems wait for each query, unlike DrivingBench’s moving car (Liang et al., 2023; Nasiriany et al., 2024).The paper also relates its reflection protocol to verbal self-reflection between trials (Shinn et al., 2023) and its knowing-doing gap to prior game and bandit findings (Paglieri et al., 2025; Schmied et al., 2026).Source: §2 · p. 3
  • Prior work reports that robot-controlled LLMs can be coerced into unsafe actions, while task framing and tool-output format affected whether models drove in DrivingBench (Robey et al., 2025; Zhang et al., 2025; Laine et al., 2024).GPT-6 Astra refused to drive until the task was reframed and the prompt iterated, despite refusing only 2 of 100 harmful robot-arm instructions in RoboHarm.Source: §2 · p. 3

3 DRIVINGBENCH SYSTEM

DrivingBench connects agent applications to a real car through an MCP server and a modified openpilot controller. Its action interface evolved toward simple, replaceable commands that expose latency and support analysis of model behavior.

3 DRIVINGBENCH SYSTEM

  • A 2022 Toyota Corolla retrofitted with a comma four provides camera views and car control through a modified openpilot system connected to the model’s laptop.The comma connects to the car’s CAN bus and provides forward-facing and wide-angle camera views.Source: §3 · p. 3
  • The open-source harness exposes observe(), set_motion(), and stop_now() through an MCP server; the controller translates motion commands into steering and speed control.observe() returns camera frames, speed, steering, and command time remaining; set_motion() specifies direction, steering percentage, speed, duration, and a reason, while stop_now() brakes immediately.Source: §3 · p. 4
  • An operator platform supports manual control and configuration, displays live camera views and tool calls, and records telemetry and tool traces for analysis.With the final tool interface and prompt framing, every model drove on every attempt.Source: §3 · p. 4

3.1 ACTION INTERFACE DESIGN

  • The interface evolved from geometric paths to timed curvature and finally to steering percentage, speed, and duration, with each change responding to earlier failure modes.The development process yielded design principles, but the authors note they did not make controlled comparisons.Source: §3.1 · pp. 4–5, Table 1 · p. 5
  • Commands take effect when accepted, and responses include timestamps and image age, making latency visible while the car continues under its running command.A development run saw the car move 2.4 m while the model planned; path re-anchoring removed a rejection but hid the delay further.Source: §3.1 · p. 4
  • Steering percentages let models compare requested and measured steering on the same scale, supporting self-calibration as steering ramps up over time.Earlier curvature and distance inputs were difficult to read from camera images and models were not calibrated enough to use them properly.Source: §3.1 · p. 5
  • The final harness clamps commands to native limits and reduces rejections, which otherwise cost a model turn while the car may keep moving or wait.Only speeds above a shared ceiling are rejected; a rejected command leaves the previous one running.Source: §3.1 · p. 5
  • New commands replace rather than queue behind running commands, and expiration brakes the car; the final interface also reduces the number of parameters and records a free-form reason for each action.The reason field captures the model’s stated intent and made failure-mode categorization easier.Source: §3.1 · p. 5

4 EVALUATION SETTING

DrivingBench evaluates four model–harness systems on a fixed real-world cone course, with up to three attempts in a shared chat. It scores course progress and records operational measures while an attentive operator supervises the car.

4.1 COURSE

  • The 127 m cone course combines a left turn, two right turns, straights, and gentle bends, ending in a 7×5 m finish zone.Source: §4.1 · p. 6

4.2 MODELS AND TRIAL SETUP

  • Four models run in native harnesses, and each one-time chat allows up to three attempts, ending at the first success; Table 2 reports progress per attempt and aggregate time, tokens, and cost.Attempts share context, with fixed reflection and continuation prompts after failures; models are compared as deployed model–harness systems.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 METRICS

  • Progress is the furthest course-centerline point reached within 4 m, expressed as a share of the 127 m route; reaching the finish zone is 100%.Progress pauses outside the 4 m boundary, and the authors note that the metric is imperfect.Source: §4.3 · p. 7

4.4 OPERATOR PROTOCOL AND SAFETY

  • An attentive operator monitors the car and brakes if it leaves the course or nears an obstacle; failed attempts end with an operator brake.The car starts from a fixed position and heading, and the operator engages openpilot after the first accepted command.Source: §4.4 · p. 7

5 RESULTS

Only GPT-6 Astra completed the course, while Fable made substantial progress but also failed; latency, perception, planning, and hardware constraints shaped outcomes. Later attempts show uneven improvement, but the shared-context design limits independent replication.

5.1 COURSE COMPLETION

  • 100%: GPT-6 Astra finished on attempt two after reaching 49% on attempt one; Fable rose from 9% to 45%, while Grok and Sol never cleared the first turn.Grok scored 8%, 11%, and 10%; Sol scored 6% on each of its three attempts.Source: §5.1 · pp. 7–8

5.2 LATENCY IS PART OF THE TASK

  • 22.2 s: Fable had the longest median interval from the previous tool call to a new motion command; the car continued moving under active commands while models reasoned.Median intervals were Astra 4.9 s, Sol 5.4 s, and Grok 12.5 s; Astra's command was accepted after the car had traveled a median of about 4 m from the prior frame.Source: §5.2 · p. 8
  • From rest, the steering wheel took a median 4.0 s to reach 90% of a requested change, versus 1.9 s while rolling, making stop-and-go costly at turns.Only Astra and Sol replaced running commands; Fable and Grok did not.Source: §5.2 · p. 8

5.3 FAILURE MODES

  • 8 of 11 attempts involved misreading the cone boundary at section A, including every Sol and Grok attempt and Fable's first two.Figure 4 pairs a failed attempt's last narrow-camera frame with the model's final command, stated reason, and outcome.Source: §5.3 · p. 8, Figure 4 · p. 10
  • Premature commitment meant acting before enough information or room for error; it affected the two furthest failed attempts, Astra's first and Fable's third.Astra accelerated before turning right at D, while Fable began that turn with a gentle 30% command at 1.2 m/s, leaving too little room.Source: §5.3 · pp. 8–9
  • The steering system's rate and range limit recovery: its tightest measured turning radius was about 16 m, and late corrections near islands or curbs may be unrecoverable.The models noticed some vehicle limits but did not always interpret or apply them correctly in later attempts.Source: §5.3 · p. 9

5.4 IN-CONTEXT LEARNING ACROSS ATTEMPTS

  • Astra improved from 49% to 100% and Fable from 9% to 45%; Grok and Sol did not materially improve or clear the first turn.Reflections often identified plausible fixes, but translating them into improved control remained a knowing-doing gap; Astra's second attempt used shorter, slower commands and more observations.Source: §5.4 · p. 9
  • Each model had only one independent trial because its attempts shared the same conversation context, limiting independent evidence about cross-attempt learning.Source: §5.4 · p. 10

6 FRAMING AND REFUSALS

Driving willingness varied with how the task and tool outputs presented the physical stakes; reassurance did not reliably resolve Astra’s refusals. The final interface and operational framing were associated with every model driving on every trial attempt.

6 FRAMING AND REFUSALS

  • Astra consistently refused when the observation tool identified the endpoint as a real vehicle, despite driving under a prompt that named the real Corolla.Its refusals cited the endpoint field directly.Source: §6 · p. 10
  • 0 of 16 conversations using the instruction to call tools without deliberating drove; 12 refusals came before “GO.”Additional assurances shifted refusals earlier, while Sol drove with the same instruction.Source: §6 · p. 10
  • Astra’s refusals could be inconsistent across turns, and telling models the task was a simulation sometimes only delayed refusal until they inferred the setting was real.The authors interpret willingness as depending on how explicitly and often the prompt and tool outputs emphasized physical stakes.Source: §6 · p. 11
  • With the final interface and task framing, every model drove on every trial attempt.The setup reduced what the model read about the stakes and described the task operationally.Source: §6 · p. 11

7 DISCUSSION AND LIMITATIONS

Astra completed the course, but the authors report a substantial speed gap from human driving and limitations in steering, sensing, and trial design. The evaluation used only one independent trial per model, with dependent in-context attempts.

7 DISCUSSION AND LIMITATIONS

  • GPT-6 Astra completed the course, but its best attempt took 4:44 versus well under a minute for a human.The authors attribute part of the gap to latency, while also identifying spatial reasoning and decision making as limitations.Source: §7 · p. 12
  • ≈16 m turning radius at the typical measured steering angle limits tight-corner tracks and the car’s ability to recover from errors.The passage also notes camera blind spots and possible startup speed overshoot, so outcomes reflect low-level execution as well as model decisions.Source: §7 · p. 12
  • Each model had only one independent trial, with up to three dependent in-context attempts; the authors plan to add statistical significance in future work.Source: §7 · p. 12

8 CONCLUSION

DrivingBench evaluates real-car control and reports that Astra alone finished the course, on its second attempt. The conclusion connects the task to broader physical-AI capabilities, while the paper documents its safety setting, released materials, and prompts.

8 CONCLUSION

  • GPT-6 Astra completed the course on its second attempt; no other model reached 50%.Most models stopped at the first corner.Source: §8 · p. 12
  • Perception or spatial reasoning, latency, and accurate self-diagnosis in control procedures matter for physical AI beyond driving.The authors leave questions about appropriate behavior and evaluation when models control real vehicles for future work, and plan more independent trials and harder courses.Source: §8 · p. 12

ETHICS STATEMENT

  • Trials used an empty parking lot at very low speeds with an attentive safety driver; the harness imposed speed limits and emergency-stop safeguards.No collisions occurred in the reported trials, and the emergency stop did not trigger.Source: ETHICS STATEMENT · p. 13

REPRODUCIBILITY STATEMENT

  • The authors release the harness, prompts, course information, traces, telemetry, transcripts, and blurred video; reproduction depends on compatible hardware and a comparable course.Results may also depend on model versions and agent applications available at trial time.Source: REPRODUCIBILITY STATEMENT · p. 13

A PROMPTS

  • Each trial starts with a main prompt; after failure, the operator sends a reflection prompt and then a continuation prompt.The prompts are reproduced verbatim in the appendix.Source: A PROMPTS · p. 16

MAIN PROMPT

  • The main prompt sets a cone-bounded backwards-U route ending in a blue-cone parking zone, with steering, speed, and duration controls.It tells the model that the car keeps moving while it thinks and to observe movement after commands and adapt.Source: MAIN PROMPT · pp. 16–17

REFLECTION PROMPT

  • After a failed attempt, the reflection prompt asks the model to consider why it ended and what to improve next time.Source: REFLECTION PROMPT · p. 17

CONTINUATION PROMPT

B.1 OPERATOR MESSAGES

Two operator-message deviations are documented: Astra received an extra continuation, while Grok’s attempt transitions used different message handling.

B.1 OPERATOR MESSAGES

  • After Astra’s attempt-2 continuation prompt, the operator interrupted a turn during reset and later sent “Continue” without other information.Source: §B.1 · p. 17
  • For Grok, the operator sent the next message instead of interrupting, because Cursor could overwrite the prior message after a stop.Source: §B.1 · p. 17
  • Grok’s extra operator messages included reflection prompts after attempts 1 and 2, then “Stop” after attempt 3.Source: §B.1 · p. 17

B.2 LATENCY EXAMPLES

Latency and command expiry can leave the car moving on stale observations or force it to restart from rest. The examples show these effects differed across attempts and models.

B.2 LATENCY EXAMPLES

  • Commands expire and brake the car, while the car may continue moving during model deliberation or between tool calls.This can leave the model acting on old frames or require it to restart from rest, which can make situations such as turns more difficult.Source: §B.2 · p. 17
  • Grok’s second attempt sent its last command 17 s after its last seen frame, while the car rolled 7.4 m.Its reflection noted that an expired command starts braking but the car can still roll into a cone.Source: §B.2 · pp. 17–18
  • Astra recognized that several seconds of interpretation and tool calls at 1–1.5 m/s meant several additional meters of travel before its strong correction.The correction began after much of the available clearance was already gone.Source: §B.2 · p. 18
  • Fable’s second attempt used a 55 s-old frame but moved only 2.1 m; the car was moving for 15% of that attempt, the lowest share across attempts.Because the car was nearly stationary, the stale observation mostly wasted time rather than covering distance.Source: §B.2 · p. 18

C VEHICLE CHARACTERIZATION

Supervised calibration of the Corolla, comma four, and openpilot identified vehicle behaviors that help explain the reported driving results.

C VEHICLE CHARACTERIZATION

  • The car-specific controller accumulated requested acceleration while stopped, so the first command from rest reached 1.6–2.4 m/s for requests of 0.8–1.5 m/s.The measured breakaway threshold was about 0.52 m/s^2 on flat pavement and up to 1.56 m/s^2 in one run.Source: C VEHICLE CHARACTERIZATION · p. 18
  • From rest, the wheel took a median 4.0 s to reach 90% of a steering change, versus 1.9 s while rolling.openpilot lateral control activates only above 0.30 m/s.Source: C VEHICLE CHARACTERIZATION · p. 18
  • Trial requests of 100% (180°) reached a median wheel angle of 129° and at most 173°; the typical plateau was about 120–130°.Toyota’s 100°/s steering-rate limit bounds turn buildup, and the turning radius at the typical plateau is about 16 m.Source: C VEHICLE CHARACTERIZATION · p. 18
  • With an earlier curvature controller, reaching full left curvature took about 22.5 m from straight, versus 12 m for full right curvature.The authors did not re-measure this asymmetry with the final controller.Source: C VEHICLE CHARACTERIZATION · p. 18

D ADDITIONAL RESULTS

Additional results materials document attempt endings and per-attempt statistics, and provide figures for course progress, observation latency, steering requests, and control traces.

D ADDITIONAL RESULTS

  • Table 3 records how each attempt ended, while Table 4 defines per-attempt statistics including duration, moving share, command replacement, median command gap, and peak speed.Table 3’s section letters refer to the course sections.Source: Table 3 · p. 19, Table 4 · p. 19
  • The trace viewer synchronizes recorded evaluation video, MCP tool use, and car telemetry by timestamp for replay.Telemetry includes steering, acceleration, and position.Source: Figure 6 · p. 20
  • Figure 7 reports furthest course progress within 4 m of the 127 m intended centerline, both per attempt and over time.Later attempts occur in the same chat after a fixed reflection prompt.Source: Figure 7 · p. 20
  • Figure 8 summarizes time since the previous tool call and distance driven between the last received frame’s capture and command acceptance.Its bars show medians.Source: Figure 8 · p. 21
  • Figure 9 compares requested and actual steering, including response delays, achieved wheel angles, and measured path curvature against a bicycle model.The figure reports a 4.0 s median for the wheel to reach 90% of its change from rest and a 16 m turning radius at the plateau.Source: Figure 9 · p. 21
  • Figure 10 shows requested steering-percent distributions by attempt and the pooled time between consecutive accepted set_motion calls.The inter-call time is plotted on a log scale.Source: Figure 10 · p. 22
  • Figures 11–14 plot requested and measured speed and steering-wheel angle over time for Astra, Fable, Grok, and Sol attempts.Shading marks periods without an active command, and dashed lines show the typical approximately ±130° EPS plateau.Source: Figure 11 · p. 23, Figure 12 · p. 24, Figure 13 · p. 25

Related papers

Loading 2609.38948v1…