PaperlayerView exact revision on arXiv

Source-linked AI summary

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

Jie Yang, Jiajun Chen, Jiazheng Zhou, Mianqiu Huang, Yining Zheng, Yuxin Wang, Xipeng Qiu

arXiv:2609.35916v2cs.MAcs.CLcs.CV

TL;DR

Driving and cockpit-interaction benchmarks are typically evaluated separately, leaving limited study of agents serving evolving passenger requests while adapting to traffic. VehicleArena unifies these functions and evaluates independent agents in shared traffic; across task groups, strong service or driving scores do not reliably coincide with arrival.

  • Problem

    Closed-loop driving benchmarks and cockpit environments are typically evaluated in isolation, separating passenger intent and cabin interaction from the physical driving process.

    Source: §1 · p. 1

  • Method

    VehicleArena combines passenger requests, cabin interaction, and LLM-controlled driving in a shared 3D urban world, including vehicles with independent objectives.

    Source: §1 · pp. 1–3

  • Results

    Across task groups, strong request, cabin, or driving scores do not consistently coincide with arrival; Qwen3.8-Max reaches 65.0% Basic arrival and 65.6% Multi-Agent arrival.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Evaluated driving policies can reduce surrounding vehicles’ arrival rates, making traffic effects beyond the focal vehicle a relevant outcome.

    Source: §6 · p. 10

Abstract

from arXiv · show

Real-world embodied agents often pursue independent objectives within a shared physical environment, where their actions can alter the conditions faced by others. Existing benchmarks, however, typically assume shared goals or explicitly prescribed interaction protocols, leaving such emergent physical coupling underexplored. We introduce VehicleArena, a 3D urban-driving benchmark for studying independently operating agents in a dynamic shared world. In VehicleArena, LLM-controlled agents must fulfill evolving passenger requests while navigating complex traffic, and each agent's driving decisions can reshape traffic flow, delays, risks, and subsequent observations for surrounding agents. The benchmark provides 112 evaluation tasks spanning single-agent and multi-agent driving. Across nine evaluated models, the highest arrival rates reach only 65.0% on single-agent tasks and 65.6% on multi-agent tasks, while strong passenger-request or cabin scores do not reliably translate into successful trip completion. Moreover, in matched multi-agent runs, every tested focal policy reduces the arrival rate of surrounding vehicles relative to the simulator's native traffic controller, revealing measurable externalities beyond the focal vehicle itself.

1 Introduction

Driving and cockpit benchmarks typically isolate road decisions from passenger interaction, limiting study of agents that must adapt to both. VehicleArena unifies these tasks and extends them to independent agents whose actions affect one another through shared traffic.

1 Introduction

  • Existing closed-loop driving benchmarks and cockpit environments are typically evaluated in isolation, separating passenger intent and cabin interaction from the physical driving process.Source: §1 · p. 1
  • Personal Agents issue route, driving-preference, and device requests, while Driving Agents execute them under traffic conditions, coupling passenger intent, cabin operations, route decisions, and vehicle motion.Source: §1 · pp. 1–2
  • Unlike benchmarks that define collaboration, competition, or communication structures, VehicleArena lets independently goal-directed vehicles influence one another through physical consequences in shared traffic.Source: §1 · p. 2
  • The paper calls this independent-objective physical coexistence: agents become coupled through persistent interaction in a shared world, raising the question of reliability under changing user demands and other agents’ actions.Source: §1 · p. 2
  • VehicleArena evaluates integrated single-agent capabilities and multi-agent physical coexistence across 112 held-out tasks, using separate metrics for driving, service, traffic impact, and inference cost.The benchmark also includes 100 development tasks.Source: §1 · p. 3

2 Related Work

Prior work studies structured agent relationships or ego-focused driving, while VehicleArena emphasizes physical coupling among independently operating vehicles and combines passenger service with driving.

2.1 Multi-Agent LLM Systems

  • Unlike multi-agent systems organized around collaboration, competition, or predefined interaction structures, VehicleArena studies independent objectives coupled through persistent physical interaction in shared traffic.Source: §2.1 · p. 3

2.2 LLM-Based Autonomous Driving

  • Driving benchmarks and closed-loop systems largely focus on ego-vehicle decisions or predefined navigation, while VehicleWorld separates device interaction from driving; VehicleArena brings these functions together in persistent traffic.Source: §2.2 · p. 3
  • In the recorded episode, DA raises its target speed to 35 km/h after a rear-traffic warning, then handles a cabin request without replacing that motion target as SUMO advances between wakes.Source: Figure 2 · p. 4

3 VehicleArena Environment

VehicleArena places passenger service and physical driving in a shared urban world, with agent roles and harnesses coordinating decisions, execution, and request verification.

3.1 Overview

  • The environment combines a shared 3D world, traffic simulation, synchronized rendering, configurable vehicle equipment, and vehicles controlled by either LLM drivers or native SUMO logic.Source: §3.1 · p. 4, §3 · p. 4
  • Personal Agents express passenger needs, Driving Agents decide how to serve them, and Judges evaluate execution; each role shares the episode state and clock but has distinct responsibilities and access.PA cannot actuate the vehicle, and Judge observes and evaluates without controlling it.Source: §3.1 · p. 4

3.2 Driving-Agent Harness

  • Driving Agents operate through repeated observe–act sessions that combine current observations and passenger updates with retained context, tools, interaction history, and a persistent Todo board.Source: §3.2 · p. 4
  • DA wakes are event-triggered, periodic, or scheduled; the world evolves between wakes under active controls, and simultaneous agents observe the same committed road state before their commands are committed.Source: §3.2 · p. 5
  • DA issues persistent motion commands and cabin-state changes, while the simulator realizes them under physical constraints and surrounding traffic; route planning guides navigation but does not automatically follow the route.Source: §3.2 · p. 5

3.3 Personal-Agent and Judge Harness

  • Personal Agents may issue, revise, or cancel immediate and delayed requests, whose acceptance criteria then remain fixed; Judges verify delayed requests independently of DA wake timing against interaction and execution histories.Source: §3.3 · p. 5

4 Benchmark

The benchmark evaluates trip completion and passenger tasks under shared-road conditions, using constructed scenarios, matched SUMO reference runs, and separate outcome metrics.

4 Benchmark

  • The benchmark compares whether agents complete their trips and passenger tasks while sharing the road, using matched SUMO runs for reference behavior and a task deadline.Source: §4 · p. 5

4.1 Task Suite and Construction

  • The held-out suite contains 80 Basic tasks with one DA-controlled vehicle and 32 MultiLLM tasks with several independently configured DAs; experts design trips, traffic situations, and passenger goals.Scenarios include intersections, merges, narrow roads, and pedestrian crossings.Source: §4.1 · p. 5

4.2 Evaluation Protocol

  • Each DA is compared with a SUMO-controlled ego vehicle in the same scene and surrounding-agent configuration, and the deadline is fixed using the later of the reference arrival and last scheduled event plus a margin.The margin is normally 10 seconds.Source: §4.2 · pp. 5–6

4.3 Evaluation Metrics

  • Six outcomes remain separate: ego trip completion and driving quality, passenger fulfillment and cabin compliance, background-traffic effects relative to SUMO, and model-token use.Source: §4.3 · p. 6

5 Experiments

Across 112 held-out tasks, nine models show a persistent gap between trip arrival and other task scores; analysis also finds varied planning efficiency, driving outcomes, and burdens on surrounding traffic. Disabling the ego Personal Agent improves arrival and cabin scores for two tested models, while passenger requests can coincide with missed obligations or delayed trips.

5.1 Experimental Setup

  • The evaluation compares nine models across 80 Basic and 32 MultiLLM tasks, with three runs per task and fixed peer assignments in MultiLLM.In Basic, other traffic uses native simulation control; in MultiLLM, the ego shares the road with three DA-controlled vehicles, whose actions respond to evolving traffic.Source: §5.1 · p. 6

5.2 Main Results

  • Qwen3.8-Max reaches 65.0% Basic and 65.6% Multi-Agent arrival, while GPT-5.6-Sol leads several other scores but arrives at only 20.0% and 9.4%, showing these outcomes can diverge.MIMO-V2.6-Pro has the highest Basic Request score, 90.0; GPT-5.6-Sol has the highest Basic Cabin score, 90.2, and the highest Multi-Agent Drive and Cabin scores, 87.2 and 94.0.Source: §5.2 · p. 6

5.3 Analysis

  • Qwen3.8-Max pairs strong outcomes with economical interaction, while frequent shallow wakes do not ensure success and similar interaction counts can yield different outcomes.For driving, neither speed nor collision count alone captures effectiveness: GLM-5.3-Flash and Kimi-K3 have similar mean speeds but different Multi-Agent arrival rates.Source: §5.3 · p. 7, Table 2 · p. 8
  • GPT-5.6-Sol makes the most skill loads, including driving_control in 105 tasks, yet has low arrival and completion-weighted outcomes; Qwen3.8-Max navigates effectively without that guide.Figure 4 reports successful skill loads by task and non-finish tool-call shares by family, using separate percentage scales for its two panels.Source: §5.3 · p. 8, Figure 4 · p. 9

5.5 Ablation

  • Disabling the ego Personal Agent raises arrival by 11.6 points for Kimi-K3 and 13.4 for GLM-5.3-Flash, while field-weighted Cabin scores rise by 30.9 and 20.2 points.With the Personal Agent enabled, Kimi-K3 and GLM miss simultaneous mandatory cabin checks in 20.5% and 18.8% of tasks; some PA-on timeouts involved low-speed passenger requests.Source: §5.5 · pp. 9–10, Table 3 · p. 9

5.4 Impact on Surrounding Vehicles

  • GPT-5.6-Sol has the largest equal-weight mean surrounding-traffic burden across the four axes and both groups, while DeepSeek-V4.1-Flash has the smallest.On Basic, GPT has the largest burden and Qwen3.8-27B the smallest; on Multi-Agent, Qwen3.8-27B has the largest and Qwen3.8-Max the smallest.Source: §5.4 · p. 9, Figure 5 · p. 10

6 Conclusion

VehicleArena studies independent agents whose private plans produce persistent, observable consequences in a shared physical world. Its results show that passenger and cabin performance can coexist with failed trips, and that driving policies can reduce other vehicles’ arrival rates.

6 Conclusion

  • VehicleArena examines independent agents whose actions create persistent, observable consequences in a shared physical world.Its 3D traffic, passenger requests, cabin actions, and non-compensatory metrics connect local decisions with system-level effects.Source: §6 · p. 10
  • Strong passenger-request or cabin scores often coexist with failed trips, while evaluated driving policies can reduce other vehicles’ arrival rates.The conclusion identifies external effects beyond the focal vehicle itself.Source: §6 · p. 10
  • Future work should test longer episodes, richer sensing, broader road cultures, and transfer to real driving data.Source: §6 · p. 10

A Experimental Settings

The appendix documents model-specific inference controls and matched MultiLLM evaluation settings. In its ablation, only the ego controller changes, with the ego PA/Judge ablation disabling those roles.

A.1 Inference Configuration

  • Table 4 records requested inference settings and documented thinking or effort controls, which are model-specific; N/A indicates a control the model does not expose.Model specifications are listed in Appendix G.Source: §A.1 · p. 12, Table 4 · p. 12
  • aMiMo exposes no adjustable effort, while Qwen3-VL uses instruct checkpoints.Source: §A.1 · p. 12
  • MultiLLM runs retain peer personalities, active PAs, and Judges; only the ego controller changes, while its PA/Judge ablation disables those ego roles.Without the ego PA, no ego passenger requests are generated; cabin scoring follows the same automatic rule path in both settings.Source: §A.1 · p. 12

A.2 Task Split

The task suite separates Basic training and test tasks from MultiLLM interaction tasks, and specifies their scenarios, routes, and environmental conditions. Passenger requests are generated and evaluated through scheduled wakes, typed triggers, and staged Judge checks.

A.2 Task Split

  • The suite has 100 Basic training tasks and 112 test tasks—80 Basic and 32 MultiLLM—with task counts including all LLM-controlled vehicles and map counts deduplicated within each split.The Basic training pool contains scenarios outside the Basic test manifest.Source: §A.2 · p. 12, Table 5 · p. 13
  • Basic tasks cover varied road interactions and maneuvers, while MultiLLM tasks combine narrow meetings, unprotected left turns, synchronized intersections, and multi-vehicle merges.Weather and illumination conditions also vary across the task families.Source: §A.2 · pp. 12–13
  • Ego route is lane-level distance from the initial pose to destination, using a frozen route or the lane planner; the time limit is a calibrated simulation deadline.Source: §A.2 · p. 13

A.3.1 Personal-Agent wakes

  • Personal Agents receive passenger-visible state, recent motion, wake category, prior requests, and passenger-visible Driving Agent replies; Judge grades and hidden state are excluded.Judge checks, tool receipts, request outcomes, and Driving Agent heartbeats do not trigger Personal Agent wakes.Source: §A.3.1 · p. 13, Table 6 · p. 13

A.3.2 Typed deferred-request triggers

  • Typed deferred-request candidates are filtered by remaining route, scheduled events, and available free-flow time; numeric parameters are state-bounded and predicates are checked at 0.1-second physics resolution.Free-text predicates are unsupported.Source: §A.3.2 · pp. 13–14, Table 7 · p. 14

A.3.3 Request lifecycle and Judge windows

  • Request contracts record criteria, request kind, response time, validity, and optional trigger; Judge checks occur at 0.1, 1, and 3 seconds after creation or trigger activation.Checks use contract, replies, tool receipts, before/after state, and physical trace; ongoing requests remain pending through the final observation window.Source: §A.3.3 · p. 14, Table 8 · p. 14
  • The request-verification loop validates and freezes criteria, updates verification schedules, records valid verdicts or explicit unscored status, and closes requests at completion or final check.Source: §A.3.3 · p. 15

A.4 Evaluation Metric Definitions

The evaluation reports trip, driving, passenger-request, cabin, and traffic-impact measures separately, with explicit rules for aggregation, unscored cases, and paired traffic comparisons. It also defines a completion-weighted index and the conventions behind skill, tool, and token counts.

A.4 Evaluation Metric Definitions

  • Delay increases are summed per vehicle, trip-completion delay includes only arrivals, and untriggered or unverified requests and failed Judge calls are reported separately without grades.The metric table reports measures separately rather than combining them into one scalar.Source: §A.4 · p. 15, Table 9 · p. 15

A.5.1 Aggregation and unscored cases

  • Request-level means pool valid scored requests, whereas scenario-level means give equal weight to each scenario with at least one scored request; empty applicable sets are unavailable, not zero.Aggregates report valid observation counts and unavailable, unscored, or invalid cases.Source: §A.5.1 · p. 16

A.5.2 Passenger-request grades

  • Judges grade only frozen passenger requirements, using fulfillment evidence rather than verdict time; promises or queued commands alone do not establish fulfillment.Clear, truthful refusals may resolve a requirement when supported by evidence of lacking the requested capability, but do not establish physical completion.Source: §A.5.2 · p. 16, Table 10 · p. 16

A.5.3 Traffic Interaction Set

  • Traffic-impact comparisons include all required non-ego trip vehicles, including fixed MultiLLM peers, and pair evaluated and SUMO runs by task and vehicle ID.Trip-completion delay is calculated only for vehicles arriving in both runs.Source: §A.5.3 · p. 16, §A.6 · p. 16

A.6 Surrounding-Vehicle Impact Measures

  • Surrounding-vehicle measures compare arrival, collisions, and delays against SUMO; delays use vehicle-seconds per task and collision and non-arrival changes use percentage points.Radar axes rescale delays and positive collision changes; collision decreases plot at zero while the underlying signed difference remains, and its coordinate average is descriptive rather than an evaluation metric.Source: §A.6 · pp. 16–17

A.7 Completion-Weighted Outcome Index

  • The completion-weighted plotted index uses Arr., Drive, Req., and Cabin metrics, with Arr. expressed as a percentage and the metrics’ unrounded values.Source: §A.7 · p. 17

B.1 Model-Level Skill, Tool, and Token Counts

  • Table 11 pools 80 Basic and 32 Multi-Agent tasks; skill entries count successful loads, tool entries count non-finish calls, and token entries total ego Driving Agent input and output tokens.Token totals are in millions and rounded to two decimal places.Source: §B.1 · p. 17

B.2 Token Cost Analysis

VehicleArena compares ego Driving Agent token use with task outcomes, showing that similar inference budgets can produce substantially different results. Token count is a proxy for inference cost, not provider pricing.

B.2 Token Cost Analysis

  • Ego DA input and output tokens per task serve as an inference-cost proxy, excluding passenger agents, judges, and fixed peer models.Token count adds an effort measure because similar wake and tool-call counts can involve different amounts of text; provider pricing is not included.Source: §B.2 · p. 17
  • On Multi-Agent, GLM-5.3-Flash outperforms Kimi-K3 and Qwen3.8-27B at nearly the same token budget, while Qwen3.8-Max has the highest outcome with moderately more tokens.At a roughly two-million-token budget, DeepSeek-V4.1-Flash outperforms GPT-5.6-Sol.Source: §B.2 · p. 17
  • On Basic, Qwen3.8-Max achieves the strongest outcome among models using roughly one to two million ego DA tokens per task, matching DeepSeek-V4.1-Flash with less than half as many tokens.Figure 6 compares mean ego DA tokens per task with a completion-weighted outcome index.Source: §B.2 · p. 18

C Passenger Requests in the PA Ablation

These sections describe passenger-request examples and the benchmark’s scoring, vehicle configuration, cabin-rule evaluation, and map inventory. Together, they specify key evaluation and runtime boundaries.

C Passenger Requests in the PA Ablation

  • Table 12 pairs six individual passenger requests from four representative ego trajectories with the road and environment context at each request.Requests are translated or abridged; numerical speeds are passenger preferences, and distance thresholds specify when an action is wanted rather than measured remaining distance.Source: C Passenger Requests in the PA Ablation · pp. 18–19, Table 12 · p. 19

D Driving-Rule Scoring Details

  • Driving quality starts at 100, falls to zero for an at-fault collision or red-light violation, and otherwise subtracts event-level penalties.Deduction types have a three-second cooldown; continuous unjustified stops and intersection blocking repeat at five-second intervals after the initial penalty, while a pedestrian encounter receives only its most severe deduction.Source: D Driving-Rule Scoring Details · p. 20, Table 13 · p. 20

E Vehicle Modules and Configuration

  • Each vehicle receives a validated, immutable capability set before simulation, governing installed modules, tool and skill catalogs, and physical limits.Unsupported module requests return an explicit capability_not_available result, distinguishing them from failed use of an installed device.Source: E Vehicle Modules and Configuration · p. 20, Table 14 · p. 21

E.1 Built-in module catalog

  • The current release registers 40 modules; vehicles receive only the subset selected by their capability set.Five world and driving modules are required by the runtime; other modules are optional or may be supplemented by a trusted extension.Source: §E.1 · p. 21, Table 15 · p. 21

E.2 Declarative rule format and evaluation

  • Cabin requirements use validated YAML rules with domain-specific triggers, optional guards, expected actions, and optional tolerances.Experiments derive automatic Cabin expectations from weather, day–night, and map transitions; PA-generated requests are evaluated separately, without user-intent Cabin checkpoints in PA-on or PA-off.Source: §E.2 · p. 21
  • The rule engine derives expected actions from current and previous world snapshots, then compares reference and actual state changes against declared tolerances.Cabin score pools correct fields over ego checkpoints; checks vary with device state, and free-form DA text or an API call without its required state change does not count as compliance.Source: §E.2 · pp. 21–22

F Supported Cities

G Model Specifications

This section documents the evaluated model set and the prompts, skills, and tools that define VehicleArena agent behavior, treating driving as physical, observation-dependent control with explicit rules for weather, day/night transitions, and unsafe passenger requests. It also specifies separate interfaces for driving, passenger-request management, and judging, with request state shared across roles and judging isolated from vehicle control.

G Model Specifications

  • In the experiments summarized in Table 17, nine models were evaluated, and Peer DA, PA, and Judge all used Qwen3.8-27B.Source: G Model Specifications · p. 23
  • Table 17 summarizes nine models evaluated in the experiments; Peer DA, PA, and Judge all use Qwen3.8-27B, while the experimental output limit is 32,768 tokens and native context is extensible to 1M tokens.Source: G Model Specifications · pp. 23–24

H Prompts

  • The prompt templates define separate roles for the Driving Agent, Personal Agent, and passenger-request Judge.The driver controls the vehicle through tools and wakes, the passenger submits observable-outcome requests, and the Judge evaluates frozen requested outcomes using observed evidence rather than accepted commands or promises alone.Source: H Prompts · pp. 24–25

I Skills

  • The optional skill catalog contains four guides, filters guidance for equipment absent from a vehicle, and expands shared weather-safety rules when loaded.The listed guide bodies are shown before vehicle-specific equipment filtering.Source: I Skills · p. 26, Table 18 · p. 26

I.1 driving_control

  • Driving-control commands advance in 0.1-second world steps, and route plans provide guidance rather than automatic junction choices or trip completion.Accepted commands are not completed maneuvers; traffic signals and road rules remain observable, and the driver chooses behavior. CameraVisual supplies optical information, while LiDAR and radar are optional installed sensors.Source: §I.1 · pp. 26–27

I.2 weather_transition

  • Weather-transition guidance acts only on a weather event, applies relevant settings to installed equipment, and specifies cleanup and cold-weather or low-visibility adjustments.It directs drivers not to preemptively change clear-weather defaults; day/night lighting requirements can override weather cleanup.Source: §I.2 · pp. 28, 30

I.3 daynight_transition

  • Day/night-transition guidance requires four actions when entering dusk, night, or dawn and reverses those settings when returning to daytime.The first tick in a dark period uses the same entering-dark-period setup.Source: §I.3 · pp. 30–31

I.4 safety_refusal

  • Safety-refusal guidance requires the agent to refuse unsafe requests, explain why, and broadcast a safety warning; system constraints block several actions while driving.Examples include opening doors or the trunk and playing video while moving; suggested alternatives include stopping before opening a door or offering audio instead of video.Source: §I.4 · p. 31

J.1 Driving Agent (DA) Tools

  • DA tools cover driving, sensing, speed limits, and signaling; API discovery reflects installed modules, while text tools omit camera-visible objects and motion calls require later confirmation.Cabin calls report state changes, and invalid calls return repairable errors.Source: §J.1 · p. 32

J.2 Personal Agent (PA) Tools

  • PA’s four non-driving calls manage the passenger request sheet shared with DA and Judge.Source: §J.2 · p. 32

J.3 Judge Tools

  • Judge submissions are validated against the frozen request and timeline, and neither change vehicle state nor wake DA.Source: §J.3 · p. 32

Related papers

Loading 2609.35916v2…