PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving

Yupeng Zhou, Can Cui, Juntong Peng, Zichong Yang, Juanwu Lu, Jitesh H Panchal, Bin Yao, Ziran Wang

arXiv:2506.14100v1cs.ROcs.SYeess.SY

TL;DR

VLMs must be adapted from broad web-scale data to safety-critical driving, while existing simulation- and dataset-based evaluation has limits in representing real-world conditions. The paper presents a hierarchical real-world test platform and reports practical feasibility through deployment on an Autoware-equipped vehicle.

  • Problem

    Adapting VLMs to safety-critical driving and evaluating them under realistic conditions remains challenging because simulations and datasets have limits in fidelity and real-world scenario representation.

    Source: scenarios · p. 2, mimic the target domain as closely as… · pp. 2–3

  • Method

    The paper presents a hierarchical real-world test platform for evaluating VLM-integrated autonomous driving systems.

    Source: §6 · p. 24

  • Results

    Deployment on an actual vehicle using the open-source Autoware stack demonstrated the platform’s practical feasibility.

    Source: §6 · p. 24

  • Takeaways & Limitations

    The platform provides a real-vehicle setting for validating and evaluating VLM-integrated autonomous-driving technology.

    Source: §6 · p. 24

    Further work should reduce latency and improve accuracy, particularly in the Human Verbal Command Processor.

    Source: and effectiveness of our approach · p. 24

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated notable promise in autonomous driving by offering the potential for multimodal reasoning through pretraining on extensive image-text pairs. However, adapting these models from broad web-scale data to the safety-critical context of driving presents a significant challenge, commonly referred to as domain shift. Existing simulation-based and dataset-driven evaluation methods, although valuable, often fail to capture the full complexity of real-world scenarios and cannot easily accommodate repeatable closed-loop testing with flexible scenario manipulation. In this paper, we introduce a hierarchical real-world test platform specifically designed to evaluate VLM-integrated autonomous driving systems. Our approach includes a modular, low-latency on-vehicle middleware that allows seamless incorporation of various VLMs, a clearly separated perception-planning-control architecture that can accommodate both VLM-based and conventional modules, and a configurable suite of real-world testing scenarios on a closed track that facilitates controlled yet authentic evaluations. We demonstrate the effectiveness of the proposed platform`s testing and evaluation ability with a case study involving a VLM-enabled autonomous vehicle, highlighting how our test framework supports robust experimentation under diverse conditions.

1 Introduction

VLMs offer multimodal reasoning for driving, but adapting them from broad web data to safety-critical conditions creates domain-shift challenges. The paper proposes a hierarchical real-world platform for controlled, reproducible testing of VLM-integrated driving systems.

detection, semantic segmentation, and depth estimation.

  • VLMs pretrained on extensive image-text data offer cross-modal representations and multimodal reasoning for driving, but must adapt to safety-critical, egocentric conditions.Driving imagery can include specialized viewpoints, motion blur, sensor noise, and road-focused scenes; fine-tuning data may be limited or synthetic.Source: detection, semantic segmentation, and… · p. 2, scenarios · p. 2

scenarios.

  • Domain shift arises between broad web-scale pretraining, narrower driving fine-tuning data, and deployment conditions, creating uncertainty and potential blind spots in VLM performance.The passages identify variation in lighting, weather, road conditions, and object appearance as sources of mismatch.Source: scenarios · p. 2

mimic the target domain as closely as possible.

  • Existing datasets and simulations do not provide a comprehensive way to evaluate whole VLM-enabled vehicles in realistic conditions with flexible scenario changes and controlled variables.Logged data cannot controllably reproduce interactions or adapt to changes in an evaluated agent’s behavior in a closed loop.Source: mimic the target domain as closely as… · p. 3

scenarios.

  • The proposed platform targets controlled, authentic, reproducible real-world testing of VLM-based driving systems before road deployment.It is presented as a hierarchical testing framework addressing the testing and validation gap.Source: scenarios · p. 3
  • Its on-vehicle middleware is lightweight, structured, low-latency, VLM-agnostic, and modular for integrating different VLMs.The contribution list also describes a clear separation of perception, planning, and control modules.Source: scenarios · p. 3
  • The platform supports customizable traffic scenarios on a closed track, and its framework is demonstrated in a case study with a real-world VLM-integrated vehicle.Source: scenarios · p. 3

2 Related Works

Related work spans modular and end-to-end driving stacks, specialized autonomous-driving platforms, and simulation or real-world testing methods. The paper positions its framework as an extension of an open-source stack for repeatable real-world VLM experimentation and evaluation.

and control algorithms, accelerating research and innovation in the field.

  • Modular stacks such as Autoware and Apollo include perception, planning, and control, while modular architectures divide the stack into components for targeted development.Autoware is built on ROS and ROS2; modular platforms let researchers focus on individual components.Source: and control algorithms, accelerating… · p. 4, §2.1 · p. 4

and collaboration.

  • End-to-end systems map sensory inputs directly to control outputs; openpilot exemplifies this approach for highway ADAS.The passage describes openpilot as providing lateral and longitudinal control across multiple vehicle platforms.Source: and collaboration · p. 4

in real-world autonomous driving applications.

and real-world testing and [55].

  • Simulation platforms such as CARLA [16] and LGSVL model vehicle, sensor, and traffic conditions and support testing across safety, communication, computation, planning, and control.Simulation also permits cost-effective testing of potentially dangerous scenarios without real-world risks.Source: and real-world testing and [55] · pp. 4–5

platforms [9] are widely used for testing and validation.

deployment, testing, and validation for state-of-the-art autonomous driving technology.

  • VLM driving research covers perception, navigation, decision-making, control, end-to-end learning, and data generation, but most cited work is tested on datasets.Only a few works extend to onboard deployment and real-world experiments [14] [15] [48].Source: deployment, testing, and validation… · p. 5
  • The proposed framework extends an open-source driving stack to support easy integration, testing, and evaluation of VLMs in repeatable real-world scenarios.The overview figure divides the platform into a testing environment and a testing vehicle.Source: deployment, testing, and validation… · p. 5, Figure 1 · p. 5

3 Methodology

The methodology combines a hierarchical VLM–autonomy stack and configurable layered vehicle platform with a real-world testing environment. It supports repeatable closed-loop evaluation of VLM-integrated driving systems across varied road, weather, and traffic conditions.

3 Methodology

  • The testing environment varies road geometry, conditions, weather, and traffic participants to create or repeat scenarios under controlled conditions.Researchers can modify elements such as traffic patterns and visibility to target particular driving conditions.Source: §3 · p. 6

3.1 Tesing Vehicle Architecture

  • The platform integrates a VLM with a classical autonomy pipeline for by-wire vehicles, closing the loop between VLM decisions and real-world driving.The architecture routes sensor and vehicle information to the VLM and sends its directives through the autonomy stack to the actuators.Source: Figure 2 · p. 6, §3.1 · p. 7

3.1.1 Strategic Driving Intelligence Layer: This layer is essentially the VLM itself, representing the top-level intelligence

  • The Strategic Driving Intelligence Layer uses an on-board or on-cloud VLM for high-level decisions from prompts combining perception, vehicle state, driving behavior, and human feedback.Its high-level strategic inference leaves lower-level perception and control to other modules.Source: §3.1.1 · p. 7, the real-time interaction of either… · p. 7

the real-time interaction of either on-board or on-cloud VLMs.

3.1.2 Prompting & Action Interface Layer: Acting as the intermediary between the Strategic Driving Intelligence Layer

  • The Prompting & Action Interface Layer packages vehicle-state and environment information into structured prompts and translates VLM commands into executable action vectors.This layer bridges high-level decisions and downstream modules through its input and output pipelines.Source: Acting as the intermediary between the… · p. 8

3.1.3 Information Process & Execution Layer: The Information Process & Execution Layer implements a two-phase

motion control in the next layer.

  • To avoid rebuilding a complete autonomous stack, the platform uses an open-source driving platform as its backbone and foundation for custom features.The passage identifies the costs and workload of redeveloping a comprehensive stack as the motivation.Source: §3.1.4 · p. 9

features can be built.

  • Plug-in submodules and adjustable perception, planning, and control interfaces let developers select, create, and fine-tune parameters, inputs, and outputs.Exposed settings include detection thresholds, bounding boxes, path smoothness, turning radius, trajectories, and controller constraints.Source: features can be built · p. 9

3.1.5 Hardware Layer: Finally, at the lowest level, the Hardware Layer encompasses the physical sensor kits, computing

  • LiDAR, radar, and cameras send raw sensor data upward, while drive-by-wire actuators execute steering, throttle, and braking; isolating hardware-specific concerns supports adaptation across vehicle platforms.The autonomous driving layer generates control signals to follow the final trajectory.Source: §3.1.5 · p. 9

3.2 Testing Enviroment Setup

  • Repeatedly staged near-crash scenarios with dummy pedestrians or remote-controlled vehicles enable physical-world testing that open-road testing and conventional simulations do not easily allow.The platform also varies road conditions and geometry to control scenario difficulty.Source: §3.2 · p. 10

to the vehicle’s perception, planning, and control systems.

  • Rain, snow, and fog can degrade sensor performance and vehicle dynamics, while adding or removing road users adjusts traffic complexity and interaction demands.Examples include camera occlusion, degraded LiDAR returns, reduced traction, and longer braking distances.Source: to the vehicle’s perception, planning,… · p. 10

situations involving multiple road users with potentially unpredictable trajectories.

  • Systematic variation of road geometry, road conditions, weather, and traffic creates repeatable scenarios from routine driving to corner cases for validating VLM-integrated systems.Controlled environmental factors also allow direct performance comparisons and help isolate each parameter’s effect.Source: situations involving multiple road… · p. 10, §3.2 · p. 10

4 Case Study

The case study deploys the hierarchical framework on a real vehicle and uses controlled, repeatable trials to examine a VLM’s decision-making and scene understanding. Its layers connect multimodal VLM decisions to vehicle behavior through interfaces that process driving information and execute commands.

4 Case Study

  • The case study reports robust VLM decision-making and scene interpretation, with capabilities consistent across diverse, dynamic scenarios.The testing environment was systematically altered across controlled, repeatable trials.Source: §4 · p. 12

4.1 Real-World Testing Vehicle Setup

  • A hybrid vehicle with by-wire control and the Autoware software stack serves as the real-world test platform.The authors selected Autoware for its open-source flexibility and ROS-based architecture.Source: §4.1 · p. 12

4.1.1 Strategic Driving Intelligence Layer. In our real-world deployment, we adopt on-cloud ChatGPT-4 [36] as our

  • The on-cloud strategic driving layer uses a VLM with multimodal image or video scene inputs and contextual natural language instructions, including temporal sequences, to guide behaviors such as steering through turns and lane changes around obstacles.Source: §4.1.1 · p. 12

4.1.2 Prompt Generation & Action Layer. The Prompt Generation and Action Layer serves as a critical intermediate

work together to translate high-level driving strategies into executable vehicle behaviors.

  • The information-processing and execution layers supply relevant driving data to the VLM, then implement its requested behavior, steering, and speed adjustments through planned trajectories and motion-control tuning.The information vector includes perception results, safety-guaranteed trajectories, vehicle states, and human verbal commands.Source: §4.1.3 · p. 14

following the decision.

After the VLM decision, dedicated processors gather and structure driving information for its reasoning, then convert its actions into vehicle-specific inputs. Checks on trajectory feasibility and control-parameter ranges are part of execution.

following the decision.

  • Information processors assemble perception, planning, localization, and human-command inputs into the driving state supplied to the VLM.The inputs include detected objects, feasible waypoint trajectories, vehicle position and velocity, map information, and speech commands transcribed with Whisper.Source: following the decision · p. 14, processing · pp. 14–15

processing.

  • Execution processors translate VLM actions into vehicle-specific controls, checking trajectory feasibility and keeping adjusted control parameters within safety-guaranteed ranges.Shell scripts automate ROS parameter updates and trigger control algorithms.Source: processing · p. 15

architectures while maintaining a clear separation between the decision-making and execution processes.

The platform uses a modular autonomous-driving stack and conventional perception and localization methods alongside the VLM. In the case study, prerecorded trajectories support repeatable trials and controlled comparisons.

architectures while maintaining a clear separation between the decision-making and execution processes.

  • Autoware.AI provides a customizable, modular backbone whose components support independent development and testing.The authors cite its open-source flexibility, comprehensive software stack, and ROS-based architecture.Source: §4.1.4 · p. 15

enhancing scalability across different hardware configurations.

  • The implementation uses YOLOv5 for object detection and NDT scan-to-map matching for vehicle pose estimation, supplemented by GNSS and IMU sensors.Detection outputs comprise bounding boxes, class labels, and confidence scores.Source: enhancing scalability across different… · p. 16

scene interpretation and decision-making by the VLM.

  • Prerecorded waypoint trajectories represent different driving behaviors and keep planned paths consistent across repeated trials, supporting controlled comparisons.The authors state that this controls experimental variables for fair comparisons and validation.Source: scene interpretation and… · p. 16

of the VLM’s decision-making and scene understanding performance under identical conditions.

The platform combines decoupled vehicle control, configurable hardware, and real-world test settings with a layered information flow between driving modules and the VLM. Its setup covers highway, intersection, and parking-lot scenarios.

of the VLM’s decision-making and scene understanding performance under identical conditions.

4.1.5 Hardware Layer. The modification on the 2019 Lexus 450h is conducted by AutonomouStuff [3]; this autonomous

overview of the vehicle hardware setup is shown in Fig. 4.

  • The sensor suite includes LiDAR, radar, front-view cameras, and RTK-enabled GNSS with an IMU; these sensors provide raw data at 10 fps.Two webcams and built-in microphones capture driver behavior and verbal commands.Source: overview of the vehicle hardware setup… · p. 17

Mbps for uploads and 50-60 Mbps for downloads.

4.2 Real-World Testing Experiment Setup

  • Highway and intersection tests use a dedicated track for high-speed maneuvers and traffic interactions, while a parking-lot track assesses low-speed maneuvering in confined space.The three testing environments are a three-lane highway, a two-lane intersection, and a parking lot track.Source: §4.2 · p. 18

with more uncertainty.

  • The test cases cover highway speed maintenance, lane changes, slowdowns and merging; intersection stops, turns and pedestrian yielding; and parking-lot obstacle avoidance and parking.The parking-lot cases include parallel parking and backing into a spot.Source: with more uncertainty · p. 18
  • The middle layer combines perception, localization, planning, and passenger-command messages, updating at a minimum frequency of 10 Hz.The speech processor publishes a new command when detected and otherwise retains the previous command.Source: with more uncertainty · p. 19
  • The VLM generates actions in the top layer, which are executed through finalized planning and adjusted motion-control parameters; cloud requests are sent every three seconds.The experiments record computational resources, timing, and messages for validation and assessment.Source: with more uncertainty · pp. 18–19

parameters.

Highway and intersection scenarios test how the VLM agent responds to passenger commands under clear and adverse weather conditions. The reported actions pair urgency with maneuvering in clear conditions and caution with control adjustments in adverse conditions.

slowdowns in traffic, and merging onto the highway from an on-ramp.

4.2.2 Intersection. On the second trip, the vehicle is driving on a snowy highway with low visibility due to a blurred

  • On a snowy, low-visibility highway, the VLM agent maintains its lane and speed, keeps a safe distance, and adjusts control parameters to reduce skidding or loss of control.It decreases longitudinal PID proportional gain and increases steering weight to support stability.Source: Figure 6 · p. 19, On the second trip, the vehicle is… · pp. 20–21

conditions.

  • At a clear intersection without immediate obstacles, the urgent “I need to catch a flight” command leads the VLM agent to follow the current path while accelerating.The action is based on the vehicle state, driving behavior, and verbal command.Source: Figure 7 · p. 20, conditions · p. 21

appropriately to maintain traffic flow, considering potential safety measures.

  • At a snowy, foggy intersection, the “Keep safe” command leads the VLM agent to maintain a safe speed and following distance while adjusting PID and MPC parameters to avoid sudden maneuvers.The described purpose is to avoid maneuvers that could lead to loss of control.Source: Figure 7 · p. 20, appropriately to maintain traffic… · pp. 21–22

weight 𝑾𝒔to implement a smooth maneuver to keep the vehicle stable.

The parking-lot examples vary available space and passenger urgency while describing the VLM agent’s adjustments to path tracking and motion-control weights. Both emphasize navigating around obstacles safely while balancing efficiency.

4.2.3 Parking Lot. The parking lot scenario evaluated the vehicle’s ability to navigate in tight spaces, avoid obstacles,

commands and conditions.

  • In a crowded parking lot at 10 km/h, the hurried passenger’s request is balanced with caution through precise path tracking around potential obstacles.The motion-control setting uses a high lateral tracking-error weight.Source: commands and conditions · p. 22
  • In a similar 10 km/h setting with more free space, the VLM agent uses a slightly lower lateral-error weight for less precise path tracking and larger acceleration.The described objective remains cautious but efficient driving.Source: commands and conditions · p. 22

5 Testing and Validation

The platform is evaluated through module-level latency, correctness, and resource-use measures. Its validation reports high correctness and low latency across modules, while identifying higher speech-processing latency and the Vision Perception Aggregator’s greater resource use.

5.1 Evaluation Metrics

the information to the next layer.

  • The Speech Command Processor takes 0.5–1.9 seconds to process commands, whose durations vary with command length.The evaluation uses processing time per word for this module.Source: the information to the next layer · p. 23

memory, and 1% GPU.

  • The resource-use measure expresses module CPU and GPU time relative to the total available, alongside memory use.The evaluation defines CPU, GPU, and memory utilization for each module.Source: the information to the next layer · p. 23

computational resource utilization.

  • Most modules reach 100% correctness, while the Speech Command Processor reaches 91.78%, attributed to audio noise and speech-recognition tools.The paper says overall module accuracy remains high despite this shortfall.Source: computational resource utilization · p. 23
  • The Action Interface averages 2.48 × 10^-3 ms latency; the Prompt Generation Interface averages 5.84 ms, with a maximum below 20 ms.The paper reports short processing times across modules and identifies the Vision Perception Aggregator as relatively slower because it handles larger image volumes.Source: computational resource utilization · p. 23
  • End-to-end Speech Command Processor latency is around 167.95 ms, largely due to PyAudio and speech-recognition library overhead, but remains within stated real-time bounds.The paper says VLM decisions reach the Execution Layer with minimal delay.Source: computational resource utilization · p. 23

into the autonomous driving pipeline.

  • The Vision Perception Aggregator uses 3% CPU, 5% memory, and 1% GPU memory; other modules generally require fewer resources.The authors report that this resource use does not significantly affect VLM or overall driving performance.Source: into the autonomous driving pipeline · pp. 23–24

5.2 Validation Results

6 Conclusion and Future Work

The conclusion presents a real-world platform that connects a high-level VLM agent with a classical autonomy stack and supports validation across designed scenarios. Vehicle experiments report latency below 20 ms with high information-processing accuracy, while future work targets latency and accuracy improvements, especially for the Human Verbal Command Processor.

6 Conclusion and Future Work

  • The hierarchical framework connects a high-level VLM agent to the classical autonomous functionality stack for validation across designed driving scenarios.The paper describes a real-world case study and customizable closed-track traffic scenarios.Source: §6 · p. 24

and effectiveness of our approach.

  • Experiments across diverse scenarios report latency below 20 ms, high information-processing accuracy, and operation with limited computational resources.The platform was deployed on an actual vehicle using the open-source Autoware software stack.Source: and effectiveness of our approach · p. 24, §6 · p. 24
  • The results indicate potential for validation of VLM-integrated driving tasks spanning semantic reasoning, decision-making, planning, and control.The authors also identify potential relevance to academic research and industrial applications.Source: and effectiveness of our approach · p. 24
  • Future work should reduce latency and enhance accuracy, with particular attention to the Human Verbal Command Processor.The authors also suggest exploring on-board VLM testing integrated with end-to-end autonomous driving approaches.Source: and effectiveness of our approach · p. 24
Loading 2506.14100v1…