PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving

Yuan Gao, Dengyuan Hua, Mattia Piccinini, Finn Rasmus Schäfer, Korbinian Moller, Lin Li, Johannes Betz

arXiv:2603.09482v1cs.RO

TL;DR

Existing VLA systems often generate generic trajectories without diverse style control or explicit kinematic supervision, limiting personalized and physically plausible autonomous driving. StyleVLA combines a style-labeled BEV/FPV instruction dataset with physics-informed fine-tuning, and reports higher composite driving scores than Gemini-3-Pro on both domains.

  • Problem

    Existing VLA models primarily generate generic trajectories, lack user-controlled driving-style conditioning, and often omit explicit vehicle-kinematic supervision.

  • Method

    StyleVLA fine-tunes a Qwen3-VL-4B model using a dataset of five driving styles and a hybrid loss combining CE, continuous regression, and kinematic consistency.

  • Results

    StyleVLA achieves composite driving scores of 0.55 on BEV and 0.51 on FPV, compared with 0.32 and 0.35 for Gemini-3-Pro.

  • Takeaways & Limitations

    A specialized, physics-informed 4B-parameter model can outperform proprietary and state-of-the-art VLA models on style-aware trajectory generation.

Abstract

from arXiv · show

Vision Language Models (VLMs) bridge visual perception and linguistic reasoning. In Autonomous Driving (AD), this synergy has enabled Vision Language Action (VLA) models, which translate high-level multimodal understanding into driving behaviors, typically represented as future trajectories. However, existing VLA models mainly generate generic collision-free trajectories. Beyond collision avoidance, adapting to diverse driving styles (e.g., sporty, comfortable) is essential for personalized driving. Moreover, many methods treat trajectory generation as naive token prediction, which can produce kinematically infeasible actions. To address these limitations, we present StyleVLA, a physics-informed VLA framework for generating diverse and physically plausible driving behaviors. We introduce a hybrid loss that combines a kinematic consistency constraint with a continuous regression head to improve trajectory feasibility. To train StyleVLA, built on Qwen3-VL-4B, we construct a large-scale instruction dataset with over 1.2k scenarios, 76k Bird's Eye View (BEV) samples, and 42k First Person View (FPV) samples, with ground-truth trajectories for five driving styles and natural-language instructions. Experiments show that our 4B-parameter StyleVLA significantly outperforms proprietary models (e.g., Gemini-3-Pro) and state-of-the-art VLA models. Using a composite driving score measuring success rate, physical feasibility, and style adherence, StyleVLA achieves 0.55 on BEV and 0.51 on FPV, versus 0.32 and 0.35 for Gemini-3-Pro. These results show that a specialized, physics-informed, lightweight model can surpass closed-source models on domain-specific tasks.

I. INTRODUCTION

VLA models extend multimodal reasoning to autonomous-driving trajectory generation, but existing systems largely produce generic collision-free behavior. StyleVLA addresses this gap by generating physics-informed trajectories conditioned on diverse user-specified driving styles.

  • StyleVLA: StyleVLA generates trajectories for Default, Balanced, Comfort, Sporty, and Safety styles in response to user instructions.The framework is designed to adapt driving behavior to specified style preferences.
  • Background: VLM-based VLA models use multimodal reasoning to map sensor inputs toward trajectory-based driving behaviors.They are presented as an evolution beyond traditional rule-based and classical end-to-end systems.
  • Motivation: Existing VLA models prioritize collision avoidance while neglecting heterogeneous human driving preferences such as sporty or comfort-oriented behavior.The paper links this limitation to the lack of large-scale datasets containing ground-truth trajectories for diverse styles.
  • Related Work: Traditional end-to-end driving architectures face challenges with rare long-tail scenarios and diverse driving behavior, motivating VLA-based approaches.The cited discussion frames VLA models as integrating vision-language reasoning into end-to-end driving.

3) Autonomous Driving Datasets:

Existing autonomous-driving datasets provide rich multimodal data but generally lack explicit, diverse driving-style supervision and physics-informed trajectory constraints. StyleVLA responds with a style-labeled instruction dataset and a physics-informed fine-tuning framework evaluated across BEV and FPV domains.

  • Autonomous Driving Datasets: Existing autonomous-driving datasets contain multimodal sensor data and ego trajectories but lack explicit annotations capturing diverse driving-style distributions.This limits research on personalized autonomous driving, including cautious and sporty behavior.
  • Critical Summary: Existing VLA models generally lack mechanisms to condition trajectory generation on user-specified styles, producing generic driving behaviors.The paper attributes this limitation to training on homogeneous driving data.
  • Critical Summary: Many VLA methods treat trajectory generation as token prediction or rely on external decoders without explicitly modeling vehicle kinematic constraints.The paper identifies this as a lack of physics-informed trajectory supervision.
  • Contributions: 1,216 scenarios, 76,030 BEV samples, and 42,084 FPV samples provide five driving styles and language instructions for StyleVLA training and evaluation.The styles are Default, Balanced, Comfort, Sporty, and Safety.
  • Contributions: StyleVLA combines CE loss, an MLP regression head, and a physics-informed kinematic loss when fine-tuning a 4B VLM.The framework targets style-aware trajectories with improved feasibility and style adherence on unseen data.
  • Evaluation: The StyleVLA dataset supports large-scale evaluation of off-the-shelf VLMs and state-of-the-art VLA methods across BEV and FPV domains.The evaluation is intended to reveal limitations in style-aware trajectory generation.

II. METHODOLOGY

StyleVLA constructs style-specific trajectory data with a multi-objective motion planner, filters it for stylistic conformance, and uses the resulting corpus to support physics-informed VLA training.

  • Trajectory Generation: The methodology generates style-specific ground-truth trajectories with Frenetix by sampling candidate end states and selecting feasible trajectories under style-conditioned costs.Candidates are checked against acceleration, curvature, and yaw-rate bounds before cost-based selection.
  • Style Conditioning: Comfort, Sporty, Safety, Balanced, and Default modes use different cost priorities to represent comfort, progress, safety margins, trade-offs, and a baseline.Comfort penalizes jerk, Sporty penalizes deviation from desired velocity, and Safety emphasizes larger obstacle buffers.
  • Scenario Corpus: The corpus contains 1,484 scenarios from 14 countries, 53,457 dynamic agents, 11.2 hours of data, and 116,400 planning instances before filtering.Each planning instance contains a BEV image and its planner-generated ground-truth trajectory.
  • Dataset Filtering: Each trajectory is summarized by six aggregate kinematic statistics, including velocity, acceleration, and jerk measures, for style-distribution filtering.The feature vector contains mean and standard-deviation velocity, RMS and peak acceleration, RMS jerk, and jerk variability.
  • Dataset Filtering: Samples are retained when their conformance score exceeds 80, producing 76,030 planning instances across 1,216 scenarios.Style-specific Gaussian distributions are fitted with the Minimum Covariance Determinant estimator, and conformance is assessed using Mahalanobis distance mapped through a chi-squared CDF.

B. Instruction Dataset Generation (BEV Domain)

The BEV instruction dataset pairs visual scenario context with language instructions, historical state information, and supervised trajectory responses in a LLaVA-style conversation format.

  • BEV Instruction Dataset: Raw kinematic data are transformed into multimodal instruction samples by pairing visual context, linguistic instructions, historical state data, and ground-truth trajectories.The dataset follows the LLaVA VQA structure of image, human instruction, and model response.
  • BEV Instruction Dataset: The model response provides a supervised learning target that connects the visual and linguistic inputs to trajectory generation.
  • BEV Instruction Dataset: Each BEV sample uses a single CR-environment image as its primary spatial context within the conversation format.

C. Instruction Dataset Generation (FPV Domain)

The FPV dataset extends the BEV setup with camera imagery to support realistic 3D end-to-end driving scenarios.

  • FPV Instruction Dataset: StyleVLA extends BEV images with FPV camera images through CARLA simulation for realistic end-to-end driving.
  • FPV Instruction Dataset: The FPV domain provides a 3D visual setting for trajectory generation alongside the original BEV domain.
  • FPV Instruction Dataset: CARLA is used to generate the FPV extension of the instruction dataset.

1) Map Conversion and Scenario Replay:

The 3D data pipeline converts maps and replays style-conditioned trajectories in CARLA, validates the rendered scenarios, and trains StyleVLA with physics-informed supervision.

  • Map Conversion and Scenario Replay: CR maps are converted to OpenDRIVE while preserving lane topology, junctions, signs, and traffic lights for CARLA compatibility.
  • Map Conversion and Scenario Replay: Each scenario is resimulated in CARLA with synchronized FPV video while ego and traffic vehicles follow replayed ground-truth trajectories.CR states are transformed into CARLA positions and orientations, with vehicle models matched by dimensions.
  • Map Conversion and Scenario Replay: A two-stage validation process removes rendering and map-conversion failures, then excludes scenarios with ambiguous visuals or incomplete traffic spawning.The resulting dataset contains 42,084 high-quality successfully replayed instances.
  • Map Conversion and Scenario Replay: FPV instructions omit external traffic states and retain only ego history, a goal point, and a style command to enforce vision-based driving.
  • Fine-Tuning Architecture: The framework fine-tunes Qwen3-VL-4B with QLoRA and uses a physics-informed hybrid loss to jointly improve geometric accuracy and kinematic plausibility.

1) Hybrid Loss Function Design:

StyleVLA replaces purely token-level trajectory prediction with a hybrid objective that combines semantic cross-entropy, continuous regression, and physics-informed kinematic consistency.

  • Standard VLMs formulate trajectory generation as next-token prediction, which can discretize continuous states and introduce quantization error.
  • An auxiliary MLP regression head projects pooled response embeddings into continuous kinematic states and minimizes geometric error against ground truth.
  • The hybrid objective balances discrete CE and continuous regression losses using learnable log-variance parameters that adaptively rescale their contributions.
  • Physics-Informed Kinematic Consistency (PIKC): The PIKC loss compares directly predicted next positions with positions extrapolated from the predicted state using discrete kinematic equations.
  • Physics-Informed Kinematic Consistency (PIKC): The final regression objective combines direct regression with a kinematic penalty, assigning higher weights to position and kinematic terms while using velocity and heading as auxiliary guides.

III. RESULTS & DISCUSSION

The results section reports the experimental setup hardware used for StyleVLA evaluation.

  • All experiments run on a Dell Alienware R15 with an Intel i7-13700KF CPU, NVIDIA RTX 4090 GPU with 24GB VRAM, and 128GB RAM.

1) Evaluated Vision Language Models:

StyleVLA evaluation uses diverse proprietary and open-source VLMs and combines trajectory accuracy, planning success, physical consistency, and inference speed metrics into a unified score.

  • Evaluated Vision Language Models: The evaluation compares Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 3 Pro, GPT 5 Nano, Qwen3-VL-4B, Qwen2.5-VL-7B, and InternVL3-9B.
  • Evaluation metrics: ADE and FDE measure mean and final Euclidean displacement from the ground-truth 2D position sequence.
  • Evaluation metrics: PSR is the percentage of trajectories with ADE < 1.0 m, while MR captures failures with FDE > 2.0 m.
  • Evaluation metrics: KCE quantifies physical violations as the discrepancy between the output position at t + 1 and the kinematically predicted position.
  • Evaluation metrics: Sfinal ∈[0, 1] combines success, reachability, accuracy, and kinematic scores, prioritizing safety and success over raw precision.

B. Experiment 1: Fine-tuning StyleVLA on BEV Domain

BEV experiments show that larger training subsets improve baseline generalization and that adding regression and kinematic consistency losses yields progressive gains in trajectory quality and planning success.

  • Data Scaling Analysis: The standard 50k-sample dataset achieves the lowest ADE at 1.17 m and highest PSR at 33.19%, outperforming the small subset’s 2.08 m ADE and 20.60% PSR.
  • Ablation Study: The ablation compares CE, CE + REG, and CE + REG + PIKC configurations on Qwen2.5-VL-7B trained with the standard dataset.
  • Qualitative Comparison: Figure 4 visualizes predicted trajectories against goals and ground truth across Default, Balanced, Comfort, Sporty, and Safety styles, with asterisks marking failed generations.
  • Ablation Study: Adding the MLP regression head reduces FDE by 0.65 m, from 3.82 m to 3.17 m, and increases PSR by 3.08%.
  • Ablation Study: Adding kinematic consistency further improves ADE from 1.21 m to 1.17 m, FDE from 3.17 m to 3.06 m, PSR by 1.11%, and Heading MAE from 0.036 rad to 0.035 rad.

2) Benchmarking across VLMs:

Benchmarking shows that fine-tuning lightweight VLMs with StyleVLA substantially improves driving-style trajectory generation over zero-shot and baseline models across BEV and FPV settings.

  • BEV benchmarking: 39.47 % success rate is achieved by fine-tuned Qwen3-VL-4B, compared with 16.38 % for the best closed-source model in BEV evaluation.Its inference time remains 1.92 s, supporting online-capable operation.
  • BEV benchmarking: 1.92 s inference is faster for fine-tuned Qwen3-VL-4B than Qwen2.5-VL-7B at 3.70 s, while its Score is higher at 0.55 versus 0.46.The comparison links smaller model size and quantization with improved efficiency in this benchmark.
  • FPV benchmarking: 91.39 s latency limits Gemini-3-Pro despite its best zero-shot FPV performance among proprietary models.SOTA methods also lack velocity or acceleration outputs, preventing kinematic consistency evaluation.
  • FPV benchmarking: 0.51 Score, 38.60 % PSR, and 36.90 % MR are achieved by the fine-tuned FPV agent, surpassing proprietary and SOTA baselines.Its 2.13 s inference time is slightly higher than the BEV model’s 1.92 s because FPV requires implicit obstacle perception from raw images.
  • Dataset and training: 1.2k scenarios, 76k BEV samples, and 42k FPV samples compose the StyleVLA instruction dataset for five driving styles.The framework uses BEV and FPV contexts and fine-tunes VLA models with a physics-informed hybrid loss.
Loading 2603.09482v1…