Source-linked AI summary
Vega: Learning to Drive with Natural Language Instructions
Sicheng Zuo, Yuxuan Li, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu
TL;DR
Existing VLA driving systems generally use language for descriptions or reasoning rather than flexible user instructions. Vega introduces the instruction-annotated InstructScene dataset and a unified vision-language-world-action model that jointly plans actions and predicts future images, achieving strong planning and instruction-following performance.
Problem
Existing VLA models mainly use language for scene descriptions or reasoning and lack flexible open-ended instruction-following capabilities for personalized driving.
Method
Vega combines the InstructScene dataset of around 100,000 instruction-annotated scenes with a unified autoregressive-diffusion model for instruction-conditioned action planning and future-image generation.
Results
Vega achieves strong instruction-following capabilities and competitive or state-of-the-art planning performance across NAVSIM benchmarks, including 86.9 EPDMS on NAVSIM v2 and 87.9 PDMS on NAVSIM v1.
Takeaways & Limitations
Joint future-image prediction and action planning provides dense supervision and supports instruction-compliant visual generation and driving actions.
Abstract
from arXiv · showhide
Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.
1. Introduction
Existing vision-language-action driving models generalize across scenarios but mainly use language for descriptions or reasoning, limiting flexible instruction following. Vega addresses this gap with InstructScene and a unified model that jointly generates future images and plans actions.
- Motivation: Existing VLA models mainly use language for scene description or reasoning and struggle with open-ended driving instructions.Many imitate an averaged expert policy or support only simple commands such as “turn left” or “go straight.”
- Dataset: InstructScene contains around 100,000 instruction-annotated driving scenes with corresponding trajectories.The dataset is built on NAVSIM to support instructional driving.
- Motivation: Direct VLA training struggles with feasible trajectories and accurate instruction following because visual-instruction inputs greatly exceed low-dimensional action supervision.The paper identifies this information disparity as a challenge for learning a generalizable mapping from instructions to actions.
- Approach: Vega jointly performs future image generation and action planning conditioned on past observations and language instructions.Future image prediction supplies dense, pixel-level supervision for relationships among instructions, actions, and visual predictions.
- Approach: Vega uses autoregressive processing for visual and instruction understanding, diffusion for image and action generation, joint attention, and a Mixture-of-Transformers design.These components unify vision-language understanding, world modeling, and action planning.
2. Related Work
Related work progresses from VLM-based scene reasoning and textual waypoints toward VLA and world-model approaches. Unified visual understanding and generation methods differ in how autoregressive and diffusion components are connected, with integrated transformers addressing information-flow limitations.
- Vision-language driving: Early autonomous-driving VLMs supported high-level scene understanding and reasoning but could not output drivable trajectories.Later methods predicted textual waypoints, but precise numerical reasoning remained difficult for LLMs.
- World models: World models predict future states from past observations and current actions, with driving applications spanning image-based, occupancy-based, and VLA-based methods.Image-based methods focus mainly on video synthesis for data generation and scene simulation, while occupancy-based methods model 3D scene evolution.
- Unified generation: Quantized autoregressive, external diffusion, and integrated transformer pipelines are the main approaches to unified visual understanding and generation.Quantized autoregressive methods generate discrete visual tokens, external diffusers use VLM-produced latent conditioning, and integrated transformers combine both mechanisms.
- Unified generation: External diffusion can restrict information flow through a narrow interface between the VLM and generation module, while autoregressive visual generation often has lower image quality and higher latency.Integrated transformers merge autoregressive and diffusion mechanisms within a single transformer.
3. Proposed Approach
Vega extends driving models from imitation toward instruction-conditioned planning by combining multimodal sequence modeling with future-image generation. Its unified autoregressive-diffusion architecture shares information across text, image, and action latents while jointly training planning and world modeling.
- From imitation to instruction: Conventional driving pipelines use perception, prediction, and planning modules, whereas VLA models map visual observations and historical actions to scene descriptions and ego actions.The conventional pipeline extracts scene representations, forecasts agent motion, and plans the ego action; both approaches are described with formal mappings.
- From imitation to instruction: Existing VLA models lack flexible instruction following, motivating a model that predicts ego actions from observations, historical actions, and user instructions.The instruction-based formulation is expressed as At = V([It−T, ..., It], [At−T, ..., At−1], Lt).
- Dataset and formulation: InstructScene provides around 100,000 instruction-annotated scenes, with instructions generated from future observations and actions using a VLM supplemented by rules.Each timestep yields image, instruction, and action triplets for training instructional driving.
- Unified model: Vega jointly learns instruction-based action planning and future image generation to address sparse action supervision and model the causal chain from instruction to action to visual outcome.The generative model receives current and historical observations with instructions, then predicts an action and the resulting future observation.
- Unified model: Figure 2 contrasts traditional single-expert trajectory prediction with Vega’s instruction-conditioned generation of diverse trajectories and future images.The figure’s key comparison is whether language can control both planning diversity and predicted visual futures.
- Joint autoregressive-diffusion architecture: Vega combines autoregressive multimodal understanding with diffusion generation in an integrated transformer rather than relying on discrete visual tokens or an external diffuser.The architecture addresses visual-generation quality, sequential latency, and the information bottleneck of limited latent-token interfaces.
- Joint autoregressive-diffusion architecture: The model concatenates encoded text, images, actions, and target noise into one sequence, applies causal attention across modalities, and decodes denoised latents into images or actions.This design enables information flow among text, image, and action representations.
- Input construction: Inputs include tokenized instructions, VAE and ViT visual latents, and normalized relative trajectory movements.Historical images and actions precede instructions, followed by noisy action or future-image targets depending on the task.
4. Experiments
Vega is evaluated on NAVSIM benchmarks, dataset construction, main results, ablations, and qualitative instruction-following behavior. Results show competitive planning, benefits from future-frame prediction and an independent action expert, and trajectories aligned with natural-language instructions.
- Dataset and evaluation: The InstructScene dataset contains around 100,000 instruction-annotated scenes, with instructions generated through visual understanding and rule-based supplementation.The annotation pipeline uses consecutive front-view frames and combines scene descriptions with speed, acceleration, and turn-rate rules.
- Main results: 86.9 EPDMS on NAVSIM v2 is comparable to state of the art, while best-of-N achieves top performance on several compliance and comfort metrics.The reported metrics include Driving Direction Compliance, Traffic Light Compliance, Lane Keeping, and History Comfort.
- Main results: 87.9 PDMS on NAVSIM v1 matches multimodal BEV methods, improving to 89.8 with best-of-N, although Vega trails state-of-the-art VLA methods on this benchmark.The paper attributes the discrepancy partly to NAVSIM v1’s imbalanced metrics and notes that competing enhancements can be added modularly.
- Ablations: Future-frame prediction improves planning, while the choice among future-frame targets has limited impact when other training settings are held constant.The ablation compares always predicting the next frame, randomly selecting one of eight future frames, and removing future prediction.
- Ablations: Interleaving image-action sequences converges faster and eventually achieves lower training loss than noninterleaved training despite higher initial loss.The experiment varies interleaved sequence lengths of 2, 4, and 6.
- Ablations: The independent action expert slightly outperforms a VLM-module planner and significantly surpasses a diffusion-module planner, supporting the proposed architecture.The alternatives reduce model size but increase computational cost because their modules operate at higher dimensionality.
- Qualitative analysis: Qualitative examples show Vega adjusts speed and jointly generates actions and future images consistently with varied user instructions.The visualizations include increased, decreased, or maintained speed and scenarios with multiple possible courses of action.
- Baseline analysis: A VLA baseline trained on the same instruction dataset achieves only approximately 60 PDMS and often fails to generate instruction-consistent trajectories.The paper attributes this limitation to sparse, low-dimensional action supervision relative to the high-dimensional visual-language inputs.
5. Conclusion
Vega addresses diverse-instruction driving by jointly generating instruction-compliant future images and planning actions. Built with instruction annotations and integrated modeling, it achieves SOTA planning performance and strong instruction-following in visual generation and action planning.
- In the illustrated scenario, three instruction sets produce three action sequences and corresponding future images consistent with those actions.
- Vega jointly generates instruction-compliant future images and plans actions for instructional driving.The joint task uses future visual generation as dense supervision for action planning.
- An instruction-annotated driving dataset enables training for instructional driving.The dataset contains around 100,000 instruction-annotated scenes with corresponding trajectories.
- The model learns relationships among instructions, actions, and visual outcomes through joint generation and planning.
- Vega achieves SOTA planning performance while demonstrating strong instruction-following in visual generation and action planning.