PaperlayerView exact revision on arXiv

Source-linked AI summary

OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision

Ruixun Liu, Lingyu Kong, Derun Li, Hang Zhao

arXiv:2509.05578v1cs.AIcs.RO

TL;DR

VLMs offer vision-language reasoning but lack robust 3D spatial understanding, while accessible representations and dense 3D supervision remain challenges. OccVLA predicts occupancy from 2D images as both output and supervision, reporting state-of-the-art nuScenes results in trajectory planning and 3D VQA.

  • Problem

    VLMs lack robust 3D spatial understanding for autonomous driving, and accessible 3D representations with dense supervision remain challenging to construct without extensive manual annotation.

    Source: §1 · p. 1

  • Method

    OccVLA integrates dense occupancy prediction from 2D images into a VLM, using occupancy as both an output and a supervisory signal.

    Source: §1 · p. 2

  • Results

    OccVLA reports state-of-the-art nuScenes results in trajectory planning and 3D VQA, with superior performance across multiple perception and planning tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Decoded occupancy provides interpretable, quantitatively evaluable outputs for fully vision-based autonomous driving.

    Source: §1 · p. 2

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown strong vision-language reasoning abilities but still lack robust 3D spatial understanding, which is critical for autonomous driving. This limitation stems from two key challenges: (1) the difficulty of constructing accessible yet effective 3D representations without expensive manual annotations, and (2) the loss of fine-grained spatial details in VLMs due to the absence of large-scale 3D vision-language pretraining. To address these challenges, we propose OccVLA, a novel framework that integrates 3D occupancy representations into a unified multimodal reasoning process. Unlike prior approaches that rely on explicit 3D inputs, OccVLA treats dense 3D occupancy as both a predictive output and a supervisory signal, enabling the model to learn fine-grained spatial structures directly from 2D visual inputs. The occupancy predictions are regarded as implicit reasoning processes and can be skipped during inference without performance degradation, thereby adding no extra computational overhead. OccVLA achieves state-of-the-art results on the nuScenes benchmark for trajectory planning and demonstrates superior performance on 3D visual question-answering tasks, offering a scalable, interpretable, and fully vision-based solution for autonomous driving.

1 INTRODUCTION

OccVLA addresses weak 3D grounding and sparse, costly annotation by predicting dense occupancy from 2D images as both an output and supervisory signal. It reports state-of-the-art nuScenes results for trajectory planning and 3D VQA, while occupancy prediction can be skipped at inference.

1 INTRODUCTION

  • VLMs lack robust 3D spatial grounding, while text-based 3D annotations are weak, sparse, and costly to produce.Adding explicit 3D inputs also leaves fine-grained spatial understanding constrained by limited large-scale 3D vision-language pretraining and detailed scene captions.Source: §1 · p. 1
  • OccVLA predicts dense 3D occupancy from 2D images as both an output and supervisory signal within a unified multimodal framework.Occupancy tokens receive visual features through cross-attention, and occupancy prediction extends the VLM while preserving 2D visual information.Source: §1 · p. 2, Figure 2 · p. 3
  • OccVLA uses only 2D visual inputs and can decode occupancy into interpretable, quantitatively evaluable outputs.The paper presents these properties as advantageous for fully vision-based autonomous driving.Source: §1 · p. 2
  • Cross-modal attention lets OccVLA skip occupancy prediction during inference without adding computational complexity.The occupancy prediction process is treated as implicit reasoning during training and can remain inactive at inference.Source: §1 · p. 2
  • OccVLA achieves state-of-the-art results on nuScenes for end-to-end trajectory planning and 3D VQA.The paper also reports superior performance across multiple perception and planning tasks.Source: §1 · p. 2

2 RELATED WORK

Related work examines VLMs for autonomous driving and occupancy-based 3D perception. OccVLA differs by using occupancy as both a model output and a dense supervision signal.

2.1 MLLMS IN AUTONOMOUS DRIVING

  • DriveVLM relies on high-quality annotated datasets, EMMA requires substantial manual labeling, and OmniDrive requires additional 3D sensors and processes large 3D inputs.OccVLA instead uses autoannotation occupancy data for dense 3D supervision.Source: §2.1 · p. 3

2.2 OCCUPANCY FOR 3D PERCEPTION

  • Unlike approaches that feed occupancy into LLMs or use it to predict future occupancy and actions, OccVLA uses occupancy as both output and supervision.The paper motivates this design in part by the risk that captions omit critical 3D information.Source: §2.2 · p. 3

3 METHOD

OccVLA combines 3D occupancy prediction and future ego-motion planning in a unified VLM framework, with an independent planning head and a three-stage training process.

3.1 OVERVIEW

  • OccVLA uses an independent planning head and a three-stage training process to combine occupancy prediction with future ego-motion planning.The overview identifies occupancy prediction and planning as core components.Source: §3.1 · p. 4

3.2 OCCUPANCY PREDICTION

  • The VLM first predicts occupancy from visual input before next-token prediction, integrating visual and textual information during perception.Occupancy tokens access VLM visual features through cross-attention; the occupancy branch maps compact latent features through a VQ-VAE decoder and classification head to occupancy predictions.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 MOTION PLANNING

  • Meta actions encode short-term driving intent through separate velocity and directional categories, giving the VLM a discrete, language-friendly space for motion reasoning.Velocity categories are Maintain speed, Accelerate, and Decelerate; directional categories include straight, turns, lane changes, and stop.Source: §3.3 · p. 5
  • CoT supervision has the VLM describe a six-view scene, infer driver intent from past meta actions, and then predict a future meta action.The automated data pipeline derives velocity labels from acceleration thresholds and classifies lane-level future trajectories for directional labels; all nuScenes meta actions are inspected, with about 20% further refined.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 TRAINING STAGE

  • The training pipeline first fine-tunes the VLM on autonomous-driving visual and text data, then jointly trains occupancy and language, and finally trains the planning head to predict coordinates from meta actions.The occupancy-language stage combines autoregressive language loss with a non-autoregressive occupancy loss computed against ground-truth occupancy labels.Source: Figure 4 · p. 6, §3.4 · p. 6

3.3 MOTION PLANNING

  • A lightweight planning head converts predicted meta actions into future coordinates, using their embeddings, prior velocity, and VLM visual tokens to predict positions for the next 3 seconds.No high-level navigation commands are provided, so planning decisions emerge from the model’s scene understanding.Source: §3.3 · p. 6

3.4 TRAINING STAGE

  • Occupancy-language joint training supervises occupancy and language together, using occupancy categories as the 3D perception target alongside autoregressive language modeling.The authors report that direct alignment of latent features is suboptimal because of biases introduced by VQ-VAE encoding.Source: §3.4 · p. 6

4 EXPERIMENT

Across nuScenes experiments, OccVLA reports strong camera-based planning and 3D question-answering results, while occupancy prediction supports object-level scene understanding and occupancy supervision improves meta-action prediction.

4.1 EXPERIMENT SETTINGS

  • Experiments use nuScenes, evaluate localization, object querying, and relational comparison, and train with PaliGemma2-3B-224px on eight NVIDIA A800 GPUs.The dataset has 700 training scenes and 150 validation scenes; the model is trained with AdamW.Source: §4.1 · pp. 6–7

4.2 RESULTS AND ANALYSIS

  • OccVLA achieves state-of-the-art average L2 distance for motion planning, competitive planning within 3 seconds, and outperforms larger 3D-input models on NuScenes-QA overall accuracy.The planning comparison covers different inputs and supervision; the QA result uses an image-only 3B model.Source: Table 1 · p. 7, §4.2 · pp. 7–8, Table 3 · p. 8
  • A 3B OccVLA model also achieves better performance than OccLLaMA (7B) while using camera input and integrating occupancy prediction into the LLM.OccVLA is also reported to perform competitively against methods using ground-truth occupancy as input.Source: §4.2 · p. 7, Table 2 · p. 8
  • About 10% mIoU is achieved for current-time 3D occupancy prediction, with strong predictions for lanes, vehicles, pedestrians, and nearby object details.Without multi-timestamp images, the model is limited on occluded regions such as buildings hidden behind trees; multi-view current-time images still support object-level scene understanding.Source: §4.2 · p. 8, Figure 5 · p. 9

4.3 ABLATION STUDY

  • Approximately 1.5% improvement in meta-action prediction accompanies occupancy supervision, which the authors attribute to strengthened 3D priors in LLM visual features.The ablation compares joint occupancy–vision–language learning with the original VLM without occupancy integration.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Without past ego trajectories, OccVLA retains competitive planning performance and predicts actions from scene understanding without high-level navigation instructions.The study notes prior concerns that ego trajectories may introduce excessive priors and risk dataset overfitting.Source: §4.3 · p. 9, Table 5 · p. 9

5 CONCLUSION

OccVLA jointly learns occupancy and vision-language representations in a parallel latent-space LLM architecture, without additional 3D inputs. It can bypass occupancy prediction at inference to address delays associated with large 3D VLM models.

5 CONCLUSION

  • OccVLA jointly learns occupancy and vision-language representations in a parallel latent-space LLM architecture, using pretrained 2D knowledge without additional 3D inputs.Occupancy prediction can be bypassed during inference to address delays associated with the large parameter counts of prior 3D VLM-based driving models.Source: §5 · p. 9

Related papers

Loading 2509.05578v1…