PaperlayerView exact revision on arXiv

Source-linked AI summary

A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving

Jingtao Sun, Xiaohai He, Yike Zhang, Dong Huang, Yaonan Wang, Ajmal Mian, Mike Zheng Shou

arXiv:2608.20890v1cs.CVcs.RO

TL;DR

Existing VLA driving systems often formulate autonomous driving as visual question answering and lack effective interaction across heterogeneous modalities, limiting reliable reasoning in complex and long-tail scenarios. The paper combines affinity-guided optimal transport, distribution-consistent modality transfer, and multi-trajectory planning with perception-oriented refinement. Experiments report improved driving reliability across challenging scenarios, including closed-loop gains over VLR-Driver and competitive inference efficiency.

  • Problem

    VLM/VLA driving methods face unreliable visual-question-answering-based decisions, heterogeneous-modal conflicts, and limited explicit safety-aware mechanisms for long-tail scenarios.

  • Method

    The framework uses Affinity-Guided Optimal Transport, Distribution-Consistent Modality Transfer, and multi-modal multi-trajectory planning with perception-oriented trajectory refinement.

  • Results

    The method improves driving decision reliability in challenging long-tail scenarios and achieves state-of-the-art closed-loop performance, surpassing VLR-Driver across the reported core metrics.

  • Takeaways & Limitations

    Structured multimodal interaction combined with online trajectory optimization supports interpretable and reliable VLA-based autonomous driving while maintaining competitive inference efficiency.

  • Takeaways & Limitations

    Optimal-transport interaction adds computational overhead, while trajectory refinement may be sensitive to perception errors.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for end-to-end autonomous driving by jointly integrating perception, reasoning, and decision making within a unified multimodal framework. However, most existing VLA models formulate end-to-end autonomous driving as a visual question answering task, leading to unreliable and less interpretable decision reasoning. In addition, they fail to establish effective multi-modal interaction across heterogeneous sensors, thereby limiting robust scene perception and reliable driving reasoning in long-tail driving scenarios. To this end, we propose a robust VLA-based end-to-end autonomous driving system that combines multi-modality interaction with multi-trajectory planning and optimization, enabling more reliable, interpretable, and safer driving decisions. Our method comprises three core components: (1) Affinity-Guided Optimal Transport for main-auxiliary modality two-way interaction; (2) Distribution-Consistent Modality Transfer for heterogeneous modality distribution transfer and cross-modal interaction; (3) Multi-modal Multi-Trajectory Planning along with Perception-Oriented Trajectory Refinement for better driving decisions to long-tail driving scenarios. Experimental results in open-loop and closed-loop datasets demonstrate improvements in safety long-horizon driving reasoning and road scene perception over existing driving systems, highlighting the ability of our mutli-modality interaction and multi-trajectory planning and optimization for scalable VLA-based systems.

1 INTRODUCTION

Existing VLA-based E2E-AD systems face unreliable reasoning, cross-modal heterogeneity, and insufficiently explicit safety-aware decisions. The paper addresses these gaps through structured modality interaction and trajectory optimization for long-tail driving.

  • Approach: The proposed system unifies scene perception and motion planning using camera, LiDAR, and natural-language inputs for causal reasoning and ego-vehicle trajectory planning.The framework is designed to satisfy efficiency and safety requirements despite heterogeneous multimodal inputs and limited 3D understanding.
  • Motivation: Existing fusion methods often combine modalities into a single hybrid BEV cue, while early and late fusion use heuristic strategies lacking structured semantics.These limitations complicate consistent scene embeddings from misaligned RGB images, LiDAR point clouds, and language instructions.
  • Approach: Affinity-Guided Optimal Transport learns pairwise modality affinities and enables two-way interaction between main and auxiliary modalities.The method adaptively learns interaction relationships rather than relying on a single fused representation.
  • Approach: Distribution-Consistent Modality Transfer maps heterogeneous modality distributions into a unified Gaussian space for cross-modal interaction.It uses a semantic-aligned normalizing flow to maintain modal distribution consistency.
  • Approach: Multi-modal Multi-Trajectory Planning and Perception-Oriented Trajectory Refinement use risk-aware optimization to improve decisions in long-tail driving scenarios.The refinement constructs a driving risk cost map and directly optimizes generated trajectories.

2 RELATED WORK

Related work spans multimodal fusion, end-to-end trajectory planning, VLM/VLA driving, and optimal transport. The paper positions semantic-aware transport and online trajectory optimization as alternatives to loosely coupled fusion and question-answering-based decisions.

  • Multi-modality Fusion: Multimodal fusion methods combine RGB imagery with depth or 3D semantics, but many rely on loosely coupled mechanisms that limit cross-modal interaction.Early, late, and feature-level fusion provide intermediate representations but do not establish the paper’s semantic-aware transport interaction.
  • End-to-End Autonomous Driving: Traditional E2E-AD systems map sensory observations to trajectories or controls, while newer systems model perception tasks and compact scene representations for planning.The cited trajectory-planning systems include approaches using single-trajectory planning and vectorized representations.
  • End-to-End Autonomous Driving: Existing E2E-AD systems mainly learn implicit mappings from sensors to controls, limiting natural-language interaction with humans.The paper integrates multimodal interaction with trajectory planning and optimization to target more reliable driving decisions.
  • VLM/VLA-based Autonomous Driving: VLM/VLA driving methods map egocentric video and language instructions to causal reasoning and ego-motion planning, but often formulate driving as visual question answering.The paper identifies missing explicit mechanisms for reliable motion decisions and addresses them through multimodal interaction and online trajectory optimization.
  • Optimal Transport: Optimal transport measures distribution discrepancy by modeling probability-mass transportation and minimizing total transportation cost.This paper applies discrete optimal transport to transfer knowledge between heterogeneous modalities for E2E-AD interaction.

3 PRELIMINARIES

The paper formulates multimodal E2E-AD as a multitask problem combining perception pretraining with language-guided trajectory prediction. It introduces optimal transport as the mathematical basis for aligning heterogeneous feature distributions.

  • Problem & Task Definition: The framework jointly addresses road-scene perception and ego-planning, including 3D object detection and BEV map segmentation as auxiliary tasks.This extends traditional sensor-to-trajectory E2E-AD toward multitask learning.
  • Problem & Task Definition: A two-stage objective first trains multimodal interaction for 3D detection and BEV segmentation, then fine-tunes trajectory prediction under language navigation guidance.The predicted ego-vehicle trajectory contains 2D waypoints over a planning horizon.
  • Optimal Transport: Optimal transport estimates a globally optimal correlation between probability distributions by transporting probability mass while minimizing total cost.In the Kantorovich relaxation, the optimization searches over joint probability distributions and transport plans.
  • Discrete Optimal Transport: Discrete optimal transport represents distributions with finite feature-token sets, a cost matrix, and a nonnegative transport-plan matrix.Entropy regularization smooths the transport matrix and converts the objective into a strictly convex optimization problem.
  • Discrete Optimal Transport: The regularized discrete optimal-transport problem admits a closed-form solution.This solution follows the entropy-regularized formulation and transport constraints described in the preliminaries.

4 METHODOLOGY

The methodology integrates affinity-guided optimal transport, distribution-consistent modality transfer, and multi-trajectory planning with perception-oriented refinement. These components align heterogeneous modalities, support cross-modal interaction, and refine candidate trajectories using interpretable driving-risk objectives.

  • LiDAR and Camera Interactive Fusion: Affinity-guided optimal transport computes modality similarities and globally aligns visual and LiDAR tokens through entropy-regularized transport.The learned affinity defines transportation costs; Sinkhorn optimization obtains the transport plan, which aggregates two-way cross-modal interactions.
  • LiDAR and Camera Interactive Fusion: A positive-semidefinite affinity kernel regularization encourages the model to learn interactive relations between main-modality and auxiliary-modality features.The regularizer promotes non-negative eigenvalues in the adaptive similarity matrix.
  • Navigation Prompts Interactive Fusion: Distribution-consistent modality transfer maps camera, LiDAR, and language tokens into a common latent Gaussian manifold before cross-modal interaction.A shared invertible normalizing flow performs the transfer, while likelihood optimization accounts for the Jacobian of the transformation.
  • Navigation Prompts Interactive Fusion: Cross-modal distribution consistency reduces Sinkhorn distances among transformed camera, LiDAR, and language distributions, while affinity-guided transport adds semantic-aware token alignment.The distribution loss addresses discrepancy between modalities; transport then transfers semantic information to camera and LiDAR representations.
  • Multi-Modal Multi-Trajectory Planning: Multi-modal planning generates candidate trajectories from visual, BEV, language, and ego-state tokens, then selects among them after risk-based refinement.The refinement uses real-time 3D detection and BEV segmentation to construct an interpretable driving-risk cost map and directly optimize trajectories without additional policy training.

5 EXPERIMENTS

Across open-loop and closed-loop evaluations, the proposed multimodal interaction and multi-trajectory optimization framework improves driving performance, scene perception, and trajectory refinement. Results also show strong transfer across datasets, interaction designs, VLA foundations, and inference settings.

  • Open-Loop Planning: 92.2 PDMS on NAVSIM, exceeding WAM-Flow’s 90.3 and demonstrating strong open-loop planning performance.The enhanced WAM-Flow variant achieves the reported 92.2 score through fine-grained safety reasoning and multi-path optimization.
  • Open-Loop Planning: 87.0 EPDMS surpasses TransFuser by 9.2 and GaussianFusion by 2.0, while diverse trajectories model multiple plausible driving intentions.Qualitative comparisons indicate that the method avoids the limited conservative solutions associated with mode collapse in conventional VLM-based planning.
  • Closed-Loop Planning: Against VLR-Driver on 44 Bench2Drive scenarios, the method improves Driving Score by 2.44%, Route Completion by 2.14%, Infraction Score by 0.03%, and Success Rate by 6.00%.It also reduces pedestrian-collision incidence by 0.14 and vehicle-collision incidence by 0.01 in subdivision infraction evaluation.
  • Open-Loop Planning: On nuScenes, the method achieves the lowest L2 error of 0.30%, collision rate of 0.21%, and intersection rate of 1.27%, while maintaining the highest diversity average metric.The reported results associate multimodal trajectory generation and optimization with more accurate and diverse trajectory hypotheses.
  • Scene Perception: The multimodal interaction framework achieves 74.0% mAP and 78.0% NDS on the nuScenes test set, exceeding BEVFormer by 1.1 and 3.9, respectively.The framework also reports best overall 3D detection performance on Argoverse 2, reaching 43.5 mAP.

6 LIMITATIONS AND FURTHER WORK

The framework has computational and robustness limitations. Future work targets more efficient transport and improved uncertainty-aware interaction modeling.

  • Optimal transport adds computational overhead, especially for dense multimodal tokens.The authors propose sparse or hierarchical transport strategies as future work.
  • Trajectory refinement may be sensitive to perception errors because it relies on perception-derived risk information.Future work will investigate joint uncertainty estimation and dynamic interaction modeling.

7 CONCLUSION

The paper presents an interpretable VLA-based framework for multimodal perception, reasoning, and planning. Its components improve multimodal interaction and driving-decision reliability in challenging long-tail scenarios while maintaining competitive inference efficiency.

  • The framework targets reliable multimodal perception, reasoning, and planning for VLA-based end-to-end autonomous driving.The authors characterize it as interpretable and designed for challenging long-tail scenarios.
  • Affinity-Guided Optimal Transport establishes structured two-way interactions between heterogeneous modalities.
  • Distribution-Consistent Modality Transfer aligns multimodal features in a unified Gaussian latent space.
  • Multi Modality Multi Trajectory Planning with Perception-Oriented Trajectory Refinement performs trajectory optimization.
  • The proposed framework improves multimodal interaction and driving-decision reliability in challenging long-tail scenarios while maintaining competitive inference efficiency.
Loading 2608.20890v1…