PaperlayerView exact revision on arXiv

Source-linked AI summary

ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, Xiang Bai

arXiv:2503.19755v1cs.CV

TL;DR

E2E driving methods still struggle with causal reasoning in interactive closed-loop evaluation, and VLMs face a gap between semantic reasoning and numerical trajectory actions. ORION addresses this gap by combining QT-Former temporal context, LLM reasoning, and a generative planner, achieving 77.74 DS and 54.62% SR on Bench2Drive while exceeding prior SOTA results.

  • Problem

    VLM-based E2E driving must bridge semantic reasoning and numerical trajectory action spaces to make effective decisions in complex closed-loop scenes.

  • Method

    ORION combines QT-Former long-term context aggregation, LLM scenario reasoning, and a generative planner that aligns reasoning with trajectory actions.

  • Results

    77.74 Driving Score (DS) and 54.62% Success Rate (SR) were achieved on Bench2Drive, exceeding DriveTransformer by +14.28 DS and +19.61% SR.

  • Takeaways & Limitations

    ORION demonstrates a flexible holistic framework for aligning vision, reasoning, and action spaces in closed-loop E2E driving.

  • Takeaways & Limitations

    ORION is limited by the high computational complexity of its scalable VLM in real-time driving scenarios.

Abstract

from arXiv · show

End-to-end (E2E) autonomous driving methods still struggle to make correct decisions in interactive closed-loop evaluation due to limited causal reasoning capability. Current methods attempt to leverage the powerful understanding and reasoning abilities of Vision-Language Models (VLMs) to resolve this dilemma. However, the problem is still open that few VLMs for E2E methods perform well in the closed-loop evaluation due to the gap between the semantic reasoning space and the purely numerical trajectory output in the action space. To tackle this issue, we propose ORION, a holistic E2E autonomous driving framework by vision-language instructed action generation. ORION uniquely combines a QT-Former to aggregate long-term history context, a Large Language Model (LLM) for driving scenario reasoning, and a generative planner for precision trajectory prediction. ORION further aligns the reasoning space and the action space to implement a unified E2E optimization for both visual question-answering (VQA) and planning tasks. Our method achieves an impressive closed-loop performance of 77.74 Driving Score (DS) and 54.62% Success Rate (SR) on the challenge Bench2Drive datasets, which outperforms state-of-the-art (SOTA) methods by a large margin of 14.28 DS and 19.61% SR.

1. Introduction

ORION addresses the difficulty of applying VLMs to closed-loop driving by aligning semantic reasoning with numerical trajectory generation. It combines long-term temporal context, VLM reasoning, and generative planning, achieving strong Bench2Drive results.

  • Motivation: Closed-loop benchmarks expose causal-reasoning weaknesses in classic E2E methods despite their promising open-loop performance.These methods integrate perception, prediction, and planning through multi-task learning but struggle with dynamic environmental interactions.
  • Motivation: Direct VLM trajectory generation is limited by weak numerical reasoning and single-result autoregressive inference, which mismatches planning uncertainty.Alternative interfaces using meta-actions decouple reasoning and action spaces, hindering collaborative optimization.
  • ORION: ORION uses a generative planner to establish a unified latent representation between VLM reasoning and trajectory actions.This lets VLM reasoning instruct trajectory generation and supports modeling relationships between scene information and driving behavior.
  • ORION: QT-Former aggregates long-term visual context through a memory bank and history queries while linking temporal information to reasoning and action spaces.The query-based design avoids simply concatenating multi-frame images, which is constrained by VLM token length and computational overhead.
  • Results: 77.74 Driving Score (DS) and 54.62% Success Rate (SR) were achieved by ORION on Bench2Drive, exceeding previous SOTA methods by 14.28 driving scores and 19.61% success rates.The evaluation measures closed-loop driving ability in interactive CARLA-based scenarios.
  • Contributions: ORION combines generative planning, long-term context, and compatibility with diverse generative models without additional bells and whistles.The paper identifies these features as sources of performance and framework flexibility.

2. Related work

Prior work integrates perception and planning or applies VLMs to visual understanding and textual trajectory prediction. ORION builds on these directions with query-based feature extraction and a vision-reasoning-action pipeline.

  • E2E Autonomous Driving: Classic E2E driving methods jointly integrate perception, prediction, and planning, with recent systems becoming increasingly planning-oriented.The related work describes UniAD and VAD as unified planning frameworks and notes probabilistic planning in VADv2.
  • Vision-Language Models: VLM research improves visual contextual understanding through instruction tuning, image decomposition, dynamic resolution, and multimodal positional encoding.These approaches address image-text alignment, visual detail comprehension, token efficiency, and diverse-modality processing.
  • ORION: ORION’s pipeline aligns vision, reasoning, and action spaces through QT-Former, an LLM, and a generative planner.The figure identifies QT-Former as the long-term context and vision-reasoning link, while the planner conditions multi-modal trajectories on a planning token.
  • Vision Feature Extraction: QT-Former uses learnable queries and cross-attention to extract multi-view image features while reducing token redundancy.It is presented as a query-based feature-extraction design inspired by Q-Former-style architectures.
  • VLMs for Driving: Prior autonomous-driving VLMs use compressed scene features for environment perception and textual trajectory prediction.OmniDrive is cited as connecting vision and reasoning spaces before producing textual trajectories.

3. Method

ORION unifies scene understanding, long-term temporal context, language reasoning, and trajectory generation in an end-to-end driving pipeline. Its generative planner aligns reasoning and action spaces, while QT-Former supplies historical context and auxiliary perception signals.

  • 3. Method: ORION combines QT-Former, an LLM, and a generative planner to align vision, reasoning, and trajectory-action spaces.The pipeline compresses scene and historical features, performs driving-scenario reasoning, and conditions trajectory generation on a planning token.
  • 3.1. QT-Former: QT-Former uses scene and perception queries with cross-attention over multi-view image features to compress visual information and support perception tasks.Perception queries feed object detection, traffic-state, and motion-prediction heads, while scene queries represent key current-scene information.
  • 3.1. QT-Former: History queries retrieve relevant information from a timestamped memory bank and interact with current scene features to model long-term temporal context.Updated history queries are stored using FIFO replacement, and the module initializes history queries to extract current features related to historical information.
  • 3.2. Large Language Model: The LLM combines scene, history, and language tokens for driving understanding and reasoning, then produces a planning token that conditions trajectory generation.Its tasks include scene description, history review, scene analysis, and action reasoning; ORION also extends Bench2Drive with automatically generated Chat-B2D VQA annotations.
  • 3.3. Generative Planner: The generative planner represents trajectories as p(a|s), aligns reasoning and trajectory distributions with a VAE, and decodes trajectories from the latent space.A GRU decoder produces trajectories, while the planner uses KL divergence for alignment and can also be paired with diffusion-based generation.
  • 3.4. Training Objectives: QT-Former training combines detection, traffic-state, and motion-prediction losses, while planner training combines VAE, MSE, collision, and boundary losses.Detection and regression terms use focal and L1 losses; the generative planner additionally uses collision and boundary losses following VAD.

4. Experiments

ORION is evaluated on Bench2Drive through closed-loop and multi-ability experiments, qualitative scenarios, paradigm comparisons, and ablations of its generative planner, QT-Former, temporal memory, and joint VQA-planning training. The experiments report strong overall performance while identifying weaker lane-changing performance and sensitivity to excessive history queries.

  • Main Results: ORION achieves +14.28 DS and +19.61% SR over DriveTransformer on Bench2Drive closed-loop evaluation.It also exceeds DriveAdapter by +13.52 DS and +21.54% SR despite DriveAdapter using camera and LiDAR inputs with expert feature distillation.
  • Main Results: ORION improves mean ability by +16.12% over DriveTransformer and +12.64% over DriveAdapter.Its reported scenario results include Overtaking (71.11%), Emergency Brake (78.33%), and Traffic Sign (69.15%).
  • Main Results: ORION falls behind DriveAdapter in Merging and Give Way, indicating difficulty with lane-changing decisions.The authors suggest that diverse lane-changing decision timing makes the relevant causal relationship harder to capture.
  • Qualitative Results: ORION captures driving-action reasoning and predicts trajectories following the reasoning instruction in two canonical closed-loop scenarios.The qualitative results associate brown with action decisions, red with influential objects, and green with prediction trajectories.
  • Planning Generation Paradigms: The plain-text paradigm records 42.23 DS, 13.14% SR, and 15.39% mean ability, while ORION gains +7.01 DS, +9.5% SR, and +6.28% mean ability over the MLP decoder paradigm.The experiments use the same sensor inputs, vision encoder, QT-Former, VLM, and training strategy, varying only the VLM output format.
  • Generative Planner Ablation: The VAE-based trajectory generator significantly outperforms the diffusion-based generator in the generative-planner ablation.The authors attribute this to more direct latent alignment with the multi-modal action space and a more stable training process; diffusion still exceeds DriveTransformer by +8.51 DS and +11.53% SR.
  • QT-Former and Temporal Memory: Adding a memory bank and historical QA supervision increases performance by +3.67 DS and +4.85% SR, reaching 77.74 DS and 54.62% SR.Increasing history queries from 16 to 32 degrades performance, which the authors associate with reduced attention to current-frame features.
  • Multi-Task Training: Joint VQA and planning training improves planning by +3.64 DS and +9.66% SR while also increasing CIDEr by +0.12 and BLEU by +1.67.Single-task training does not provide both reasoning and planning capabilities according to the reported ablation.

5. Conclusion

ORION bridges vision-reasoning and action spaces through a generative planner and long-term visual context, improving closed-loop planning evaluation. Its current limitation is high computational complexity in real-time driving scenarios.

  • ORION uses a generative planner and long-term visual context to bridge vision, reasoning, and action spaces.
  • ORION demonstrates significant improvements in closed-loop planning evaluation over state-of-the-art methods.
  • High computational complexity limits ORION’s use in real-time driving scenarios despite strong Bench2Drive closed-loop simulation performance.

A. Details on the Chat-B2D dataset

Chat-B2D extends Bench2Drive with automatically generated VQA pairs to support VLM application in closed-loop simulated driving across diverse tasks.

  • Chat-B2D extends Bench2Drive with automatically generated VQA pairs because high-quality scene-text annotations were unavailable.
  • The dataset is designed to promote VLM application in closed-loop simulated driving scenarios and covers diverse tasks.

A.1. Data Annotation Pipeline

The Chat-B2D annotation pipeline selects behaviorally important objects, generates multimodal descriptions, preserves historical context, and converts the outputs into VQA pairs.

  • The pipeline begins by selecting crucial objects that may affect the ego vehicle’s driving behavior.Selection includes potential three-second collisions, leading vehicles, active traffic signals, and vulnerable road users.
  • Video clips, ego status, and selected-object ground truth are provided to Qwen2VL-72B for scene, object-impact, and action-reasoning generation.
  • A queue preserves environmental dynamics and ego-motion characteristics from prior frames during description generation.
  • Generated descriptions and historical information are combined with question templates to create VQA pairs.
  • Chat-B2D contains 2.11M training VQA pairs and 0.12M validation pairs across four primary categories.

B. Training Details

ORION uses staged training to progressively align visual features, language reasoning, and trajectory actions, followed by end-to-end fine-tuning.

  • Training proceeds through three stages, with each stage inheriting the previous stage’s weights and using six epochs with batch size 32.
  • 3D Vision-Language Alignment: 3D Vision-Language Alignment trains QT-Former components to align the vision space with the reasoning space.
  • Table A1 compares open-loop planning in nuScenes and identifies textual LLM processing and high-level-command usage conditions.
  • Language-Action Alignment: Language-Action Alignment unfreezes the generative planner to predict trajectories while transmitting knowledge from reasoning to action space.
  • End-to-End Fine-tuning: End-to-End Fine-tuning follows the preceding stage’s training settings to continue joint optimization.

C.1. Experiments on nuScenes dataset

On nuScenes, ORION is compared with established end-to-end methods, but its relative performance is weaker than other VLM-based approaches. The paper therefore emphasizes Bench2Drive closed-loop evaluation.

  • ORION is compared with prior state-of-the-art end-to-end autonomous driving methods on nuScenes.nuScenes is used for detection and open-loop planning evaluation, with L2 error and collision rate as planning metrics.
  • ORION is suboptimal compared with other VLM-based methods on nuScenes.The authors attribute this to differences between the trajectory distributions of nuScenes and Bench2Drive.
  • The paper primarily evaluates ORION’s closed-loop performance on Bench2Drive rather than its nuScenes open-loop performance.The authors note that simple MLP decoders with ego status can already achieve strong open-loop planning performance on nuScenes.

C.2. More Ablation Studies on Bench2Drive

The Bench2Drive ablation study evaluates progressive vision-language-action space alignment and shows substantial gains from adding successive training stages. The complete strategy achieves the best reported ablation performance.

  • Training pipeline: Progressive space-alignment training is designed to facilitate alignment across vision, language, and action spaces.The ablation removes collision loss and the long-term memory bank with history queries from QT-Former.
  • Training pipeline: +7.14 DS and +12.51% SR are achieved by second-stage training compared with direct planning training without the first stage.This comparison is reported between training configurations ID-2 and ID-1.
  • Training pipeline: 74.65 DS and 49.32 SR are achieved after third-stage training, producing the optimal ablation performance.The result is reported for configuration ID-3 after completing the third training stage.

C.3. More Qualitative Results

The qualitative results illustrate ORION’s historical retrieval, scene understanding, critical-object identification, and action reasoning outputs. The appendix also documents the prompts and VQA templates used to generate these responses.

  • Historical information memory and retrieval: ORION retrieves historical information and perceives changes in critical elements across previous and current times.The example specifically highlights changes such as traffic-light status through the long-term memory bank and history queries.
  • Scene understanding: ORION describes nighttime, wet-road conditions and identifies nearby vehicles relevant to braking distance.The qualitative example places a stationary car directly ahead of the ego vehicle and notes another vehicle with illuminated brake lights.
  • Action reasoning: ORION recommends rapid slowing and lane following because a pedestrian crossing requires immediate attention.It also considers the relative motion of same-lane and opposite-lane vehicles when rejecting a lane change.
  • Qualitative results and VQA templates: The qualitative figures mark critical objects in red and present scene understanding and reasoning alongside multi-view inputs and BEV traffic conditions.The appendix includes prompt templates and diverse VQA question templates for scene, historical, and action-related outputs.
Loading 2503.19755v1…