PaperlayerView exact revision on arXiv

Source-linked AI summary

ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, Xiang Bai

arXiv:2503.19755v1cs.CV

TL;DR

E2E driving methods still struggle with causal reasoning in interactive closed-loop evaluation, and VLMs face a gap between semantic reasoning and numerical trajectory actions. ORION addresses this gap by combining QT-Former temporal context, LLM reasoning, and a generative planner, achieving 77.74 DS and 54.62% SR on Bench2Drive while exceeding prior SOTA results.

  • Problem

    VLM-based E2E driving must bridge semantic reasoning and numerical trajectory action spaces to make effective decisions in complex closed-loop scenes.

    Source: §1 · pp. 1–2

  • Method

    ORION combines QT-Former long-term context aggregation, LLM scenario reasoning, and a generative planner that aligns reasoning with trajectory actions.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    77.74 Driving Score (DS) and 54.62% Success Rate (SR) were achieved on Bench2Drive, exceeding DriveTransformer by +14.28 DS and +19.61% SR.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    ORION demonstrates a flexible holistic framework for aligning vision, reasoning, and action spaces in closed-loop E2E driving.

    Source: §1 · p. 2, §5 · p. 9

    ORION is limited by the high computational complexity of its scalable VLM in real-time driving scenarios.

    Source: §5 · p. 9

Abstract

from arXiv · show

End-to-end (E2E) autonomous driving methods still struggle to make correct decisions in interactive closed-loop evaluation due to limited causal reasoning capability. Current methods attempt to leverage the powerful understanding and reasoning abilities of Vision-Language Models (VLMs) to resolve this dilemma. However, the problem is still open that few VLMs for E2E methods perform well in the closed-loop evaluation due to the gap between the semantic reasoning space and the purely numerical trajectory output in the action space. To tackle this issue, we propose ORION, a holistic E2E autonomous driving framework by vision-language instructed action generation. ORION uniquely combines a QT-Former to aggregate long-term history context, a Large Language Model (LLM) for driving scenario reasoning, and a generative planner for precision trajectory prediction. ORION further aligns the reasoning space and the action space to implement a unified E2E optimization for both visual question-answering (VQA) and planning tasks. Our method achieves an impressive closed-loop performance of 77.74 Driving Score (DS) and 54.62% Success Rate (SR) on the challenge Bench2Drive datasets, which outperforms state-of-the-art (SOTA) methods by a large margin of 14.28 DS and 19.61% SR.

1. Introduction

ORION addresses the difficulty of applying VLMs to closed-loop driving by aligning semantic reasoning with numerical trajectory generation. It combines long-term temporal context, VLM reasoning, and generative planning, achieving strong Bench2Drive results.

Motivation

  • Closed-loop benchmarks expose causal-reasoning weaknesses in classic E2E methods despite their promising open-loop performance.These methods integrate perception, prediction, and planning through multi-task learning but struggle with dynamic environmental interactions.Source: §1 · p. 1
  • Direct VLM trajectory generation is limited by weak numerical reasoning and single-result autoregressive inference, which mismatches planning uncertainty.Alternative interfaces using meta-actions decouple reasoning and action spaces, hindering collaborative optimization.Source: §1 · pp. 1–2

ORION

  • ORION uses a generative planner to establish a unified latent representation between VLM reasoning and trajectory actions.This lets VLM reasoning instruct trajectory generation and supports modeling relationships between scene information and driving behavior.Source: §1 · p. 2
  • QT-Former aggregates long-term visual context through a memory bank and history queries while linking temporal information to reasoning and action spaces.The query-based design avoids simply concatenating multi-frame images, which is constrained by VLM token length and computational overhead.Source: §1 · p. 2

Results

  • 77.74 Driving Score (DS) and 54.62% Success Rate (SR) were achieved by ORION on Bench2Drive, exceeding previous SOTA methods by 14.28 driving scores and 19.61% success rates.The evaluation measures closed-loop driving ability in interactive CARLA-based scenarios.Source: §1 · p. 2

Contributions

  • ORION combines generative planning, long-term context, and compatibility with diverse generative models without additional bells and whistles.The paper identifies these features as sources of performance and framework flexibility.Source: §1 · p. 2

2. Related work

Prior work integrates perception and planning or applies VLMs to visual understanding and textual trajectory prediction. ORION builds on these directions with query-based feature extraction and a vision-reasoning-action pipeline.

E2E Autonomous Driving

  • Classic E2E driving methods jointly integrate perception, prediction, and planning, with recent systems becoming increasingly planning-oriented.The related work describes UniAD and VAD as unified planning frameworks and notes probabilistic planning in VADv2.Source: §2.1 · p. 2

Vision-Language Models

  • VLM research improves visual contextual understanding through instruction tuning, image decomposition, dynamic resolution, and multimodal positional encoding.These approaches address image-text alignment, visual detail comprehension, token efficiency, and diverse-modality processing.Source: §2.2 · p. 2

ORION

  • ORION’s pipeline aligns vision, reasoning, and action spaces through QT-Former, an LLM, and a generative planner.The figure identifies QT-Former as the long-term context and vision-reasoning link, while the planner conditions multi-modal trajectories on a planning token.Source: Figure 2 · p. 3

Vision Feature Extraction

  • QT-Former uses learnable queries and cross-attention to extract multi-view image features while reducing token redundancy.It is presented as a query-based feature-extraction design inspired by Q-Former-style architectures.Source: §2.2 · p. 3

VLMs for Driving

  • Prior autonomous-driving VLMs use compressed scene features for environment perception and textual trajectory prediction.OmniDrive is cited as connecting vision and reasoning spaces before producing textual trajectories.Source: §2.3 · p. 3

3. Method

ORION unifies scene understanding, long-term temporal context, language reasoning, and trajectory generation in an end-to-end driving pipeline. Its generative planner aligns reasoning and action spaces, while QT-Former supplies historical context and auxiliary perception signals.

3. Method

  • ORION combines QT-Former, an LLM, and a generative planner to align vision, reasoning, and trajectory-action spaces.The pipeline compresses scene and historical features, performs driving-scenario reasoning, and conditions trajectory generation on a planning token.Source: §3 · p. 3

3.1. QT-Former

  • QT-Former uses scene and perception queries with cross-attention over multi-view image features to compress visual information and support perception tasks.Perception queries feed object detection, traffic-state, and motion-prediction heads, while scene queries represent key current-scene information.Source: §3.1 · pp. 3–4
  • History queries retrieve relevant information from a timestamped memory bank and interact with current scene features to model long-term temporal context.Updated history queries are stored using FIFO replacement, and the module initializes history queries to extract current features related to historical information.Source: §3.1 · p. 4

3.2. Large Language Model

  • The LLM combines scene, history, and language tokens for driving understanding and reasoning, then produces a planning token that conditions trajectory generation.Its tasks include scene description, history review, scene analysis, and action reasoning; ORION also extends Bench2Drive with automatically generated Chat-B2D VQA annotations.Source: §3.2 · p. 4

3.3. Generative Planner

  • The generative planner represents trajectories as p(a|s), aligns reasoning and trajectory distributions with a VAE, and decodes trajectories from the latent space.A GRU decoder produces trajectories, while the planner uses KL divergence for alignment and can also be paired with diffusion-based generation.Source: §3.3 · pp. 4–5

3.4. Training Objectives

  • QT-Former training combines detection, traffic-state, and motion-prediction losses, while planner training combines VAE, MSE, collision, and boundary losses.Detection and regression terms use focal and L1 losses; the generative planner additionally uses collision and boundary losses following VAD.Source: §3.4 · p. 5

4. Experiments

ORION is evaluated on Bench2Drive through closed-loop and multi-ability experiments, qualitative scenarios, paradigm comparisons, and ablations of its generative planner, QT-Former, temporal memory, and joint VQA-planning training. The experiments report strong overall performance while identifying weaker lane-changing performance and sensitivity to excessive history queries.

Main Results

  • ORION achieves +14.28 DS and +19.61% SR over DriveTransformer on Bench2Drive closed-loop evaluation.It also exceeds DriveAdapter by +13.52 DS and +21.54% SR despite DriveAdapter using camera and LiDAR inputs with expert feature distillation.Source: §4.3 · p. 6
  • ORION improves mean ability by +16.12% over DriveTransformer and +12.64% over DriveAdapter.Its reported scenario results include Overtaking (71.11%), Emergency Brake (78.33%), and Traffic Sign (69.15%).Source: §4.3 · p. 6
  • ORION falls behind DriveAdapter in Merging and Give Way, indicating difficulty with lane-changing decisions.The authors suggest that diverse lane-changing decision timing makes the relevant causal relationship harder to capture.Source: §4.3 · p. 6

Qualitative Results

  • ORION captures driving-action reasoning and predicts trajectories following the reasoning instruction in two canonical closed-loop scenarios.The qualitative results associate brown with action decisions, red with influential objects, and green with prediction trajectories.Source: §4.4 · p. 7, Figure 4 · p. 7

Planning Generation Paradigms

  • The plain-text paradigm records 42.23 DS, 13.14% SR, and 15.39% mean ability, while ORION gains +7.01 DS, +9.5% SR, and +6.28% mean ability over the MLP decoder paradigm.The experiments use the same sensor inputs, vision encoder, QT-Former, VLM, and training strategy, varying only the VLM output format.Source: §4.5 · p. 7

Generative Planner Ablation

  • The VAE-based trajectory generator significantly outperforms the diffusion-based generator in the generative-planner ablation.The authors attribute this to more direct latent alignment with the multi-modal action space and a more stable training process; diffusion still exceeds DriveTransformer by +8.51 DS and +11.53% SR.Source: §4.5 · p. 8, Table 3 · p. 8

QT-Former and Temporal Memory

  • Adding a memory bank and historical QA supervision increases performance by +3.67 DS and +4.85% SR, reaching 77.74 DS and 54.62% SR.Increasing history queries from 16 to 32 degrades performance, which the authors associate with reduced attention to current-frame features.Source: §4.5 · pp. 8–9

Multi-Task Training

  • Joint VQA and planning training improves planning by +3.64 DS and +9.66% SR while also increasing CIDEr by +0.12 and BLEU by +1.67.Single-task training does not provide both reasoning and planning capabilities according to the reported ablation.Source: §4.5 · p. 9

5. Conclusion

ORION bridges vision-reasoning and action spaces through a generative planner and long-term visual context, improving closed-loop planning evaluation. Its current limitation is high computational complexity in real-time driving scenarios.

  • ORION uses a generative planner and long-term visual context to bridge vision, reasoning, and action spaces.Source: §5 · p. 9
  • ORION demonstrates significant improvements in closed-loop planning evaluation over state-of-the-art methods.Source: §5 · p. 9
  • High computational complexity limits ORION’s use in real-time driving scenarios despite strong Bench2Drive closed-loop simulation performance.Source: §5 · p. 9

A. Details on the Chat-B2D dataset

Chat-B2D extends Bench2Drive with automatically generated VQA pairs to support VLM application in closed-loop simulated driving across diverse tasks.

A.1. Data Annotation Pipeline

The Chat-B2D annotation pipeline selects behaviorally important objects, generates multimodal descriptions, preserves historical context, and converts the outputs into VQA pairs.

  • The pipeline begins by selecting crucial objects that may affect the ego vehicle’s driving behavior.Selection includes potential three-second collisions, leading vehicles, active traffic signals, and vulnerable road users.Source: §A.1 · p. 13
  • Video clips, ego status, and selected-object ground truth are provided to Qwen2VL-72B for scene, object-impact, and action-reasoning generation.Source: §A.1 · p. 13
  • A queue preserves environmental dynamics and ego-motion characteristics from prior frames during description generation.Source: §A.1 · p. 13
  • Generated descriptions and historical information are combined with question templates to create VQA pairs.Source: §A.1 · p. 13
  • Chat-B2D contains 2.11M training VQA pairs and 0.12M validation pairs across four primary categories.Source: §A.2 · p. 13

B. Training Details

ORION uses staged training to progressively align visual features, language reasoning, and trajectory actions, followed by end-to-end fine-tuning.

  • Training proceeds through three stages, with each stage inheriting the previous stage’s weights and using six epochs with batch size 32.Source: B. Training Details · p. 13

3D Vision-Language Alignment

  • Table A1 compares open-loop planning in nuScenes and identifies textual LLM processing and high-level-command usage conditions.Source: Table A1 · p. 14

Language-Action Alignment

  • Language-Action Alignment unfreezes the generative planner to predict trajectories while transmitting knowledge from reasoning to action space.Source: B. Training Details · p. 14

End-to-End Fine-tuning

  • End-to-End Fine-tuning follows the preceding stage’s training settings to continue joint optimization.Source: B. Training Details · p. 14

C.1. Experiments on nuScenes dataset

On nuScenes, ORION is compared with established end-to-end methods, but its relative performance is weaker than other VLM-based approaches. The paper therefore emphasizes Bench2Drive closed-loop evaluation.

  • ORION is compared with prior state-of-the-art end-to-end autonomous driving methods on nuScenes.nuScenes is used for detection and open-loop planning evaluation, with L2 error and collision rate as planning metrics.Source: §C.1 · p. 14
  • ORION is suboptimal compared with other VLM-based methods on nuScenes.The authors attribute this to differences between the trajectory distributions of nuScenes and Bench2Drive.Source: §C.1 · p. 14
  • The paper primarily evaluates ORION’s closed-loop performance on Bench2Drive rather than its nuScenes open-loop performance.The authors note that simple MLP decoders with ego status can already achieve strong open-loop planning performance on nuScenes.Source: §C.1 · p. 14

C.2. More Ablation Studies on Bench2Drive

The Bench2Drive ablation study evaluates progressive vision-language-action space alignment and shows substantial gains from adding successive training stages. The complete strategy achieves the best reported ablation performance.

Training pipeline

  • Progressive space-alignment training is designed to facilitate alignment across vision, language, and action spaces.The ablation removes collision loss and the long-term memory bank with history queries from QT-Former.Source: §C.2 · pp. 14–15
  • +7.14 DS and +12.51% SR are achieved by second-stage training compared with direct planning training without the first stage.This comparison is reported between training configurations ID-2 and ID-1.Source: §C.2 · p. 15
  • 74.65 DS and 49.32 SR are achieved after third-stage training, producing the optimal ablation performance.The result is reported for configuration ID-3 after completing the third training stage.Source: §C.2 · p. 15

C.3. More Qualitative Results

The qualitative results illustrate ORION’s historical retrieval, scene understanding, critical-object identification, and action reasoning outputs. The appendix also documents the prompts and VQA templates used to generate these responses.

Historical information memory and retrieval

  • ORION retrieves historical information and perceives changes in critical elements across previous and current times.The example specifically highlights changes such as traffic-light status through the long-term memory bank and history queries.Source: §C.3 · p. 15

Scene understanding

  • ORION describes nighttime, wet-road conditions and identifies nearby vehicles relevant to braking distance.The qualitative example places a stationary car directly ahead of the ego vehicle and notes another vehicle with illuminated brake lights.Source: §C.3 · p. 16

Action reasoning

  • ORION recommends rapid slowing and lane following because a pedestrian crossing requires immediate attention.It also considers the relative motion of same-lane and opposite-lane vehicles when rejecting a lane change.Source: §C.3 · p. 16

Qualitative results and VQA templates

  • The qualitative figures mark critical objects in red and present scene understanding and reasoning alongside multi-view inputs and BEV traffic conditions.The appendix includes prompt templates and diverse VQA question templates for scene, historical, and action-related outputs.Source: Figure A3 · p. 16, Table A3 · p. 17, §C.3 · pp. 17–18

Related papers

Loading 2503.19755v1…