PaperlayerView exact revision on arXiv

Source-linked AI summary

Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, Xinggang Wang

arXiv:2410.22313v1cs.CVcs.RO

TL;DR

End-to-end driving models have strong planning ability but limited commonsense, while LVLMs are poorly suited to precise numerical prediction. Senna combines them by using language-based high-level decisions to condition an end-to-end trajectory model, achieving state-of-the-art planning performance with reported gains from DriveX pre-training and nuScenes fine-tuning.

  • Problem

    End-to-end models can struggle with commonsense in complex scenarios, while LVLM-based trajectory or control prediction is limited by imprecise numerical computation.

  • Method

    Senna decouples high-level language-based planning decisions from low-level trajectory prediction, combining Senna-VLM, multi-image scene understanding, planning-oriented QAs, and three-stage training.

  • Results

    Senna achieves state-of-the-art planning performance on nuScenes and DriveX, with experiments reporting improved high-level planning and trajectory planning over comparison methods.

  • Takeaways & Limitations

    Language-based planning provides a way to combine LVLM commonsense and reasoning with end-to-end models’ precise trajectory prediction, supporting cross-scenario generalization and transferability.

  • Takeaways & Limitations

    Large-model inference speed may hinder real-time autonomous-driving requirements, and the reported results indicate that performance has not yet reached a ceiling as training data grows.

Abstract

from arXiv · show

End-to-end autonomous driving demonstrates strong planning capabilities with large-scale data but still struggles in complex, rare scenarios due to limited commonsense. In contrast, Large Vision-Language Models (LVLMs) excel in scene understanding and reasoning. The path forward lies in merging the strengths of both approaches. Previous methods using LVLMs to predict trajectories or control signals yield suboptimal results, as LVLMs are not well-suited for precise numerical predictions. This paper presents Senna, an autonomous driving system combining an LVLM (Senna-VLM) with an end-to-end model (Senna-E2E). Senna decouples high-level planning from low-level trajectory prediction. Senna-VLM generates planning decisions in natural language, while Senna-E2E predicts precise trajectories. Senna-VLM utilizes a multi-image encoding approach and multi-view prompts for efficient scene understanding. Besides, we introduce planning-oriented QAs alongside a three-stage training strategy, which enhances Senna-VLM's planning performance while preserving commonsense. Extensive experiments on two datasets show that Senna achieves state-of-the-art planning performance. Notably, with pre-training on a large-scale dataset DriveX and fine-tuning on nuScenes, Senna significantly reduces average planning error by 27.12% and collision rate by 33.33% over model without pre-training. We believe Senna's cross-scenario generalization and transferability are essential for achieving fully autonomous driving. Code and models will be released at https://github.com/hustvl/Senna.

I. INTRODUCTION

Senna addresses limitations in end-to-end driving and LVLM-based planning by separating language-based high-level decisions from precise trajectory prediction. It combines surround-view understanding, planning-oriented data, and staged training to improve planning performance and transferability.

  • Motivation: End-to-end models can lack interpretability and commonsense, while LVLMs struggle with precise numerical trajectory prediction.These limitations motivate a structured division between decision-making and trajectory execution.
  • Training data: Planning-oriented QAs cover scene descriptions, traffic participants, signals, and meta-action planning, and can be generated automatically at scale.The QAs are designed specifically to expose planning-related clues in driving scenes.
  • Training strategy: Senna-VLM uses mixed pre-training, driving fine-tuning, and planning fine-tuning instead of general-purpose pre-training followed by driving fine-tuning alone.The paper reports that this three-stage strategy achieves the best planning performance while preserving commonsense knowledge.
  • Structured planning: Senna-VLM predicts high-level planning decisions in natural language, and Senna-E2E uses them to generate concrete trajectories.The decisions are encoded into high-dimensional features before being provided to Senna-E2E.
  • Scene understanding: Senna employs efficient multi-image encoding and surround-view prompts to support spatial understanding of driving scenarios.The driving vision adapter compresses image tokens, while view-specific prompts distinguish features from different perspectives.
  • Results: Experiments on nuScenes and DriveX report state-of-the-art planning performance, with pre-training improving cross-scenario generalization and transferability.The reported gains are attributed to the structured planning approach, architecture, and training strategy.

II. RELATED WORK

Related work spans end-to-end autonomous driving, large vision-language models, and their integration for driving. Senna builds on these directions while using language-based decisions rather than precise LVLM trajectory prediction.

  • End-to-End Autonomous Driving: End-to-end driving maps perception directly to planning outputs and benefits from the growth of large-scale driving data.Prior systems also use auxiliary tasks to support planning and related predictions.
  • Large Vision-Language Models: LVLMs extend language-model understanding and conversation capabilities to visual inputs through multimodal pre-training.This line of work provides the visual-language foundation used by driving-oriented systems.
  • LVLMs and Autonomous Driving: Prior LVLM-driving systems predict trajectories or control signals, while DriveVLM combines an LVLM with an end-to-end model to refine low-frequency trajectories.The paper identifies precise mathematical prediction as a limitation of using LVLMs for trajectory points.
  • LVLMs and Autonomous Driving: Senna instead predicts high-level decisions in natural language and conditions the end-to-end model on those decisions to produce final trajectories.This design aligns the LVLM task with language prediction while retaining end-to-end trajectory accuracy.

III. SENNA

Senna processes scene information through Senna-VLM and Senna-E2E in a structured pipeline. The first module produces language-based decisions and meta-action features, while the second predicts the final trajectory using those features and scene inputs.

  • Architecture: Senna receives multi-view image sequences, user instructions, and navigation commands as scene information.User instructions are used as prompts, while the remaining inputs are sent to both modules.
  • Architecture: Senna-VLM encodes image and text information, generates high-level decisions, and converts them into high-dimensional features through a Meta-action Encoder.The language model processes image and text tokens before producing the decisions.
  • Architecture: Senna-E2E predicts the final planning trajectory from scene information and the meta-action features generated by Senna-VLM.This connects language-based planning decisions with low-level trajectory prediction.
  • Training: Planning-oriented QAs are generated through an automatic labeling pipeline without human annotation and are used to train Senna-VLM.The paper presents these QAs as a scalable source of driving-scene training supervision.

A. Driving Scene Understanding

Senna’s driving scene understanding uses planning-oriented QAs covering scene context, traffic signals, vulnerable road users, motion intentions, and meta-action planning. These representations provide structured information for high-level decisions.

  • Planning-oriented QAs: Planning-oriented QAs are designed to expose driving-scene factors relevant to safe and accurate planning.Their inputs can come from automatically annotated 3D detections, tracking trajectories, and generated descriptions.
  • Scene Description: Scene-description prompts summarize traffic, environment, road type, weather, time, and road conditions without redundant information.The prompts target concise descriptions of factors relevant to planning.
  • Traffic Signal Detection: Traffic-signal QAs classify the signal state as red, green, yellow, or none.The category none indicates that no traffic light is detected in front of the ego vehicle.
  • Vulnerable Road User Identification: Vulnerable-road-user QAs describe detected categories and their lateral and longitudinal distances relative to the ego vehicle.The approach uses ground-truth 3D detection results and predicts only integer distance parts to reduce learning complexity.
  • Motion Intention Prediction: Motion-intention prediction represents the future behavior of surrounding vehicles with meta-actions to improve dynamic-scene understanding.These predictions are intended to support more informed planning decisions.
  • Meta-action Planning: Ego-vehicle meta-actions encode lateral decisions as Left, Straight, or Right and longitudinal decisions as Accelerate, Keep, Decelerate, or Stop.This representation avoids requiring the LVLM to predict precise trajectories directly.

B. Senna-VLM

Senna-VLM encodes multi-view driving images into compressed image tokens, combines them with text instructions, and predicts high-level decisions that Senna-E2E can use for trajectory planning.

  • Architecture: Senna-VLM processes multi-view image sequences with a Vision Encoder and Driving Vision Adapter, producing compressed image tokens for the language model.The adapter encodes and compresses image features before they enter the LLM.
  • Architecture: The LLM combines image and text tokens to predict high-level driving decisions, which the Meta-action Encoder converts into Senna-E2E features.The resulting meta-action features have dimension D, the feature dimension of Senna-E2E.
  • Multi-view understanding: Excessive image tokens can cause model collapse and decoding failure, motivating the adapter’s image-feature compression.The paper reports that compression does not degrade model performance in its experiments.
  • Multi-view understanding: Multi-view prompts label each image perspective so Senna-VLM can distinguish views and build spatial understanding for driving scenarios.The prompt places a view-specific tag, such as <FRONT VIEW>, before the corresponding image token.
  • Decision interface: Meta-action embeddings provide a one-to-one mapping from formatted planning decisions to features consumed by Senna-E2E.The learnable embedding group has N_act entries and output dimension D.

C. Senna-E2E

Senna-E2E extends VADv2 with perception, motion prediction, and planning modules, while incorporating Senna-VLM’s meta-action features into trajectory planning.

  • Architecture: Senna-E2E extends VADv2 and accepts multi-view images, navigation commands, and meta-action features as inputs.Meta-action features are integrated as an additional interaction token.
  • Architecture: Its perception module detects dynamic objects and generates a local map, while its motion prediction module forecasts future trajectories.These modules provide scene information for the planning module.
  • Trajectory planning: The planning module uses planning tokens that interact with scene features through attention to predict the final trajectory.The trajectory process is conditioned on the navigation command and meta-action features.

D. Training Strategy

Senna-VLM is trained through a three-stage strategy that progresses from mixed single-image pre-training to driving and planning fine-tuning. Senna-E2E uses ground-truth meta-actions during training and Senna-VLM predictions during inference.

  • Three-stage training: Senna-VLM uses Mix Pre-training, Driving Fine-tuning, and Planning Fine-tuning as its three training stages.Mix Pre-training trains the Driving Vision Adapter with mixed instruction-following and driving description data; later stages use surround-view inputs and planning-oriented QAs.
  • Cross-module training: During Senna-E2E training, ground-truth meta-actions are inputs; during inference, predicted meta-actions from Senna-VLM are used.
  • Planning-oriented QAs: Planning-oriented QA evaluation distinguishes description, explanation, vulnerable road user identification, and motion intention prediction components.The supplied table caption defines these abbreviations and QA categories.
  • Input design: Surround-view input and compressed image tokens are treated as training-strategy variables in the ablation studies.The supplied table captions identify ablations of input image views and the number of image tokens.

IV. EXPERIMENTS

Experiments evaluate Senna on nuScenes and the large-scale DriveX dataset using decision, description, and trajectory-planning metrics. Additional ablations examine dataset size and inference latency.

  • Datasets: Senna is evaluated on nuScenes, which contains 1,000 driving scenes of about 20 seconds with surround-view cameras, LiDAR, and rich annotations.
  • Datasets: DriveX contains 1000K three-second driving clips, with 800k used for training and 200k for validation.The dataset provides diverse, balanced driving data, surround-view images, and varied annotations.
  • Metrics: High-level driving decisions are evaluated with accuracy and F1 Score, scene descriptions with BLEU-4, CIDEr, and METEOR, and trajectories with displacement error and collision rate.
  • Ablations: Ablations cover training dataset size and inference latency, with latency measured on an NVIDIA GeForce RTX 4090 GPU.

B. Main Results

Senna achieves strong results in high-level planning, trajectory planning, and ablation studies across DriveX and nuScenes. Its reported gains support the value of structured planning, planning-oriented QAs, surround-view input, efficient encoding, and staged training.

  • High-level planning: 10.44%: Senna improves planning accuracy over the best results from other methods after fine-tuning comparisons on DriveX.
  • High-level planning: 17.67%: Senna raises the deceleration F1 score from 52.68 to 61.99.
  • Trajectory planning: 40.28%: Senna reduces average planning displacement error, while collision rate decreases by 45.45% versus VAD after DriveX pre-training and nuScenes fine-tuning.
  • Trajectory planning: 14.27%: Senna reduces average planning displacement error on DriveX and delivers the best planning performance among the compared methods.
  • Ablation study: All planning-oriented QA types together achieve optimal planning performance, while removing individual QA types reduces decision accuracy.Removing Meta Action QA particularly affects planning accuracy; removing scenario-description or traffic-light QAs affects acceleration and deceleration decisions.
  • Ablation study: 64.91% to 71.21%: incorporating surround-view multi-image input increases Senna’s planning accuracy.
  • Ablation study: 256 image tokens enable normal output after 576 and 512 tokens cause decoding failure or model collapse.
  • Ablation study: Senna’s decision accuracy improves steadily as the training dataset grows, while relatively small datasets do not cause significant degradation.

D. Qualitative Results

Qualitative results show Senna describing driving scenes, making high-level decisions, explaining plans, and generalizing zero-shot to nuScenes.

  • Qualitative evaluation: Senna captures dynamic and static scene elements such as pedestrians crossing, traffic lights, and traffic cones in qualitative evaluations.The evaluation covers scene description, high-level decision-making, planning explanation on DriveX, and zero-shot performance on nuScenes.

V. LIMITATION

Senna’s main deployment limitation is that large-model inference speed may prevent meeting autonomous-driving real-time requirements. The authors also report that performance continues improving with more data, indicating the system has not reached its performance ceiling.

  • Deployment constraint: Large-model inference speed may hinder Senna from meeting autonomous-driving real-time requirements.The authors suggest smaller vision-language models and hardware-level optimization as potential mitigations.
  • Data-scaling boundary: Senna’s planning performance improves significantly as dataset size increases, without signs of diminishing returns.The authors therefore believe limited training data leaves room for further improvement.
  • Scope and future direction: Senna demonstrates cross-scenario generalization and transferability through language-based integration of an LVLM and an end-to-end model.The conclusion identifies non-predefined, granular language instructions for controlled trajectory planning as future work.
Loading 2410.22313v1…