PaperlayerView exact revision on arXiv

Source-linked AI summary

SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving

Xuesong Chen, Linjiang Huang, Tao Ma, Rongyao Fang, Shaoshuai Shi, Hongsheng Li

arXiv:2505.16805v1cs.CV

TL;DR

Autonomous driving systems need better reasoning and interpretability, while VLM integrations remain costly and difficult to deploy in real time. SOLVE combines VLMs and E2E models through shared visual features, progressive trajectory refinement, and temporal decoupling, achieving state-of-the-art planning results on nuScenes.

  • Problem

    End-to-end driving faces learning, reasoning, interpretability, and causal-confusion challenges, while VLM-based approaches remain computationally expensive for real-time decision-making.

    Source: §1 · p. 1

  • Method

    SOLVE shares visual features between VLM and E2E components, progressively refines trajectories with T-CoT, and asynchronously uses VLM outputs to support E2E planning.

    Source: §1 · p. 2

  • Results

    SOLVE achieves state-of-the-art open-loop planning results on the nuScenes benchmark.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework provides integrated VLM-E2E knowledge and planning cooperation while maintaining efficient real-time system operation.

    Source: §1 · p. 2

Abstract

from arXiv · show

The integration of Vision-Language Models (VLMs) into autonomous driving systems has shown promise in addressing key challenges such as learning complexity, interpretability, and common-sense reasoning. However, existing approaches often struggle with efficient integration and realtime decision-making due to computational demands. In this paper, we introduce SOLVE, an innovative framework that synergizes VLMs with end-to-end (E2E) models to enhance autonomous vehicle planning. Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components. We propose a Trajectory Chain-of-Thought (T-CoT) paradigm, which progressively refines trajectory predictions, reducing uncertainty and improving accuracy. By employing a temporal decoupling strategy, SOLVE achieves efficient cooperation by aligning high-quality VLM outputs with E2E real-time performance. Evaluated on the nuScenes dataset, our method demonstrates significant improvements in trajectory prediction accuracy, paving the way for more robust and reliable autonomous driving systems.

1. Introduction

SOLVE addresses limitations of end-to-end driving and prior VLM integrations by combining feature-level knowledge sharing, trajectory refinement, and temporally decoupled cooperation. It achieves state-of-the-art open-loop planning results on nuScenes.

Motivation

  • End-to-end driving faces learning complexity, limited common-sense reasoning and interpretability, while VLMs add reasoning benefits but remain costly and weak in 3D perception.These limitations complicate efficient integration into real-time autonomous driving systems.Source: §1 · p. 1
  • Previous VLM-E2E approaches cooperate mainly at the trajectory level, limiting information sharing between VLM reasoning and E2E 3D perception.The E2E model cannot fully use VLM understanding of critical objects, while the VLM cannot access precise E2E scene perception.Source: §1 · p. 2

Method

  • A shared visual encoder enables feature-level synergy by compressing objects, maps, and environmental cues into unified visual tokens for both models.The Q-Former sequentially collects and aligns features with perception queries, reducing redundant extraction and easing scene understanding.Source: §1 · p. 2
  • Trajectory Chain-of-Thought progressively refines candidate trajectories from a predefined bank, using each selected trajectory as the basis for the next refinement.A trajectory adapter converts candidates into tokens aligned with text and visual tokens before selection.Source: §1 · p. 2
  • Temporal decoupling uses high-quality VLM trajectories asynchronously as priors for the E2E model, preserving real-time operation despite VLM computation cost.The memory-based mechanism stores VLM trajectories and accesses them as initialization priors despite temporal offsets from the current state.Source: §1 · p. 2

Contributions

  • SOLVE achieves state-of-the-art open-loop planning results on the nuScenes benchmark through integrated VLM and E2E knowledge and trajectory cooperation.The framework combines shared feature representations, T-CoT trajectory refinement, and temporal decoupling.Source: §1 · p. 2

2. Related Work

Related work spans modular and end-to-end autonomous driving, with end-to-end models improving unified planning while retaining interpretability and reasoning challenges. VLM-based systems address some of these challenges but often remain computationally expensive and incompletely integrated with E2E models.

2.1. End-to-end Autonomous Driving

  • End-to-end driving maps sensor inputs directly to planning outputs in a differentiable framework, with Transformer-based and BEV methods improving perception and planning.Representative systems add multi-task supervision, vectorized scene representations, uncertainty modeling, or richer environmental context.Source: §2.1 · p. 2

2.2. VLMs for Autonomous Driving

  • End-to-end autonomous driving remains limited by learning complexity, insufficient common-sense reasoning, limited interpretability, and causal confusion.These limitations motivate efforts to improve end-to-end systems for real-world applications.Source: §2.2 · pp. 2–3
  • VLM-based driving methods improve explainability, reasoning, and generalization, but most require substantial computation and cannot achieve real-time decision-making.Examples include control prediction with explanations, text-based decisions, 3D vision-language planning, graph-based VQA, and latent-space trajectory forecasting.Source: §2.2 · p. 3
  • DriveVLM combines VLMs with E2E architectures through trajectory-level cooperation, but it overlooks high-level feature complementarity between the models.Its VLM trajectories are refined by an E2E model to produce final planning outputs.Source: §2.2 · p. 3

3. SOLVE

SOLVE integrates VLM and E2E planning through shared visual features, trajectory-level reasoning, time-decoupled cooperation, and staged training. Its components progressively condense scene information, refine trajectories, and align VLM accuracy with E2E efficiency.

3. SOLVE

  • SOLVE shares SQ-Former visual features between the VLM and E2E model, enabling feature-level knowledge integration alongside trajectory-level cooperation.The shared visual encoder condenses scene cues before both components use them for planning.Source: §3 · p. 3

3.1. SQ-Former

  • SQ-Former sequentially compresses static scene, dynamic participant, and map cues into condensed visual queries using ordered perception-task decoders and temporal memory.Collector queries are progressively enriched, then adapted to the VLM with an appended ego token for trajectory prediction.Source: §3.1 · pp. 3–4
  • Using 384 queries, SQ-Former outperforms OmniDrive’s 512-query design while reducing the computational cost of the following VLM.The sequential design is reported to capture rich, condensed visual cues for cross-task synergy.Source: §3.1 · p. 4

3.2. Trajectory Chain-of-Thought

  • T-CoT selects and refines trajectories progressively from a bank of clustered candidates, using trajectory tokens to align candidate paths with VLM text and visual inputs.The bank contains 36 candidates per navigation command, and refinement generates waypoints from the selected reference trajectory.Source: §3.2 · pp. 4–5

3.3. Time-decoupled Synergy Prediction

  • Time-decoupled cooperation stores VLM trajectories so the real-time E2E model can asynchronously use long-horizon predictions as additional initialization.The VLM predicts beyond its latency and the E2E prediction horizon, allowing delayed trajectories to support collaborative prediction.Source: §3.3 · p. 5

3.4. Training Strategy

  • SOLVE trains its shared components in stages: QA pretraining, trajectory-adapter and E2E-head training, joint VLM/E2E optimization, and historical VLM trajectory refinement.The final trajectory-level stage refines the E2E planner using VLM forecasts from historical frames.Source: §3.4 · p. 5

4. Experiments

SOLVE is evaluated on nuScenes through benchmark comparisons, qualitative planning examples, and component ablations. The experiments report state-of-the-art performance, improved complex-scene trajectories, and trade-offs for visual encoding and query design.

4.2. Comparison with State-of-the-art Methods

  • SOLVE-E2E and SOLVE-VLM achieve state-of-the-art results in their respective nuScenes open-loop planning tracks.SOLVE-E2E improves L2 error over UniAD, VAD, and BEV-planner by 0.15, 0.06, and 0.04, while SOLVE-VLM surpasses OmniDrive and DriveVLM by 0.05 and 0.03 L2 error.Source: §4.2 · p. 6, Table 1 · p. 6
  • SOLVE-VLM’s shared SQ-Former and joint training improve visual feature selection and planning supervision, while temporal initialization makes SOLVE-E2E (Async) 0.01 L2 better than SOLVE-E2E.The shared encoder emphasizes planning-relevant objects and suppresses less relevant visual information, while VLM history initializes the asynchronous E2E planner.Source: §4.2 · p. 6

4.3. Qualitative Results

  • In complex scenes, VLM trajectories stay nearer the lane center than E2E trajectories, and VLM-assisted E2E-Async produces better trajectories than E2E.In simple straight-ahead scenes, both VLM and E2E match the expert trajectory.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Ablation Studies

  • Using 384 SQ-Former queries balances visual representation, performance, and computation better than 256 or 512 queries.Too few queries lose visual features, whereas too many add redundant information and computational cost.Source: §4.4 · p. 7, Table 2 · p. 8
  • Encoding image features improves L2 by 0.7, while encoding detection queries before lane queries improves L2 by 0.3.The selected order is image, detection query, then lane query, because scene-level and task-order information both affect planning.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • Trajectory CoT improves L2 by 1.1, six reference trajectories balance diversity and computation, and shared SQ-Former features reduce E2E L2 error by 1.5 cm.VLM performance also improves by 0.6 through E2E integration, whereas DriveVLM-style post-processing initialization shows no significant VLM planning improvement.Source: §4.4 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5. Conclusion

SOLVE combines VLM and E2E planning through shared knowledge and planning integration. It uses T-CoT and temporal decoupling, and achieves state-of-the-art results on nuScenes.

5. Conclusion

  • SOLVE integrates VLM and E2E models through shared knowledge and planning integration, using T-CoT and temporal decoupling for trajectory cooperation.T-CoT progressively refines trajectories from a predefined trajectory bank through chain-style reasoning.Source: §5 · p. 8

Related papers

Loading 2505.16805v1…