PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Actions from Discrete Minds: Latent-Aligned Planning for End-to-End Autonomous Driving

Ruoyu Yao, Yusen Xie, Qingzhao Liu, Pei Liu, Zewei Yang, Yipeng Zhu, Xiaolong Wang, Jun Ma

arXiv:2609.04070v1cs.CVcs.RO

TL;DR

Grounding semantic reasoning in continuous, physics-constrained driving control remains a challenge. LaPla predicts continuous action latents with a unified VLA model and decodes them through a frozen residual VQ-VAE prior. It reports competitive open-loop performance and superior closed-loop success, with lower long-horizon UniAD L2 error than AutoVLA.

  • Problem

    Grounding high-dimensional semantic representations in the continuous, physics-constrained control required for autonomous driving remains a fundamental bottleneck.

    Source: §I · p. 1

  • Method

    LaPla predicts continuous latent vectors from multimodal context and decodes them with a frozen residual VQ-VAE as a kinematic prior, bypassing discrete codebook lookups.

    Source: §I · p. 2

  • Results

    LaPla achieves competitive open-loop accuracy and superior closed-loop success; at 3 s, its UniAD L2 error is 0.98 m versus AutoVLA’s 1.16 m.

    Source: 1) Comparative Studies · p. 5, §V · p. 8

  • Takeaways & Limitations

    The frozen decoder provides a kinematic prior that steers outputs toward the kinematically plausible manifold while bypassing quantization bottlenecks.

    Source: §I · p. 2

Abstract

from arXiv · show

Bridging the gap between the discrete reasoning of Vision-Language Models and the continuous, physics-constrained nature of autonomous driving remains a significant challenge. In this work, we introduce LaPla, a unified Vision-Language-Action (VLA) framework featuring latent-aligned planning to seamlessly ground semantic understanding in precise motion execution. We first design an action tokenizer based on a residual vector-quantized variational autoencoder (VQ-VAE), capturing vehicle kinematics and encoding trajectory features into a structured latent space. Rather than discrete codebook lookups that inevitably introduce quantization errors, LaPla repurposes this representation as a physical prior to bridge the modality gap between high-dimensional semantics and the raw action space. Specifically, given multimodal inputs integrating multi-view images, historical actions, and textual instructions, LaPla incorporates concurrent action queries to causally attend to the multimodal context in a single forward pass, projecting hidden states directly into the pretrained VQ-VAE latent space. The frozen decoder then translates these continuous latents into actions, effectively eliminating quantization errors and ensuring physically plausible trajectories while bypassing time-consuming autoregressive generation. Extensive experiments on the nuScenes benchmark demonstrate that LaPla achieves competitive open-loop performance, reducing long-horizon L2 error by 15.52% compared to state-of-the-art VLA methods. Closed-loop evaluations on the NVIDIA AlpaSim simulator further confirm its superior capability in ensuring smooth driving progress, improving the success rate by 33.34 percentage points with significantly reduced inference latency.

I. Introduction

VLA models bring pretrained knowledge to autonomous driving, but grounding semantic representations in continuous, physics-constrained control remains a bottleneck. LaPla plans in a frozen, kinematically structured latent space, and the paper reports competitive open-loop results and closed-loop improvements.

Research gap

  • Current approaches face distinct trade-offs: action tokenization introduces quantization errors, dual-system designs disconnect semantic intent from planning, and diffusion heads incur iterative-denoising latency.The paper also notes that diffusion heads can suffer training instability and knowledge erosion when jointly optimized with a VLM backbone.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Learned latent action spaces can encode plausible driving behaviors, but co-learning the space and policy burdens the VLA with discovering both a coordinate system and a driving policy.The paper argues that a frozen, kinematically sound latent structure lets the VLA focus on executing policies within an invariant space.Source: §I · p. 2

LaPla

  • LaPla maps multimodal context directly into a continuous latent space learned by a pretrained residual VQ-VAE, then uses its frozen decoder to generate smooth actions without discrete codebook lookups.The discrete bottleneck is used during VQ-VAE pretraining to cluster trajectories into plausible primitives, while planning bypasses quantization.Source: §I · p. 2

Evaluation overview

  • LaPla achieves competitive open-loop performance on nuScenes and advances closed-loop success rate and inference speed on AlpaSim.The paper describes its open-loop planning as physically coherent and as mitigating long-horizon drift observed in state-of-the-art methods.Source: §I · p. 2

II. Related Work

End-to-end autonomous driving maps sensor inputs to trajectories, while VLA models add language-based reasoning and action generation. Existing VLA designs unify these functions or separate them across systems, but action generation still lacks structural physical grounding.

A. End-to-End Autonomous Driving

  • End-to-end autonomous driving maps raw sensor inputs directly to continuous trajectories, while language-enabled VLA models add explicit commands and open-world knowledge to end-to-end systems.Modular pipelines separate perception, prediction, planning, and control but can accumulate errors and have inconsistent optimization objectives.Source: A. End-to-End Autonomous Driving · p. 2

B. Vision-Language-Action Models for Autonomous Driving

  • Unified VLA methods combine semantic reasoning and action generation, with approaches ranging from language-token trajectories and action tokenization to generative action heads.AutoVLA uses action tokenization for dedicated representation learning.Source: B. Vision-Language-Action Models for… · p. 2
  • Dual-system VLA methods cascade high-level reasoning into low-level execution, but both unified and dual-system architectures face a lack of structural physical grounding in action generation.Examples include DriveVLM’s slow-fast system and Senna’s cascaded VLM decision-maker and VA planner.Source: B. Vision-Language-Action Models for… · p. 3

III. Methodology

LaPla separates learning a trajectory prior from downstream policy learning: a pretrained residual VQ-VAE supplies a frozen decoder, and the VLA predicts continuous latents for trajectory generation. Its latent prior encodes motion context and hierarchically represents maneuvers.

III. Methodology

  • LaPla first pretrains a trajectory VQ-VAE and freezes its decoder as a motion prior, then predicts continuous latents to drive trajectory generation while jointly performing VQA.The separation enables end-to-end differentiable planning without discrete token generation.Source: §III · p. 3

A. Latent Prior Learning

  • A frozen trajectory decoder biases outputs toward smooth driving patterns, relieving the downstream policy of learning kinematic regularities from scratch.Direct waypoint regression would require rediscovering vehicle kinematics and multimodal maneuver structures from limited driving data.Source: A. Latent Prior Learning · p. 3
  • The VQ-VAE encodes per-step displacement and heading features with historical motion context, so each latent carries current motion and the recent trajectory trend.The recent-motion feature is formed by pooling over a local time window and concatenating it with the current-step embedding.Source: A. Latent Prior Learning · p. 3
  • VQ-VAE training minimizes reconstruction and commitment losses, and the codebooks use exponential moving-average updates with dead-code revival.The decoder reconstructs motion features from quantized latents.Source: A. Latent Prior Learning · p. 4
  • A coarse-to-fine residual design decomposes maneuvers into macroscopic motion primitives and fine-grained corrections, producing a densely populated, combinatorially expressive latent space that remains semantically smooth.The passage estimates approximately 6.7 × 10^7 possible codes.Source: A. Latent Prior Learning · p. 4

B. Unified VLA Framework with Latent-Aligned Planning

LaPla uses a single causal transformer to predict future trajectories through latent-aligned planning and answer driving questions through VQA. Its VLM backbone processes a sequence integrating visual, action, and textual modalities.

B. Unified VLA Framework with Latent-Aligned Planning

1) Multimodal Sequence Formulation:

LaPla forms a multimodal sequence from visual observations, historical motion, and navigation instructions, then uses learnable action queries to predict trajectories in the frozen prior’s continuous latent space.

1) Multimodal Sequence Formulation:

  • Historical motion is encoded through residual quantization, while future trajectories are predicted as continuous latents to avoid discretization precision loss.The representation uses discrete quantization as an information bottleneck for historical context and continuous future latents for high-precision decoding.Source: Figure 3 · p. 4, 1) Multimodal Sequence Formulation · p. 4
  • Learnable action queries for the future are inserted after the multimodal context, with an optional VQA suffix used during training.The sequence is processed by a causal transformer.Source: 1) Multimodal Sequence Formulation · p. 4
  • The causal attention operation uses query, key, and value vectors, with the key dimensionality setting its scaling.The historical-step count is H, and time indices t≤0 are relative to the current planning moment.Source: 1) Multimodal Sequence Formulation · p. 4
  • The trajectory prediction head operates entirely in the continuous latent space of the frozen prior decoder.The sequence includes visual tokens from multiple camera views, encoded historical actions, and navigation instructions.Source: 2) Latent-Aligned Action Generation · p. 4, 1) Multimodal Sequence Formulation · p. 4

2) Latent-Aligned Action Generation:

LaPla projects action-query hidden states into the frozen decoder’s latent input space, creates temporally consistent candidate perturbations, and decodes them into future waypoints.

2) Latent-Aligned Action Generation:

  • All future action queries are processed in one forward pass, and their hidden states are projected into the frozen decoder’s latent input space.This produces N diverse trajectory candidates from the same semantically grounded representations.Source: 2) Latent-Aligned Action Generation · pp. 4–5
  • Learned basis-vector perturbations create diverse latent candidates, with each perturbation broadcast across timesteps to maintain temporal consistency.Each perturbed latent sequence is decoded into predicted displacement and heading.Source: 2) Latent-Aligned Action Generation · p. 5
  • Decoded displacements are cumulatively summed from the current ego position to reconstruct absolute waypoints.The decoder outputs displacement and heading at each future step.Source: 2) Latent-Aligned Action Generation · p. 5
  • Best-of-N SmoothL1 waypoint supervision drives projected latents toward decoder-manifold regions producing correct trajectories, without auxiliary latent-space supervision.Continuous latent prediction allows fine-grained adjustment and avoids discretization precision loss.Source: 2) Latent-Aligned Action Generation · p. 5
  • A multiround VQA dialogue can be appended after the action placeholders to support visual question answering.The VQA suffix is used during training.Source: 3) VQA as an Auxiliary Task · p. 5

3) VQA as an Auxiliary Task:

The VQA task is incorporated into the unified framework through answer generation conditioned on preceding dialogue context and a joint training loss.

3) VQA as an Auxiliary Task:

  • The model generates each answer autoregressively, conditioned on all preceding tokens.The context includes cached hidden states, questions through the current round, previous answers, and earlier tokens in the current answer.Source: 3) VQA as an Auxiliary Task · p. 5
  • The VQA training examples are randomly extracted from annotated question-answer pairs to expose the model to varying dialogue contexts.The passage states this is intended to enhance generalization capacity.Source: 3) VQA as an Auxiliary Task · p. 5
  • The unified VLA framework is trained end-to-end with a joint loss.Source: 4) Training Objectives · p. 5

4) Training Objectives:

LaPla training leaves the pretrained VQ-VAE frozen while adapting selected VLA components, and the joint-loss weighting parameter balances two terms.

4) Training Objectives:

IV. Experiment

The experiments evaluate LaPla in open-loop nuScenes tests and closed-loop NVIDIA AlpaSim simulation, using different datasets, metrics, and evaluation protocols.

A. Experiment Setup

  • nuScenes open-loop evaluation uses about 30K frames at 2 Hz, measuring L2 distance and collision rate, with VQA annotations added to enhance reasoning.Source: A. Experiment Setup · p. 5
  • AlpaSim closed-loop tests use 60 representative scenarios and measure collisions, off-road events, route completion, success rate, and FPS.The scenarios span about 20 seconds at 10 FPS; 20K frames are collected for fine-tuning.Source: A. Experiment Setup · p. 5
  • Open-loop baselines come from prior work, while closed-loop baselines are reproduced locally; LaPla runs without perturbations or BoN for deterministic evaluation.AlpaSim runs on four RTX-4090 GPUs and communicates with the planner over gRPC.Source: A. Experiment Setup · p. 5

B. Open-Loop Evaluations

Open-loop nuScenes results compare LaPla with prior methods, while ablations and qualitative analyses examine latent alignment, VQA, and trajectory behavior.

B. Open-Loop Evaluations

  • 0.60 m UniAD average L2 error for LaPla compares with 0.66 m for OpenDriveVLA-7B and 0.70 m for AutoVLA.At 3 s, LaPla records 0.98 m UniAD L2 error versus AutoVLA’s 1.16 m, and 0.52 m ST-P3 L2 error versus EMMA’s 0.54 m.Source: 1) Comparative Studies · p. 5, Table I · p. 6

1) Comparative Studies:

  • M3 (full LaPla) reaches 0.60 m UniAD L2 error and 0.52% collision rate, compared with M1’s 1.81 m and 1.48%.Adding the pretrained decoder as a kinematic prior in M2 reduces UniAD L2 error from 1.81 m to 1.13 m and collision rate from 1.48% to 1.04%.Source: 2) Ablation Studies · p. 6, Table II · p. 6

2) Ablation Studies:

  • Removing VQA degrades overall performance, supporting its effectiveness for internalizing planning-oriented reasoning.Source: 2) Ablation Studies · p. 6

3) Qualitative Analysis:

  • In rain and nighttime scenes LaPla suppresses jitter, while bus bypass and urban overtaking show expected lateral movement within the first second.Near-stationary traffic shows mild directional ambiguity; the visualizations span six driving conditions.Source: 3) Qualitative Analysis · p. 6, Figure 4 · p. 7
  • Res-VQ with the {64, 64, 128, 128} configuration achieves 0.0038 m ADE; adding history reduces ADE from 0.0097 m to 0.0038 m and FDE from 0.0162 m to 0.0046 m.Spiral-trajectory reconstruction improves with residual quantization and historical information.Source: Table III · p. 6, 4) Validation of Action Quantization · p. 7, Figure 5 · p. 7

4) Validation of Action Quantization:

Closed-loop AlpaSim results show higher success and route completion for LaPla than the AutoVLA variants, alongside faster inference; the conclusion attributes the approach to continuous latent prediction through a frozen decoder.

C. Closed-Loop Evaluations

V. Conclusion

  • LaPla predicts continuous latent vectors in one forward pass through a frozen residual VQ-VAE decoder, bypassing quantization errors and generation latency.Source: §V · p. 8

Related papers

Loading 2609.04070v1…