PaperlayerView exact revision on arXiv

Source-linked AI summary

Planning in the Backbone: DiffAdapterVLA for Native Continuous Trajectory Generation with Driving VLMs

Changxin Lu, Xiaoliang Meng, Yu Wu, Rui Huang, Honglin Li, Tao Chen, Kaixuan Zhou, Yadong Shao

arXiv:2609.15322v3cs.ROcs.AI

TL;DR

Driving VLMs provide rich driving representations, but existing approaches separate their condition computation from continuous trajectory formation. DiffAdapterVLA introduces trajectory state into selected late VLM layers for recursive refinement with depth-wise conditions. On NAVSIM, it reports higher PDMS and lower latency than Alpamayo’s second-stage action expert, using few trainable parameters.

  • Problem

    Existing planning interfaces start trajectory evolution after VLM condition computation or turn continuous planning into discrete generation, leaving depth-wise conditions outside continuous trajectory formation.

    Source: §1 · p. 2

  • Method

    DiffAdapterVLA inserts explicit trajectory state into selected late VLM layers, where lightweight layer-wise DiffAdapters refine it with evolving conditions and asymmetric attention preserves the condition stream.

    Source: §1 · p. 3

  • Results

    On NAVSIM, PDMS improves from 71.1 to 88.3 over Alpamayo’s second-stage action expert while latency falls from 1003.8 to 80.4 ms per sample.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    NAVSIM results support continuous planning in which trajectory state participates in backbone condition computation, with high-quality, low-latency planning using few trainable parameters.

    Source: §1 · p. 3, §6 · p. 10

    Planning is bounded by backbone-available conditions, and the explicit state covers ego motion while other participants’ multimodal futures and uncertainty remain primarily implicit.

    Source: D LIMITATIONS AND FUTURE WORK · p. 19

Abstract

from arXiv · show

Pretrained driving vision-language models (VLMs) integrate visual, route, language, and driving context into rich driving priors, yet their representation objectives remain separated from continuous driving planning. Existing methods typically begin trajectory generation only after the VLM has formed a final condition, leaving depth-wise condition computation outside the stepwise formation of trajectory state. We introduce DiffAdapterVLA, which realizes Planning in the Backbone: it injects explicit trajectory tokens into selected VLM late layers, bringing trajectory state into backbone forward computation, where it co-evolves with driving conditions at different depths. Lightweight layer-wise DiffAdapters organize this computation into recursive trajectory refinement, while asymmetric joint attention preserves directed guidance from the condition stream to trajectory planning. By placing planning within existing backbone computation rather than relying on an independent trajectory planner, DiffAdapterVLA adapts only lightweight trajectory modules to turn existing driving priors into efficient continuous planning capability. NAVSIM results show that it achieves high-quality closed-loop planning with low end-to-end latency using few trainable parameters, and demonstrate that jointly evolving trajectory state and depth-wise driving conditions in VLM late-layer computation effectively realizes continuous trajectory planning.

1 INTRODUCTION

Driving VLMs contain rich driving representations, but their objectives do not naturally produce precise continuous trajectories. DiffAdapterVLA brings explicit trajectory state into late-layer backbone computation, where it evolves with driving conditions through lightweight refinement.

1 INTRODUCTION

  • Existing interfaces either start trajectory evolution after VLM condition computation or reformulate continuous planning as discrete action-token generation.Loosely coupled and final-condition approaches defer trajectory formation; autoregressive VLAs serialize actions or trajectories as tokens.Source: §1 · p. 2
  • Driving VLMs progressively integrate visual evidence, route, language, motion history, and traffic interactions, but final-output planning cannot use those depth-wise signals as trajectory state forms.The paper calls its proposed interface “backbone-native continuous planning,” or “Planning in the Backbone.”Source: §1 · p. 2
  • DiffAdapterVLA recursively refines continuous trajectory states within selected backbone layers, enabling low-latency planning without an external trajectory planner.The architecture uses lightweight layer-wise DiffAdapters.Source: §1 · p. 3
  • The interface places an explicit trajectory state in late-layer computation, where it forms progressively from driving conditions at different depths.This makes trajectory formation part of the pretrained VLM’s representational process.Source: §1 · pp. 2–3
  • DiffAdapterVLA is reported to achieve high-performance, low-latency continuous planning with few trainable parameters.Source: §1 · p. 3
  • Condition-preserving asymmetric joint attention directs information flow from trajectory denoising to backbone condition computation while preserving the original condition stream.Source: §1 · p. 3

2 RELATED WORK

Prior approaches connect driving representations to planning through shared scene features, downstream planners or experts, and discrete action tokens. DiffAdapterVLA instead evolves a continuous trajectory state within a driving VLM’s late-layer computation.

2 RELATED WORK

  • Prior E2E and continuous-planning work shares scene representations or uses generative methods, but trajectory generation generally remains a head or module after scene encoding.The cited approaches include multimodal fusion, latent-world models, diffusion, and flow matching.Source: §2 · p. 3
  • Driving VLAs pass representations to planners or action components, or predict discrete action tokens; DiffAdapterVLA instead evolves a trajectory state within frozen late layers.Source: §2 · p. 3
  • Other native continuous-action policies jointly model actions with visual, linguistic, or historical context, including approaches that combine language generation with flow-matching trajectories.Examples include noisy action tokens alongside context tokens, joint action-video modeling, DITA, and MindVLA-U1.Source: §2 · p. 3

3 PRELIMINARIES

The planning setup predicts a continuous future trajectory from multimodal driving conditions. A frozen driving VLM encodes those conditions, while conditional diffusion trains and samples the trajectory prediction.

3 PRELIMINARIES

  • Planning is framed as predicting a continuous trajectory of H future waypoints from driving conditions at time t.The conditions include visual observations from M cameras, route or navigation information, and vehicle history or other available driving context; each waypoint has state dimension d.Source: §3 · p. 4
  • A pretrained driving VLM with L Transformer layers propagates condition tokens through depth, with its backbone parameters kept frozen during adaptation.The specified backbone is Cosmos-Reason2-8B from the released Alpamayo-1.5-10B driving VLA.Source: §3 · p. 4
  • Conditional diffusion adds Gaussian noise to a clean trajectory, then trains a denoiser to predict that added noise from the noisy trajectory, diffusion step, and driving condition.Source: §3 · pp. 4–5
  • At inference, the trajectory starts from Gaussian noise and is iteratively recovered by deterministic x0 projection using DDPM parameterization and reduced-step DDIM sampling.Conventional conditional generation uses the final VLM representation as a fixed condition in an external denoiser.Source: §3 · p. 5

4 METHODOLOGY

DiffAdapterVLA inserts trajectory tokens into selected late VLM layers, where layer-wise adapters refine trajectory state alongside driving conditions. Asymmetric attention preserves the condition stream while allowing trajectory tokens to read valid conditions.

4 METHODOLOGY

  • Trajectory tokens enter selected late VLM layers and are recursively refined alongside driving conditions, with a lightweight readout and deterministic x0 projection producing the planned trajectory.The model alternates frozen late-layer computation and layer-specific DiffAdapters; denoising uses five deterministic x0-projection steps and reuses cached conditions.Source: §4 · p. 5, §4.2 · pp. 6–7

4.1 HISTORY TRAJECTORY ENCODER

  • The history encoder represents ego positions, local motion, and validity in waypoint features, then masked-pools and transforms them into shared history context gt.This context is shared across selected late-layer DiffAdapters.Source: §4.1 · p. 5

4.2 DIFFADAPTER

  • After early frozen layers, appended trajectory tokens traverse selected late layers, where independent adapters update trajectory state using timestep modulation and condition and history inputs.Adapters use temporal convolution and self-attention to mix waypoint information, then read condition tokens through cross-attention.Source: §4.2 · p. 6

4.3 CONDITION-PRESERVING ASYMMETRIC JOINT ATTENTION

  • Condition queries cannot read noisy trajectory tokens, while trajectory tokens read valid condition tokens and interact bidirectionally with one another.This directed attention preserves the condition stream’s original propagation while giving trajectory updates access to driving representations at the current depth.Source: §4.3 · p. 7

5 EXPERIMENTS

DiffAdapterVLA is evaluated on NAVSIM navtest with a frozen driving VLM and lightweight trainable modules. Results cover planning quality, inference efficiency, component ablations, and how trajectory state evolves with depth-wise conditions.

5.1 EXPERIMENTAL SETUP

  • Evaluation uses NAVSIM navtrain’s 1,192 scenes and navtest’s 136 scenes; the frozen VLM backbone is adapted through trajectory embedding, DiffAdapters, history encoding, and trajectory readout.The model takes front-camera observations, route information, and four historical ego states, then predicts eight future poses.Source: §5.1 · p. 7

5.2 MAIN RESULTS AND ABLATION STUDY

  • Against Alpamayo’s second-stage action expert, DiffAdapterVLA raises PDMS from 71.1 to 88.3 while reducing end-to-end latency from 1003.8 to 80.4 ms per sample, a 12.5× speedup.It also improves PDMS by 6.4 points over a lightweight DiT planner at comparable latency: 80.4 versus 67.6 ms.Source: §5.2 · p. 8, Table 2 · p. 9
  • The full model reaches 86.9 PDMS versus 85.3 without the history encoder and 85.6 with bidirectional interaction; omitting both DiffAdapter and history encoding lowers PDMS to 58.2.The full design’s EP is 80.5, the highest among the ablations.Source: §5.2 · p. 8, Table 3 · p. 9
  • Across two NAVSIM navtest scenes, DiffAdapterVLA follows ground-truth trajectories more closely than OneVL and ReCogDrive, particularly through the roundabout curve.The comparison is qualitative.Source: §5.2 · p. 8, Figure 3 · p. 9
  • PDMS rises from 80.2 with final-condition generation to 85.2 with layer-wise conditions and 86.9 when trajectory tokens jointly enter late-layer computation.The three interface settings hold backbone, adapter capacity, training data, and sampling schedule fixed.Source: §5.2 · pp. 8–9, Table 4 · p. 9
  • Figure 4 tracks update RMS and trajectory-to-condition residuals across adapters and denoising steps; late-adapter updates grow near t=0, while residuals peak at adapters 4–5 mid-denoising.At t=80, adapters 0 and 1 make relative updates of about 68% and 70%.Source: Figure 4 · p. 9, §5.2 · p. 9

6 CONCLUSION

DiffAdapterVLA brings continuous trajectory planning into a driving VLM’s backbone, where trajectory state evolves with depth-wise driving conditions. NAVSIM results support this approach as a lightweight, low-latency route to planning with existing driving priors.

6 CONCLUSION

  • DiffAdapterVLA inserts explicit trajectory state into selected late layers, where it evolves with depth-wise driving conditions throughout denoising.Lightweight DiffAdapters turn this computation into layer-wise trajectory refinement.Source: §6 · p. 9
  • The framework trains only a small set of parameters and achieves high-quality, low-latency continuous planning without an independent trajectory planner.The authors report NAVSIM results as validation of its planning performance and efficiency.Source: §6 · p. 10
  • The paper concludes that trajectory state participating in backbone condition computation benefits continuous planning compared with reading the final condition once.The proposed adaptation is presented as a way for existing driving priors to shape future motion.Source: §6 · p. 10

AI USE STATEMENT

The authors disclose AI assistance in research and manuscript preparation, and state that the work was reviewed by the authors. The appendix provides implementation, evaluation, results, limitation, and qualitative-comparison material.

AI USE STATEMENT

  • AI tools assisted with related-work retrieval, research ideation, implementation, and manuscript editing for grammar, wording, and clarity.The authors say they reviewed and verified all AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • The authors state that generative AI was not used to generate synthetic datasets or prove mathematical claims.Source: AI USE STATEMENT · p. 10

A APPENDIX

  • The appendix supplies full implementation and evaluation details, additional results, limitations and future directions, and further qualitative trajectory comparisons.Source: A APPENDIX · p. 16

B FULL IMPLEMENTATION DETAILS

The appendix details NAVSIM scoring, training and inference protocols, and component and interface studies. It also defines the activation statistics used to characterize layer- and timestep-dependent trajectory refinement.

Evaluation Metric

  • PDMS evaluates closed-loop planning by combining no-fault collision, drivable-area compliance, time-to-collision, comfort, and ego progress.Values are averaged over navtest and reported on a percentage scale; driving-direction compliance has zero weight in this configuration.Source: B FULL IMPLEMENTATION DETAILS · p. 16
  • The rollout checks drivable-area compliance throughout, evaluates TTC by short-horizon motion extrapolation, and scores comfort across six acceleration, jerk, and yaw quantities.Progress is safety-gated and normalized against the prediction and PDM reference proposals.Source: B FULL IMPLEMENTATION DETAILS · p. 16

Training Protocol for Main Results

  • SFT trains the trajectory modules for 80 epochs with a DDPM noise-prediction objective; RFT then applies DiffGRPO for 10 epochs to the same trainable modules.The twelve DiffAdapters operate on the final K = 12 VLM layers, and the VLM backbone remains frozen.Source: B FULL IMPLEMENTATION DETAILS · p. 16

Inference Protocol for Table 2

  • Inference uses five equally spaced deterministic x0-projection steps, while the shared Table 2 protocol reports model and end-to-end latency per sample after warm-up.End-to-end latency includes input preparation, trajectory post-processing, and the evaluation interface; early-layer condition states are cached across denoising steps.Source: B FULL IMPLEMENTATION DETAILS · p. 17

Protocols for Component and Interface Studies

  • Table 3 variants share navtrain data, 50 epochs, inputs, DDPM objective, optimizer configuration, and five-step inference; Table 4 varies only the trajectory–VLM interface.The ablations remove history context, replace directed with bidirectional attention, or remove layer-wise trajectory updates.Source: B FULL IMPLEMENTATION DETAILS · p. 17

Layer-wise Dynamic Analysis

  • Figure 4 panel (a) reports log normalized update RMS and panel (b) residual RMS; together these statistics characterize depth- and timestep-dependent refinement and condition access.The update is ΔZ_l,s; the residual is the trajectory-to-condition cross-attention residual R_cross.Source: B FULL IMPLEMENTATION DETAILS · p. 17

C MORE RESULTS

DiffAdapterVLA’s extended NAVSIM v2 evaluation reports a higher EPDMS than two comparison models. Results also show a late-layer span sweet spot, a five-step efficiency trade-off, and stable condition interaction during training.

Late-layer Span for Backbone-Native Planning

  • PDMS rises from 73.0 at K = 4 to 86.9 at K = 12, then falls to 82.1 at K = 16 and 81.1 at K = 20.The strongest interval is in later backbone layers, while earlier layers form driving conditions.Source: C MORE RESULTS · pp. 17–18, Table 5 · p. 18

Quality–Efficiency Trade-off across Denoising Steps

  • Five denoising steps retain near-saturated quality: from five to ten steps, PDMS changes from 87.00 to 87.07 while latency rises from 80.8 to 126.8 ms per sample.A single step does not recover a usable trajectory; the gain from two to five steps is steady.Source: C MORE RESULTS · p. 18, Table 6 · p. 18

Extended Predictive Driver Model Score

  • EPDMS 86.3 outperforms DriveVLA-W0 by 0.2 points and ReCogDrive by 2.7 points under NAVSIM v2 extended evaluation.The model also reports 88.2 EP, 97.7 TTC, 99.1 DDC, and 99.8 TLC.Source: C MORE RESULTS · p. 18, Table 7 · p. 18

Condition Interaction during Training

  • Condition-interaction RMS rises from about 0.28 to 0.36–0.37 and remains stable after the SFT flow loss converges.The curves use separate vertical axes.Source: C MORE RESULTS · p. 19, Figure 5 · p. 19

D LIMITATIONS AND FUTURE WORK

The method is bounded by the information available to the backbone’s conditions, and the explicit trajectory state represents only ego motion. Evaluation beyond NAVSIM’s planning-oriented closed-loop setting remains important for deployment assessment.

Limitations and Future Work

  • Planning remains bounded by backbone conditions combining visual observations, route and language prompts, and motion history.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • The explicit state covers only ego future motion; other participants’ multimodal futures and uncertainty remain primarily implicit in the condition representation.The proposed direction is to maintain interactive environment states alongside ego trajectories.Source: D LIMITATIONS AND FUTURE WORK · p. 19
  • NAVSIM provides planning-oriented closed-loop evaluation, while broader road distributions, sensor degradation, and end-to-end system latency remain unassessed settings for deployment evaluation.Source: D LIMITATIONS AND FUTURE WORK · p. 19

E ADDITIONAL QUALITATIVE RESULTS

The selected examples come from remaining top-ranked contrastive cases and compare three models against the same ground-truth trajectory.

Additional Qualitative Results

  • Each row compares OneVL, ReCogDrive, and DiffAdapterVLA with the same ground-truth trajectory in selected remaining top-ranked cases.The two scenes shown in the main paper are excluded.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 19

Related papers

Loading 2609.15322v3…