Source-linked AI summary
DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Driving
Shuyao Shang, Bing Zhan, Yunfei Yan, Yuqi Wang, Yingyan Li, Yasong An, Xiaoman Wang, Jierui Liu, Lu Hou, Lue Fan, Zhaoxiang Zhang, Tieniu Tan
TL;DR
Driving VLA models need reasoning that captures fine-grained world evolution without the redundancy and latency of textual or visual CoT. DynVLA introduces compact, decoupled Dynamics CoT before action generation and, across multiple benchmarks, demonstrates effectiveness while retaining efficient inference. Its main limitation is that inaccurate dynamics traces can propagate into suboptimal actions in complex or uncertain scenarios.
Problem
Textual CoT lacks fine-grained spatiotemporal understanding, while Visual CoT introduces redundant prediction and high inference latency in driving.
Method
DynVLA uses a Dynamics Tokenizer with decoupled ego-centric and environment-centric representations, then trains models to generate dynamics tokens before actions through SFT and RFT.
Results
Across NAVSIM, Bench2Drive, and a large in-house dataset, DynVLA outperforms non-CoT VLA, Textual CoT, and Visual CoT methods.
Takeaways & Limitations
Dynamics CoT provides compact, physically meaningful, and planning-oriented intermediate reasoning for autonomous-driving VLA models.
Takeaways & Limitations
Under highly complex or uncertain driving scenarios, inaccurate Dynamics CoT traces may propagate errors into subsequent action generation.
Abstract
from arXiv · showhide
We propose DynVLA, a driving VLA model that introduces a new CoT paradigm termed Dynamics CoT. DynVLA forecasts compact world dynamics before action generation, enabling more informed and physically grounded decision-making. To obtain compact dynamics representations, DynVLA introduces a Dynamics Tokenizer that compresses future evolution into a small set of dynamics tokens. Considering the rich environment dynamics in interaction-intensive driving scenarios, DynVLA decouples ego-centric and environment-centric dynamics, yielding more accurate world dynamics modeling. We then train DynVLA to generate dynamics tokens before actions through SFT and RFT, improving decision quality while maintaining latency-efficient inference. Compared to Textual CoT, which lacks fine-grained spatiotemporal understanding, and Visual CoT, which introduces substantial redundancy due to dense image prediction, Dynamics CoT captures the evolution of the world in a compact, interpretable, and efficient form. Extensive experiments on NAVSIM, Bench2Drive, and a large-scale in-house dataset demonstrate that DynVLA consistently outperforms Textual CoT and Visual CoT methods, validating the effectiveness and practical value of Dynamics CoT. Project Page: https://yaoyao-jpg.github.io/dynvla.
1. Introduction
Existing driving VLA CoT methods struggle to balance fine-grained spatiotemporal reasoning with efficient inference. DynVLA addresses this by reasoning over compact, decoupled future dynamics before generating actions, and reports consistent gains across benchmarks.
- Textual CoT provides high-level decision logic but struggles with the fine-grained spatiotemporal relationships required by physically constrained driving.
- Visual CoT improves spatiotemporal representation but redundantly predicts decision-irrelevant background and texture details, increasing learning difficulty and computational cost.
- Dynamics CoT compresses future dynamics into a small number of tokens before action generation, shortening reasoning traces and reducing inference latency by over an order of magnitude.
- The Dynamics Tokenizer decouples ego-centric motion from environment-centric changes and applies physically grounded regularization to reduce ambiguous dynamics representations.
- Across NAVSIM, Bench2Drive, and a large in-house dataset, Dynamics CoT outperforms non-CoT, Textual CoT, and Visual CoT methods.
2. Related Works
Prior VLA research has explored language-based and visual reasoning, including CoT methods for robotics and autonomous driving. DynVLA extends this direction by structuring driving reasoning around compact future dynamics.
- VLA research has progressed from LLM-based systems toward VLM-based policies for robotics and end-to-end autonomous driving.
- Robotics CoT methods add structured textual reasoning, adaptive slow reasoning, or visual future prediction before action generation.
- DynVLA’s structured Dynamics CoT first generates discrete dynamics tokens and then action tokens, providing a distinct alternative to textual or pixel-based reasoning.
3. Method
DynVLA learns compact, decoupled dynamics tokens and trains the model to generate them before actions, combining supervised fine-tuning with reinforcement fine-tuning. Its tokenizer reconstructs future observations while using action-based regularization and cross-view decoding to separate ego and environment dynamics.
- Training pipeline: DynVLA trains a Dynamics Tokenizer, applies SFT to structured Dynamics CoT, and refines the policy with RFT.The training pipeline proceeds from dynamics-token learning to dynamics-before-action generation and trajectory-level optimization.
- Dynamics encoding: The encoder uses small sets of learnable queries to produce separate ego-centric and environment-centric dynamics tokens from adjacent observations.The two token branches represent ego motion and external scene changes, while keeping the dynamics representation compact.
- Dynamics encoding: Separate VQ codebooks discretize the continuous ego-centric and environment-centric representations into dynamics tokens used by the decoder.Nearest-neighbor codebook assignment produces discrete tokens, which are mapped back to continuous embeddings for decoding.
- Dynamics reconstruction: Action-based regularization aligns ego-centric dynamics with ego motion, encouraging disentanglement from environment-centric dynamics.The regularizer penalizes the discrepancy between predicted and ground-truth actions.
- Dynamics reconstruction: The decoder reconstructs future image and BEV states from shared dynamics representations, with cross-view consistency encouraging the same scene evolution across modalities.Image and BEV decoders are conditioned on the same dynamics representation and their respective current-state patch sequences.
- Dynamics-before-action policy learning: Dynamics CoT SFT orders future dynamics tokens before action tokens, while RFT uses trajectory and format rewards with GRPO and KL regularization.The structured target places dynamics reasoning between BOD/EOD markers before action generation, and RFT preserves that template during policy optimization.
4. Experiments
DynVLA is evaluated across real-world, closed-loop, and large-scale in-house driving settings, where it consistently outperforms strong baselines and alternative CoT designs. Further analyses show that Dynamics CoT improves effectiveness and efficiency, while decoupled tokenizer design supports transferable dynamics representations and safer, more foresighted planning.
- Main benchmark results: DynVLA achieves the highest PDMS on NAVSIM, outperforming traditional end-to-end and recent VLA-based methods.
- Main benchmark results: DynVLA achieves the best performance across all metrics on Bench2Drive’s long-horizon, interactive driving scenarios.
- Main benchmark results: DynVLA achieves the lowest ADE and Collision Rate on the large-scale in-house dataset, indicating more reliable motion prediction and safer maneuver decisions.
- Interpretability and planning behavior: Transferred dynamics tokens preserve ego motion and govern surrounding-agent motion, while Dynamics CoT supports intent-aware, foresighted, and road-geometry-aware planning.
- CoT design and efficiency: Dynamics CoT delivers the best overall performance without substantial latency overhead, whereas visual and optical-flow reasoning incur higher latency.
- Tokenizer ablations: Dynamics decoupling increases activated VQ codes, prevents codebook collapse, and improves planning when combined with dual image-BEV supervision.
5. Conclusion
DynVLA introduces Dynamics CoT for VLA-based autonomous driving, using disentangled dynamics tokens to reason about future scene evolution before generating actions. SFT and RFT train this process, with experiments across multiple benchmarks supporting its effectiveness.
- DynVLA introduces Dynamics CoT, which reasons over compact future dynamics before action generation.The approach aims to reduce reasoning redundancy and latency while retaining spatiotemporal understanding.
- Its Dynamics Tokenizer disentangles ego-centric and environment-centric dynamics and regularizes them with physically grounded supervision.The representations are trained with ego action supervision and cross-view consistency.
- SFT enables explicit future-dynamics reasoning, while RFT further improves decision quality.
- Experiments across multiple benchmarks demonstrate DynVLA's effectiveness and identify Dynamics CoT as a promising direction for reasoning-based VLA models.
A. Datasets and Metrics
The evaluation uses NAVSIM, Bench2Drive, and a large-scale in-house dataset, combining metrics for safety, efficiency, rule compliance, ability, displacement, and collisions.
- NAVSIM: NAVSIM evaluates urban driving with PDMS, a scalar metric aggregating safety, efficiency, temporal-risk, and comfort measures.PDMS integrates No At-Fault Collision, Drivable Area Compliance, Ego Progress, Time-to-Collision, and Comfort.
- NAVSIM: In PDMS, NC and DAC gate safety, while EP and TTC measure efficiency and temporal risk margins, and C measures comfort.
- Bench2Drive: Bench2Drive provides closed-loop interactive-traffic evaluation through Success Rate, Driving Score, and mean Multi-Ability.Multi-Ability averages Merging, Overtaking, Emergency Brake, Give Way, and Traffic Sign performance.
- In-House Dataset: The in-house dataset contains 700k frames with diverse driving distributions, and evaluation emphasizes safety-critical and long-horizon cases.Reported metrics include Average Displacement Error and Collision Rate within 3 seconds.
B. Implementation Details
Implementation uses compact, factorized dynamics tokens, reconstruction and regularization objectives, and staged SFT followed by RFT for Dynamics CoT training.
- Dynamics Tokenizer: Each scene uses 8 dynamics tokens: 4 ego-centric and 4 environment-centric, with 64-entry codebooks and 32-dimensional VQ embeddings.The tokenizer uses a Transformer architecture with hidden dimension 1024.
- Dynamics Tokenizer: The tokenizer reconstructs future observations while conditioning on current observations, and combines MSE, LPIPS, VQ, and action-regularization losses.Action regularization uses relative ego motion between frames, while BEV maps provide supervision.
- Dynamics CoT SFT: Dynamics CoT SFT uses a pretrained EMU3 base and a 128-type dynamics codebook alongside FAST action tokens.The model vocabulary reserves separate token ranges for action and dynamics tokens.
- Dynamics CoT RFT: RFT is applied on top of the SFT model using trajectory and format rewards with equal weighting.Training runs for 6k steps on 6 NVIDIA H800 GPUs.
C. More Ablation Studies
Ablations show that Dynamics CoT benefits from moderate temporal lookahead and a compact, sufficiently expressive token bottleneck, with decoded futures capturing ego and surrounding-agent motion.
- Prediction Horizon: Each future dynamics token encodes the transition between adjacent images separated by 1 second.The prediction horizon K counts the number of future dynamics steps.
- Prediction Horizon: Dynamics CoT consistently improves PDMS over no-CoT baselines across prediction horizons.
- Prediction Horizon: Increasing the horizon from K = 1 to K = 2 yields clear gains, while longer horizons produce diminishing returns and higher inference latency.Longer horizons introduce greater future uncertainty and less reliable predicted dynamics.
- Number of Dynamics Tokens: Using 8 dynamics tokens achieves the best PDMS, indicating that a compact but sufficiently expressive dynamics bottleneck is important for planning-relevant scene evolution.
- Decoded Dynamics: Dynamics Tokenizer visualizations show decoded future images and BEV maps alongside current and ground-truth views, faithfully capturing ego motion and surrounding-agent dynamics.
D. More Qualitative Comparisons
Qualitative visualizations show that Dynamics CoT captures transferable motion representations and improves planning by anticipating future scene evolution.
- Decoded futures closely align with ground-truth evolution in image and BEV space, reflecting ego motion and surrounding-agent behavior.These visualizations indicate that the Dynamics Tokenizer compresses planning-relevant dynamics.
- Cross-scenario visualizations show that learned dynamics capture disentangled and transferable motion representations that generalize across scenes.
- Additional qualitative examples demonstrate the effect of Dynamics CoT on planning behavior.
- Compared to models without Dynamics CoT, dynamics reasoning enables anticipation of future scene evolution and improves decision-making.
E. Limitations and Future Works
Dynamics CoT benefits safety-critical planning but can produce suboptimal decisions when its reasoning is incorrect; future work considers richer priors and fast-slow deployment.
- Limitations: Incorrect Dynamics CoT reasoning traces may induce suboptimal decisions despite providing structured intermediate reasoning for safety-critical planning.
- Future Works: Additional visualizations examine the transferability of learned dynamics across scenarios.
- Future Works: A fast-slow driving architecture could pair lower-frequency long-horizon dynamics reasoning with a lightweight planner that acts every control cycle.
F. Failure Cases
Failure cases arise when inferred future dynamics misrepresent other agents, road geometry, or scene evolution under degraded visual observations, producing suboptimal or unsafe plans.
- Agent intentions: Misinference of other agents’ intentions can cause the planner to predict continued forward motion and follow with a straight trajectory.
- Road geometry: During complex turning, incorrect future road-structure reasoning can make the model treat a non-drivable parking area as drivable.Predicting road geometry visible only from a novel perspective is challenging.
- Degraded observations: Severely degraded observations, such as heavy rain causing camera occlusion and blur, can make inferred dynamics ambiguous and lead to unsafe decisions.
- Failure overview: Representative failure cases include incorrect vehicle-intention inference, drivable-area misidentification during large turns, and ambiguous dynamics reasoning from degraded observations.