Source-linked AI summary
AUGUstus: Autonomous Driving VLA Model with Unified Generation and Understanding
Yuqi Ye, Caigao Jiang, Zhuonan Wang, Congyuan Pei, Chao Qu, Changhao Peng, Junhong Lin, Qize Yu, Zetian Song, Guangzhao Fu, Chongruo Wu, Shiyu Huang, Shuchang Zhou, Honggou Yang, Wei Gao
TL;DR
Current driving VLAs struggle to combine VLM understanding with low-latency action generation. AUGUstus unifies diffusion-based action generation and autoregressive CoT understanding, with action-first ordering and joint action–CoT optimization. It reports strong results across NAVSIMv1, NAVSIMv2, and nuScenes, and demonstrates practical scalability through internal validation and real-vehicle deployment.
Problem
Current VLAs either underuse VLM understanding in low-latency action generation or serialize CoT before actions, making real-time deployment difficult.
Source: §1 · p. 2Method
AUGUstus unifies diffusion-based action generation and autoregressive CoT understanding in one model, using action-first ordering and AUGUstus-GRPO to jointly optimize action and CoT.
Source: §1 · pp. 2–3Results
94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2 without a scorer module; nuScenes results are 0.27 m average L2 error and 0.05% average collision rate, best among compared methods.
Source: §5 · p. 9Takeaways & Limitations
AUGUstus demonstrates practical scalability through large-scale training, internal driving-dataset validation, and real-vehicle deployment.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Vision-language-action (VLA) models show strong potential for end-to-end autonomous driving by leveraging world knowledge and understanding to excel in long-tail scenarios. However, current VLAs have two limitations: 1) Degeneration into vision-action (VA) models. For low latency, regression/flow-based methods directly generate actions but underuse VLM understanding, causing unsafe, unexplainable actions. 2) Autoregressive high latency. Autoregressive methods sequentially generate chain-of-thought (CoT) and action tokens, fully leveraging VLM understanding, but high latency hinders deployment. To enable low-latency deployment while using CoT to assist action generation, we propose AUGUstus, an Autonomous-driving VLA model with Unified Generation and Understanding. We unify diffusion-based action generation and autoregressive CoT understanding in one model, where understanding only serves generation. Joint CoT supervision preserves VLM spatial understanding in shared representation for safer, more accurate trajectories. AUGUstus-mode orders action generation before CoT, so CoT supervises reliable actions during training while inference needs only action output, enabling low latency. AUGUstus-GRPO, a unified GRPO algorithm, jointly optimizes action and CoT, improving action performance. Experiments show AUGUstus achieves 94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2, SOTA planning without scorer module. On nuScenes, it achieves average L2 error 0.27 m and average collision rate 0.05%, best among compared methods. To validate AUGUstus practicality, we train on in-house data with 512 NVIDIA H800 GPUs for ~2 days, evaluate on an internal 1,000 km dataset, and preliminarily deploy on a real vehicle, achieving 12 Hz operation. Project page: https://augustus-ad.github.io
1 Introduction
AUGUstus addresses the tension between VLM-informed action generation and low-latency control by jointly training diffusion actions and autoregressive CoT, with action-first inference. The paper reports strong benchmark performance and preliminary real-vehicle operation.
1 Introduction
- Current VLA designs either underuse VLM understanding in direct or diffusion-based action generation, or serialize CoT before actions and incur high latency.Earlier end-to-end driving models also face limited model capacity and training data, restricting policy world knowledge for long-tail and semantically complex scenes.Source: §1 · pp. 1–2
- AUGUstus combines diffusion-based action generation and autoregressive CoT understanding in one model, with joint CoT supervision preserving VLM spatial understanding for action generation.AUGUstus-mode orders action before CoT, so inference needs only the action output.Source: §1 · pp. 2–3
- Among explored CoT designs, descriptions of other agents’ positions and future orientations were most effective for action.The paper frames understanding as serving safer, more accurate trajectory generation rather than being trained for its own sake.Source: §1 · p. 2
- 94.6 PDMS on NAVSIMv1 and 90.7 EPDMS on NAVSIMv2 are reported without a scorer module; nuScenes results are 0.27 m average L2 error and 0.05% average collision rate.The nuScenes results are reported as best among compared methods.Source: §1 · p. 2
- The paper reports preliminary real-vehicle operation at 12 Hz after training on in-house data with 512 NVIDIA H800 GPUs for approximately two days.The model was evaluated on an internal 1,000 km driving dataset.Source: §1 · p. 3
- AUGUstus-GRPO jointly optimizes action and CoT, rewarding reasoning for serving planning and thereby improving action performance.It combines closed-form Gaussian importance-sampling ratios and KL terms for JaT-GRPO with token-level GRPO for CoT-GRPO.Source: §1 · p. 3
2 Related Work
Prior driving VLA designs trade off VLM semantics, action distributions, and latency. AUGUstus instead places action and CoT in one transformer, with reasoning trained to serve action generation.
2 Related Work
- Query-based policies are low-latency but lack action distributions for RL and weakly leverage VLM semantics; external flow-matching experts model continuous action distributions but weaken VLM semantics.Autoregressive policies expose semantic reasoning but serialize CoT and action tokens, incurring high latency.Source: §2 · p. 3
- AUGUstus unifies continuous action tokens and CoT understanding in one transformer, trains understanding to serve action generation, and outputs actions directly at deployment.This design addresses the related-work trade-off between VLM understanding and low latency.Source: §2 · p. 3
- Driving VLA post-training includes direct action supervision and token-level output supervision; AUGUstus-GRPO jointly optimizes CoT and action under a shared group-relative advantage.The passage notes that direct action supervision can be prone to reward hacking and token-level methods slow training and inference.Source: §2 · p. 3
3 Methodology
AUGUstus unifies continuous action generation and autoregressive CoT understanding in one transformer, with training and reinforcement learning designed to make the reasoning stream serve planning while inference can skip CoT.
3.1 Unified framework
- AUGUstus combines continuous flow-based trajectory generation and autoregressive CoT in one transformer; action-first ordering lets inference evaluate only actions for low latency.The action block is bidirectional and cannot attend to CoT tokens, while CoT attends to the complete action block.Source: §3 · p. 3, §3.1 · p. 4, Figure 2 · p. 4
3.2 Supervised Fine-Tuning for AUGUstus
- Supervised fine-tuning jointly trains conditional flow matching for actions and teacher-forced language modeling for CoT.Training noises the target trajectory, then predicts a clean trajectory estimate and the next CoT token.Source: §3.2 · p. 4, §3.1 · p. 4
- CoT predicting key-object bounding boxes and future positions is reported as more beneficial for safe, efficient actions than perception or planning-reasoning CoT.Additional CoT experiments are provided in the supplementary material.Source: §3.2 · p. 5
3.3 Reinforcement Learning for AUGUstus
- AUGUstus-GRPO pairs action and CoT rollouts under a shared normalized group-relative advantage, optimizing actions with JaT-GRPO and CoT with token-level likelihoods.JaT-GRPO applies GRPO to the continuous action branch under the x0 parameterization.Source: §3.3 · p. 5
- Stage I uses PDMS as the planning reward for progress; Stage II switches to terminal and safety rewards while retaining the learned progress and prediction reward.Both stages use the shared reward and advantage to update JaT-GRPO and CoT-GRPO; only the planning reward changes.Source: §3.3 · p. 6
4 Experiments
AUGUstus is evaluated on public driving benchmarks, an internal 1,000 km dataset, and a preliminary real-vehicle test. The reported results include strong benchmark scores, favorable internal safety, comfort, and efficiency performance, and real-time operation.
4.1 Benchmarks and Baselines
- Public evaluation uses NAVSIM and nuScenes; the in-house evaluation uses data collected without public datasets and includes a preliminary Guangzhou real-vehicle test.Training used 512 NVIDIA H800 GPUs for approximately two days with a global batch size of 2048; public benchmark inputs are 2-second, 4-frame clips, with benchmark-specific waypoint horizons.Source: §4.1 · p. 6
4.2 Main Results
- AUGUStus reports 94.0/94.6 PDMS with 1/3-step denoising on NAVSIMv1, 90.7 EPDMS on NAVSIMv2 with the filter (87.1 without), and nuScenes 0.27 m average L2 error/0.05% average collision rate.NAVSIMv2 and nuScenes use only 1-step denoising; the no-scorer condition applies to NAVSIMv1.Source: §4.2 · p. 7
- AUGUstus performs best on internal safety, comfort, and efficiency, while flow matching is slightly better on compliance; operation reaches 13.6 Hz on one H800 and 12.0 Hz on the real vehicle.The internal 1,000 km evaluation reports macro-averages across safety, comfort, efficiency, and compliance, with AUGUstus fixed at 0%.Source: §4.2 · p. 7, Table 4 · p. 8, Table 5 · p. 8
4.3.1 Ablation Study of AUGUstus-mode and CoT
- With AUGUstus-mode, adding CoT to SFT improves performance and safety, while the mode retains comparable performance with real-time inference.The CoT-attention visualization reports 46% in-box attention on the nearby risk object with CoT supervision versus 0% without it.Source: §4.3.1 · pp. 8–9, Table 6 · p. 8, Figure 6 · p. 9
4.3.2 Ablation Study of AUGUstus-GRPO
- AUGUstus-GRPO stage I improves PDMS by +4 and stage II adds +0.6, reaching 94.6 PDMS.The ablation jointly optimizes JaT-GRPO and CoT-GRPO.Source: §4.3.2 · p. 9, Table 7 · p. 9
4.4 Visualization
- In the illustrated right-turn scene, the no-CoT model collides after failing to yield, whereas CoT supervision yields a safer trajectory; AUGUstus avoids collision while maintaining high progress against other methods.Figure 7 shows trajectory visualizations with and without CoT and comparisons with other methods.Source: §4.4 · p. 9, Figure 7 · p. 9
5 Conclusion and Future Work
AUGUstus unifies diffusion-based action generation and autoregressive CoT understanding, ordering actions before CoT for low-latency inference and jointly optimizing both with AUGUstus-GRPO. It reports state-of-the-art results on NAVSIM and nuScenes, plus preliminary real-vehicle operation; broader vehicle testing and robot-VLA applications remain future work.
5 Conclusion and Future Work
- AUGUstus combines diffusion-based action generation and autoregressive CoT in one transformer, using action-first AUGUstus-mode and joint AUGUstus-GRPO optimization.The reported benchmark results are 94.6 PDMS on NAVSIMv1, 90.7 EPDMS on NAVSIMv2, and nuScenes average L2 error of 0.27 m with average collision rate of 0.05%.Source: §5 · p. 9
- The authors propose more extensive real-vehicle testing and exploring application to robot VLA as future work.The paper reports preliminary real-vehicle deployment and real-time operation.Source: §5 · p. 10
A Supplementary Architecture Details
The appendix details how AUGUstus represents and processes waypoint actions alongside multimodal and CoT tokens, then describes the reverse-time sampling and GRPO components. It also specifies the exploration-noise condition and settings used for single-step GRPO.
A.1 Backbone and action projections
- Eight noisy future waypoints and flow time are embedded as action tokens, processed after multimodal context, and decoded to metric waypoint coordinates; the action block is bidirectional and deployment evaluates only that block.CoT tokens may read action states but action tokens cannot read CoT tokens; during training the CoT loss can shape action representations. The decoder supports one-step output or repeated ODE/SDE rollout.Source: §A.1 · p. 12
B.1 Reverse-time SDE kernel in x0-parameterization
- The reverse-time SDE uses an Euler–Maruyama step, and its action-transition schedule supports one-step deployment and a clean terminal trajectory.The proposed schedule is finite at the pure-noise endpoint and vanishes at the clean endpoint; the schedule enters the exact ratio and KL through c0 and σ_s.Source: §B.1 · p. 13, Table 8 · p. 13
B.2 Closed-form log-ratio and KL
- The GRPO derivation expresses the action-policy log-ratio and KL in closed form, with a per-element quadratic form and a corresponding reference-policy KL and deployed one-step policy.The implementation uses per-dimension quantities for x, y, and heading.Source: §B.2 · p. 14
B.3 Gradient decomposition and comparison
- The gradient combines an advantage-weighted noise-following term with a trust-region-like pull toward the old prediction; the appendix contrasts this with DiffusionNFT and Flow-GRPO.The cited comparison states that Flow-GRPO has no closed-form KL and a diverging t=1 schedule.Source: §B.3 · p. 14
B.4 Necessity of 𝜆> 0 for Single-Step GRPO
- For single-step GRPO with N=1, positive exploration variance λ is necessary because λ=0 makes all group rollouts identical and their advantages undefined or zero.The passage states sufficiency follows because the Gaussian kernel has full support.Source: §B.4 · p. 14
B.5 Exploration Noise Engineering
- Exploration noise is applied per action dimension in quantile-normalized space and smoothed along the trajectory; the implementation uses noise level 5×10^-2 and groups of G=8.The passage reports that heteroscedastic noise and waypoint reweighting can create an effective step-size ratio of about 10^3, collapsing effective sample size to a few near-deterministic slots.Source: §B.5 · p. 14
C More detail about CoT
The CoT ablations favor action-relevant prediction targets, while attention and trajectory examples show how CoT relates to hazard focus and progress. The reward curriculum switches from PDMS to endpoint-and-safety shaping in Stage II.
C More detail about CoT
- Removing CoT lowers PDMS 1.4 (89.2 vs. 90.6), chiefly DAC (−1.1) and EP (−0.7).The full metric and supervision-format breakdown is reported in Table 9.Source: C More detail about CoT · p. 15, Table 9 · p. 15
- Prediction CoT beats perception (+0.3 PDMS), planning (+0.5), and their union (+0.3); CoT-first reaches the same 90.6 as action-first.The ordering changes deployment cost, not the quality gain attributed to CoT supervision.Source: C More detail about CoT · p. 15
C.1.1 What each CoT teaches
- The three task-oriented CoT targets are competing supervision alternatives on the same scene and action, not stages in a serial pipeline.Perception labels boxes and classes, prediction adds future positions, and planning describes the ego maneuver.Source: §C.1.1 · p. 15, Figure 9 · p. 17
- Prediction CoT supplies current conflict locations and likely future bearings; free-form reasoning scores 89.6 PDMS, while atomic compression scores 89.1 versus 89.2 without CoT.Planning CoT scores 90.1 PDMS; its eight waypoints describe the ego trajectory rather than the visual evidence.Source: C More detail about CoT · p. 15, §C.1.1 · p. 16
C.2 CoT attention visualization
- With CoT, action queries attend to nearby task-relevant vehicles and vulnerable road users instead of focusing narrowly on the lead vehicle.The authors relate the qualitative attention difference to the NC and DAC gains reported in Table 9.Source: §C.2 · pp. 17–18, Figure 10 · p. 18
C.3 Trajectory visualization: collision reduction
- CoT raises NC from 98.8 to 99.0, DAC from 96.8 to 97.9, and TTC from 95.8 to 96.5; Figure 11 contrasts collision trajectories.The figure’s cases show the no-CoT trajectory entering a conflict region while the CoT trajectory keeps feasible clearance.Source: §C.3 · p. 19, Figure 11 · p. 19
C.4 Trajectory visualization: progress preservation
- Prediction CoT raises EP from 83.5 to 84.2 and PDMS from 89.2 to 90.6 while retaining full comfort; Figure 12 shows representative progress-preserving cases.The examples are illustrative, not a pointwise guarantee.Source: §C.4 · p. 19, Figure 12 · p. 20
D.1 Reward Design Details and Curriculum Rationale
- Stage II replaces PDMS with endpoint-and-safety shaping because late-stage PDMS is nearly constant within groups (std∼0.02) and direct optimization slightly trades NC/TTC for EP.The endpoint-plus-shield shaping preserves a dense waypoint signal while reimposing safety ceilings; the softest of four tested soft-error shapings worked best.Source: §D.1 · pp. 20–21
D.2.1 SFT Training Detail
SFT jointly trains the action and language heads from noisy trajectories and structured CoT labels. CoT loss updates shared action states without making action inference depend on generated text.
D.2.1 SFT Training Detail
- The SFT model uses a Qwen3.5-2B backbone, eight action tokens of hidden dimension 2048, and a three-layer SiLU ActionEncoder with time embeddings.The ActionDecoder ends in a three-dimensional output layer; the default loss weights are λdiff=10 and λlm=1.Source: §D.2.1 · p. 21
- Each example mixes a target trajectory with sampled Gaussian noise at a sampled flow time, then runs one joint AUGUstus-mode forward pass.The action head predicts the clean trajectory while the language head teacher-forces the structured CoT.Source: §D.2.1 · p. 21
- CoT cross-entropy back-propagates through action states and the shared trunk, but action tokens never read CoT tokens, so inference needs no generated text.The action loss is a dimension-normalized flow-matching L1 loss; the default weights are λdiff=10 and λlm=1.Source: §D.2.1 · p. 21
D.2.2 RL Training Detail
AUGUstus-GRPO trains action and CoT from shared rollout outcomes in two stages. The procedures specify both on-policy optimization and action-only deployment, with CoT decoding optional.
D.2.2 RL Training Detail
- RL rollouts use 32 prompts with 8 rollouts each (256 sequences), shared initial noise per group, and a 3-step SDE; PPO updates use a 10^-6 learning rate.Both heads use a 0.2 clipping threshold, with gradient clipping norm 1.0.Source: §D.2.2 · p. 21
- The shared reward combines prediction-CoT reward with a weighted planning reward: Stage I uses PDMS, while Stage II uses endpoint-plus-safety shaping.The action objective is JaT-GRPO and the language objective is token-level CoT-GRPO; both use the same outcome signal and native KL terms.Source: §D.2.2 · pp. 21–22
D.3 Complete Training and Inference Algorithms
- SFT trains on noisy actions and teacher-forced prediction CoT, whereas deployment denoises from Gaussian noise and returns the action without CoT decoding unless requested.The two-stage GRPO procedure freezes the SFT reference, shares initial noise within rollout groups, and jointly updates the clipped action and CoT objectives.Source: §D.3 · p. 22, §D.2.2 · p. 22, §D.4 · pp. 22–23
D.5 Three-stage training trace
The training trace describes a progression from SFT through two RL stages, while deployment profiling emphasizes a short action-only path. Post-hoc CoT is disabled in the speed-critical configuration.
D.5 Three-stage training trace
- CoT prediction accuracy improves from 0.75 at SFT to 0.85 after RL; Stage I improves progress and Stage II retains it while increasing the safety margin.Figure 13 illustrates the three-stage path changes as qualitative traces; aggregate values are reported in Table 7.Source: §D.5 · p. 23, Figure 13 · p. 24
D.6 Deployment Complexity
- CUDA graph replay reduces the LLM stage from 91.83 to 33.19 ms (2.77×), while the visual stage remains unchanged.The eager profile launches about 3.7k small kernels and performs roughly 200 pageable host-to-device copies per sample.Source: §D.6 · p. 24
- The action-only path takes 73.6 ms on an H800 and 83 ms on the real vehicle (12 Hz); a 166-token post-hoc explanation takes about 4.5 s.Generating 32–64 candidates multiplies the expensive VLM pass in a VLA-plus-scorer design.Source: §D.6 · pp. 24–25
E.1 Public benchmark examples
The appendix pairs qualitative comparisons on public and internal scenes with real-vehicle examples. These examples show collision-free, route-compliant public trajectories and smooth one-step outputs in nighttime and multi-camera conditions.
E.2 Internal 1,000 km data
- The internal 1,000 km evaluation illustrates long-tail interactions and camera conditions absent from NAVSIM; category-level aggregates are reported in Table 4.Figures 16 and 17 show twelve representative cases.Source: §E.2 · p. 25, Figure 16 · p. 28, Figure 17 · p. 29
E.3 Regression and flow-matching baselines
- Internal comparisons hold scene input fixed across VLM+regression and VLM+flow matching, isolating action parameterization and shared CoT training rather than sensor changes.Figures 18 and 19 present the comparisons.Source: §E.3 · p. 25, Figure 18 · p. 30, Figure 19 · p. 31
E.1 Public benchmark examples
- Public examples compare ReCogDrive, AutoDrive-P3, and AUGUstus on the same scenes; the shown AUGUstus trajectories are collision-free and route compliant.Figures 14 and 15 contain twelve representative cases.Source: §E.1 · p. 25, Figure 14 · p. 26, Figure 15 · p. 27
E.4 Real-vehicle visualization
- Real-vehicle examples show smooth one-step outputs in nighttime and multi-camera conditions, using the action-only deployment path.The deployed path runs at 12 Hz and omits CoT decoding from the loop.Source: §E.4 · p. 32, Figure 20 · p. 32