PaperlayerView exact revision on arXiv

Source-linked AI summary

SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

Changhui Jing, Haochen Tian, Hongchen Li, Tianyu Li, Caojun Wang, Zihao Zhang, Yanhao Liu, Chengen Xie, Bailin Li, Hao Zhao, Zhenjie Yang

arXiv:2610.13781v1cs.CV

TL;DR

Driving logs provide future videos only for recorded trajectories, leaving alternative candidates without matched visual supervision. SIMVTA renders candidate-matched videos and jointly trains video prediction and trajectory planning; it improves results across NAVSIM and HUGSIM benchmarks.

  • Problem

    Driving logs contain future videos only for recorded trajectories, leaving alternative post-training candidate trajectories without matched visual supervision.

    Source: §1 · p. 1

  • Method

    SIMVTA renders reconstructed scenes along selected candidate paths and jointly trains video prediction with trajectory GRPO, prioritizing higher-error futures among high-reward rendered candidates.

    Source: §1 · p. 2

  • Results

    91.3 PDMS from 86.2 (+5.1) on NAVSIM v1, 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM are reported for SIMVTA.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Ablations show that video supervision improves planning when it updates shared representations, and that prioritizing higher-error futures among high-reward candidates further improves performance.

    Source: §5 · p. 9

    Robustness under shifted observation histories and reactive traffic remains for future evaluation.

    Source: G LIMITATIONS · p. 21

Abstract

from arXiv · show

Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3 (+5.1). It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 38.8 in closed-loop evaluation on HUGSIM. Ablations show that updating shared representations with video supervision improves planning, while prioritizing higher-error futures among high-reward trajectories further improves performance.

1 INTRODUCTION

Driving logs provide future videos only for recorded trajectories, leaving alternative policy trajectories without matched visual targets. SIMVTA uses rendered candidate-matched videos alongside trajectory optimization to post-train driving WAMs.

1 INTRODUCTION

  • Driving WAMs combine future-scene prediction with ego-trajectory planning, using dense visual supervision to complement sparse trajectory targets.Recent driving models retain video prediction during training but predict trajectories directly at inference.Source: §1 · p. 1
  • Driving logs contain future videos only for recorded trajectories, so alternative candidate paths lack matched video targets during post-training.Reusing a logged video for a different candidate can create an incorrect trajectory-video pair.Source: §1 · p. 1
  • SIMVTA uses World Engine to render reconstructed real-world scenes along executed candidate paths, providing trajectory-matched targets independent of the model’s predictions.The simulator uses camera poses along candidate paths in scenes reconstructed with 3D Gaussian Splatting.Source: §1 · p. 2
  • Joint video flow-matching and trajectory GRPO train visual predictions of candidate outcomes while favoring trajectories with higher driving rewards.Video loss can update the model’s shared representations.Source: §1 · p. 2
  • SIMVTA renders high-reward candidates and prioritizes higher-error futures among them for video supervision, while inference predicts trajectories without simulator calls or future-video generation.This separates trajectory exploration from video-target selection under a limited rendering budget.Source: §1 · p. 2
  • 91.3 PDMS from 86.2 (+5.1), 88.2 EPDMS, and 38.8 average HD-Score are reported for SIMVTA on NAVSIM v1, NAVSIM v2, and HUGSIM, respectively.The NAVSIM v1 comparison starts from a WAM initialization.Source: §1 · p. 2

2 RELATED WORK

Related work covers world-action models, driving simulation, and reinforcement-learning post-training. SIMVTA focuses on trajectory-matched video supervision during driving WAM post-training.

2 RELATED WORK

  • Video-based world-action models jointly predict future frames and trajectories, while Fast-WAM and SimWAM retain video prediction during training but omit it at inference.SIMVTA focuses on trajectory-matched video supervision during driving WAM post-training.Source: §2 · pp. 2–3
  • 3DGS renders reconstructed scenes from specified camera poses, and World Engine renders observations along simulated ego motion for policy post-training.SIMVTA uses World Engine to render video targets along selected candidate trajectories; CausalDrive instead uses a learned renderer.Source: §2 · p. 3
  • Driving policies are post-trained with reinforcement learning or adaptation, and prior work applies GRPO to diffusion and autoregressive planners.SIMVTA jointly updates shared representations through video prediction and trajectory GRPO.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

SIMVTA alternates candidate-trajectory execution, simulator rendering of selected paths, and joint trajectory/video updates. It matches rendered video targets to executed motions and prioritizes high-reward candidates with larger prediction errors for video supervision.

3 METHOD

  • SIMVTA’s post-training loop samples candidates, executes them for paths and rewards, renders selected paths, and updates the model before sampling again.Source: §3 · p. 3

3.1 TRAJECTORY SAMPLING

  • From shared history features, the SDE policy samples G candidate trajectories; the PDM executor converts them into ego paths and rewards, and all candidates can train the trajectory policy.Source: §3.1 · p. 4

3.2 VIDEO RENDERING

  • World Engine renders eligible high-reward candidates along executed paths, while video prediction uses the same path; online rendering keeps targets aligned with changing policy trajectories.Source: §3.2 · p. 4

3.3 VIDEO SUPERVISION

  • Among high-reward rendered candidates, SIMVTA supervises those with at-least-mean prediction error using matched-video flow matching, whose loss trains video prediction and shared planning representations.The error combines frame reconstruction and temporal-change errors.Source: §3.3 · pp. 4–5

3.4 TRAINING AND INFERENCE

  • Training combines GRPO trajectory optimization, selected-video loss, and teacher-planner behavior-cloning regularization; inference predicts trajectories directly without simulator calls or video generation.The teacher is the frozen initial trajectory network.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate a NAVSIM-adapted Epona initialization on NAVSIM v1/v2 and HUGSIM, averaging benchmark results over three random-seed runs. The section reports benchmark outcomes and ablations of video supervision and rendering choices.

4.2 MAIN RESULTS

  • 91.3 PDMS on NAVSIM v1 (from 86.2), 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM, where SIMVTA leads compared methods.On HUGSIM, it also achieves the highest average RC (51.5), exceeding reported DrivoR results by 1.7 RC and 3.1 HD-Score points.Source: Table 3 · p. 6, §4.2 · p. 7
  • In illustrated failures, SIMVTA avoids Epona’s collision and road departure, and avoids its extra-vehicle video inconsistency.Figure 4 compares planned paths and renderings; Figure 6 shows the video inconsistency in an illustrated scene.Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 6 · p. 9
  • Compared with trajectory GRPO alone, SIMVTA reduces FID from 20.4 to 13.1 and FVD from 105.4 to 46.3.Lower FID and FVD indicate better video quality.Source: Table 4 · p. 7, §4.2 · p. 7

4.3 ABLATION STUDIES

  • 91.3 PDMS results when video gradients update the shared encoder, versus 89.1 when restricted to VideoDiT and for trajectory GRPO alone.The shared-encoder setting improves PDMS by 2.2 points over Exp1.Source: §4.3 · p. 8, Table 5 · p. 8
  • Among four target-selection settings, high-reward, high-error Q2 supervision yields 91.3 PDMS, versus 91.0 for Q1+Q2 under the same high-reward rendering pool.The mixed high/low-reward Q2+Q3 and Q1–Q4 variants each achieve 90.9 PDMS.Source: §4.3 · p. 8, Table 6 · p. 8
  • With the same rendering budget, selective Q2 flow-matching supervision reaches 91.3 PDMS, compared with 91.0 when video GRPO is applied to all valid rendered candidates.Both variants allow video gradients to update the shared encoder.Source: §4.3 · pp. 8–9, Table 7 · p. 8
  • Refreshing current-policy video targets scores 91.3 PDMS versus 90.4 with a fixed initial-policy cache; shuffling trajectory-video pairs lowers PDMS to 83.1 from 91.3.The shuffled-target comparison tests the value of preserving trajectory-video correspondence.Source: Table 8 · p. 8, §4.3 · p. 9, Table 9 · p. 9
  • Doubling the rendering budget provides no PDMS gain: Exp7 scores 91.2 versus 91.3 for Exp2.Both use the same target-selection strategy.Source: §4.3 · p. 9, Table 10 · p. 9

5 CONCLUSION

SIMVTA combines trajectory GRPO with simulator-based video supervision to post-train driving world action models. Its reported benchmark gains accompany ablation evidence for shared-representation updates and higher-error target selection among high-reward trajectories.

5 CONCLUSION

  • SIMVTA combines trajectory GRPO with video supervision, rendering futures along current-policy trajectories to provide visual targets beyond logged action-future pairs.The full framework improves NAVSIM v1 PDMS from 86.2 to 91.3 (+5.1), reaches 88.2 EPDMS on NAVSIM v2, and obtains 38.8 average HD-Score on HUGSIM.Source: §5 · p. 9

A TRAINING PROCEDURE

The training procedure initializes from a NAVSIM-adapted Epona checkpoint and iteratively samples, scores, filters, renders, and trains on candidate trajectories and matched videos.

Initialization

  • SIMVTA starts from a NAVSIM-fine-tuned nuPlan-pretrained Epona checkpoint; the described defaults apply to Exp2.The adapted checkpoint initializes both Exp0 and Exp2.Source: A TRAINING PROCEDURE · p. 14

Candidate Sampling and Reward

  • For each observation, the procedure samples G trajectories, executes them with PDM, computes rewards, and normalizes rewards within the observation group.Exp2 weights EP, TTC, and Comfort by 10, 5, and 2; DDC screens candidates but has zero reward weight, and video prediction error is excluded from trajectory rewards.Source: A TRAINING PROCEDURE · p. 14

Rendering

  • World Engine renders selected paths, and the procedure retains each candidate’s correspondence to its rendered video.Eligible candidates are ranked by reward, with the first K selected for rendering; fewer than K may be rendered if fewer qualify.Source: A TRAINING PROCEDURE · p. 14

Training Update

  • Teacher-forced errors classify rendered candidates; the procedure selects video targets, generates a detached BC teacher chain, and combines trajectory loss, video loss, and BC regularization for updates.When no video target participates across devices, VideoDiT parameters remain unchanged for that update.Source: A TRAINING PROCEDURE · pp. 14–15

Architecture and Temporal Settings

  • Epona’s shared spatiotemporal Transformer encodes four observed frames; TrajDiT predicts eight waypoints over four seconds, while video pose conditions use 2 Hz samples.PDM evaluates the path at 10 Hz, and one sampled future frame is trained per update.Source: A TRAINING PROCEDURE · p. 14

Candidate Validity and Safety

  • Candidates must pass execution, pose-support, and safety checks before reward ranking selects the rendering pool; these filters do not remove candidates from trajectory learning.For Exp2, NC, DAC, TTC, and DDC scores must each be at least 0.999; failed renders are excluded from video supervision.Source: A TRAINING PROCEDURE · p. 14

B CONDITIONAL VIDEO PREDICTION

Conditional video prediction pairs rendered targets with WAM predictions along the same executed pose sequence. Video supervision and trajectory optimization use selected rendered futures, shared representations, and a teacher-based regularizer.

B CONDITIONAL VIDEO PREDICTION

  • Rendered targets and WAM predictions use the same executed pose sequence, aligning each prediction with the candidate path it is meant to represent.The pose sequence conditions the WAM internally during post-training.Source: B CONDITIONAL VIDEO PREDICTION · p. 15
  • For frame prediction, the rolling context uses preceding observed or rendered frames and embedded pose changes, excluding the clean current target.Rendered latents and pose inputs are fixed, while the shared encoder and motion embedding remain trainable.Source: B CONDITIONAL VIDEO PREDICTION · p. 15
  • Prediction-error scoring uses teacher-forced inputs, fixed flow time, shared Gaussian noise, and a temporal-change term, with the complete error computation detached.The motion weight is broadcast over channels; the stated hyperparameters are H and γv = 0.85.Source: B CONDITIONAL VIDEO PREDICTION · pp. 15–16

C VIDEO SUPERVISION AND POLICY OPTIMIZATION

  • Reward labels follow the ranked rendering pool, while candidates are classified as high error at or above the scene’s mean error.Exp2 uses the high-reward pool; it trains on valid Q2 candidates and cycles the trained future frame across the horizon.Source: C VIDEO SUPERVISION AND POLICY… · p. 16
  • Video gradients can update shared parameters through a scaled gradient path, while trajectory GRPO optimizes group-relative advantages with a clipped transition-score objective.In Exp2, the video-gradient scale is η = 0.2; the BC teacher parameters remain fixed while teacher chains are regenerated from the current shared representation.Source: C VIDEO SUPERVISION AND POLICY… · pp. 16–17
  • The trajectory sampler uses stochastic Gaussian transitions on a decreasing grid, clips final action coordinates before reward evaluation, and updates the shared encoder and TrajDiT through transition-mean gradients.Sampled states and advantages are detached, and the trajectory objective uses no current-to-sampling-policy probability ratio.Source: C VIDEO SUPERVISION AND POLICY… · pp. 16–17

D EXPERIMENTAL VARIANTS AND BUDGET ACCOUNTING

The experiments compare video-target selection, candidate pools, rendering schedules, optimization objectives, and budget choices. Exp2 is the strongest of the reported variants across the three tested random seeds.

D EXPERIMENTAL VARIANTS AND BUDGET ACCOUNTING

  • Exp0 uses trajectory GRPO and BC without video supervision, while Exp1 restricts video gradients to VideoDiT and Exp2 also lets them update shared parameters.Exp1 and Exp2 both use Q2 targets; Exp3 instead shuffles rendered targets across trajectory conditions within a scene when a valid permutation exists.Source: D EXPERIMENTAL VARIANTS AND BUDGET… · p. 18
  • Exp4 and Exp6 use high- and low-reward candidate pools with different safety treatments, while Exp5 compares Q1+Q2 training against Exp2’s Q2 supervision.Exp7 raises the rendering budget from four to eight while retaining Exp2’s Q2 selection and video-training rule.Source: D EXPERIMENTAL VARIANTS AND BUDGET… · pp. 18–19
  • With a fixed offline cache, Exp9 reaches 90.4 PDMS versus 89.1 for Exp0 and 91.3 for Exp2.Exp9 reuses videos for initial-policy trajectories; Exp2 refreshes targets for current-policy trajectories.Source: D EXPERIMENTAL VARIANTS AND BUDGET… · p. 18
  • Exp8 replaces flow matching with video GRPO on all valid rendered high-reward candidates, using stochastic future latents and group-normalized advantages.Its video reward does not enter trajectory advantages, and groups with fewer than two valid candidates or reward standard deviation below 10^-6 receive no video-policy update.Source: D EXPERIMENTAL VARIANTS AND BUDGET… · p. 19
  • The rendering cap K applies to rendered candidates, whereas video-training costs depend on the selected subset and objective; Exp9 renders its cache once and Exp2 renders each sampling round.Total training cost also includes trajectory sampling, PDM evaluation, video optimization, and optimizer updates.Source: Table 12 · p. 18, D EXPERIMENTAL VARIANTS AND BUDGET… · p. 19
  • Exp2 outperforms Exp0, Exp5, and Exp7 on NAVSIM v1 PDMS for each of three random seeds.Table 14 reports the mean and sample standard deviation over the three runs.Source: D EXPERIMENTAL VARIANTS AND BUDGET… · p. 19, Table 14 · p. 19

E VIDEO SUPERVISION AND EVALUATION

The evaluation separates prediction consistency, renderer fidelity, and video-generation quality, alongside comparisons with recent WAM results and qualitative path examples. Renderer measurements are scoped to logged viewpoints, while video predictions are evaluated along logged ground-truth trajectories.

E VIDEO SUPERVISION AND EVALUATION

  • Teacher-forced prediction error measures consistency against rendered simulation targets given preceding reference frames, separately from renderer fidelity, video generation quality, and action-conditioning accuracy.The renderer comparison uses logged ground-truth trajectories on NAVSIM v1 NavTest, with eight future frames over four seconds at 2 Hz.Source: E VIDEO SUPERVISION AND EVALUATION · p. 19
  • Renderer fidelity is measured with FID on frames and FVD on clips against real observations, using matched preprocessing and feature extractors.These measurements cover logged viewpoints; novel-view coverage is discussed in Appendix G.Source: E VIDEO SUPERVISION AND EVALUATION · p. 19
  • Exp0 and Exp2 video predictions are compared with World Engine renderings along the same logged ground-truth trajectory and observation history.The comparison uses the same dataset, clip length, preprocessing, and feature extractors as the renderer evaluation.Source: E VIDEO SUPERVISION AND EVALUATION · pp. 19–20
  • Figures 5 and 6 show alternative executed paths and compare World Engine renderings with SIMVTA and Epona predictions at matched future times.Table 15 compiles published WAM results while retaining each source’s checkpoint and evaluation convention; training data, initialization, and evaluators differ across methods.Source: E VIDEO SUPERVISION AND EVALUATION · p. 20, Table 15 · p. 20

F CROSS-BENCHMARK RESULTS

SIMVTA improves the reported NAVSIM v1 and v2 comparisons, while HUGSIM results describe final-planner performance rather than an isolated post-training gain. The reported NAVSIM v2 aggregate improvement coexists with lower traffic-light compliance and extended comfort scores.

F CROSS-BENCHMARK RESULTS

  • SIMVTA raises NAVSIM v1 PDMS from 86.2 to 91.3, a 5.1-point improvement over the initialization.The Exp0 comparison, 89.1 to 91.3, isolates the additional gain when video supervision is added within the framework.Source: F CROSS-BENCHMARK RESULTS · p. 20
  • On NAVSIM v2, SIMVTA reaches 88.2 EPDMS versus 85.0 for the NAVSIM-adapted Epona initialization, a +3.2 difference.The aggregate improvement coexists with traffic-light compliance falling from 99.7 to 99.0 and extended comfort from 67.6 to 60.0.Source: F CROSS-BENCHMARK RESULTS · p. 20
  • On HUGSIM, SIMVTA has the highest reported average scores among the compared methods and leads on hard scenarios, while DrivoR scores higher on extreme-scenario RC and HD-Score.These closed-loop results assess the final planner; isolating post-training gain requires initialization and Exp0 under matching routes, controller, and rollout settings.Source: F CROSS-BENCHMARK RESULTS · p. 20
  • Published WAM comparisons retain each source’s checkpoint and evaluation convention, and differ in training data, initialization, and evaluators.The paper describes these results as complementary to within-backbone ablations.Source: F CROSS-BENCHMARK RESULTS · p. 20

G LIMITATIONS

The limitations concern the quality and coverage of rendered targets, robustness under distribution shifts, and the architecture and training-cost scope of validation.

G LIMITATIONS

  • Occlusions and limited pose coverage can degrade off-path renderings, and artifacts may remain despite safety and pose-support checks.High prediction error can reflect either model limitations or poor-quality simulation targets; evaluation at alternative poses remains necessary.Source: G LIMITATIONS · p. 20
  • Robustness under shifted observation histories and reactive traffic remains unevaluated, while log-based histories and rendered targets constrain supervision.Teacher-forced selection does not capture autoregressive error accumulation.Source: G LIMITATIONS · p. 21
  • Validation uses the Epona backbone, leaving generalization to alternative WAM architectures for future work, and training incurs additional cost.Online rendering, error scoring, and video optimization increase training cost; inference uses direct trajectory prediction.Source: G LIMITATIONS · p. 21

Related papers

Loading 2610.13781v1…