PaperlayerView exact revision on arXiv

Source-linked AI summary

ACT-Bench: Towards Action Controllable World Models for Autonomous Driving

Hidehisa Arai, Keishi Ishihara, Tsubasa Takahashi, Yu Yamaguchi

arXiv:2412.05337v1cs.CVcs.LGcs.RO

TL;DR

Driving world models need reproducible evaluation of whether generated scenes follow specific action instructions, not only visual or downstream-task assessment. ACT-BENCH provides an open action-fidelity benchmark and TERRA baseline, showing improved instruction fidelity over Vista. The framework also exposes estimator-domain and causal-misalignment limitations relevant to simulation use.

  • Problem

    Driving world-model research lacks open, reproducible evaluation of fidelity to specific action-conditioned instructions.

    Source: §1 · p. 2

  • Method

    ACT-BENCH pairs nuScenes context scenes with annotated actions and trajectories, uses ACT-ESTIMATOR for evaluation, and introduces trajectory-controllable TERRA trained on three datasets.

    Source: §1 · p. 2

  • Results

    44.11% versus 30.72%: TERRA achieves a higher instruction-execution match rate than Vista, whose generated scenes do not fully adhere to instructions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ACT-BENCH enables open, systematic comparison of action fidelity and supports reproducible research on controllable driving world models.

    Source: §1 · p. 2, §6 · p. 8

    The evaluation uses DROID-SLAM despite a target-domain gap, and observed causal misalignment can unintentionally affect other agents.

    Source: §4.2 · p. 6, §5.3 · p. 8

Abstract

from arXiv · show

World models have emerged as promising neural simulators for autonomous driving, with the potential to supplement scarce real-world data and enable closed-loop evaluations. However, current research primarily evaluates these models based on visual realism or downstream task performance, with limited focus on fidelity to specific action instructions - a crucial property for generating targeted simulation scenes. Although some studies address action fidelity, their evaluations rely on closed-source mechanisms, limiting reproducibility. To address this gap, we develop an open-access evaluation framework, ACT-Bench, for quantifying action fidelity, along with a baseline world model, Terra. Our benchmarking framework includes a large-scale dataset pairing short context videos from nuScenes with corresponding future trajectory data, which provides conditional input for generating future video frames and enables evaluation of action fidelity for executed motions. Furthermore, Terra is trained on multiple large-scale trajectory-annotated datasets to enhance action fidelity. Leveraging this framework, we demonstrate that the state-of-the-art model does not fully adhere to given instructions, while Terra achieves improved action fidelity. All components of our benchmark framework will be made publicly available to support future research.

1. Introduction

ACT-BENCH addresses limited, non-reproducible evaluation of action fidelity in driving world models by providing an open benchmark and baseline model. Its studies find that Vista does not fully follow instructions, while TERRA improves action fidelity.

Motivation

  • World models can supplement scarce or dangerous real-world data and support closed-loop autonomous-driving evaluations, but reliable simulations require fidelity to action-conditioned instructions.These needs motivate evaluating controllability in addition to visual or downstream-task performance.Source: §1 · p. 1

Benchmark motivation and contribution

  • Open-access benchmarks and models are needed because existing action-fidelity evaluations depend on non-public evaluators or datasets, limiting reproducibility.The paper positions accessible evaluation as a way to support standardized and reproducible studies.Source: §1 · p. 2
  • ACT-BENCH provides an open benchmark for evaluating action fidelity using nuScenes scenes annotated with high-level driving actions and trajectories.The framework supports systematic comparison of how generated scenes execute specified instructions.Source: Figure 1 · p. 1, §1 · p. 2
  • ACT-ESTIMATOR compares intended instructions with estimated actions and ground-truth trajectories to quantify execution consistency and spatial accuracy.It enables automatic fidelity assessment for generated driving scenes.Source: §1 · p. 2

Baseline model

  • TERRA is an open-access trajectory-controllable world model following GAIA-1’s architecture and trained on OpenDV-YouTube, nuScenes, and CoVLA.Its training uses more trajectory-annotated datasets than Vista.Source: §1 · p. 2

Empirical findings

  • 44.11% versus 30.72%: TERRA exceeds Vista in instruction-execution match rate, although both models achieve adequate quality across other metrics.The comparison evaluates how well existing world models follow driving instructions.Source: §1 · p. 2

2. Related Work

Related work frames driving world models as neural simulators with increasingly diverse conditioning and evaluation goals. Existing metrics cover visual quality, action fidelity, layout fidelity, and downstream tasks, but open action-fidelity evaluation remains limited.

2.1. Driving World Model

  • World models represent environments internally so agents can simulate futures and evaluate action sequences across applications including reinforcement learning and model predictive control.This predictive role underlies their use as learned simulators.Source: §2.1 · p. 3
  • Driving world models increasingly emphasize realistic video, consistent multi-view outputs, and conditional inputs such as text, bounding boxes, and maps.These capabilities broaden their use for training and testing autonomous systems.Source: §2.1 · p. 3

2.2. Evaluation Metrics for Driving World Model

  • Driving world-model evaluation includes FID, FVD, and CLIPSIM for visual quality, 3D detection for instructed layouts, and downstream-task performance for planning or perception.These metrics assess different outputs and conditioning types.Source: §2.2 · p. 3
  • GenAD and Vista evaluate action fidelity by estimating instructions from generated videos, but their evaluation components or datasets are not publicly available.This lack of access prevents open evaluation and limits reproducibility.Source: §2.2 · p. 3

3. ACT-Bench

ACT-Bench combines an action-conditioned driving dataset, fidelity metrics, an automated estimator, and the open-access TERRA baseline to evaluate whether generated scenes follow trajectory instructions.

3. ACT-Bench

  • ACT-Bench provides an open benchmark with annotated driving scenes, fidelity evaluation, and TERRA as a reference world model.The benchmark is designed for evaluating action fidelity in generated driving scenes.Source: §3 · p. 3

3.1. Benchmark Dataset

  • Nine trajectory-template categories pair 10-frame front-camera context videos with varied curvature and speed instructions, yielding 2,286 filtered video-trajectory pairs.Pairs are filtered by initial-speed consistency and removal of front-object interactions; templates also support per-frame action inputs and nonlinear trajectories.Source: §3.1 · p. 4

3.2. Benchmark Metrics

  • IEC measures instruction-execution agreement, while TA measures distance between intended and estimated trajectories using metrics such as ADE and FDE.Lower trajectory distance indicates closer alignment.Source: §3.2 · pp. 4–5

3.3. ACT-Estimator

  • ACT-ESTIMATOR jointly classifies high-level driving actions and predicts vehicle trajectories from generated camera frames using shared spatiotemporal representations.Its architecture combines an I3D backbone and self-attention with classification and autoregressive trajectory-prediction heads.Source: §3.3 · p. 5

3.4. TERRA: A Baseline World Model

  • TERRA is an open-access, trajectory-controlled world model following GAIA-1’s design principles while training on OpenDV-YouTube, nuScenes, and CoVLA.Trajectory instructions can be supplied at each frame for precise control; its overview omits text conditioning and a video decoder for simplicity.Source: §3.4 · p. 5, Figure 3 · p. 5

4. Experiment: Estimator Validation

ACT-ESTIMATOR validation covers high-level action classification and vehicle trajectory estimation. It achieves high classification accuracy and outperforms adapted DROID-SLAM for trajectory alignment, despite a target-domain gap in the baseline comparison.

4.1. High-level Action Classification

  • 94.03% overall accuracy shows ACT-ESTIMATOR reliably classifies high-level driving actions on the validation dataset.Diagonal matrix entries represent correct predictions, while offdiagonal entries represent mismatches.Source: Figure 4 · p. 6, §4.1 · p. 6
  • ACT-ESTIMATOR maintains consistently high performance across action classes, indicating robustness in distinguishing different driving maneuvers.Source: §4.1 · p. 6

4.2. Vehicle Trajectory Estimation

  • The trajectory evaluation uses ADE and FDE against ground truth, with adapted DROID-SLAM serving as the baseline despite a target-domain gap.Vista and GenAD use undisclosed XVO-based mechanisms, motivating the selected baseline.Source: §4.2 · p. 6
  • Our model outperforms adapted DROID-SLAM on both ADE and FDE, with consistently lower errors across action categories.The comparison evaluates estimated vehicle trajectories from video sequences; HS and LS denote high- and low-speed cases.Source: §4.2 · p. 6, Table 3 · p. 6

5. Action Controllability Exploration

ACT-BENCH evaluates whether open-access driving world models execute instructed actions and follow target trajectories. TERRA generally outperforms Vista, but both show fidelity failures including abrupt transitions and causal misalignment.

5.1. Instruction-Execution Consistency Evaluation

  • 44.11% versus 30.72%: TERRA achieves a higher instruction-execution match rate than Vista, although both remain imperfect and instruction frequency may influence the comparison.TERRA accepts instructions every frame, whereas Vista accepts one instruction per generation round.Source: §5.1 · p. 7

5.2. Trajectory Alignment Evaluation

  • TERRA outperforms Vista in ADE and FDE across most high-level action classes, indicating closer alignment with conditioned trajectories.The trajectory-alignment table reports ADE and FDE as measures of deviation from intended paths.Source: §5.2 · p. 7, Table 4 · p. 7
  • TERRA follows curving trajectories more closely than Vista, whose generated paths are more straight-like, although TERRA still struggles to maintain consistent travel distance.Frame-level conditioning supports more responsive curving and directional adherence.Source: §5.2 · p. 7, Figure 7 · p. 7

5.3. Noteworthy Findings

  • Vista can produce abrupt, unnatural motion changes at generation-round transitions because action conditioning is limited to each round.The example shows nearly straight first-round motion followed by a significant rightward shift in the second round.Source: §5.3 · p. 8, Figure 8 · p. 8
  • Causal misalignment occurs when instructions for the ego vehicle inadvertently alter other agents, such as causing a leading car to stop unexpectedly.This behavior deviates from real-world dynamics and challenges use of world models as simulators.Source: §5.3 · p. 8, Figure 9 · p. 8
  • TERRA can deliberately generate crash scenes, highlighting its potential for producing difficult-to-collect action-conditioned scenarios.The conclusion also reports diverse action-conditioned scenes, including accidental collisions.Source: §5.3 · p. 8, Figure 10 · p. 8

6. Conclusion

The paper introduces ACT-BENCH, ACT-ESTIMATOR, and TERRA to advance action-controllable driving world models. It reports improved action fidelity for TERRA while identifying instruction failures and diverse generated scenarios, including collisions.

6. Conclusion

  • ACT-BENCH combines an annotated dataset, ACT-ESTIMATOR, and TERRA to quantify and improve action fidelity in driving world models.The framework is presented as an open-access evaluation framework.Source: §6 · p. 8
  • TERRA exhibits improved action fidelity, while the state-of-the-art model does not fully adhere to given instructions.The framework also reveals potential to generate diverse action-conditioned scenes, including accidental collisions.Source: §6 · p. 8

A. Dataset Construction for ACT-Estimator

The ACT-ESTIMATOR dataset uses four-second nuScenes CAM FRONT segments with local-coordinate ego trajectories. Rule-based thresholds then assign high-level action labels consistently across trajectory windows.

A. Dataset Construction for ACT-Estimator

  • The dataset slices nuScenes CAM FRONT sequences into overlapping 44-frame, approximately four-second segments and transforms ego trajectories into local coordinates.A stride of one frame maximizes dataset size while preserving temporal coherence.Source: A. Dataset Construction for… · p. 12
  • A rule-based algorithm assigns trajectories to eleven action classes using waypoint-distance changes, curvature, and trajectory-tangent angle thresholds.Thresholds are empirically calibrated to balance class distribution and support consistent ACT-ESTIMATOR training and evaluation.Source: A. Dataset Construction for… · p. 12

B. ACT-Estimator Design

The paper describes the ACT-ESTIMATOR and TERRA components used for action-fidelity evaluation and controllable video generation. TERRA autoregressively predicts future image tokens conditioned on trajectory sequences and refines decoded videos for resolution and temporal consistency.

B. ACT-Estimator Design

  • The ACT-ESTIMATOR combines an I3D backbone, Transformer Encoder, and task-specific heads for high-level action classification and trajectory regression.Left- and right-shifting classes are excluded from prediction targets because each represents less than 1% of the dataset.Source: B. ACT-Estimator Design · p. 12, Figure 11 · p. 12, Table 5 · p. 12

C. Template Instruction Trajectory

D. TERRA World Model Design

  • TERRA is an autoregressive Transformer world model that takes discretized image tokens and trajectory instructions to predict future image-token sequences.An image tokenizer, action embedder, frame-wise decoder, and post-hoc video refiner form the principal pipeline.Source: D. TERRA World Model Design · p. 13

D.1. World Modeling through Token Prediction

  • Image frames are encoded into discrete latent codes, flattened in raster-scan order, and interleaved with trajectory inputs for autoregressive prediction.At inference, predicted frame tokens and subsequent trajectory inputs are inserted sequentially to generate future frames.Source: §D.1 · pp. 14–15
  • Training uses next-token prediction on image-frame tokens, excluding trajectory tokens from the loss.The model uses learnable temporal and spatial positional embeddings for the interleaved sequence.Source: §D.1 · p. 15

D.2. Video Refiner

  • A latent-diffusion Video Refiner uses decoded and upscaled frames as conditioning to improve video resolution and temporal consistency.The refiner is built by fine-tuning Stable Video Diffusion and denoising conditioned latent representations.Source: §D.2 · p. 15

E. Implementation Details of TERRA

TERRA uses fixed image and video dimensions, six-step trajectory instructions, and a large Transformer embedding configuration. Its Video Refiner operates on upscaled frames and compressed latent variables.

E.1. Hyper-parameter Settings

  • TERRA processes 288 × 512 images in 25-frame clips, representing 2.5 seconds at 10 Hz.The Image Tokenizer uses Lookup-Free Quantization.Source: §E.1 · p. 15
  • The tokenizer uses codebook size K = 262,144, downscaling D = 16, and 576 discrete tokens per image.Six trajectory vectors produce a total training sequence length of 14,550 tokens, with embedding dimension d = 2048.Source: §E.1 · p. 16
  • Trajectory padding uses an all−1.0 matrix, while trajectory-token timing values differ between CoVLA and nuScenes.The supplied settings specify 0.45 + 0.5 × (l − 1) for CoVLA and 0.5 × l for nuScenes.Source: §E.1 · p. 16
  • The Video Refiner upsamples frames to 384 × 640 and uses latent dimensions H_r = 48 and W_r = 80, with T_r = 25 and C_r = 4.These latent variables come from the pretrained SVD Autoencoder.Source: §E.1 · p. 16

E.2. Training Procedure

TERRA training prepares three trajectory-annotated video sources as tokenized 25-frame chunks, then trains the world model and Video Refiner separately.

E.2. Training Procedure

  • OpenDV-YouTube, nuScenes, and CoVLA videos are converted to 10 Hz image sequences, paired with sampled six-point trajectory instructions, and segmented into 25-frame chunks.The resulting datasets contain 1.67 million, 25,000, and 0.23 million chunks, respectively.Source: §E.2 · p. 16
  • The world model is a randomly initialized Llama-based Transformer trained for 40k steps on 56 H100 GPUs with AdamW and cosine-decay learning rate scheduling.Each GPU uses batch size 1 with four gradient-accumulation steps.Source: §E.2 · p. 16
  • The Video Refiner is trained separately for 800k steps on eight H100 GPUs, conditioning each frame on coarse Frame-wise Decoder predictions rather than Vista’s dynamic prior.The coarse predictions are concatenated with noise along the channel axis.Source: §E.2 · p. 16
  • The training setup is summarized in the world-model hyper-parameter table.Source: Table 7 · p. 16

F. Video Generation Settings

Video generation uses frame-specific trajectory conditioning for TERRA and corrected per-round trajectories for Vista’s multi-round generation.

F. Video Generation Settings

  • Vista’s multi-round generation is modified to use the trajectory assigned to the frame starting each round instead of repeating one instruction.The corrected target trajectories specify position and orientation for faithful motion at each future timestep.Source: F. Video Generation Settings · p. 17
  • Vista video generation uses the parameter settings listed in its generation hyper-parameter table.Source: Table 8 · p. 17
  • TERRA appends the trajectory instruction corresponding to each frame to that frame’s image-token sequence throughout generation.Inference uses vLLM with temperature = 0.9, top p = 1.0, and top k = −1.Source: F. Video Generation Settings · p. 17

G. Visualization

Visualizations show that TERRA generally follows trajectory instructions while retaining some deviation, whereas Vista exhibits causal irregularities near instruction transitions. TERRA examples also show surrounding vehicles responding to ego-vehicle deceleration.

G. Visualization

  • TERRA-generated movements demonstrate reasonable adherence to instructed trajectories, although they do not exactly follow them.Figure 13 compares the provided instruction trajectory with generated frames across time.Source: G. Visualization · p. 17, Figure 13 · p. 18
  • Vista examples show preceding, oncoming, and parallel vehicles moving unnaturally, especially where instructions are inserted between generation rounds.The cited examples include unexpected starts and acceleration near these transitions.Source: G. Visualization · p. 17, Figure 14 · p. 19
  • TERRA examples show oncoming and parallel vehicles gradually decelerating and stopping as the ego vehicle reduces speed.The parallel vehicle initially travels faster than the ego vehicle before responding similarly.Source: G. Visualization · p. 17, Figure 15 · p. 19
  • Figure 13 presents the instruction trajectory alongside generated frames at multiple time steps.Source: Figure 13 · p. 18
  • Figures 14 and 15 provide paired examples of causal misalignment in Vista and TERRA, respectively.Source: Figure 14 · p. 19, Figure 15 · p. 19

Related papers

Loading 2412.05337v1…