PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

Shenghe Zheng, Wenbo Li, Jiyao Zhang, Bin Xia, Haoyang Huang, Nan Duan, Jiaya Jia

arXiv:2609.38059v1cs.ROcs.CV

TL;DR

Robot learning faces costly experience collection, while visual simulators need action-faithful prediction despite scarce, embodiment-specific control data. WorldLine separates transferable dynamics learning from image-space action grounding and combines it with interaction-focused training and few-step distillation. Across held-out and out-of-domain evaluations, it reports gains in prediction, policy evaluation, and planning, including improved task success without RoboTwin training or adaptation.

  • Problem

    Visual simulators must predict faithful control responses and coherent robot–object dynamics, but action supervision is scarce and embodiment-specific controls differ in ways that complicate sharing.

    Source: §1 · p. 2

  • Method

    WorldLine learns transferable dynamics from action-free robot videos, grounds heterogeneous controls through an image-space interface, and uses multi-view, failure-enriched training and few-step distillation.

    Source: §1 · pp. 2–3

  • Results

    Across held-out and out-of-domain settings, WorldLine reports strong visual quality and robot-motion agreement, 74% mean trajectory-success classification accuracy, and up to 21.4 percentage points higher RoboTwin task success without training or adaptation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    WorldLine provides a scalable, lower-latency visual simulator that remains useful for robot policy evaluation and embodied planning.

    Source: §7 · p. 10

    Few-step rollouts are less reliable when planning depends on fine distinctions among candidate outcomes, and whether evaluating more candidates offsets ranking errors remains open.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scalable foundation for visual simulators that predict action outcomes before physical execution. Yet they often favor visual plausibility over accurate action following and coherent robot--object dynamics, while action-conditioned simulators depend on scarce, embodiment-specific data that are difficult to share across incompatible control spaces. We introduce WorldLine, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding. WorldLine learns manipulation dynamics from more than 10,000 hours of action-free robot videos and grounds them using over 2,000 hours of action trajectories across more than ten embodiments. An image-space action representation provides a shared control interface across embodiments, while multi-view and failure-enriched training with relational regularization improves interaction-sensitive prediction. Robot-focused few-step distillation enables efficient causal rollout while preserving action-critical motion. Across held-out and out-of-domain settings, WorldLine maintains strong visual quality and robot-motion agreement; on failed trajectories, it improves robot-mask IoU by 0.1626 over the strongest baseline. It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline. Without RoboTwin training or adaptation, its rollouts improve task success by up to 21.4 percentage points over direct policy execution. Together, these capabilities make WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning. More results are available at \href{https://zhengsh123.github.io/WorldLine/}{project page}.

1 INTRODUCTION

WorldLine addresses costly, limited robot experience and incompatible embodiment-specific controls by separating transferable dynamics learning from action grounding. Its evaluations report action-faithful prediction and gains in policy evaluation and planning.

1 INTRODUCTION

  • At N = 32, WorldLine rollouts improve RoboTwin task success by 19.1 and 21.4 percentage points for π0.5 and LingBot-VLA, respectively, over direct policy execution.The planning evaluation used no RoboTwin training, adaptation, or checkpoint selection.Source: §1 · p. 2
  • WorldLine achieves 74% mean trajectory-success classification accuracy across RoboTwin and AgiBot, versus 73% for the strongest baselines.Rollouts are evaluated using extracted visual evidence and deterministic task-specific rules.Source: §1 · p. 2
  • On failed trajectories, robot-mask IoU improves by 0.1626 over the strongest baseline, alongside strong visual quality and robot-motion agreement in held-out and out-of-domain settings.Source: §1 · p. 2
  • WorldLine learns transferable robot–object dynamics from action-free videos, then grounds heterogeneous controls through a geometry-preserving image-space interface.This separates dynamics learning from scarce action-labeled trajectories and avoids mapping incompatible native controls into a common vector space.Source: §1 · p. 2
  • Synchronized multi-view observations, failure-enriched trajectories, relational regularization, and few-step distillation target interaction outcomes and efficient causal rollout.The simulator is intended for policy evaluation and embodied planning.Source: §1 · pp. 2–3

2 RELATED WORK

Prior visual prediction work spans synthetic-data generation, policy learning, planning, and evaluation, but predictive simulation requires faithful control response and coherent robot–object interactions. Existing control representations and data dependencies motivate WorldLine’s separated dynamics learning and image-space grounding.

2 RELATED WORK

  • Generated video has supported robot learning through visual prediction, synthetic data, policy-learning assistance, and candidate-action outcome prediction for planning and evaluation Finn & Levine (2017); Dasari et al. (2020); Yang et al. (2024); Wang et al. (2026); Zhu et al. (2024).Source: §2 · p. 3
  • Predictive simulation requires robot-control accuracy and coherent robot–object interactions, not visual plausibility alone Guo et al. (2026); Quevedo et al. (2026).Source: §2 · p. 3
  • Native actions are tied to embodiment-specific kinematics and coordinates, latent actions need grounding, and spatial conditions alone do not solve limited action-labeled data.Existing simulators commonly learn visual dynamics and action semantics jointly from trajectories whose control spaces and interaction distributions vary across embodiments O’Neill et al. (2024); Wu et al. (2026b).Source: §2 · p. 3
  • WorldLine instead learns transferable dynamics from action-free video and grounds heterogeneous controls through a shared image-space representation, combining complementary supervision.Source: §2 · p. 3

3 DATA PIPELINE

WorldLine builds its training pipeline from complementary action-free videos and geometrically verified action trajectories, then separates dynamics learning, action grounding, and causal rollout distillation. The data pipeline also reserves distinct in-domain and out-of-distribution evaluation sets.

3 DATA PIPELINE

  • The action-free corpus contains more than 10,000 hours of robot videos from six collections, spanning more than ten embodiments and over 3,500 manipulation tasks.Stage I does not use available source-dataset state or action annotations as model inputs or conditioning; “action-free” describes the training setup.Source: §3 · pp. 3–4
  • The pipeline filters action-free videos for dynamics pretraining and checks action-labeled trajectories for calibration and projection before pairing failure trajectories with geometrically aligned controls.Source: Figure 2 · p. 4, §3 · p. 4
  • WorldLine trains in three stages: action-free dynamics learning, image-space control grounding with multi-view and failure-enriched training, and efficient causal rollout distillation.Source: Figure 2 · p. 4
  • The action-grounding corpus contains over 2,000 hours of trajectories across more than ten embodiments, with calibrated camera geometry and temporally aligned controls.Verified controls become camera-aligned action maps; the corpus also includes a synchronized multi-view subset and approximately 200 hours of failure trajectories.Source: §3 · p. 4
  • A scene-disjoint AgiBotWorld subset is reserved for in-domain evaluation, while DROID is excluded from training and used for out-of-distribution evaluation.DROID tests generalization across unseen environments, tasks, and robot embodiments.Source: §3 · p. 4

4 METHOD

WorldLine separates learning transferable robot–object dynamics from grounding heterogeneous controls, then adds interaction-focused training and distillation for efficient causal rollout. Its three stages progress from action-free video pretraining to action-grounded post-training and robot-focused distillation.

4.1 OVERVIEW

  • WorldLine trains in three stages: action-free dynamics pretraining, action-grounded post-training with interaction-coherence training, and efficient causal-rollout distillation.The overview describes the stages as learning dynamics, grounding controls, and distilling the model, respectively.Source: §4.1 · p. 4

4.2 STAGE I: ROBOT DYNAMICS PRETRAINING

  • Stage I adapts a video model to robot videos to learn a transferable prior over robot–object dynamics without action supervision.That prior initializes Stage II.Source: §4.2 · p. 5

4.3 STAGE II: ACTION-GROUNDED POST-TRAINING

  • Stage II converts the dynamics prior into an action-grounded simulator using geometrically verified action–video trajectories and conditioning on initial observations and controls.It removes task-text conditioning.Source: §4.3 · p. 5

4.3.1 CROSS-EMBODIMENT SPATIAL ACTION CONDITIONING

  • Camera-projected action maps encode controls in a shared image-space interface, preserving spatial and temporal alignment with video tokens across embodiments.The maps use embodiment-specific kinematics and camera frames to represent control effects.Source: §4.3.1 · p. 5

4.3.2 MULTI-VIEW AND FAILURE-ENRICHED TRAINING

  • Synchronized head and wrist views broaden interaction observations, while training on successful and failed trajectories exposes the model to unsuccessful outcomes without outcome labels.Trajectories missing a required wrist view are excluded from the multi-view objective.Source: §4.3.2 · p. 5

4.3.3 RELATIONAL DYNAMICS REGULARIZATION

  • Relational regularization matches student and teacher feature relations across consecutive frames within views and synchronized frames across views.An optimization-step-dependent weighting transfers the teacher’s temporal and cross-view relational structure to WorldLine.Source: §4.3.3 · pp. 5–6

4.4 STAGE III: ROBOT-FOCUSED CAUSAL DISTILLATION

  • A block-autoregressive student replaces fixed-sequence iterative ODE sampling with four-step causal generation that accepts new controls at each block.Each block spans four latent frames (16 RGB frames); robot-focused auxiliaries support low-latency rollout and online updates.Source: §4.4 · p. 6

5 EXPERIMENTS

WorldLine is evaluated for action-conditioned video prediction, policy evaluation, embodied planning, and causal rollout efficiency. Across these settings, it generally preserves action-relevant robot–object outcomes, with ablations examining the contribution of its data, representation, and training components.

5.1 ACTION-CONDITIONED VIDEO GENERATION

  • On successful AgiBotWorld trajectories, WorldLine ranks first in SSIM, LPIPS, and robot IoU and second in PSNR; on failures, it leads all four metrics and improves robot IoU by 0.1626 over the strongest baseline.On out-of-domain DROID, it achieves the highest robot IoU (0.2739), exceeding the best prior result by 0.0690.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2.1 POLICY EVALUATION

  • 74% mean policy-evaluation accuracy makes WorldLine highest on RoboTwin and AgiBot, ahead of GE-Sim-V2 at 73%.Causal WorldLine remains within one point of Stage II at lower latency.Source: Figure 4 · p. 8, §5.2.1 · p. 8

5.3 CAUSAL FEW-STEP ACCELERATION

  • Causal distillation cuts sampling from 35 to four steps and 129-frame generation from 90 to 21 seconds, a 4.3× speedup, while preserving the main robot–object evolution.Few-step rollouts are less reliable when selection depends on fine candidate differences; whether evaluating more candidates with saved runtime offsets ranking errors remains open.Source: Figure 5 · p. 8, §5.3 · pp. 8–9

5.2.2 EMBODIED PLANNING

  • At N = 32, rollout selection narrows the policies’ direct-execution success gap from 17.9 to 15.6 percentage points, and four-step causal rollouts retain most gains at lower cost.These gains persist on RoboTwin without WorldLine training or adaptation; imperfect candidate scoring means gains need not be monotonic with rollout budget.Source: §5.2.2 · p. 8, Figure 4 · p. 8

5.4 ABLATION STUDIES

  • Removing relational regularization worsens visual quality and robot-mask IoU but has little effect on planning success, consistent with gains in sequence-level temporal and cross-view geometric consistency.Figure 6 illustrates differences in robot motion and interaction outcomes.Source: Figure 6 · p. 9, §5.4 · p. 10
  • With matched training compute, removing action-free data degrades visual quality and planning; success-only data barely changes π0.5 performance but lowers LingBot-VLA success by 10.8 points.Head-view-only training also reduces robot-mask overlap, particularly on AgiBotWorld; wrist views add close-range contact and object-motion cues.Source: Table 3 · p. 9, §5.4 · p. 10
  • Replacing image-space action maps with FiLM-injected native action vectors reduces robot IoU, worsens LPIPS, and lowers planning success across both datasets and policies.Under matched RoboTwin training and a 32-rollout budget, WorldLine also outperforms GE-Sim-V2 and Masked Visual Actions across both policies.Source: Figure 7 · p. 10, §5.4 · p. 10

6 FUTURE WORK

Future work will test WorldLine beyond its current settings and explore its use in policy training and test-time planning.

6 FUTURE WORK

  • Future evaluations will extend to rare manipulation patterns, deformable objects, diverse cameras, and longer action-conditioned horizons.These settings can test whether image-space action grounding and cross-embodiment dynamics remain reliable under greater variation.Source: §6 · p. 10
  • The authors plan to study WorldLine as an RL environment for training robot policies and as a test-time scaling tool that allocates more candidate rollouts to difficult tasks..Source: §6 · p. 10
  • Future work will examine whether uncertainty estimates and closed-loop replanning help planners identify unreliable predictions and allocate computation efficiently.It will also test whether improved simulated outcomes translate into more reliable real-robot behavior.Source: §6 · p. 10

7 CONCLUSION

WorldLine separates broad robot–object dynamics learning from action grounding across embodiments. Its data pipeline uses action-free videos for Stage I and trajectory controls with spatial conditions and synchronized views for Stage II.

7 CONCLUSION

  • WorldLine scales dynamics learning beyond scarce action-labeled data by separating action-free pretraining from image-space action grounding, then supports action-conditioned prediction and downstream robotic decisions.Four-step causal distillation retains these capabilities at substantially lower latency.Source: §7 · p. 10

A.1 STAGE-WISE DATA COMPOSITION

  • Stage I treats robot videos as action-free to learn general robot motion and object-interaction dynamics; Stage II uses reliable state or action sequences and camera geometry to ground motion through spatial action conditions.Stage II jointly processes synchronized head, left-wrist, and right-wrist observations when all required views are available.Source: §A.1 · p. 15

A.2 VIDEO SAMPLING AND MOTION FILTERING

  • Training clips span 33–129 frames, with larger temporal strides in Stage I and denser sampling in Stage II; source-specific criteria remove static, corrupted, and chassis-motion-dominated clips..Source: §A.2 · p. 15

A.3 ACTION GEOMETRY AND MULTI-VIEW CONSTRUCTION

  • Stage II projects end-effector poses into camera views to form a nine-channel spatial action condition, masking invalid or out-of-view projections.Its three camera slots represent head or top, left wrist, and right wrist; trajectories missing a required wrist view are excluded from multi-view training.Source: §A.3 · p. 15

A.4 FAILURE DATA AND EVALUATION SEPARATION

WorldLine trains on successful and failed action–video trajectories, while separating training scenes from evaluation scenes and excluding DROID from model development.

A.4 FAILURE DATA AND EVALUATION SEPARATION

  • Failure trajectories include unsuccessful interactions such as missed grasps, object slippage, unintended collisions, and incomplete manipulation, and are trained as ordinary action–video pairs without explicit outcome labels.The model must learn unsuccessful outcomes from the observed trajectories rather than predefined failure categories.Source: §A.4 · p. 15
  • AgiBotWorld evaluation scenes and associated trajectories are excluded from training, while DROID is excluded from training, adaptation, and checkpoint selection.DROID’s different viewpoints and single-arm settings provide an out-of-distribution test of viewpoint and embodiment-configuration generalization.Source: §A.4 · pp. 15–16

B.1 MODEL INITIALIZATION AND STAGE-I TRAINING

  • WorldLine initializes from pretrained Cosmos3-Nano weights and uses full-parameter Stage-I training on action-free robot videos conditioned on the initial observation and task description.Stage I adapts the video prior to robot motion and manipulation dynamics without robot actions.Source: §B.1 · p. 16

B.2 STAGE-II ACTION-GROUNDED POST-TRAINING

The section describes WorldLine’s action-grounding, relational training, and conversion to a causal few-step simulator. It also details evaluation protocols and reports that a selector identifies successful trajectories increasingly well as candidate banks grow.

B.2 STAGE-II ACTION-GROUNDED POST-TRAINING

  • Stage II zero-initializes an action-conditioning branch and adds spatially patchified image-space action tokens to aligned video tokens, while freezing other pretrained components.The rendered action maps encode projected position, depth, orientation, and gripper state; the branch is progressively introduced during post-training.Source: §B.2 · p. 16

B.3 RELATIONAL REGULARIZATION

  • Relational regularization aligns student–teacher token similarities across adjacent frames within a view and between synchronized head and wrist views.A frozen V-JEPA2 encoder supplies the relational teacher; the objectives are dynamically scaled relative to flow-matching loss.Source: §B.3 · pp. 16–17

B.4.1 CAUSAL TEACHER-FORCING ADAPTATION

  • Causal teacher-forcing adaptation generates four latent frames per block from one initial latent frame, conditioning on preceding ground-truth blocks and matching action maps.Each generated block corresponds to 16 RGB frames.Source: §B.4.1 · p. 17, §B.4 · p. 17

B.4.2 FOUR-STEP TRAJECTORY REGRESSION

  • Four-step trajectory regression trains the student to follow the frozen teacher’s denoising trajectory and preserve its final clean prediction.The student regresses each next node and uses a separate terminal objective.Source: §B.4.2 · p. 17

B.4.3 SCORE-BASED DISTRIBUTION MATCHING

  • Score-based distribution matching supplements trajectory regression by aligning student-generated samples with the real-data distribution estimated by a frozen teacher.The stage also includes latent-anchor, boundary-consistency, perceptual, and robot-focused objectives; robot-region losses use projected masks and emphasize wrist views.Source: §B.4.3 · pp. 17–18

B.4.4 BLOCK-AUTOREGRESSIVE ROLLOUT

  • Block-autoregressive rollout jointly generates three camera views and bounds long-horizon memory by retaining a persistent sink block and the three most recent blocks.Generated blocks use cached history for continuity rather than repeating the preceding final frame.Source: §B.4.4 · p. 18

B.5 INFERENCE CONFIGURATION

  • 21 seconds per 129-frame rollout gives Causal WorldLine a 4.3× speedup over Stage II’s 90 seconds; evaluation tests held-out AgiBotWorld and unadapted DROID settings.Causal WorldLine uses four denoising steps per block versus 35 for Stage II; video-generation metrics are averaged across trajectories and three views, while robot-mask IoU uses the head view.Source: §B.5 · p. 18, §C.1 · pp. 18–19

C.2 EMBODIED PLANNING

  • 89.3% selected-trajectory success is reached at N=32 on ground-truth RoboTwin rollouts, versus 69.0% at N=2.The intermediate results rise through 79.3%, 82.8%, 82.8%, and 85.9% at N=4, 8, 16, and 24.Source: §C.2 · p. 20

C.3 POLICY EVALUATION

The evaluation estimates trajectory success from visual evidence extracted from generated rollouts and deterministic task-specific rules. It reports mean accuracy across RoboTwin and AgiBot and describes matched data and ablation protocols.

C.3 POLICY EVALUATION

  • The evaluation uses balanced successful and failed trajectory sets from RoboTwin and AgiBot, with five distinct video-generation seeds per trajectory.All simulators receive the same matched base trajectories.Source: §C.3 · p. 20
  • WorldLine generates future videos from initial observations and recorded actions; Qwen3VL-8B extracts structured visual evidence, which deterministic rules convert into success or failure predictions.The model itself does not output a success label.Source: §C.3 · p. 20
  • 74% mean accuracy across RoboTwin and AgiBot measures agreement between final predictions and recorded success or failure labels, averaged across five generation seeds.Each seed pools 100 trajectories from the two datasets.Source: §C.3 · p. 22

C.4 ABLATION PROTOCOL

  • Ablations retain corresponding main-experiment protocols, and the no-action-free-data variant matches the full model’s total training-compute budget.This comparison more directly isolates access to action-free data than an unmatched-compute baseline.Source: §C.4 · p. 22

D.1 REPRESENTATIVE-TASK PLANNING RESULTS

WorldLine improves representative-task planning success over direct policy execution, though gains vary by task and causal-variant performance remains uneven. Supplementary comparisons visualize predictions across held-out scenes, unseen embodiments, and long-horizon rollouts.

D.1 REPRESENTATIVE-TASK PLANNING RESULTS

  • 24.3 percentage points is WorldLine’s average gain for π0.5 on five representative tasks; LingBot-VLA gains 21.6 points on the same subset.The averages rise from 53.3% to 77.6% for π0.5 and from 26.4% to 48.0% for LingBot-VLA.Source: §D.1 · pp. 22–23, Table 4 · p. 23
  • The causal four-step variant reaches 65.6% for π0.5 and 44.0% for LingBot-VLA, but neither variant improves LingBot-VLA on Lift Pot.The causal variant provides little improvement on Handover Block for π0.5; gains are strongest on tasks with visually identifiable state transitions.Source: §D.1 · p. 23

E.1 ADDITIONAL COMPARISONS ON AGIBOTWORLD AND DROID

  • Held-out AgiBotWorld and out-of-domain DROID comparisons show synchronized ground truth and predictions, emphasizing commanded motion, contact, and object-state changes.DROID examples involve unseen environments, viewpoints, and robot embodiments.Source: §E.1 · p. 23

E.2 THREE-VIEW WORLDLINE PREDICTIONS ON AGIBOTWORLD

  • Synchronized head and wrist views let readers examine temporal alignment of robot motion, contact, and object changes in held-out AgiBotWorld scenes and unadapted DROID trajectories.The DROID evaluation uses no training, adaptation, or checkpoint selection on DROID data.Source: §E.2 · p. 23, §E.3 · p. 23

E.4 LONG-HORIZON CAUSAL ROLLOUTS

  • Long-horizon causal rollouts are compared with ground truth across synchronized views to inspect motion, object-state changes, and cross-view consistency over successive generation blocks.Source: §E.4 · p. 24

E.5 QUALITATIVE ABLATIONS

Qualitative ablations compare how training data, action control, and modeling choices affect predicted interactions. Selected failures illustrate residual issues, while the paper identifies training coverage and downstream candidate quality as scope boundaries.

E.5 QUALITATIVE ABLATIONS

  • AgiBotWorld ablation examples compare the full model with variants under identical initial observations and actions, highlighting motion, contact, object displacement, and failure-state differences.The visualizations complement aggregate metrics by showing interaction details that may not be fully captured by image metrics.Source: §E.5 · p. 24, Figure 12 · p. 29

F FAILURE CASES

  • Selected failures include OOD DROID scene distortion and blur, action misalignment, and inconsistent object or container states; the examples do not estimate failure frequency.Source: F FAILURE CASES · p. 30, Figure 13 · p. 30

G.1 LIMITATIONS

  • WorldLine’s coverage remains shaped by training-interaction diversity, with rare patterns, deformable objects, unusual cameras, and longer horizons identified as areas for broader supervision.Planning improvement is also bounded by the underlying policy’s trajectory diversity and the multimodal selector’s quality.Source: §G.1 · p. 30

G.2 BROADER IMPACT

  • The authors state that visualizing and evaluating candidate behaviors before execution may reduce repeated physical interaction, hardware wear, and data-collection costs.Source: §G.2 · p. 30
Loading 2609.38059v1…