WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

• Video models leave 3D dynamics implicit, action-based models face costly supervision and embodiment-tied representations, and flow-only motion training provides limited physical-dynamics supervision.Dense 3D scene flow is computationally expensive; coarse spatial grids can waste tokens on irrelevant regions and limit fine motion-boundary fidelity.Source: §1 · p. 2
• WorldFlow jointly predicts future video and query-conditioned 3D trajectories using a unified MoT backbone and rectified-flow objective.Its continuous displacement tokens encode normalized frame-to-frame motion, include initial point geometry, and align spatiotemporally with video tokens.Source: §1 · p. 2
• WorldFlow achieves the lowest ADE and FDE across all three evaluation sets; versus flow-only, it reduces ADE by 48.5% on Sim7 and 55.9% on SpatialCog, and FDE by 68.5% on SpatialCog.Source: §1 · p. 2
• The training corpus contains approximately 1.44 million object-level samples from simulated and real-world domains, and WorldFlowBench evaluates three sets including SpatialCog for spatial reasoning.The corpus combines existing annotations with automated trajectory extraction, unified representations, source-specific quality control, and fixed domain-level normalization.Source: §1 · pp. 2–3

2.1 Predictive World Models: From Pixels to Actions

• Predictive world models range from video generators that forecast pixels to action-conditioned models that jointly predict observations and actions for robotic behavior and control.Unified multimodal models also combine visual understanding with generation, including language, pixels, and actions.Source: §2.1 · p. 3

2.2 3D Motion Modeling: From Tracking to Forecasting

• 3D motion research spans pixel-level tracking, video-generation-and-tracking pipelines, and direct geometric forecasting, with query-conditioned object-point prediction among recent approaches.Trackers rely on immediate visual evidence and can accumulate errors under occlusion or non-rigid deformation; direct forecasting predicts future geometry without synthesizing video.Source: §2.2 · p. 3

3.1 Problem Formulation

• Given a language instruction, observed RGB frame, and 3D query points, WorldFlow predicts future RGB frames and 3D flow in the initial camera coordinate system.Flow is each query point’s frame-to-frame displacement, and trajectories are recovered by accumulation.Source: §3.1 · p. 4

3.2 Architecture

• A shared Mixture-of-Transformers generator jointly denoises future video and 3D flow, then decodes video and accumulates rescaled flow into metric trajectories without a separate tracking stage.It is conditioned on language, observed-frame latents, and query geometry.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 3D Flow Tokenization

• Each query point’s future displacement is a continuous-valued token augmented with initial 3D geometry and spatiotemporal positional encoding aligned to video tokens.The shared positional convention encodes relative spatiotemporal relationships between flow and video tokens in multimodal attention.Source: §3.3 · pp. 4–5
• Displacements are domain-normalized and arranged frame-major, while query geometry encodes absolute and centroid-relative positions before combination with noisy flow and conditioning embeddings.The geometry embedding is shared across future tokens for the same query point.Source: §3.3 · pp. 4–5

3.4.1 Joint flow matching

• A weighted rectified-flow objective jointly supervises video-latent and normalized-flow vector fields, using shared per-sample noise and excluding conditioning positions from supervision.Losses are computed separately for each modality.Source: §3.4.1 · p. 6

3.4.2 Auxiliary motion objectives

• Auxiliary objectives penalize frame-to-frame displacement, cumulative direction, and relative endpoint errors on reconstructed clean normalized flow.Small-net-displacement points are excluded from endpoint supervision, and noise-dependent weighting emphasizes lower-noise samples.Source: §3.4.2 · p. 6

4.1 Datasets

• The training corpus contains approximately 1.44 million object-level samples, and WorldFlowBench contains 1,678 held-out samples across simulated, real-world, and spatial-reasoning subsets.The evaluation subsets contain 396 Sim7, 882 Real3, and 400 SpatialCog samples; evaluation samples are excluded from training.Source: §4.1 · pp. 6–7

4.2 Training Configuration and Strategy

• Training initializes from pretrained Cosmos-Nano checkpoints, then uses a 51,100-step heterogeneous-data curriculum with simulation, real-world adaptation, and SpatialCog refinement.The first stage trains on SpatialCog for 32,000 steps; the second stage retains cross-domain replay while prioritizing each phase’s target domain.Source: §4.2 · p. 7

4.3.1 Evaluation Metrics

• ADE averages position error across query points and future frames, FDE measures error at the final frame, and PWT averages hit rates at 1, 2, and 5 cm thresholds.ADE and FDE are reported in meters; higher PWT indicates more point–frame pairs within the distance thresholds.Source: §4.3.1 · p. 7

4.3.3 Ablation Studies

• Joint video-and-flow generation reduces ADE versus a flow-only variant by 48.5% on Sim7, 19.8% on Real3, and 55.9% on SpatialCog.On SpatialCog, FDE falls from 0.248 to 0.078 m, a 68.5% reduction; FDE is lower on all three subsets.Source: §4.3.3 · p. 8

4.3.2 Quantitative Evaluation on WorldFlowBench

• WorldFlow achieves the lowest ADE and FDE among evaluated learning-based methods on all three WorldFlowBench subsets.Relative to each subset’s strongest external learned baseline, ADE falls by approximately 46.9% on Sim7, 4.7% on Real3, and 54.2% on SpatialCog.Source: §4.3.2 · p. 8, Table 1 · p. 9

4.3.3 Ablation Studies

• Directly generated flow achieves lower ADE and FDE and higher PWT than Track4World and 4RC tracking of WorldFlow-generated videos on all three subsets.On SpatialCog, direct generation achieves 0.060 m ADE, versus 0.083 m for Track4World and 0.098 m for 4RC.Source: §4.3.3 · pp. 8–9
• Auxiliary motion objectives improve ADE, FDE, and PWT on all three subsets; on Sim7, ADE falls from 0.063 to 0.051 m, a 19.0% reduction.The objectives supervise displacement, cumulative motion direction, and trajectory endpoints through reconstructed clean flow.Source: §4.3.3 · p. 9

4.4 Flow-Guided Robotic Manipulation

• Across 36 physical-robot trials, WorldFlow achieves 97.2% flow-prediction success and 66.7% robot-execution success, versus 25.0% and 11.1% for µ0.The flow predictor is the only changed component; execution failures also arise from grasping, trajectory optimization, and unstable stacking placement.Source: §4.4 · p. 10, Table 2 · p. 10

5 Conclusion

• WorldFlow jointly generates future video and metric 3D flow from a single observation, language instruction, and 3D query points.Continuous flow tokens are grounded in video coordinates and initial point geometry, and both modalities are jointly denoised by a shared rectified-flow generator.Source: §5 · p. 10
• Experiments across diverse embodied domains show improved motion forecasting over direct flow predictors and video-generation-then-tracking pipelines.Source: §5 · p. 10
• The current formulation requires predefined query points and inherits noise from automatically generated trajectory supervision.Source: §5 · p. 10

AI Use Statement

• Generative AI tools assisted with translation, experimental-methodology feedback, and interpretation of results.Source: AI Use Statement · p. 11
• Gemini 2.5 Pro generated captions for Sim7 and Real3.Source: AI Use Statement · p. 11
• The authors reviewed AI-assisted content and checked technical descriptions, numerical comparisons, and conclusions against experimental records.Source: AI Use Statement · p. 11

A.1 Data Processing Details

• For Sim7, Real3, and SpatialCog, records use 49 RGB frames and 64 query points, with tracked pixels back-projected into the first-frame camera coordinate system.The first-frame object mask defines the visible partial surface for query-point selection, and frames and trajectories share temporal indices.Source: §A.1 · p. 16
• MolmoMotion-1M reuses released trajectories, standardizes point counts and clip lengths, and discards records with fewer than eight valid object-point candidates.Intervals with 8–48 frames are extended by repeating the final frame and positions, producing zero-displacement padded transitions included in training.Source: §A.1 · p. 16
• Quality control checks geometric validity, temporal continuity, observation completeness, and motion plausibility before temporal padding.Source: §A.1 · p. 16
• Motion and spatial thresholds vary by source: SpatialCog and Real3 cap peak displacement at 3 m, while YT-VIS uses relative statistics without absolute displacement or speed thresholds.Simulation records are screened for complete geometry and valid task annotations.Source: §A.1 · p. 16
• SpatialCog has 82,588 valid manifest records from 82,831 candidates, a 99.7% retention rate; 400 evaluation records are excluded from training.Counts precede the training/evaluation split and include the benchmark samples.Source: §A.1 · pp. 16–17, Figure I · p. 17

A.2 Dataset Statistics

• 1,438,379 standardized object clips contain 70,480,571 frame instances and 4,418,700,288 3D displacement vectors, before Sim7 runtime exclusion.The inventory also reports 92,056,256 point trajectories.Source: §A.2 · p. 17
• The reported approximately 1,223.6 hours measure standardized clip volume, not deduplicated source-video duration.Different object records may share the same underlying video interval.Source: §A.2 · p. 17
• MolmoMotion-1M sources exhibit substantial variation in native temporal coverage; Figure II reports interval lengths before conversion to a common 49-frame representation and the fraction of records requiring tail padding.Source: §A.2 · p. 17
• Component RMS statistics characterize motion-target numerical scale before normalization, including zero-motion transitions from temporal padding, rather than physical speed or endpoint displacement.The statistics include per-record distributions and source-level aggregate RMS.Source: §A.2 · p. 17

B Training Details

• Table I summarizes the composition of the processed data inventory across sources.Real3 and SpatialCog exclude benchmark evaluation records, while Sim7 uses its QC inventory before runtime exclusion.Source: Table I · p. 18
• Each object clip contains 49 frames standardized to 16 FPS, and duration is computed from the reported clip count.The calculation is Nclips × 49/(16 × 3600).Source: Table I · p. 18
• Reported standardized hours measure object-clip volume rather than deduplicated source-video duration because records may share video intervals.Source: Table I · p. 18

B.1 Data Sampling and Curriculum

• MolmoMotion-1M subsets are converted to 49 frames by padding shorter intervals and uniformly subsampling longer ones; motion-scale statistics include padding-induced zero-motion transitions.The statistics describe numerical target scale, not physical speed or endpoint displacement.Source: Figure II · p. 19, Figure III · p. 19
• Second-stage training uses five sampling domains and a three-phase curriculum, with cross-domain replay and transitions between domain-sampling distributions.Within-domain source sampling uses source counts with a sublinear exponent to reduce larger sources’ dominance.Source: §B.1 · pp. 18–20
• Training prompts add origin, actor, camera-view, and scene context fields, using sample metadata when available and source defaults otherwise.Training and inference share the format, which excludes dataset identifiers and flow-normalization values.Source: §B.1 · p. 20
• The second-stage schedule warms up to 10^-4, plateaus at 8 × 10^-5 through step 42,100, then decays during late adaptation and refinement.The 100-step Outdoor interval remains at 8 × 10^-5.Source: §B.1 · p. 20, Figure IV · p. 20

C Auxiliary Motion Objectives

• Auxiliary objectives supervise frame-to-frame displacement, cumulative displacement direction, and final displacement in normalized coordinates.Source: C Auxiliary Motion Objectives · p. 21

C.1 Clean-Flow Reconstruction

• The model reconstructs clean displacements from its predicted rectified-flow vector field, and auxiliary losses act on that reconstructed sequence.Squared displacement supervision is a noise-dependent reweighting of vector-field squared error.Source: §C.1 · p. 21

C.2 Motion Supervision

• Motion supervision compares local displacement, cumulative direction, and relative endpoint error to ground truth, constraining both stepwise motion and trajectory-level consistency.Direction error is weighted by ground-truth cumulative displacement magnitude.Source: §C.2 · p. 21
• Relative endpoint loss applies only when net displacement is at least δ, excluding stationary or near-returning trajectories where relative error is unstable.The threshold is defined in normalized coordinates.Source: §C.2 · p. 21