Source-linked AI summary

Learning the 4D World through Point Trajectory Forecasting

Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin

arXiv:2610.28594v1cs.CV

TL;DR

Multi-view forecasts should describe the same evolving 3D scene, but visual plausibility alone does not establish coherent 3D motion. VPM jointly predicts future videos and persistent point trajectories; point supervision improves visual forecasting and carries over to robot action learning.

  • Problem

    Visual plausibility alone does not establish whether forecasts describe coherent 3D motion across cameras.

    Source: §1 · p. 2
  • Method

    VPM uses posed multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and persistent 3D point trajectories.

    Source: §1 · p. 2
  • Results

    Point supervision improves visual forecasting on simulated and real benchmarks and carries over to robot action learning.

    Source: §7 · p. 10
  • Takeaways & Limitations

    Persistent point trajectories provide a shared description of motion across cameras, relating appearance changes to identifiable points’ movement.

    Source: §1 · p. 2

    Simulation depth does not reproduce real sensor noise, and rendering profiles are fixed rather than calibrated to real camera responses.

    Source: §D.6 · p. 29

Abstract

from arXiv · show

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.

1 INTRODUCTION

Multi-view forecasts must describe one coherent evolving 3D scene, not merely look plausible from each camera. VPM jointly predicts future videos and persistent 3D point trajectories, with datasets and benchmarks supporting training and evaluation.

1 INTRODUCTION

  • Reliable multi-view forecasting requires cameras to depict the same future event despite differences in perspective and visibility.Persistent trajectories track the same scene point over time, providing a common motion description across cameras.Source: §1 · p. 2
  • VPM-Data pairs synchronized multi-view videos with point trajectories, camera geometry, visibility, and language descriptions; VP-WorldBench evaluates simulated and real environments.Its simulation component assesses image quality and motion accuracy on the same held-out events.Source: §1 · p. 2
  • Point supervision improves visual forecasting, and point supervision in the action-enabled model also improves task success across simulated benchmarks and real-robot experiments.The evaluation covers future-video quality and 3D trajectory accuracy on VPM’s benchmark and existing benchmarks.Source: §1 · p. 2
  • VPM jointly forecasts multi-view videos and persistent 3D point trajectories, using point-motion supervision to improve visual prediction from arbitrary viewpoints.It processes posed histories and language instructions, relating appearance changes to the movement of identifiable scene points.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses video geometry, multi-view generation, motion tracking, and action learning through varied targets and tasks. VPM’s distinction is forecasting persistent metric 3D point motion beyond observed frames and examining its relation to video and action prediction.

Geometry- and Motion-Aware Video Prediction

  • Unlike novel-view generation with multi-view tracking, VPM forecasts beyond observed frames, lifts supervision from 2D to metric 3D, and extends it to action learning (Lee et al., 2026b).MVTrack4Gen supervises novel-view generation with multi-view point tracking (Lee et al., 2026b).Source: §2 · p. 3
  • VPM uses persistent 3D trajectories to retain point identity across views and time, unlike single-view or per-frame geometry and motion targets.Related methods include RGB and pointmap forecasting, geometry or trajectory-field distillation, and multi-view tracking supervision.Source: §2 · p. 3

Video World Models and Action Learning

  • Existing world models predict future observations and actions, while VPM supervises multi-view forecasting with future 3D point motion and examines whether it benefits action learning.Related approaches include VideoVLA and Cosmos 3 (Shen et al., 2026; Agarwal et al., 2026), and multi-view RGB with spatial action heatmaps in SpatialVAM (Li et al., 2026b).Source: §2 · p. 3

3D Motion Forecasting and Benchmarks

  • Tracking benchmarks annotate correspondences or language-conditioned motion, whereas VPM’s benchmark pairs visual and geometric supervision to evaluate future appearance and point motion.The passage contrasts TAPVid-MV and PointMotionBench with VPM’s paired evaluation target.Source: §2 · p. 3

3 METHOD

VPM predicts future videos from arbitrary viewpoints alongside the 3D trajectories of points queried in observed frames. Its camera-conditioned model combines multi-view video processing with trajectory and correspondence supervision, then jointly trains and decodes the forecasts.

3.1 PROBLEM FORMULATION

  • Given posed video histories, observed and requested camera parameters, and a language instruction, VPM predicts future frames at arbitrary viewpoints and trajectories of queried points in a fixed reference frame.Each point query specifies a time, camera, and image location in the history.Source: §3.1 · p. 3

3.2 PRELIMINARIES: MIXTURE-OF-TRANSFORMERS FOR WORLD MODELING

  • The Mixture-of-Transformers backbone uses separate reasoner and generator towers that interact through joint attention; generator tokens attend to both instruction and generator sequences.In forecasting, reasoner attention is causal and restricted to its instruction sequence.Source: §3.2 · p. 3

3.3 CAMERA-CONDITIONED MULTI-VIEW PREDICTION

  • Camera-conditioned multi-view prediction packs camera video tokens into a shared sequence and encodes ray geometry and view identity to distinguish streams.Camera-dependent attention rotations use the geometry associated with each token, while a fourth positional-encoding axis distinguishes cameras sharing temporal and spatial coordinates.Source: §3.3 · p. 4

3.4 JOINT VIDEO AND TRAJECTORY LEARNING

  • Joint learning forecasts point locations from intermediate generator features and directly supervises attention features so the shared video representation learns geometric motion and correspondence.The head refines per-view point estimates over space, time, and views; it trains on partially denoised future features and applies position, displacement, log-depth, and confidence losses to visible points.Source: §3.4 · pp. 4–5

3.5 TRAINING AND INFERENCE

  • Training adapts the generation pathway and camera embeddings, trains the trajectory head with other parameters frozen, then jointly optimizes the video backbone and trajectory head.At inference, future video tokens are denoised from Gaussian noise while history tokens remain clean, and trajectories are decoded from predicted-clip features in the reference frame.Source: §3.5 · p. 5

4 DATASET AND BENCHMARK

VPM-Data pairs synchronized multi-view video with persistent 3D point trajectories, while VP-WorldBench evaluates forecasting across simulated and real environments. Together, they support aligned visual and geometric training and evaluation across tabletop, urban, indoor, and real-world scenes.

4 DATASET AND BENCHMARK

  • VPM-Data contains 10K clips pairing synchronized multi-view videos with persistent 3D trajectories for tabletop manipulation and urban dynamics.The clips include camera geometry, visibility annotations, and language descriptions; points are propagated through motion and occlusion and checked for geometric alignment and annotation completeness.Source: §4 · p. 5, §4.1 · p. 6

4.2 VP-WORLDBENCH

  • VP-WorldBench evaluates tabletop, indoor, and urban scenes in simulation and the real world, including held-out episodes and environments.Its simulation splits include new tabletop locations, unseen kitchens, and urban dynamics absent from training; hard clips include targets hidden from individual cameras while visible in at least one camera.Source: §4 · p. 5, §4.2 · p. 6
  • Real-world evaluation includes 100 teleoperated robot-manipulation episodes and 100 ten-second, 48-camera sequences of people and object interactions.General-scene sequences pair multi-view video with 4D reconstructions, and ground-truth tracks are triangulated from cross-view 2D tracks and retained when depth-consistent.Source: §4.2 · pp. 6–7

5 EXTENSION: ROBOT CONTROL AND JOINT ACTION LEARNING

The robot-control extension adds action prediction to VPM’s shared video-and-trajectory backbone. It trains action tokens jointly with forecasting and denoises video and action tokens together at inference.

5 EXTENSION: ROBOT CONTROL AND JOINT ACTION LEARNING

  • The action-enabled VPM predicts robot commands alongside multi-view video and point trajectories using action tokens in the shared predictive backbone.Action sequences are represented as an additional token modality, allowing action learning to draw on representations trained with point-motion supervision.Source: §5 · p. 7
  • Action tokens are corrupted with independently sampled Gaussian noise and their denoiser output is trained jointly with the forecasting objective.The action loss is applied in the final training stage.Source: §5 · p. 7
  • At inference, video and action tokens are initialized from noise and denoised jointly using only observations and the instruction.Source: §5 · p. 7

6 EXPERIMENTS

Across VP-WorldBench, zero-shot 4D generation, and robot manipulation, point-trajectory supervision improves video fidelity, geometric consistency, and action-enabled task success. The experiments also examine how trajectory and action supervision contribute to these outcomes.

6.2 VP-WORLDBENCH RESULTS

  • VPM (V+T) achieves the best performance on every VP-WorldBench split and metric, improving video fidelity and geometric consistency over VPM (V).Both variants use the same training budget; VPM (V) also improves over the Cosmos3-Nano backbone.Source: §6.2 · p. 8

6.3 ZERO-SHOT 4D GENERATION RESULTS

  • VPM (V+T) achieves the best results on ReCamMaster and CMU Panoptic, and outperforms all baselines on RoboCasa Kitchen.Compared with VPM (V), point supervision improves video fidelity and 3D motion accuracy; on RoboCasa Kitchen, it also surpasses Robot4DGen in multi-view consistency.Source: §6.3 · pp. 8–9

6.4 ROBOT MANIPULATION RESULTS

  • VPM (V+T+A) achieves the highest success rates across the reported simulation benchmarks and every real-world task, outperforming VPM (V+A).The results indicate that forecasting persistent 3D point trajectories provides a useful learning signal for robot actions.Source: Table 5 · p. 8, §6.4 · p. 9

6.5 ANALYSIS AND ABLATION STUDIES

  • Adding trajectories yields the largest gains in video fidelity and 3D motion accuracy on every split, while adding actions yields modest improvements.Training with video, trajectories, and actions performs best on all splits and metrics, indicating that action supervision complements point supervision.Source: Table 6 · p. 8, §6.5 · p. 9
  • The trajectory head predicts point motion from generator features that subsequent blocks use to produce frames, directly shaping the video-generation representation.Because each trajectory follows one physical point, videos following those trajectories remain consistent across views.Source: §6.5 · p. 9, Figure 4 · p. 9
  • Robot trajectories encode 3D motion across views, densely constrain future robot configuration, and can specify manipulated-object destinations when available.Converting predicted trajectories into commands with inverse kinematics completed unseen real-robot tasks zero-shot, although the rollouts were not consistent enough for quantitative evaluation.Source: §6.5 · pp. 9–10, Figure 4 · p. 9

7 CONCLUSION AND LIMITATIONS

VPM jointly forecasts future videos and persistent 3D point trajectories from posed video histories and language instructions. Its scope is limited by partial point coverage, omitted independent cloth and hair motion, calibrated-camera requirements, and costs that grow with the number of views.

7 CONCLUSION AND LIMITATIONS

  • Point supervision covers the robot and annotated objects rather than the full scene, and skeletal annotations omit independent cloth and hair motion.Source: §7 · p. 10
  • VPM assumes known camera intrinsics and poses at inference, and its cost grows with the number of views because camera streams share one token sequence.The paper identifies real multiview trajectory supervision, denser non-rigid annotations, and relaxed camera-calibration requirements as future work.Source: §7 · p. 10

A.1 TRAINING DETAILS

  • Training proceeds in three stages: generation pathway, camera embeddings, and view frequencies; trajectory head and view frequencies; then joint video-backbone and trajectory-head optimization.The reasoner remains frozen throughout, and the training pipeline can forecast without future camera parameters.Source: §A.1 · p. 17

A.2 EVALUATION DETAILS

The evaluation specifies video, geometric-consistency, and robot-manipulation protocols. It also tests whether predicted point trajectories can directly drive robot commands without action labels or robot-specific training.

A.2 EVALUATION DETAILS

  • Video evaluation predicts 64 frames at 480p from either one input frame or 16 frames of multi-view history; single-view baselines generate each view independently.Models use their default denoising steps.Source: §A.2 · p. 17
  • FVD and PSNR are averaged across views, while geometric consistency uses cross-view MEt3R and δ3D_avg.Cross-view MEt3R measures deviation from ground-truth consistency because raw MEt3R can mis-rank multi-view videos when views have little overlap.Source: §A.2 · pp. 17–18

A.3 ROBOT MANIPULATION TASKS

  • Three pick-and-place tasks require multi-view scene fusion, obstacle avoidance, and instruction grounding among same-category distractors.Each trial scores 1 for reaching the goal without touching the box pile, 0.5 for a successful pick-and-place that hits the pile, and 0 otherwise.Source: §A.3 · p. 19

A.4 DETAILS OF ROBOT MANIPULATION THROUGH 3D POINT TRAJECTORY

  • Predicted robot trajectories are triangulated into 3D, fitted to the robot’s kinematics, and executed receding-horizon without action labels or robot-specific training.The method samples 128 robot-surface query points and reads their tracks at the third of four denoising steps, where the training noise-level distribution has its mode.Source: §A.4 · p. 20

B DATASET CURATION

  • VPM training draws on 600K samples from simulation, robot-manipulation datasets with kinematics-grounded trajectories, and open-source video datasets.Source: B DATASET CURATION · p. 21

B.1 DATA STATISTICS

The training collection combines VPM-Data, robot datasets, and open-source clips. Robot-video point trajectories are constructed from known geometry and kinematics, then filtered for image-space alignment.

B.1 DATA STATISTICS

  • The 600K training samples include 10K VPM-Data samples, 91K robot episodes from DROID and HRDexDB, and clips sampled from open-source datasets.Source: §B.1 · p. 21

B.2 3D POINT EXTRACTION PIPELINE FROM ROBOT VIDEOS

  • Robot and object surface points are sampled once in mesh-local coordinates and transformed by recorded poses, preserving point identity across the episode.The pipeline uses robot CAD or visual meshes, calibrated cameras, and known transforms between the robot base and camera frames.Source: §B.2 · pp. 21–22
  • Robot-data episodes are retained only when CAD-mask recall is at least 0.9, filtering cases where projected geometry disagrees substantially with the observed robot mask.Recall is preferred because image masks may include non-robot regions or merge the robot with nearby objects.Source: §B.2 · p. 22
  • The output pairs synchronized multi-view RGB with persistent surface-point trajectories, joint states, and camera metadata, so each index identifies the same physical location across time and views.Source: §B.2 · p. 22

C TRAINING DATA GENERATION

The paper describes separate simulation pipelines for training data and VP-WorldBench, with synchronized views and persistent geometric annotations. The benchmark includes tabletop and urban clips, including subsets designed to test incomplete per-camera visibility.

C.1 CAPTURE SETTINGS AND ENVIRONMENTS

C.2 TABLETOP GENERATION; C.3 URBAN CAPTURE

  • Tabletop training replays simulated pick-and-place episodes with targets, receptacles, and distractors; urban capture records moving pedestrians or vehicles across synchronized camera views.Tabletop distractors are placed relative to robot and object swept volumes, while urban clips balance five clip-length buckets.Source: §C.2 · p. 23, §C.3 · pp. 23–24

C.4 GEOMETRIC AND LANGUAGE ANNOTATIONS

  • Geometric annotations preserve surface-point identities over time and record per-view visibility, camera parameters, and language captions.Tabletop clips include robot points; urban visibility distinguishes visible, out-of-frustum, and occluded targets.Source: §C.4 · p. 24

C.5 VISIBILITY-CONTROLLED CLIPS

  • Training hard clips create incomplete observations in individual cameras while retaining target coverage across the camera set, and labels are verified from measured visibility.Urban hard clips use departure and re-acquisition phases; failed attempts are replaced.Source: §C.5 · pp. 24–25

C.6 VALIDATION AND RENDERING

  • Validation replaces failed samples, and the training profile records departure–re-acquisition as a label without requiring it for acceptance.The benchmark profile instead enforces departure–re-acquisition as an acceptance condition.Source: §C.6 · p. 25

D.1 COMPOSITION AND CAPTURE SETTINGS

  • VP-WorldBench’s simulation component contains 500 tabletop and 500 urban clips, totaling 661,071 images and 716,544 persistent point trajectories.Images and annotations align to the same simulation timesteps, and point identities remain defined during occlusion.Source: §D.1 · p. 26

D.2 TABLETOP CAPTURE; D.3 URBAN CAPTURE; D.4 GROUND-TRUTH ANNOTATIONS

  • Benchmark capture synchronizes urban sensors at each simulator tick and stores persistent object points, camera geometry, and coordinates for tabletop and urban scenes.Urban targets use skeletal deformation annotations, which follow the skeleton rather than independent cloth or hair motion.Source: §D.3 · p. 27, §D.4 · p. 27

D.5 VISIBILITY-CONTROLLED SUBSETS

  • No hard tabletop frame lacks target visibility across all views; among 17,195 hard urban frames, none lacks visibility under the strict labeling array.One hard urban frame has no visible camera under bounding-box visibility.Source: §D.5 · p. 28

D.6 VALIDATION AND LIMITATIONS

The benchmark validates simulation annotations and documents their scope limits. It also adds a real-world human benchmark and positions its simulation resource alongside existing datasets with complementary annotations and tasks.

D.6 VALIDATION AND LIMITATIONS

  • Urban surface reconstructions have 90th-percentile reprojection errors of 0.00 px for pedestrians and 1.00 px for vehicles; 500 of 793 judged clips pass the strict profile and are selected.Transparent and unsupported surface sections are excluded from validation, and all selected urban clips have complete tracks without breaks.Source: §D.6 · p. 28
  • Simulation depth does not reproduce real sensor noise, rendering profiles are not calibrated to real camera responses, and seeds alone cannot guarantee bit-exact regeneration.The hard urban subset also over-represents pedestrians, and skeletal annotations omit independent cloth and hair motion.Source: §D.6 · p. 29

D.7 REAL-WORLD BENCHMARK DETAILS

  • The real-world human split contains 100 synchronized 10-second sequences from three participants, with reconstructed 3D surfaces and camera parameters.The sequences capture everyday activities in a furnished 23.1 m2 home-like space.Source: §D.7 · p. 29

D.8 RELATION TO EXISTING RESOURCES

  • The simulation benchmark pairs future multi-view appearance with metric surface-point trajectories, camera geometry, depth, visibility, and language on the same clips.Related resources provide complementary annotations and tasks, including human-body geometry, multi-view re-rendering, point tracking, and language-conditioned point-motion forecasting.Source: §D.8 · p. 30

E.1 DETAILED EXPERIMENTS ON VP-WORLDBENCH

Across VP-WorldBench and other forecasting settings, VPM’s point-trajectory supervision improves video fidelity and 3D motion accuracy; ablations show that temporal identity, camera conditioning, and cross-view attention matter to these gains.

E.1 DETAILED EXPERIMENTS ON VP-WORLDBENCH

  • VPM (V+T) achieves the best performance on every VP-WorldBench split and metric, improving video fidelity and 3D motion accuracy over VPM (V).The comparison uses equal training budgets; VPM (V) also improves over its Cosmos3-Nano backbone.Source: Table 16 · p. 31

E.3 COMPARISON WITH PER-FRAME GEOMETRY SUPERVISION

  • 3D trajectory supervision outperforms per-frame depth or pointmap supervision on every split, with the largest gap on dynamic scenes with substantial motion and occlusion.Per-frame supervision recovers only 19% (depth) and 28% (pointmap) of VPM (V+T)’s FVD improvement over VPM (V).Source: §E.3 · p. 33, Table 22 · p. 33

E.4 PERFORMANCE OF 3D POINT TRAJECTORY HEAD

  • VPM’s trajectory head achieves the lowest ADE on HOT3D and the best results on all three metrics on WorldTrack, while lowering ADE and FDE versus MolmoMotion-AR on every dataset.With camera conditioning, VPM also surpasses Cosmos3-Nano on every VP-WorldBench split in all-view V2V; its predicted tracks improve tracking accuracy, including δ3D avg from 64.75 to 67.22 on Real-world Robot.Source: §E.1 · pp. 31–32, §E.4 · p. 33

E.5 PERFORMANCE ON SINGLE-VIEW WORLD MODELING

  • On single-view benchmarks, VPM leads PAIBench-G’s Overall and Domain scores, ties Quality, and ranks second on RBench; its V2V score of 52.1 beats three baselines but trails Magi-1 and Cosmos3-Super.The V2V comparison is reported in Table 25.Source: §E.5 · p. 34, Table 25 · p. 34

E.6 ADDITIONAL RESULTS ON MODALITY-WISE ABLATION STUDY

  • On zero-shot ReCamMaster and CMU Panoptic, adding point trajectories reduces FVD from 103.12 to 79.63 and raises δ3D avg from 34.72 to 42.03; action supervision complements trajectory supervision.The benchmarks contain no robots and were not used for training; adding camera conditioning to VPM (V+T+A) further improves all metrics on both.Source: Table 26 · p. 34, §E.6 · p. 35

E.7 ANALYSIS ON MULTI-VIEW VIDEO GENERATION

  • VPM’s attention maps align a queried hand across views, unlike Cosmos3-Nano’s scattered attention and visibly distorted generated arm.The authors interpret this as evidence that VPM learns and uses cross-view correspondence.Source: §E.7 · p. 35, Figure 8 · p. 35

E.8 CLOCK-TIME ANALYSIS / E.9 DATA SCALABILITY

  • For four views, decoding trajectories and actions adds 0.9 s and 0.2 s, respectively; point supervision improves over video-only training at every data scale, with VPM (V+T) trained on 100K samples outperforming VPM (V) trained on 600K on every split.The reported example is 70.61 vs. 78.52 FVD on Simulation Robot; the authors characterize point supervision as more data-efficient than scaling video data alone.Source: §E.8 · p. 36, §E.9 · p. 36

F.1–F.5 ABLATION STUDIES

  • Camera-dependent rotation improves both metrics on every split; all objectives combined and the full curriculum each perform best on every split, while removing Stage 2 is more harmful than Stage 3-only training.On Simulation General, adding the Plücker embedding reduces FVD from 128.41 to 84.60.Source: §F.1 · p. 37, §F.2 · p. 37, §F.3 · p. 38

F.6 ABLATION STUDY FOR TRAINING DATA

Training sources contribute most to the domains they cover, and each source supports performance across evaluation splits. The accompanying figures show multi-view forecasts, video comparisons, and robot-task rollouts.

F.6 ABLATION STUDY FOR TRAINING DATA

  • Removing any training source degrades every split, and each source mainly benefits the evaluation domain it covers.Removing VPM-Data causes the largest simulation degradation, especially on Simulation General (+48.3% FVD and −9.16 δ3D avg); removing robot datasets mainly affects Real-world Robot (+18.2% FVD and −12.39 δ3D avg), while removing open-source videos mainly affects Real-world General (+12.2% FVD and −6.73 δ3D avg).Source: §F.6 · p. 39

G.1 MULTI-VIEW VIDEO AND POINT TRAJECTORY PREDICTION

  • The multi-view qualitative examples overlay predicted point tracks on two-view and three-view forecasts, with conditioning frames omitted.Additional examples cover robot and general scenes in both real and simulated VP-WorldBench settings.Source: Figure 11 · p. 40, Figure 12 · p. 41

G.2 PREDICTED VS. GROUND-TRUTH VIDEOS

  • The video comparisons place each predicted view next to its ground truth, with conditioning frames omitted.Source: Figure 13 · p. 42

G.3 EXAMPLE ROLLOUTS OF ROBOT MANIPULATION TASKS

  • The rollout figure shows successful policy episodes on six manipulation tasks from two exterior cameras and a wrist camera, sampled uniformly over each episode.Source: Figure 14 · p. 43
Loading 2610.28594v1…