PointCast: One World Model for Rigid, Articulated, and Deformable Object Manipulation

I. INTRODUCTION

• Prior approaches trade regime-specific physics structure or pixel-space breadth against geometric point-motion models that add appearance features or material labels.The open need is a model given only object point locations that covers rigid, articulated, and deformable objects under one recipe.Source: §I · p. 1
• POINTCAST models persistent 3D object and end-effector points, supervises each point on its own trajectory, and uses alternating local/global attention with end-effector cross-attention.The same recipe covers rigid objects, cloth, rope, and multi-joint cabinets, or can train on real robot episodes.Source: §I · p. 1
• POINTCAST is best on three of four simulated regimes and second on rigid, and on real data is best in four of six categories, second in two, and outperforms PGND in all six.Its simulation checkpoints are also best on two of four zero-shot captures.Source: §I · pp. 1–2
• Frozen inside sampling-based MPC, POINTCAST plans all four tasks at one network evaluation per window, competitively with or better than every baseline on each.The experiments span four simulated regimes, six real-world categories, and four zero-shot captures.Source: §I · p. 2

Physics-embedded Models and Digital Twins

• Physics-embedded models and digital twins commit to a solver’s equations, restricting accuracy to the modeled material and requiring parameters identified for each new object.Examples include particle-grid and material-point methods,,, spring-mass reconstructions, Gaussian representations, and reduced-order elastic bases.Source: §II · p. 2

Learned Point Dynamics

• Graph-based dynamics learn particle interactions through hand-set proximity thresholds, while point-track and 3D-flow forecasters predict motion without conditioning on robot actions.AdaptiGraph estimates physical properties online, but its graph thresholds constrain which interactions it can learn.Source: §II · p. 2
• Transformer world models condition point sets or tokens on actions, but ParticleFormer uses unpaired shape losses and test-time material labels, while PointWorld [7] adds frozen-encoder appearance features.POINTCAST instead keeps only points and learns their interactions and response to end-effector motion under one recipe across rigid, articulated, and deformable objects.Source: §II · p. 2

Video World Models

• Video world models generate future frames autoregressively, spending capacity on illumination, texture, and viewpoint rather than the geometric state consumed by a planner.POINTCAST applies the generative formulation to geometry, predicts a whole window in one Euler step, and runs inside sampling-based MPC.Source: §II · p. 2

A. Problem Setting

• Each interaction uses persistent object control points selected once by farthest point sampling, with correspondence supplied by simulation or real-data tracking.The input combines current geometry, its one-step displacement, and the end-effector trajectory over the next F frames; the model predicts a distribution over those future object states.Source: A. Problem Setting · p. 2
• The setup uses K=128 object points, M=16 actor points, and a five-frame prediction window, with positions scaled by the initial object bounding-box diagonal.Positions are expressed in the world frame.Source: A. Problem Setting · p. 2

B. Architecture

• The diffusion transformer denoises all F future positions per point at once, alternating kNN-local attention over 16 neighbors with global attention and using register tokens.Each object-point token fuses its noisy future window with observed position and displacement; registers belong to no point and are excluded from local attention.Source: B. Architecture · p. 2, Figure 2 · p. 3, Figure 3 · p. 3
• Actor-point cross-attention carries end-effector position and motion information so object points can resolve nearby end-effector geometry and infer contact.The end-effector’s within-window finger motion and rotation are not observed directly.Source: B. Architecture · pp. 2–3

C. Objective and Inference

• Flow matching interpolates between Gaussian noise and the clean future window, with flow time sampled toward the noisy end of the path.The model regresses the clean window under three loss terms.Source: C. Objective and Inference · p. 3
• The velocity and distance terms use λv=0.1 and λa=0.01 in every regime.Source: C. Objective and Inference · p. 3
• The objective adds a velocity term matching the prediction-implied velocity to the true path velocity and a distance-preservation prior on neighboring points.The prior holds each point at its initial distances from six nearest neighbors; on cabinets, neighbors are restricted to the same part.Source: C. Objective and Inference · p. 3
• At inference, one Euler step integrates from τ=0 to τ=1 and returns the predicted clean window.Rollout advances by appending the predicted frames to history and reading the next actor-trajectory stretch from the controller.Source: C. Objective and Inference · p. 3

A. Setup

• Simulation randomizes object geometry, material parameters, and commanded motion, with evaluation objects held out from training.Training pools contain 112k rigid, 10.2k cloth, 10.2k rope, and 5.0k cabinet interactions; 15% are held out for validation.Source: A. Setup · p. 4
• The real dataset is the six-category PGND benchmark of multi-view RGB-D teleoperated interactions, using its own splits and metric.Real object points use released CoTracker3 tracks, and actor points follow the benchmark’s commanded end-effector trajectory.Source: A. Setup · p. 4
• AdaptiGraph and PGND retain their authors’ recipes, while PTv3, ParticleFormer, and POINTCAST train from scratch per regime and category.Models use one checkpoint per regime or category, selected on a validation slice; training runs 200 epochs of 800 steps at batch size 64.Source: A. Setup · p. 4
• Simulation uses mean rollout error over the carried dense cloud in millimeters, while real data use mean displacement error at the 3 s horizon.Cabinet simulation scores only the actuated joint; simulation rollouts follow ground-truth end-effector trajectories.Source: A. Setup · p. 4

1) Simulation:

• POINTCAST is best on three of four simulated regimes and second on rigid by mean rollout error.PTv3 loses every regime at matched budget and capacity: 19.5M parameters versus POINTCAST’s 19.8M.Source: 1) Simulation · p. 4
• POINTCAST leads ParticleFormer by 16.6% on cloth and 32.6% on rope.Source: 1) Simulation · p. 4
• On the cabinet actuated joint, POINTCAST records 6.84 against PTv3’s 10.34; rigid is the loss, 16.9% behind ParticleFormer.The passage attributes the gap to the per-point loss rather than the backbone.Source: 1) Simulation · p. 4

• AdaptiGraph is excluded from both real-data evaluations because its released evaluation advances only while the end-effector moves, whereas teleoperated captures pause.Source: 2) Real World · p. 4
• POINTCAST has the lowest mean error in four of six PGND categories and ranks second in the other two, uniquely improving on PGND in all six.Source: Table II · p. 5, 2) Real World · p. 5
• PGND scoring uses benchmark step 30, the end of its 3 s horizon, and reproduces published values within 1 mm in five of six categories.Source: 2) Real World · p. 5
• Zero-shot checkpoints are tested on four captures matched to training actions; POINTCAST leads Push-T and Franka, while ParticleFormer leads rope and cloth.Source: Table III · p. 5, 2) Real World · p. 5
• On PGND episodes with little end-effector motion, rope and cloth errors are 92.45 and 134.33 mm; re-striding reduces them to 65.25 and 51.87 mm.The resulting gaps over the in-domain rows are 2.0× on cloth and 3.3× on rope.Source: 2) Real World · p. 5

• The ablation rows keep the training seed and one-step scoring fixed, except for the 25-step and direct-regression rows.Source: C. Ablations · p. 5
• Replacing per-point supervision with a shape-level objective makes the cabinet actuated joint 79% worse, while improving the all-points cabinet error from 9.66 to 6.51.The swap improves rigid by 14.5% but worsens cloth by 3.5%; the per-point model still beats ParticleFormer on cloth, rope, and cabinet under either loss.Source: C. Ablations · p. 5
• Removing the distance prior costs 18% on the cabinet joint, while improving cloth by 3.1% and rope by 2.1%; the cabinet part label is worth 15.6% on that joint.Source: C. Ablations · p. 6
• Either single attention pattern is worse than alternation on rigid by 4.9% or 8.8%, while changing cloth, rope, and cabinet-joint results by less than 7% in either direction.Source: C. Ablations · p. 6
• Twenty-five sampling steps buy under 1% on cloth and nothing on rope, but worsen rigid by 5% and the cabinet joint by 16%; a shared four-regime checkpoint is up to 3.8 times worse.The shared checkpoint uses four times the budget and remains within 3.3% on rigid and cloth, but is 15% worse on rope.Source: C. Ablations · p. 6
• One-step window prediction takes 16.2 ms of model compute on an RTX 4090 at batch size one, versus 384.8 ms for 25 steps; free-running rollouts run 18–77× faster than real time.Source: C. Ablations · p. 6

D. Planning with World Model

• The planner rolls candidate action sequences through POINTCAST in one evaluation per window, executes one window, then replans.It uses 32, 16, and 8 candidates for push, cabinet, and drag tasks, respectively, and looks three, two, and one windows ahead for pose push, sequential articulation, and drags.Source: D. Planning with World Model · p. 6
• All four tasks use unseen object geometry and point-set goals, covering pose push, sequential cabinet articulation, cloth drag, and rope drag.Pose push sends a T-block to a planar target with a commanded yaw; the drags move an object from a scripted grasp to a translated target.Source: D. Planning with World Model · pp. 6–7
• The evaluation reports mean distance error over 16 episodes and planning time per window; Fig. 7 shows one episode per task, while Fig. 8 plots goal distance by replanning step.Fig. 8 uses log axes and a bootstrap 95% interval over 16 episodes; POINTCAST's curve settles lowest after the re-attach.Source: D. Planning with World Model · pp. 6–7, Figure 7 · p. 7, Figure 8 · p. 7
• POINTCAST is competitive with or outperforms every baseline across all four planning tasks.It beats PTv3 and AdaptiGraph on pose push and both drags, PGND on pose push and cloth drag, and ParticleFormer on rope drag; remaining margins are within run-to-run spread.Source: Table V · p. 6, D. Planning with World Model · p. 7

V. CONCLUSION

• POINTCAST treats manipulation dynamics as motion of a persistent point set, supervised per point.A rollout that feeds predictions back into the model compounds error over the horizon.Source: §V · p. 7
• POINTCAST is best on three of four simulated regimes and second on rigid, and best in four of six real-robot categories and second in the other two.Inside sampling-based MPC, it plans four simulated tasks competitively with or better than every baseline on each.Source: §V · p. 7
• POINTCAST predicts one window at a time, and feeding predictions back compounds rollout error over the horizon.The pose-push planner already looks three windows ahead; extending rollouts further remains an open question.Source: §V · p. 7
• The conclusion presents geometry alone, without hand-specified material, as sufficient to predict rigid, articulated, and deformable motion.It identifies self-forcing and diffusion forcing in video models as approaches that could also be applied to point-set states.Source: §V · p. 7