PaperlayerView exact revision on arXiv

Source-linked AI summary

PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

Wenlong Huang, Yu-Wei Chao, Arsalan Mousavian, Ming-Yu Liu, Dieter Fox, Kaichun Mo, Li Fei-Fei

arXiv:2601.03782v1cs.ROcs.AIcs.CV

TL;DR

Robotic world models need to predict action-conditioned 3D changes from sparse observations while handling varied robot embodiments and open-world interactions. PointWorld addresses this with a large pre-trained 3D point-flow model and dataset, and a single checkpoint enables diverse real-world manipulation from one RGB-D capture. Its scope is bounded by rigid-body and fully realized trajectory assumptions.

  • Problem

    Robots need models that predict how unstructured environments evolve from perceptual observations and intended body actions in 3D.

  • Method

    PointWorld represents scene state and robot action as 3D point flows and trains on large-scale real and simulated manipulation data with high-quality 3D annotations.

  • Results

    A single pre-trained checkpoint enables a real robot to perform pushing, deformable and articulated manipulation, and tool use from one in-the-wild RGB-D capture without demonstrations or post-training.

  • Takeaways & Limitations

    PointWorld provides a real-time 3D world model that can be integrated with model-predictive control for diverse manipulation tasks.

  • Takeaways & Limitations

    The formulation assumes rigid robot links and a known, fully realized joint trajectory, so it does not model robot deformation or whether controllers can realize the commanded path.

Abstract

from arXiv · show

Humans anticipate, from a glance and a contemplated action of their bodies, how the 3D world will respond, a capability that is equally vital for robotic manipulation. We introduce PointWorld, a large pre-trained 3D world model that unifies state and action in a shared 3D space as 3D point flows: given one or few RGB-D images and a sequence of low-level robot action commands, PointWorld forecasts per-pixel displacements in 3D that respond to the given actions. By representing actions as 3D point flows instead of embodiment-specific action spaces (e.g., joint positions), this formulation directly conditions on physical geometries of robots while seamlessly integrating learning across embodiments. To train our 3D world model, we curate a large-scale dataset spanning real and simulated robotic manipulation in open-world environments, enabled by recent advances in 3D vision and simulated environments, totaling about 2M trajectories and 500 hours across a single-arm Franka and a bimanual humanoid. Through rigorous, large-scale empirical studies of backbones, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling, we distill design principles for large-scale 3D world modeling. With a real-time (0.1s) inference speed, PointWorld can be efficiently integrated in the model-predictive control (MPC) framework for manipulation. We demonstrate that a single pre-trained checkpoint enables a real-world Franka robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or post-training and all from a single image captured in-the-wild. Project website at https://point-world.github.io/.

1. Introduction

PointWorld frames action-conditioned world modeling as predicting 3D scene evolution from perceptual inputs and robot actions in a shared physical representation. The paper contributes a large-scale dataset, a pre-trained model, and real-robot demonstrations using a single checkpoint.

  • World modeling aims to predict how unstructured environments evolve from what a robot observes and intends to do with its body.
  • PointWorld represents state and action as 3D point flows, predicting per-point scene displacements from partially observed scene points and temporal robot points.
  • The dataset spans real and simulated single-arm, bimanual, and whole-body interactions, with 3D annotations built using metric depth, camera pose estimation, and point tracking.
  • A single pre-trained checkpoint enables real-world pushing, deformable and articulated manipulation, and tool use from one in-the-wild RGB-D capture without demonstrations or post-training.

2. Related Work

Related work spans world models, robotic dynamics, and flow-based manipulation representations. PointWorld positions 3D point flows as a geometry-centered, embodiment-agnostic interface for action-conditioned prediction from sparse RGB-D observations.

  • World Modeling: World models use representations including pixels, meshes, explicit surfaces, radiance fields, Gaussians, particles, and hybrid structures, with actions ranging from low-level joint-space commands.
  • PointWorld: PointWorld converts robot actions and scene observations into a shared point-cloud interaction geometry before predicting full-scene 3D point flows.
  • World Modeling: PointWorld emphasizes contact and geometry rather than appearance, conditions on robot-specific 3D action geometry, reasons beyond visible regions, and operates in a single real-time forward pass.
  • Dynamics Models in Robotics: Robotic dynamics models support planning, policy learning, model-based reinforcement learning, exploration, safety filtering, design, verification, and policy evaluation.
  • 2D and 3D Flows for Manipulation: Flow representations provide correspondences across space and time for policy learning, reward modeling, goal specification, and visual servoing.

3. Method

PointWorld formulates manipulation dynamics as multi-step full-scene 3D point-flow prediction conditioned on partially observed RGB-D and robot geometry. Its method combines embodiment-agnostic robot flows, point-cloud processing, movement-weighted uncertainty-regularized training, and MPC-based action inference.

  • Dynamics Prediction: PointWorld predicts future scene states over a horizon from a static scene point cloud and a temporal sequence of robot point-flow actions.The model uses chunked prediction over H = 10 steps with 0.1s per step.
  • State Representation: Scene state is represented by 3D points with positions and time-constant features, emphasizing physical interactions, partial observability, and displacement-based training.
  • Action Representation: Robot actions are represented as fully observable 3D point flows generated from joint configurations and robot geometry through forward kinematics and a URDF.This representation supports heterogeneous embodiments and contact in occluded regions.
  • Dynamics Prediction: The model concatenates scene points with time-stacked robot points, using frozen DINOv3 scene features, temporal robot embeddings, and a point-cloud backbone.
  • Training Objective: Training reweights moving points and predicts per-point uncertainty with a Huber residual loss to address sparse motion supervision and noisy real-world annotations.Invisible points from the 2D tracker are excluded from supervision.
  • Action Inference: PointWorld supports sampling-based MPC by rolling out candidate action trajectories, accumulating task and control costs, and iteratively refining the nominal trajectory.

4. Dataset Curation and Evaluation Protocol

PointWorld is trained and evaluated on large-scale real and simulated manipulation data with 3D point-flow supervision, using reconstructed real-world annotations and moving-point metrics.

  • Dataset construction: The dataset combines diverse real-world and simulated manipulation, including single-arm, bimanual, whole-body, and mobile interactions.
  • Real-world annotation: Over 60% of DROID, nearly 200 hours of raw teleoperation, receive reliable tracked 3D point-flow annotations through the reconstruction pipeline.The pipeline replaces sensor depth, refines camera extrinsics, and lifts 2D tracks into 3D while excluding occluded points from supervision.
  • Real-world annotation: The annotation pipeline achieves 1.8 cm median translation error and 1.9 degrees median rotation error for optimized extrinsics relative to a proxy reference.
  • Simulation data: Simulation data supplies ground-truth 3D point flows from known state, filtered to short-horizon trajectories with active contacts and nonzero object motion.
  • Evaluation protocol: Evaluation uses per-point, per-timestep ℓ2 distance on moving scene points, emphasizing dynamic fidelity because most scene points remain static.The metric covers one-second prediction horizons and can reveal rollout differences that task-level success rates may miss.

5. Experiments

Experiments identify scalable architectures, robust objectives, spatial action representations, chunked prediction, and varied-observability training as effective design choices. The resulting model generalizes across domains and supports zero-shot real-world manipulation through MPC.

  • Training objectives: Movement weighting, uncertainty regularization, and Huber loss stabilize noisy real-world training and improve accuracy over an unweighted ℓ2 baseline.
  • Scaling roadmap: 50M-to-1B parameter scaling yields smooth, log-linear gains in 3D world-model prediction accuracy.
  • Scaling roadmap: Modernizing the backbone, stabilizing objectives, adding pretrained features, and scaling model size produce consistent accuracy gains over the GBND baseline.PTv3 avoids GBND’s rapidly growing memory use and limitations from purely local message passing under partial observability.
  • Action representations: Gripper-only point-flow actions outperform low-dimensional and whole-body alternatives, supporting precise contact reasoning and positive transfer across heterogeneous embodiments.Whole-body flows either provide insufficient contact resolution or add inactive points, compute overhead, and sparse noisy learning signals.
  • Ablations: Chunked 10-step prediction reduces rollout drift while improving compute efficiency, and randomized camera counts provide the most robust performance across test observability settings.
  • Generalization and transfer: POINTWORLD generalizes within domains, transfers across real and simulated data after efficient finetuning, and benefits from mixed real-simulation pretraining.Held-out real-world generalization persists under changes in background, lighting, objects, and possibly motion distribution.
  • Real-world action inference: A single pre-trained checkpoint enables zero-shot MPC manipulation of rigid, deformable, articulated, and tool-use tasks in real-world environments.

6. Conclusion

PointWorld is a large pre-trained 3D world model that predicts environment dynamics from RGB-D inputs and robot actions using shared 3D point flows. A single pre-trained model supports practical real-world manipulation behaviors across several interaction types.

  • PointWorld predicts 3D environment dynamics from in-the-wild RGB-D captures and robot actions represented as shared 3D point flows.
  • The model is trained on a large-scale action-conditioned dataset with high-quality depth maps, camera poses, and 3D tracks.
  • The authors study backbone designs, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling laws.
  • A single pre-trained PointWorld model enables non-prehensile pushing, deformable and articulated object manipulation, and tool use in the real world.

A.1. Extended Discussions on Limitations

The extended discussion identifies limitations concerning initialization, action inference, fine-scale supervision, causal interpretation, appearance, robot embodiment, actuation, and physics priors. These boundaries define where PointWorld’s current formulation does not fully model the environment or robot.

  • PointWorld assumes a static world at the observation instant because it receives no prior frames or velocities.Fully dynamic initial conditions would require externally tracked trajectories or recurrent state.
  • Action inference currently requires manually specified reward or cost functions in the 3D point-flow state-action space.Automatic reward specification and learning a policy through model-based reinforcement learning are left for future work.
  • Thin or very small objects are difficult to annotate accurately because depth and extrinsic errors can approach the object thickness.Mis-registered ground-truth flows can cause confusion between gripper and nearby scene motions.
  • PointWorld captures correlations in the observed action-conditioned distribution rather than disentangling causal effects from exogenous factors.The experiments therefore evaluate predictive fidelity and planning under the observed distribution.
  • The model predicts 3D point displacements but not photometric changes such as lights or screens turning on and off.Appearance models may be needed when such dynamics are critical.
  • The rigid-link robot representation ignores deformation of soft, tendon-driven, or compliant structures.The current model reasons about scene motion from forecasted robot geometry, not deformation of the robot body itself.
  • Known, fully realized robot trajectories leave actuation limits, controller behavior, and contact-induced tracking errors outside the model.The formulation describes what the environment does if the robot follows a path, rather than whether the robot can realize it.
  • The data-driven formulation does not incorporate explicit physics priors such as Newtonian mechanics or conservation-law constraints.Physics-informed regularization or hybrid simulators could improve generalization and extrapolation.

A.2. DROID 3D Annotation Pipeline

The DROID annotation pipeline uses synchronized multi-camera robot data and 3D vision methods to recover scene flows. It processes episodes regardless of task success because world modeling depends on observed interactions rather than manipulation outcomes.

  • DROID provides synchronized stereo RGB streams, camera intrinsics, robot joint states, and a known robot kinematic model.
  • The pipeline uses two externally mounted cameras to recover 3D scene flows from human-teleoperated manipulation episodes.
  • Episodes are retained regardless of task success or failure because the modeling target is observed interaction rather than task outcome.
  • Processing aligns timestamps, downsamples by 2, estimates metric depth, refines camera extrinsics, tracks points in 2D, and reconstructs 3D trajectories.

A.2.1. Depth Estimation

Depth estimation uses FoundationStereo and sanitizes its outputs with a trusted range and validity masking. This addresses inaccurate estimates in distant, especially texture-less, regions.

  • FoundationStereo provides per-view metric depth without the minimum valid-depth threshold typical of sensor depth.
  • Depth estimates are clamped to the trusted range [0, 4] m and paired with a per-pixel validity mask.
  • Distant, especially texture-less regions such as walls can produce inaccurate estimated depth.

A.2.2. Camera Pose Estimation

Camera extrinsics are initialized from multi-view pose estimation, robot kinematics, and a rigid wrist-camera assumption, then jointly refined by robot-depth reprojection. The procedure aligns external cameras to the robot base frame using trusted robot pixels across time.

  • Camera Pose Estimation: The pipeline does not use dataset-provided extrinsics; instead, it combines VGGT initialization with robot-geometry alignment to estimate camera poses.This two-stage procedure leverages metric depth and recorded robot joint states.
  • Camera Pose Initialization: VGGT initializes external-camera poses from synchronized external and wrist-camera images, while forward kinematics supplies gripper poses in the robot base frame.The wrist camera is assumed rigidly mounted to the end effector, enabling a known gripper-to-wrist transform.
  • Camera Pose Initialization: Averaging valid wrist-frame estimates yields a reference-camera pose in the base frame, which is converted into common base-frame extrinsics for other external cameras.Relative transforms from the multi-view estimator connect each external camera to the reference camera.
  • Camera Pose Refinement: All external-camera poses are jointly refined with small 6-DoF updates by minimizing robot-depth reprojection error over valid robot pixels.Predicted depths come from projecting robot surface points through the current extrinsics and are compared with observed depths.
  • Camera Pose Refinement: The refinement uses observed depths from 0.3–2.0 m, requires at least 2,000 valid robot points per camera–frame pair, and discards failing frames.Optimization runs for 100 iterations with learning rate 10^-3, using a first-order optimizer.

A.2.3. Benchmark Metrics for 3D Annotation

The annotation pipeline evaluates camera alignment with depth reprojection and two-view point-cloud F1, then constructs short, interaction-focused 3D-flow clips from real and simulated data. Filtering, tracking, and postprocessing target reliable dynamic annotations, while mixed real–simulation training produces the most balanced qualitative behavior.

  • Benchmark Metrics: Depth reprojection loss measures L1 disagreement between rendered robot depth and observed depth over valid pixels, averaged across frames and reported per scene.Depth values outside 0.3–2.0 m and out-of-bounds robot pixels are discarded.
  • Benchmark Metrics: Two-view F1 at 5/20 mm evaluates symmetric nearest-neighbor precision and recall after back-projecting depth, removing robot pixels, and cropping to the workspace.Points count as true positives when their nearest counterpart lies within the selected threshold.
  • Clip Filtering: Real-data episodes are sliced into overlapping one-second, 16-frame clips retained when gripper state changes or end-effector motion exceeds open- or closed-gripper thresholds.The thresholds are 0.005 m and 0.10 rad when open, versus 0.002 m and 0.05 rad when closed.
  • 3D Point-Flow Construction: Dense tracking operates on selected short clips, followed by DBSCAN outlier removal and per-frame normal estimation to improve point-flow quality.Clip-wise tracking avoids mixing trajectories across widely separated intervals.
  • Dataset Sources: BEHAVIOR-1K contributes approximately 10,000 photorealistic simulated episodes across 50 household tasks using a bimanual mobile robot and three virtual cameras.Episodes average approximately 6.6 minutes before replay and short-clip extraction.
  • Clip Filtering: Simulation clips are filtered for visible workspace objects, robot motion, contact indicators, and discontinuities before accepting interaction-relevant windows.The acceptance logic includes object motion associated with non-base motion, gripper-finger contact, and negative clips where the robot moves without object motion.
  • 3D Point-Flow Construction: Accepted simulation clips derive compact 3D point flows from rendered depth, link-level segmentation, and exact rigid-link trajectories without exposing simulator state to the model.The representation exploits rigid-object structure and known link trajectories.
  • Training-Mixture Effects: Models trained on both real and simulated data show the most balanced deployment behavior, combining realistic contact handling with generalization to novel real-world scenes.Real-only models tend to remain static at contact, whereas simulation-only models may move cluttered background points with the target.
Loading 2601.03782v1…