PaperlayerView exact revision on arXiv

Source-linked AI summary

Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics

Jad Abou-Chakra, Krishan Rana, Feras Dayoub, Niko Sünderhauf

arXiv:2406.10788v1cs.RO

TL;DR

The paper targets world representations that struggle to encode physical priors while predicting future states for robotic interaction. It couples physics-driven particles with attached 3D Gaussians, using rendered-image discrepancies to correct particle states visually. The resulting model supports forward simulation and online correction, while remaining synchronized with reality in tabletop experiments and outperforming baselines in all but simulated Pickup tasks.

  • Problem

    Existing representations such as pointclouds, images [1], and latent descriptors [3] [4] cannot explicitly encode physical priors needed to constrain predicted future states.

    Source: §1 · p. 1

  • Method

    The method couples PBD-simulated particles with attached 3D Gaussians, using photometric discrepancies to generate physics-resolved visual correction forces.

    Source: §1 · p. 1, §3 · p. 4, §3.2 · p. 5

  • Results

    The system supports realtime prediction and correction from 3 cameras and outperforms all baselines except simulated Pickup tasks, where physical-prior mismatch degrades performance.

    Source: §4 · pp. 6–7

  • Takeaways & Limitations

    The unified representation can model physical and visual world states together for future-state prediction, visual synchronization, and downstream robotic planning or policy learning.

    Source: §3 · p. 4, §6 · p. 9

    The method is demonstrated in a tabletop setting and assumes predicted states remain close enough for visual forces to correct them; rapidly moved objects can violate this assumption.

    Source: §3.1 · p. 5, §6 · p. 9

Abstract

from arXiv · show

For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geometry, physics, and visual observations - that informs perception, planning, and control algorithms. We propose a novel dual Gaussian-Particle representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates visual forces that correct the particle positions while respecting known physical constraints. By integrating predictive physical modelling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. Our system runs in realtime at 30Hz using only 3 cameras. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/.

1 INTRODUCTION

The paper addresses the difficulty of predicting physically plausible future states with representations that cannot explicitly encode physical priors. It introduces a dual Gaussian-Particle representation that combines physics-based prediction with visual correction from observations.

  • Particle-based physics simulators provide forward dynamics, but the proposed method initializes particles from RGBD observations and corrects accumulated errors using RGB observations.Source: §1 · p. 1
  • Attached 3D Gaussians render predicted views, allowing photometric discrepancies with camera images to generate visual forces that move the bonded particles.Source: §1 · p. 1, Figure 1 · p. 2
  • A dual Gaussian-Particle representation unifies geometry, physics, and visual appearance for modeling the physical world.Particles capture the physical aspect, while attached 3D Gaussians represent visual appearance through rendering.Source: §1 · pp. 1–2
  • The system contributes RGBD-and-instance-map initialization and realtime visual-feedback correction, with an overview shown in Fig. 1.Source: §1 · p. 2, Figure 1 · p. 2

2 PRELIMINARIES

The paper builds on Position-Based Dynamics for physically constrained particle simulation and Gaussian Splatting for differentiable visual rendering. Together, these preliminaries provide the physical constraints, shape modeling, viewpoint rendering, and segmentation-aware initialization used by the proposed representation.

2.1 Particle-Based Physics Simulation

  • PBD represents oriented particles with positions, velocities, orientations, forces, radii, masses, and optional resting positions within shapes.The formulation is designed for realtime, robust robotics simulation.Source: §2.1 · p. 2
  • PBD minimizes ground, collision, and shape constraints with a Jacobi solver that aggregates positional corrections to satisfy physical properties.Ground constraints prevent penetration, collision constraints model particle contacts, and shape matching preserves object structure.Source: §2.1 · pp. 2–3
  • Shape matching maintains the structure of rigid and deformable objects, using one shape for rigid objects and multiple particle-neighborhood shapes for deformable objects.The shape stiffness parameter controls the constraint.Source: §2.1 · p. 3

2.2 Gaussian Splatting

  • Gaussian Splatting represents visual state with 3D Gaussians parameterized by position, orientation, scale, opacity, and color, then renders pixels by depth-sorted splatting from a viewpoint.The projected covariance is obtained from the viewpoint transform, projection Jacobian, and Gaussian covariance.Source: §2.2 · p. 3
  • Differentiable splatting supports photometric optimization, while segmentation ids and a segmentation loss are used specifically during initialization.Figure 2 presents the initialization procedure and real-world reconstruction results.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 METHOD

The method couples particles governed by physics with bonded 3D Gaussians for realtime prediction and visual correction. Initialization builds the representation from RGBD data, while online prediction and correction synchronize it with observations.

3 METHOD

  • Gaussian-Particle bonds couple PBD particles representing physical state with splatted Gaussians representing visual state for realtime simulation and correction.Each bond is a rigid transform linking a Gaussian to its parent particle; the system uses three cameras for visual synchronization.Source: §3 · p. 4

3.1 Initialization

  • Initialization fills depth- and mask-derived object boxes with Gaussians, optimizes their appearance and positions, then creates particles and shape-matching connections.The procedure supports rigid or deformable objects and produces collision-free, ground-aligned approximations of observed geometry.Source: §3.1 · pp. 4–5

3.2 Online Prediction and Correction

  • Figure 3 illustrates the sequential prediction-and-correction steps used to update the representation in realtime.The method first predicts with PBD and then applies visual correction through Gaussian splatting optimization.Source: §3.2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6
  • Prediction runs PBD at 30Hz, while correction compares camera renders with RGB observations and converts Gaussian position updates into forces on particles.The prediction step takes approximately 5 ms on an NVIDIA 3090; Gaussians are reset after five optimization steps, and particle motion remains physics-resolved.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments evaluate tracking and unseen-view reconstruction on simulated and real tabletop scenes using separate tracking, physics-only, and visually optimized baselines. The full method outperforms the baselines in all but simulated pickup scenarios, where mismatched physical priors reduce performance.

Datasets

  • Evaluation uses 25 simulated and 25 real tabletop scenarios spanning pushing, pickup, pushover, and deformable-object interactions, with three cameras supplying visual forces.Two additional cameras are used only for evaluation, and real scenes use Aruco markers to obtain groundtruth trajectories.Source: §4 · p. 6

Baselines

  • The comparisons cover 2D and 3D tracking against Cotracker and D3DGS, plus forward prediction against a physics simulator without visual forces.No single realtime, correctable world-model baseline was available, so the method is compared with systems that separately track or predict.Source: §4 · p. 6

Metrics

  • Tracking error is measured on known point trajectories, while foreground photometric reconstruction is evaluated from unseen viewpoints.Predicted 3D trajectories are projected into camera views for 2D evaluation, and initial points are queried with Cotracker.Source: §4 · p. 7

Results

  • The method outperforms all baselines in 45 of 50 experiments, with the exception of simulated Pickup tasks whose physical priors differ substantially from the scene dynamics.The system nevertheless recovers and acquires the final pickup state.Source: §4 · p. 7, Table 2 · p. 7

5 RELATED WORK

The paper positions its contribution as the first particle-and-3D-Gaussian representation for a correctable robotics world model that fuses physical and visual priors. It distinguishes this approach from slower or less predictive combinations based on NeRFs, point clouds, SDFs, or auxiliary Gaussian losses.

Contribution

  • The authors identify their representation as the first to combine particles and 3D Gaussians for fusing physical and visual priors in a correctable robotics world model.Related systems combine particle physics with other visual representations, but not this Gaussian-Particle pairing for the stated purpose.Source: §5 · p. 7

Particle-based visual models

  • ParticleNeRF combines physics-driven particles with a renderable NeRF but is too slow and camera-intensive for realtime real-world robotics, whereas Gaussian splatting enables correction from three cameras.ParticleNeRF used upwards of 10 cameras and primarily employed collision constraints.Source: §5 · p. 8

Pointcloud and SDF methods

  • Surgical-robotics methods pair PBD particles with pointcloud corrections, while this method uses faster Gaussian-splatting visual feedback for a more general realtime setting.The comparison distinguishes the visual feedback modality and claimed operating scope.Source: §5 · p. 8

Gaussian-based methods

  • D3DGS regularizes Gaussian motion with auxiliary losses but cannot predict future states as a world model, unlike the physics-based representation proposed here.D3DGS can track and reconstruct well with groundtruth masks or many cameras and ample optimization time, conditions that are difficult in robotics.Source: §5 · pp. 8–9

6 LIMITATIONS AND CONCLUSIONS

The paper presents a hybrid particle–Gaussian world model that predicts future states and corrects them from observations. Its demonstrated scope is tabletop scenarios, with limitations when modelling errors accumulate faster than visual correction or when object structure must be updated online.

Limitations

  • The method is demonstrated in tabletop scenarios where physical priors can be heavily exploited, while extension to open-world settings remains future work.Source: §6 · p. 9
  • Visual correction assumes predicted states remain close enough to ground truth; rapidly moved objects can cause modelling errors to accumulate faster than correction.The authors identify global visual information as a plausible way to close this loop.Source: §6 · p. 9
  • The current method does not update objects’ physical structure or parameters after modelling, so new observations cannot correct modelling mistakes.Online structure correction and system identification are identified as extensions.Source: §6 · p. 9

Conclusion

  • The hybrid representation combines particles and 3D Gaussians to predict future states and correct predictions from observed data for robotic world modelling.The authors suggest using the resulting world model for object-state extraction, reinforcement- or imitation-based policies, and model predictive control.Source: §6 · p. 9

Supplementary Material

The supplementary material documents a five-camera tabletop setup with a Franka robot and teleoperated object manipulation. All five cameras initialize the system, while three D455 cameras are used during prediction and correction.

A Experimental Setup

  • The real-world setup uses a Franka Emika robot with a gripper for pickup scenarios and a pusher for other scenarios.Source: A Experimental Setup · p. 12
  • Five cameras observe the tabletop, comprising three Intel RealSense D455 and two D435 cameras jointly calibrated with hand-eye calibration.Source: A Experimental Setup · p. 12
  • All five cameras are used for initialization, but only the three D455 cameras operate during prediction and correction.Source: A Experimental Setup · p. 12
  • The robot is teleoperated to manipulate tabletop objects, while camera streams are encoded as HEVC videos and evaluated at 640x360 resolution.Source: A Experimental Setup · p. 12

B Implementation

Implementation alternates between initialization and prediction/correction: particles and Gaussians are created for each object, then PBD predicts particle states and Gaussian-derived forces correct them. The system uses calibrated scene, robot, and object initialization with realtime-oriented parameters.

Implementation

  • Initialization creates particles and Gaussians for each object, followed by PBD simulation and Gaussian-based corrective forces during prediction and correction.Source: B Implementation · p. 12

Static Scene Initialization

  • The tabletop is represented with thin-disk Gaussians from five RGBD cameras, while a pointcloud estimates the ground plane.Source: B Implementation · p. 12

Robot Initialization

  • Robot particles are manually fitted to links, linked for forward kinematics, and paired with Gaussians trained from multi-view Blender renders.Source: B Implementation · p. 13

Object Initialization

  • Object initialization fills depth- and instance-mask-derived bounding boxes with Gaussians, creates particles, assigns masses and constraints, and bonds each Gaussian to its closest particle.Object modelling takes approximately 20 to 40 seconds and is performed once per scenario.Source: B Implementation · p. 13

Prediction Step

  • PBD predicts particle and Gaussian locations at 30 Hz using 20 substeps and Jacobi constraint solving.Source: B Implementation · p. 13

Correction Step

  • Visual-force correction optimizes Gaussian parameters from rendered-versus-observed RGB images, with position changes converted into forces on parent particles.Five Adam iterations are used, and displacements below 2 mm are ignored for stability.Source: B Implementation · pp. 13–14, Figure 2 · p. 14

C Ablations

Ablations show that physical priors improve simulated tracking, while visual-force settings trade tracking accuracy against stability and runtime. The chosen 640x360 resolution balances modest resolution effects with computational cost.

Physical Priors

  • Using all physical priors produces the lowest average tracking error and better matches object dynamics than configurations omitting collision, ground, or gravity constraints.Removing collision constraints harms performance during intense impacts, while removing gravity produces oscillatory motion.Source: C Ablations · p. 14

Cameras

  • Increasing camera count yields diminishing returns, while higher image resolution lowers tracking error only slightly between 1280x720, 640x360, and 320x180.The system selects 640x360 because 1280x720 visual-force computation takes approximately 40 ms versus 20 ms at lower resolutions.Source: C Ablations · p. 15
  • Below 640x360, resolution no longer limits performance and provides no tracking gain.Source: C Ablations · p. 15

Visual Forces

  • High visual gain or Gaussian position learning rate causes oscillations, whereas low values harm tracking.The oscillations resemble an undamped spring, motivating careful tuning.Source: C Ablations · pp. 15–16
  • Reducing Adam iterations is a realtime trade-off that can preserve synchronization when the physics timestep takes longer than expected.Source: C Ablations · p. 16

D Failure Modes

The system’s corrections fail when its visual rendering or physical model diverges substantially from reality, while visual and geometric symmetries can leave persistent tracking errors.

Visual Correction Failure

  • Large rendering discrepancies prevent visual forces from producing meaningful corrections.The correction mechanism depends on the rendered scene remaining sufficiently similar to the groundtruth image.Source: D Failure Modes · p. 16

Physical Prediction Failure

  • Physical-model mismatch can make the enforced physical priors harm tracking, as in the Pushover 5 T-Block failure.The T-Block could not be pushed over and escaped the visual forces’ radius of convergence.Source: D Failure Modes · p. 16

Visual Symmetries

  • In the Rope 1 scenario, rotational symmetry in texture and geometry permits a steady-state tracking error.The rope can rotate around its spine without changing its observed geometry or texture.Source: D Failure Modes · p. 16, Figure 5 · p. 16

E Experimental Results

The supplementary evaluation reports 3D tracking performance for the proposed system and baselines across all simulated and real scenarios.

3D Tracking

  • 3D tracking performance is reported for the proposed system across all scenarios.The evaluation covers the complete set of simulated and real scenarios shown in the supplementary material.Source: E Experimental Results · p. 16, Figure 6 · p. 17
  • Supplementary Figure 6 is the visual summary of the reported 3D tracking evaluation.The figure covers the scenario-wide tracking comparison rather than a single test case.Source: E Experimental Results · p. 16, Figure 6 · p. 17
  • The supplementary comparison includes the system’s baselines across all simulated and real scenarios.Supplementary Figure 6 presents the 3D tracking performance of both the proposed system and its baselines.Source: Figure 6 · p. 17

Related papers

Loading 2406.10788v1…