PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluation

Sichao Liu, Zekun Wang, Lixuan Tang, Yiming Li, Xiaohan Wang, Hanzhi Zhang, Daqiang Guo, Peng Zhou, Lihui Wang

arXiv:2609.33872v1cs.ROcs.AIcs.LG

TL;DR

Robot-GST addresses limited explicit outcome evaluation for long-horizon manipulation by reconstructing a geometry-aware environment and evaluating candidate behaviours before execution. Across reported tasks, simulation-based replanning is associated with higher success rates in simulation and real-world deployment.

  • Problem

    Many manipulation policies lack explicit mechanisms to predict task outcomes and assess whether generated actions achieve desired final states during long-horizon execution.

    Source: §V · p. 7

  • Method

    Robot-GST combines metrically aligned 3D Gaussian scenes and SAM3D object geometry with keypoint-affordance planning, candidate-behaviour simulation, and Gaussian-aware final-state estimation.

    Source: A. 3D Gaussian Splatting for robotic… · p. 3, C. Geometry-aware spatio-temporal… · p. 4, §V · p. 7

  • Results

    With simulation-based replanning, average success rates across tasks rise from 66.7% to 93.3% in simulation and from 55.3% to 80% in real-world deployment.

    Source: §IV · p. 7

  • Takeaways & Limitations

    The results support using geometry-aware reconstruction and simulation to evaluate manipulation behaviours across rigid, soft, and deformable object tasks.

    Source: §IV · p. 6, §V · p. 7

Abstract

from arXiv · show

Robotic manipulation policies are advancing rapidly with increasing reliance on vision-language models for end-to-end decision making. However, reliable deployment remains challenging because many policies lack explicit mechanisms for predicting task outcomes and evaluating whether generated actions will achieve desired final states, causing execution errors to accumulate during long-horizon manipulation. We present Robot-GST, a geometry-aware spatio-temporal behaviour representation and evaluation framework that constructs a Gaussian-SAM robotic environment for real-to-sim policy verification and improves the reliability of real-world manipulation deployment. Our approach constructs a high-fidelity robotic environment from RGB-D observations using 3D Gaussian Splatting and SAM3D, enabling ``simulation and evaluation before acting''. It integrates visual observations and language instructions with spatio-temporal reasoning for long-horizon task planning using large vision-language models. To bridge high-level planning and real-world execution, we introduce Gaussian-aware final-state estimation through geometric sampling and state-based trajectory planning. Before execution, candidate action sequences are simulated and evaluated in the Gaussian-SAM environment to filter infeasible behaviours. We validate our approach on representative manipulation tasks involving rigid, soft, and deformable objects, including cube placing, toy packing, and duck rearrangement, demonstrating that geometry-aware spatio-temporal reasoning and state-aware execution improve manipulation reliability across different object categories. Our results suggest that combining geometry-aware reconstruction with high-quality rendering and simulation provides a scalable approach for evaluating robotic manipulation behaviours. Website: https://robot-gst.github.io

II. Related Work

Related work uses language and vision-language models to decompose instructions, plan with geometric constraints, and execute diverse tasks, but often lacks dense object-state representations or explicit final-state modeling.

A. Language task decomposition and relational reasoning

  • LLMs decompose instructions into executable subgoals and task dependencies, while VLA policies use pretraining to perform diverse real-world tasks from language and visual inputs [18], [19],, [21].Source: A. Language task decomposition and… · p. 2
  • ReKep, MOKA, and SoFar respectively use relational keypoint constraints, image-space marks, and semantic orientations, but generally do not model object final states explicitly [8], [9],.This limits their application to tasks requiring collision-aware planning or contact-rich manipulation.Source: A. Language task decomposition and… · p. 2

B. Robotic scene representation and reconstruction

Scene-representation work contrasts explicit, efficient 3D Gaussian Splatting with implicit radiance fields, while noting that current Gaussian representations offer limited temporal and relational reasoning for interaction planning.

B. Robotic scene representation and reconstruction

C. Policy simulation and evaluation

Real-to-sim pipelines use Gaussian reconstructions and dynamics to evaluate candidate behaviours, but existing approaches often treat reconstruction, reasoning, and simulation as separate components.

C. Policy simulation and evaluation

  • Recent 3DGS pipelines combine real-world scene reconstruction with approximate dynamics to estimate sim-to-real correlation and select or refine candidate behaviours before deployment,.Source: C. Policy simulation and evaluation · p. 2
  • Benchmark and environment-generation efforts systematise real-to-sim evaluation, typically using 3DGS for rendering and auxiliary geometric proxies or physics engines for contact-rich simulation,.Source: C. Policy simulation and evaluation · p. 2
  • Existing frameworks often struggle with complex real-world dynamics because they rely on predefined physical parameters or approximate interaction models [7].Source: C. Policy simulation and evaluation · p. 2
  • Prior approaches typically focus on reconstruction, task reasoning, or simulation individually rather than integrating geometry-aware representation, long-horizon reasoning, and pre-execution evaluation.Source: C. Policy simulation and evaluation · p. 3

III. Methods

Robot-GST combines Gaussian scene reconstruction and SAM3D object understanding to build a geometry-aware environment for manipulation planning and evaluation. It pairs photorealistic rendering with explicit geometry for collision reasoning.

III. Methods

  • Robot-GST addresses high-fidelity geometric and semantic scene representation and evaluation of rigid and deformable manipulation behaviours in a reconstructed Gaussian Splatting environment.Source: §III · p. 3

A. 3D Gaussian Splatting for robotic scene reconstruction

III. Methods

  • SAM3D uses image masks and calibrated depth to reconstruct metrically scaled objects, refining a coarse voxel shape into shape and texture before decoding meshes and Gaussians.The mesh supports collision reasoning, while Gaussians support efficient photorealistic rendering and scene composition.Source: A. 3D Gaussian Splatting for robotic… · p. 3

B. Keypoint affordance constraints

The section turns language tasks into staged keypoint and geometric constraints, then optimizes manipulation stages and checks candidate release poses for feasibility.

B. Keypoint affordance constraints

  • Language tasks are decomposed into stages with relational keypoints and geometric constraints, which a vision-language model turns into executable constraint representations.Constraints encode subgoals such as distance, alignment, orientation, or visibility, alongside path-feasibility requirements.Source: B. Keypoint affordance constraints · pp. 3–4
  • Each stage optimizes end-effector pose and grasp parameters by penalizing constraint violations while regularizing the solution.The objective weights inequality-constraint penalties and may regularize for smoothness or deviation from a nominal pose or grasp.Source: B. Keypoint affordance constraints · p. 4
  • The target object is associated by nearest-neighbour matching, and its relative transform to the end-effector is assumed fixed after grasping.This assumption approximates stable object transport.Source: C. Geometry-aware spatio-temporal… · p. 4
  • Bounded position and rotation perturbations generate candidate release poses, which are collision-checked against scene geometry, ranked, and passed to a motion planner.Feasibility uses a signed distance field and a collision margin; feasible candidates are ranked with a stability-oriented heuristic.Source: C. Geometry-aware spatio-temporal… · p. 4

D. Robot behaviour simulation and evaluation

The section verifies candidate manipulation trajectories in a metrically aligned real-to-sim environment with rendering and geometric collision representations. Simulated trajectories are accepted for real execution only when they are executable, collision-safe, and satisfy the terminal task goal.

D. Robot behaviour simulation and evaluation

  • The real-to-sim environment preserves the real scene's scale and robot base frame, pairing a Gaussian scene for rendering with a geometric proxy for collision and contact reasoning.The reconstruction combines multi-view RGB-D observations with SAM3D segmentation, DINOv3 correspondence, CLIP language alignment, and depth geometry.Source: D. Robot behaviour simulation and… · p. 4, Figure 2 · p. 5
  • Candidate trajectories specify end-effector position and orientation plus gripper commands, then use inverse kinematics and simulator transitions to roll out state sequences.The transition model combines robot kinematics with physics and contact simulation.Source: D. Robot behaviour simulation and… · p. 4
  • Rigid objects use rigid-body dynamics and reconstructed collision meshes, while deformable objects use a soft-body model to approximate deformation and contact response.Source: D. Robot behaviour simulation and… · pp. 4–5
  • A trajectory is feasible only if executable, collision-safe, and successful under a task-specific terminal predicate; only feasible trajectories are executed on the real robot.Collision safety is checked against reconstructed scene geometry with a safety margin.Source: D. Robot behaviour simulation and… · p. 5

IV. Experiments

The experiments evaluate reconstruction quality and manipulation scenarios involving rigid, soft, and deformable objects; the reported rendering metrics show strong performance against baselines.

IV. Experiments

  • The evaluation covers cube placing, soft-toy packing, and deformable duck rearrangement using a UR5 arm with wrist and side RealSense cameras.The setup uses D435 and D405 cameras and an NVIDIA GeForce RTX 4090 workstation.Source: §IV · p. 5
  • RGB-D views, SAM3D masks, DINOv3 correspondence, CLIP language features, and depth constraints build an object-centric scene reconstructed with 3D Gaussian Splatting.The static scene, robot, and objects are aligned in a shared world frame.Source: §IV · p. 5
  • ICP aligns the reconstructed scene with the robot model so real robot poses and policy commands can update the simulated robot for task evaluation.Forward kinematics updates joint and Cartesian states in the Gaussian scene.Source: §IV · p. 5
  • The reconstruction benchmark reports PSNR, SSIM, and LPIPS against GScream, VR-GS, Decoupled GS, POGS, D3DGS, and Embodied GS.Table I summarizes average rendering performance on robotic-scene reconstruction.Source: §IV · p. 5, Table I · p. 5
  • SSIM of 0.963 and LPIPS of 0.044 outperform the baselines, while PSNR of 24.64 ranks second.The qualitative comparison reports fewer object and scene artefacts than POGS and Embodied GS.Source: §IV · p. 6, Figure 3 · p. 6

Geometry-aware spatio-temporal reasoning and plan-

Robot-GST plans manipulation in 3D using language-conditioned keypoint affordances, then evaluates candidate trajectories in a reconstructed simulator before real-world execution. Simulation-based replanning improves task success, though contact differences can still separate simulated and real outcomes.

Planning

  • Language-conditioned keypoint candidates and three text-described subtasks use affordances, depth, and pose relations to generate manipulation trajectories.Feasible final object states are estimated by sampling end-effector poses while holding the object.Source: §IV · p. 6

Policy evaluation

  • The simulator evaluates grasp-to-placement trajectories, modeling soft-object deformation when the sloth contacts the basket; successful simulated policies can proceed to the robot.The sloth’s main body is placed at a target point while other body parts can contact the basket edge and deform within it.Source: §IV · p. 6

Simulation and real-world evaluation

  • Simulation and real-world success rates are positively correlated, but simulated contact forces can differ from reality, particularly for deformable-object body-part contact.Cube placement uses point-cloud distances for collision checking; the real cube-placement rollout succeeds.Source: §IV · p. 6

Replanning

  • A failed real-world sloth-packing policy triggers a scene reset and replanning; the simulator evaluates the new plan before its successful robot rollout.The initial attempt leaves part of the sloth outside the basket.Source: §IV · p. 6, Figure 6 · p. 7

Evaluation

  • With simulation-based replanning, average task success rises from 66.7% to 93.3% in simulation and from 55.3% to 80% in real-world deployment.Replanning increases success by 20 percentage points for cube and duck manipulation and 40 percentage points for sloth packing.Source: §IV · p. 7, Table II · p. 7

Long-horizon evaluation

  • Duck rearrangement requires five placements with three control steps per duck, and subtask success degrades as the long-horizon task progresses.Replanning yields higher subtask-level consistency in simulation and real-world deployment.Source: §IV · p. 7

V. Conclusions

Robot-GST combines geometry-aware planning with simulation-based policy verification before execution. Experiments across rigid, soft, and deformable tasks show improved closed-loop reliability, while reconstruction and physics fidelity remain limitations.

V. Conclusions

  • Robot-GST verifies manipulation behaviours in simulation before execution, integrating geometry-aware spatio-temporal representation with policy evaluation.The framework reconstructs metrically aligned photorealistic scenes from short RGB-D captures and uses 3D keypoint affordances and relational constraints for language-specified tasks.Source: §V · p. 7
  • Experiments on rigid, soft, and deformable tasks demonstrate improved closed-loop reliability and a positive relationship between simulation and real-world performance.Gaussian-aware final-state estimation supplies target states for low-level trajectory planning, reducing error accumulation.Source: §V · p. 7
  • Reconstruction and physics fidelity are limited under occlusion, dynamic scene changes, and imperfect contact or deformation models.The conclusion identifies improved physical modelling and uncertainty-aware planning as future directions.Source: §V · p. 7

Related papers

Loading 2609.33872v1…