PaperlayerView exact revision on arXiv

Source-linked AI summary

ChainSplat: A Physics-Inspired Screw-Theoretic Model for Learning Deformable Linear Object Dynamics from Multi-View RGB Videos

Seungyeon Kim, Noémie Jaquier

arXiv:2608.28570v1cs.RO

TL;DR

DLO dynamics and geometry are difficult to infer for robotic manipulation because configurations are high-dimensional, materials vary, and existing pipelines often depend on noisy depth and multiple stages. CHAINSPLAT addresses this with a screw-theoretic open-chain model integrated with Gaussian splatting and optimized from multi-view RGB videos. Real-world experiments report state-of-the-art modeling, real-time estimation, and effective model-based manipulation.

  • Problem

    DLO dynamics learning must handle high-dimensional configurations, varying material properties, and limitations of multi-stage depth-dependent approaches for accurate robotic manipulation.

  • Method

    CHAINSPLAT models DLOs as revolute-jointed rigid-link chains with screw-theoretic kinematics and link-wise Gaussians, then uses two-stage differentiable optimization on multi-view RGB videos.

  • Results

    Experiments report state-of-the-art DLO digital-twin modeling, accurate dynamics, faithful 3D reconstruction, high-quality RGB rendering, and effective real-time manipulation.

  • Takeaways & Limitations

    The compact differentiable model supports efficient trajectory optimization and real-time RGB-based state estimation for model-based DLO manipulation.

Abstract

from arXiv · show

Identifying the underlying dynamics and 3D geometry of deformable linear objects (DLOs), such as cables, ropes, and hoses, is essential for accurate robotic manipulation, but remains challenging due to their high-dimensional configuration spaces and diverse behaviors arising from varying material properties. Existing methods often rely on multi-stage pipelines and auxiliary depth inputs, which are prone to errors under dynamic interactions, while their high-dimensional state representations make model-based control computationally expensive. In this paper, we introduce ChainSplat, a physics-inspired framework that jointly learns the 3D geometry, appearance, kinematics, and dynamics of DLOs solely from multi-view RGB videos. ChainSplat represents a DLO as an open-chain structure of rigid links connected by revolute joints, yielding an analytic, screw-theoretic model with a compact state representation parameterized by joint configurations. By integrating this formulation with Gaussian splatting, ChainSplat jointly recovers DLO dynamics, kinematics-aware 3D geometry, and appearance, while enabling high-fidelity RGB rendering from arbitrary states. Through real-world experiments, we demonstrate that ChainSplat achieves state-of-the-art performance in dynamics predictions, 3D geometry reconstruction, and RGB rendering across dynamic interactions. ChainSplat further enables real-time state and force estimation, as well as accurate model-based trajectory optimization, highlighting its practical utility for real-world robotic manipulation of DLOs. Accompanying source code and video are available at: https://chainsplat.github.io.

I. INTRODUCTION

CHAINSPLAT addresses DLO dynamics learning from multi-view RGB videos with a compact, physics-inspired representation that jointly models geometry, appearance, kinematics, and dynamics. Its differentiable model supports efficient estimation, rendering, trajectory optimization, and real-world manipulation.

  • DLO manipulation requires physical dynamics models despite high-dimensional configurations and behavior variation from unknown material properties.
  • Existing physics-inspired methods often use multi-stage RGB-depth pipelines that accumulate errors during dynamic 3D trajectory estimation and decouple rendering from dynamics learning.Depth-based lifting is especially vulnerable during fast motions, while decoupled modeling can reduce RGB rendering fidelity.
  • CHAINSPLAT represents a DLO as rigid links connected by revolute joints, combining screw-theoretic kinematics with link-wise Gaussian splatting.The representation jointly captures 3D geometry, appearance, kinematics, and dynamics while using RGB observations alone.
  • A two-stage differentiable optimization first recovers kinematics-aware geometry, appearance, and joint trajectories, then identifies dynamic parameters from simulated-versus-estimated trajectories.The resulting state is parameterized by the open-chain joint configurations, producing a low-dimensional dynamics model.
  • CHAINSPLAT provides efficient trajectory optimization and real-time RGB-based state estimation through differentiability and high-fidelity rendering from arbitrary states.Rendered-observed image discrepancies are minimized iteratively for state correction and closed-loop control.
  • Experiments report state-of-the-art DLO digital-twin modeling and effective real-time manipulation, including target hitting under frictional interactions.Open-loop trajectory optimization successfully accounts for friction-induced shape changes in tabletop manipulation.

1) Special Euclidean Group:

This section introduces SE(3), screw theory, and product-of-exponentials kinematics for representing rigid-body and open-chain motions. These tools provide link poses and Gaussian transformations for CHAINSPLAT.

  • 1) Special Euclidean Group:: SE(3) represents rigid-body motions in three-dimensional space using a rotation matrix and translation vector.
  • 1) Special Euclidean Group:: A twist in se(3) combines angular and linear velocity components and parameterizes joint motion or link spatial velocity.
  • 1) Special Euclidean Group:: A wrench in se(3)* represents spatial force through torque and force components, with its pairing with a twist giving scalar power.
  • 2) Screw Theory:: For revolute joints, screw axes specify a unit rotation axis and its associated linear component, and CHAINSPLAT uses exclusively revolute joints.
  • 2) Screw Theory:: A joint displacement transforms a rigid-body configuration through a screw-axis matrix exponential, while repeated exponentials compose open-chain motion.
  • 3) Forward Kinematics of Open-Chain Systems:: The product-of-exponentials formula computes end-effector and intermediate-link poses from joint configurations and home-frame screw axes.These transformed link poses also determine the locations of CHAINSPLAT’s associated Gaussian primitives.

B. Dynamics of Open-chain Systems

Open-chain rigid-body dynamics are expressed with SE(3) velocities, wrenches, spatial inertia, and adjoint mappings. These operators support coordinate changes and velocity-dependent coupling terms.

  • 2) Single Rigid-Body Dynamics:: The open-chain formulation builds on single-rigid-body dynamics and uses link frames attached to joints, with base and end-effector frames fixed at the chain boundaries.
  • 1) Large and Small Adjoint Mappings:: The large adjoint transforms twist coordinates between rigid-body frames and is used for spatial velocities and applied forces.
  • 1) Large and Small Adjoint Mappings:: The small adjoint describes infinitesimal twist changes and supplies Coriolis and centrifugal coupling terms in spatial rigid-body dynamics.
  • 2) Single Rigid-Body Dynamics:: A body-fixed rigid-body model uses spatial velocity, mass, center-of-mass inertia, spatial inertia, and externally applied torque-force wrenches.
  • 2) Single Rigid-Body Dynamics:: Reference-frame changes relate wrenches and velocities through adjoint mappings and transform the corresponding spatial inertia matrix.

3) Lie Group-based Dynamics of Open-chains:

Lie group-based dynamics formulate serially connected links with screw axes, spatial inertias, and recursive Newton–Euler computations. The resulting manipulator dynamics and Gaussian rendering form the physical and visual components used by CHAINSPLAT.

  • 3) Lie Group-based Dynamics of Open-chains:: Each serial link has a joint-attached frame, spatial inertia, and transformed screw axis, with fixed-base gravity and end-effector wrench assumptions.
  • 3) Lie Group-based Dynamics of Open-chains:: Recursive Newton–Euler forward iteration propagates positions, velocities, and accelerations from the base to the end link.
  • 3) Lie Group-based Dynamics of Open-chains:: Backward iteration propagates forces and moments from the end link to the base, yielding the open-chain inverse dynamics.
  • 3) Lie Group-based Dynamics of Open-chains:: The equivalent manipulator equation combines mass-inertia, Coriolis and centrifugal, gravity, and end-effector wrench terms.
  • 3) Lie Group-based Dynamics of Open-chains:: A block lower-triangular adjoint matrix captures kinematic propagation along the chain and supports a compact closed-form gravity term.
  • 3) Lie Group-based Dynamics of Open-chains:: Gaussian primitives encode link-wise geometry and appearance, and differentiable alpha-blending with rendering losses optimizes CHAINSPLAT from multi-view RGB videos.

IV. CHAINSPLAT: INTEGRATING LIE GROUP DYNAMICS WITH 3D GAUSSIANS

CHAINSPLAT represents a DLO as a serial open chain of rigid links and revolute screw joints, associating link-aware Gaussians with compact physical and configurational parameters.

  • The formulation assumes synchronized, calibrated multi-view RGB observations and known robot end-effector poses, with the base following the robot trajectory.
  • CHAINSPLAT models DLO kinematics and dynamics with serially connected rigid links and non-actuated revolute screw joints.
  • Each screw joint is parameterized by a screw axis, spring stiffness, and damping coefficient.
  • Each link uses mass, rotational inertia, and supporting-surface friction to parameterize its dynamics.
  • Link-aware Gaussians represent DLO geometry and appearance while assigning each Gaussian probabilities over the rigid links.
  • The DLO state at timestep k is represented by its joint configuration θk.

B. Differentiable RGB Rendering with CHAINSPLAT

CHAINSPLAT renders link-aware Gaussian primitives differentiably from joint configurations while incorporating robot-controlled base motion and physics-based joint and friction dynamics.

  • Hardly assigning each Gaussian to its most probable link is non-differentiable, so CHAINSPLAT uses a differentiable rendering formulation instead.
  • Base-link poses include the robot end-effector transformation, while other links use product-of-exponentials forward kinematics from θk.
  • Replicated Gaussians inherit scale and appearance from their source Gaussian, while opacity is modulated by link-assignment probability.
  • The differentiable renderer maps a joint configuration θ to an RGB image I = π(θ), enabling joint optimization and state estimation.
  • Joint dynamics model passive revolute joints as torsional spring–dampers and include viscous friction when the DLO contacts a supporting surface.
  • The resulting dynamics enforce physical consistency while using a low-dimensional state parameterized solely by joint configurations.

V. OPTIMIZING THE PARAMETERS OF CHAINSPLAT FROM MULTI-VIEW RGB VIDEOS

CHAINSPLAT parameters are learned from synchronized multi-view RGB videos through two optimization stages: kinematics-aware geometry recognition followed by dynamics identification.

  • The learning procedure consists of kinematics-aware geometry recognition and dynamics identification.
  • Algorithm 1 takes multi-view RGB videos as input and performs differentiable CHAINSPLAT rendering during optimization.
  • Dynamics identification integrates CHAINSPLAT under the prescribed base trajectory and updates mass, damping, and friction parameters by minimizing dynamics loss.
  • The complete procedure compares simulated joint-configuration trajectories with those recovered during the first stage.

A. Kinematics-aware Geometry Recognition

Kinematics-aware geometry recognition optimizes CHAINSPLAT geometry, appearance, and joint configurations from object-masked multi-view RGB videos using differentiable rendering.

  • The stage optimizes geometry, appearance, and kinematics by minimizing discrepancies between rendered and ground-truth RGB videos.
  • Ground-truth observations are object-masked RGB videos, with masks obtained using the Segment Anything Model.

1) Optimization Variables for Kinematics-aware Geometry Recognition:

CHAINSPLAT identifies DLO dynamics by optimizing kinematics-aware geometry, joint configurations, and physical parameters within a differentiable open-chain model. The procedure uses link-wise Gaussian representations, trajectory fitting, and stable implicit integration.

  • Kinematics-aware geometry recognition: Kinematics-aware geometry recognition jointly optimizes link-aware Gaussian parameters and joint configuration trajectories from multi-view RGB videos.Gaussian opacities and link-assignment probabilities are periodically reset during training to improve optimization stability.
  • Dynamics identification: The recovered joint configuration trajectory is collected across timesteps and used to identify CHAINSPLAT’s dynamics parameters.Dynamics identification minimizes discrepancy between simulated and recognized joint trajectories.
  • Dynamics identification: Dynamics optimization estimates joint damping, link masses, and tabletop friction while assuming uniform-density rods and negligible elasticity.Under the negligible-elasticity assumption, spring stiffness is fixed to zero.
  • Dynamics simulation: Explicit Euler updates joint velocities and configurations using the simulation timestep, but can become unstable at camera-rate timesteps when damping introduces velocity-dependent forces.Reducing the timestep through trajectory downsampling is computationally expensive.
  • Dynamics simulation: CHAINSPLAT instead uses implicit-in-velocity Euler integration, enabled efficiently by a closed-form acceleration Jacobian that avoids numerical differentiation.The method approximates the implicit update through a first-order Taylor expansion in velocity.

3) Optimization Details:

CHAINSPLAT formulates model-based manipulation as differentiable trajectory optimization over a constrained end-effector path and task-specific workspace objectives. The hitting-target task differs between tabletop and free-space settings.

  • Optimization framework: The manipulation framework uses CHAINSPLAT’s low-dimensional joint-configuration state and differentiable rendering to support gradient-based trajectory optimization.The optimized trajectory is expressed relative to the robot-controlled base trajectory.
  • Trajectory parameterization: The end-effector trajectory is restricted to one dimension and parameterized with radial basis functions to encourage exploiting DLO dynamics rather than directly moving the object.Other parameterizations, including splines, are also compatible with the framework.
  • Objective construction: CHAINSPLAT computes task objectives from differentiable 3D points extracted from link-aware Gaussians and transformed using the simulated joint trajectory.Each Gaussian is assigned to the link with the highest assignment probability.
  • Hitting-target objectives: Tabletop optimization minimizes distance from any reconstructed DLO point to the target while regularizing motion smoothness.Tabletop dynamics are characterized by dominant frictional interactions.
  • Hitting-target objectives: Free-space optimization encourages the ungrasped end to reach the target at least once because stabilization at a fixed location is generally infeasible.Free-space dynamics are primarily governed by gravity.

B. State Estimation

CHAINSPLAT estimates DLO state from RGB images by optimizing joint configurations against rendered observations, then uses the recovered state in force-estimation procedures. The paper frames this as a building block toward closed-loop manipulation, while trajectory optimization itself remains open loop.

  • Scope: The paper presents state estimation and force estimation as components toward closed-loop control, but the demonstrated trajectory optimization remains inherently open loop.The authors identify modeling errors, disturbances, and sensing noise as motivations for closed-loop control.
  • State estimation: The proposed state-estimation framework recovers DLO joint configurations from a single RGB image using the optimized CHAINSPLAT kinematic structure.Training requires multi-view RGB videos, whereas subsequent state estimation uses only one RGB image.
  • State estimation: Joint configurations are estimated by minimizing rendering loss between the observed image and CHAINSPLAT’s rendered image.The rendering function maps a joint configuration to an RGB image.
  • State estimation: Initializing each optimization with the previous frame’s joint configuration provides a strong temporal prior and enables near-real-time estimation from streaming RGB observations.The method relies on smooth variation between consecutive video frames.
  • External force estimation: CHAINSPLAT’s physics-based dynamics model supports external-force estimation without dedicated force/torque sensors.The formulation targets contact-aware interactions with the environment or humans.
  • External force estimation: The force-estimation experiment assumes a fixed DLO base, a translational force at one contact point, and a sufficiently slow human hand for quasi-static analysis.The contact Jacobian depends on the contact position and the associated articulated link.

1) Real-World Experimental Setup:

Real-world experiments evaluate ChainSplat on three ropes across tabletop and free-space interactions, comparing its geometry, appearance, dynamics, and efficiency with PGND and PhysTwin. The results show stronger overall accuracy and computational efficiency, plus practical state and force estimation capabilities.

  • Experimental Setup: Experiments use a humanoid robot, three cameras, and three ropes of lengths 20 cm, 30 cm, and 40 cm.The scenarios include tabletop interactions dominated by friction and free-space interactions primarily governed by gravity.
  • Experimental Setup: Each rope is evaluated on four 5-second trajectories generated by sinusoidal and trapezoidal end-effector motions at 0.25 Hz and 0.4 Hz.The motions use 15 cm amplitude and distinct lateral movement profiles.
  • Evaluation and Baselines: ChainSplat uses eight screw joints and nine links, while comparisons include PGND and PhysTwin variants based on 3D particle states and RGBD inputs.Evaluation measures Chamfer Distance, IoU, PSNR, SSIM, and LPIPS for dynamics, geometry, and appearance.
  • Dynamics and Rendering Results: In tabletop scenarios, ChainSplat generally provides the best dynamics predictions and RGB appearance fidelity, outperforming baselines across most metrics.PGND increasingly deviates over time, while PhysTwin can be qualitatively comparable in some cases.
  • Dynamics and Rendering Results: In free-space scenarios, ChainSplat outperforms all baselines across all metrics and DLOs while accurately capturing gravity-driven behavior.PhysTwin performs better than PGND but retains larger prediction errors than ChainSplat.

D. Model-based Manipulation Results

CHAINSPLAT supports model-based manipulation of DLOs in tabletop and free-space settings, generating trajectories that reach target points while accounting for object dynamics. The discussion also identifies two-stage optimization, limited generalization, and the need for closed-loop extensions as important boundaries.

  • Tabletop manipulation: CHAINSPLAT successfully brings DLOs into contact with target points and stabilizes them in tabletop manipulation.The reported open-loop optimization must account for friction-induced changes in DLO shape.
  • Free-space manipulation: In free space, CHAINSPLAT generates motions that use gravity-induced oscillations to bring the DLO tip into contact with the target.The authors report that this captures gravity-driven DLO dynamics for the hitting task.
  • Free-space manipulation: CHAINSPLAT produces different optimized behaviors for DLOs of different lengths: shorter ropes move faster over shorter spatial paths, whereas longer ropes move slower over larger paths.For the reported 20 cm and 40 cm ropes, the behavior is consistent with differing natural frequencies and oscillation periods.
  • Limitations and future extensions: The current two-stage optimization can bias dynamics learning because errors in estimated kinematic parameters propagate into dynamic-parameter estimates.The paper proposes unified kinodynamic optimization as a possible improvement.
  • Limitations and future extensions: A single model spanning diverse objects and interaction scenarios would require extensive data, motivating adaptive control and informative trajectory design.The discussion notes that collecting trajectories across all combinations of interactions, environments, and DLO properties may be prohibitively expensive.
  • Framework capabilities: The manipulation framework combines a low-dimensional dynamics model for efficient trajectory optimization with high-fidelity rendering for real-time state estimation.Its differentiability enables low-cost gradient-based trajectory optimization.
Loading 2608.28570v1…