PaperlayerView exact revision on arXiv

Source-linked AI summary

ChainSplat: A Physics-Inspired Screw-Theoretic Model for Learning Deformable Linear Object Dynamics from Multi-View RGB Videos

Seungyeon Kim, Noémie Jaquier

arXiv:2608.28570v1cs.RO

TL;DR

DLO dynamics and geometry are difficult to infer for robotic manipulation because configurations are high-dimensional, materials vary, and existing pipelines often depend on noisy depth and multiple stages. CHAINSPLAT addresses this with a screw-theoretic open-chain model integrated with Gaussian splatting and optimized from multi-view RGB videos. Real-world experiments report state-of-the-art modeling, real-time estimation, and effective model-based manipulation.

  • Problem

    DLO dynamics learning must handle high-dimensional configurations, varying material properties, and limitations of multi-stage depth-dependent approaches for accurate robotic manipulation.

    Source: §I · p. 1

  • Method

    CHAINSPLAT models DLOs as revolute-jointed rigid-link chains with screw-theoretic kinematics and link-wise Gaussians, then uses two-stage differentiable optimization on multi-view RGB videos.

    Source: §I · p. 2, A. Components of CHAINSPLAT · p. 7

  • Results

    Experiments report state-of-the-art DLO digital-twin modeling, accurate dynamics, faithful 3D reconstruction, high-quality RGB rendering, and effective real-time manipulation.

    Source: §I · p. 2, D. Model-based Manipulation Results · p. 15

  • Takeaways & Limitations

    The compact differentiable model supports efficient trajectory optimization and real-time RGB-based state estimation for model-based DLO manipulation.

    Source: §I · p. 2, §IX · p. 17

Abstract

from arXiv · show

Identifying the underlying dynamics and 3D geometry of deformable linear objects (DLOs), such as cables, ropes, and hoses, is essential for accurate robotic manipulation, but remains challenging due to their high-dimensional configuration spaces and diverse behaviors arising from varying material properties. Existing methods often rely on multi-stage pipelines and auxiliary depth inputs, which are prone to errors under dynamic interactions, while their high-dimensional state representations make model-based control computationally expensive. In this paper, we introduce ChainSplat, a physics-inspired framework that jointly learns the 3D geometry, appearance, kinematics, and dynamics of DLOs solely from multi-view RGB videos. ChainSplat represents a DLO as an open-chain structure of rigid links connected by revolute joints, yielding an analytic, screw-theoretic model with a compact state representation parameterized by joint configurations. By integrating this formulation with Gaussian splatting, ChainSplat jointly recovers DLO dynamics, kinematics-aware 3D geometry, and appearance, while enabling high-fidelity RGB rendering from arbitrary states. Through real-world experiments, we demonstrate that ChainSplat achieves state-of-the-art performance in dynamics predictions, 3D geometry reconstruction, and RGB rendering across dynamic interactions. ChainSplat further enables real-time state and force estimation, as well as accurate model-based trajectory optimization, highlighting its practical utility for real-world robotic manipulation of DLOs. Accompanying source code and video are available at: https://chainsplat.github.io.

I. INTRODUCTION

CHAINSPLAT addresses DLO dynamics learning from multi-view RGB videos with a compact, physics-inspired representation that jointly models geometry, appearance, kinematics, and dynamics. Its differentiable model supports efficient estimation, rendering, trajectory optimization, and real-world manipulation.

  • DLO manipulation requires physical dynamics models despite high-dimensional configurations and behavior variation from unknown material properties.Source: §I · p. 1
  • Existing physics-inspired methods often use multi-stage RGB-depth pipelines that accumulate errors during dynamic 3D trajectory estimation and decouple rendering from dynamics learning.Depth-based lifting is especially vulnerable during fast motions, while decoupled modeling can reduce RGB rendering fidelity.Source: §I · p. 2, B. Physics-inspired Dynamics Models… · pp. 3–4
  • CHAINSPLAT represents a DLO as rigid links connected by revolute joints, combining screw-theoretic kinematics with link-wise Gaussian splatting.The representation jointly captures 3D geometry, appearance, kinematics, and dynamics while using RGB observations alone.Source: §I · p. 2
  • A two-stage differentiable optimization first recovers kinematics-aware geometry, appearance, and joint trajectories, then identifies dynamic parameters from simulated-versus-estimated trajectories.The resulting state is parameterized by the open-chain joint configurations, producing a low-dimensional dynamics model.Source: §I · p. 2
  • CHAINSPLAT provides efficient trajectory optimization and real-time RGB-based state estimation through differentiability and high-fidelity rendering from arbitrary states.Rendered-observed image discrepancies are minimized iteratively for state correction and closed-loop control.Source: §I · p. 2
  • Experiments report state-of-the-art DLO digital-twin modeling and effective real-time manipulation, including target hitting under frictional interactions.Open-loop trajectory optimization successfully accounts for friction-induced shape changes in tabletop manipulation.Source: §I · p. 2

1) Special Euclidean Group:

This section introduces SE(3), screw theory, and product-of-exponentials kinematics for representing rigid-body and open-chain motions. These tools provide link poses and Gaussian transformations for CHAINSPLAT.

1) Special Euclidean Group:

2) Screw Theory:

  • For revolute joints, screw axes specify a unit rotation axis and its associated linear component, and CHAINSPLAT uses exclusively revolute joints.Source: 2) Screw Theory · p. 4
  • A joint displacement transforms a rigid-body configuration through a screw-axis matrix exponential, while repeated exponentials compose open-chain motion.Source: 2) Screw Theory · p. 4, Figure 2 · p. 4

3) Forward Kinematics of Open-Chain Systems:

  • The product-of-exponentials formula computes end-effector and intermediate-link poses from joint configurations and home-frame screw axes.These transformed link poses also determine the locations of CHAINSPLAT’s associated Gaussian primitives.Source: 3) Forward Kinematics of Open-Chain… · pp. 4–5

B. Dynamics of Open-chain Systems

Open-chain rigid-body dynamics are expressed with SE(3) velocities, wrenches, spatial inertia, and adjoint mappings. These operators support coordinate changes and velocity-dependent coupling terms.

2) Single Rigid-Body Dynamics:

1) Large and Small Adjoint Mappings:

2) Single Rigid-Body Dynamics:

  • A body-fixed rigid-body model uses spatial velocity, mass, center-of-mass inertia, spatial inertia, and externally applied torque-force wrenches.Source: 2) Single Rigid-Body Dynamics · p. 5
  • Reference-frame changes relate wrenches and velocities through adjoint mappings and transform the corresponding spatial inertia matrix.Source: 2) Single Rigid-Body Dynamics · p. 5

3) Lie Group-based Dynamics of Open-chains:

Lie group-based dynamics formulate serially connected links with screw axes, spatial inertias, and recursive Newton–Euler computations. The resulting manipulator dynamics and Gaussian rendering form the physical and visual components used by CHAINSPLAT.

3) Lie Group-based Dynamics of Open-chains:

IV. CHAINSPLAT: INTEGRATING LIE GROUP DYNAMICS WITH 3D GAUSSIANS

CHAINSPLAT represents a DLO as a serial open chain of rigid links and revolute screw joints, associating link-aware Gaussians with compact physical and configurational parameters.

B. Differentiable RGB Rendering with CHAINSPLAT

CHAINSPLAT renders link-aware Gaussian primitives differentiably from joint configurations while incorporating robot-controlled base motion and physics-based joint and friction dynamics.

V. OPTIMIZING THE PARAMETERS OF CHAINSPLAT FROM MULTI-VIEW RGB VIDEOS

CHAINSPLAT parameters are learned from synchronized multi-view RGB videos through two optimization stages: kinematics-aware geometry recognition followed by dynamics identification.

  • The learning procedure consists of kinematics-aware geometry recognition and dynamics identification.Source: §V · p. 8
  • Algorithm 1 takes multi-view RGB videos as input and performs differentiable CHAINSPLAT rendering during optimization.Source: §V · p. 9
  • Dynamics identification integrates CHAINSPLAT under the prescribed base trajectory and updates mass, damping, and friction parameters by minimizing dynamics loss.Source: §V · p. 9
  • The complete procedure compares simulated joint-configuration trajectories with those recovered during the first stage.Source: §V · p. 9

A. Kinematics-aware Geometry Recognition

Kinematics-aware geometry recognition optimizes CHAINSPLAT geometry, appearance, and joint configurations from object-masked multi-view RGB videos using differentiable rendering.

1) Optimization Variables for Kinematics-aware Geometry Recognition:

CHAINSPLAT identifies DLO dynamics by optimizing kinematics-aware geometry, joint configurations, and physical parameters within a differentiable open-chain model. The procedure uses link-wise Gaussian representations, trajectory fitting, and stable implicit integration.

Kinematics-aware geometry recognition

  • Kinematics-aware geometry recognition jointly optimizes link-aware Gaussian parameters and joint configuration trajectories from multi-view RGB videos.Gaussian opacities and link-assignment probabilities are periodically reset during training to improve optimization stability.Source: 1) Optimization Variables for… · p. 9, 2) Optimization Details · p. 9

Dynamics identification

Dynamics simulation

  • Explicit Euler updates joint velocities and configurations using the simulation timestep, but can become unstable at camera-rate timesteps when damping introduces velocity-dependent forces.Reducing the timestep through trajectory downsampling is computationally expensive.Source: 2) Dynamics Simulation · p. 10
  • CHAINSPLAT instead uses implicit-in-velocity Euler integration, enabled efficiently by a closed-form acceleration Jacobian that avoids numerical differentiation.The method approximates the implicit update through a first-order Taylor expansion in velocity.Source: 2) Dynamics Simulation · p. 10

3) Optimization Details:

CHAINSPLAT formulates model-based manipulation as differentiable trajectory optimization over a constrained end-effector path and task-specific workspace objectives. The hitting-target task differs between tabletop and free-space settings.

Optimization framework

  • The manipulation framework uses CHAINSPLAT’s low-dimensional joint-configuration state and differentiable rendering to support gradient-based trajectory optimization.The optimized trajectory is expressed relative to the robot-controlled base trajectory.Source: §VI · p. 10, A. Model-based Trajectory Optimization · pp. 10–11

Trajectory parameterization

  • The end-effector trajectory is restricted to one dimension and parameterized with radial basis functions to encourage exploiting DLO dynamics rather than directly moving the object.Other parameterizations, including splines, are also compatible with the framework.Source: A. Model-based Trajectory Optimization · pp. 10–11

Objective construction

  • CHAINSPLAT computes task objectives from differentiable 3D points extracted from link-aware Gaussians and transformed using the simulated joint trajectory.Each Gaussian is assigned to the link with the highest assignment probability.Source: A. Model-based Trajectory Optimization · p. 11

Hitting-target objectives

B. State Estimation

CHAINSPLAT estimates DLO state from RGB images by optimizing joint configurations against rendered observations, then uses the recovered state in force-estimation procedures. The paper frames this as a building block toward closed-loop manipulation, while trajectory optimization itself remains open loop.

Scope

  • The paper presents state estimation and force estimation as components toward closed-loop control, but the demonstrated trajectory optimization remains inherently open loop.The authors identify modeling errors, disturbances, and sensing noise as motivations for closed-loop control.Source: B. State Estimation · p. 11

State estimation

  • The proposed state-estimation framework recovers DLO joint configurations from a single RGB image using the optimized CHAINSPLAT kinematic structure.Training requires multi-view RGB videos, whereas subsequent state estimation uses only one RGB image.Source: B. State Estimation · p. 11
  • Joint configurations are estimated by minimizing rendering loss between the observed image and CHAINSPLAT’s rendered image.The rendering function maps a joint configuration to an RGB image.Source: B. State Estimation · p. 11
  • Initializing each optimization with the previous frame’s joint configuration provides a strong temporal prior and enables near-real-time estimation from streaming RGB observations.The method relies on smooth variation between consecutive video frames.Source: B. State Estimation · p. 11

External force estimation

  • CHAINSPLAT’s physics-based dynamics model supports external-force estimation without dedicated force/torque sensors.The formulation targets contact-aware interactions with the environment or humans.Source: C. External Force Estimation · p. 12
  • The force-estimation experiment assumes a fixed DLO base, a translational force at one contact point, and a sufficiently slow human hand for quasi-static analysis.The contact Jacobian depends on the contact position and the associated articulated link.Source: C. External Force Estimation · p. 12

1) Real-World Experimental Setup:

Real-world experiments evaluate ChainSplat on three ropes across tabletop and free-space interactions, comparing its geometry, appearance, dynamics, and efficiency with PGND and PhysTwin. The results show stronger overall accuracy and computational efficiency, plus practical state and force estimation capabilities.

Experimental Setup

  • Experiments use a humanoid robot, three cameras, and three ropes of lengths 20 cm, 30 cm, and 40 cm.The scenarios include tabletop interactions dominated by friction and free-space interactions primarily governed by gravity.Source: 1) Real-World Experimental Setup · p. 12, 2) Dataset Collection · p. 12
  • Each rope is evaluated on four 5-second trajectories generated by sinusoidal and trapezoidal end-effector motions at 0.25 Hz and 0.4 Hz.The motions use 15 cm amplitude and distinct lateral movement profiles.Source: 2) Dataset Collection · p. 12

Evaluation and Baselines

Dynamics and Rendering Results

  • In tabletop scenarios, ChainSplat generally provides the best dynamics predictions and RGB appearance fidelity, outperforming baselines across most metrics.PGND increasingly deviates over time, while PhysTwin can be qualitatively comparable in some cases.Source: B. Kinematics-aware Geometry… · pp. 13–14
  • In free-space scenarios, ChainSplat outperforms all baselines across all metrics and DLOs while accurately capturing gravity-driven behavior.PhysTwin performs better than PGND but retains larger prediction errors than ChainSplat.Source: B. Kinematics-aware Geometry… · p. 14

D. Model-based Manipulation Results

CHAINSPLAT supports model-based manipulation of DLOs in tabletop and free-space settings, generating trajectories that reach target points while accounting for object dynamics. The discussion also identifies two-stage optimization, limited generalization, and the need for closed-loop extensions as important boundaries.

Tabletop manipulation

  • CHAINSPLAT successfully brings DLOs into contact with target points and stabilizes them in tabletop manipulation.The reported open-loop optimization must account for friction-induced changes in DLO shape.Source: D. Model-based Manipulation Results · p. 15

Free-space manipulation

  • In free space, CHAINSPLAT generates motions that use gravity-induced oscillations to bring the DLO tip into contact with the target.The authors report that this captures gravity-driven DLO dynamics for the hitting task.Source: D. Model-based Manipulation Results · pp. 15–16
  • CHAINSPLAT produces different optimized behaviors for DLOs of different lengths: shorter ropes move faster over shorter spatial paths, whereas longer ropes move slower over larger paths.For the reported 20 cm and 40 cm ropes, the behavior is consistent with differing natural frequencies and oscillation periods.Source: D. Model-based Manipulation Results · p. 16

Limitations and future extensions

  • The current two-stage optimization can bias dynamics learning because errors in estimated kinematic parameters propagate into dynamic-parameter estimates.The paper proposes unified kinodynamic optimization as a possible improvement.Source: 1) End-to-End Optimization of · pp. 16–17
  • A single model spanning diverse objects and interaction scenarios would require extensive data, motivating adaptive control and informative trajectory design.The discussion notes that collecting trajectories across all combinations of interactions, environments, and DLO properties may be prohibitively expensive.Source: 2) Generalizable CHAINSPLAT across… · p. 17

Framework capabilities

  • The manipulation framework combines a low-dimensional dynamics model for efficient trajectory optimization with high-fidelity rendering for real-time state estimation.Its differentiability enables low-cost gradient-based trajectory optimization.Source: §IX · p. 17

Related papers

Loading 2608.28570v1…