PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects

Mengyuan Yan, Yilin Zhu, Ning Jin, Jeannette Bohg

arXiv:1911.06283v3cs.ROcs.CVcs.LG

TL;DR

The paper addresses efficient, generalizable manipulation of deformable linear objects despite their high-dimensional states, complex dynamics, and costly real-image annotations. It estimates rope states with self-supervision and learns physics-informed state-space dynamics, achieving lower prediction error with limited data and more efficient manipulation in simulation and on a real robot.

  • Problem

    Deformable objects have high-dimensional states and complex nonlinear dynamics, making state estimation challenging, forward prediction expensive, and real-image annotations costly.

  • Method

    The method combines self-supervised rope state estimation with coarse-to-fine hierarchical STNs and a physics-informed state-space dynamics model.

  • Results

    The dynamics model reduces prediction errors by up to 68% using 3% of total training data versus a pixel-space baseline, while the method enables efficient goal matching.

  • Takeaways & Limitations

    Explicit physical states support physics-informed dynamics and transfer across visually different ropes, enabling model-based manipulation of visually specified goals.

Abstract

from arXiv · show

We demonstrate model-based, visual robot manipulation of linear deformable objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is in contrast to dynamics learned in pixel space or latent space, where generalization to visual differences are not guaranteed. Challenges in taking the state-space approach are the estimation of the high-dimensional state of a deformable object from raw images, where annotations are very expensive on real data, and finding a dynamics model that is both accurate, generalizable, and efficient to compute. We are the first to demonstrate self-supervised training of rope state estimation on real images, without requiring expensive annotations. This is achieved by our novel self-supervising learning objective, which is generalizable across a wide range of visual appearances. With estimated rope states, we train a fast and differentiable neural network dynamics model that encodes the physics of mass-spring systems. Our method has a higher accuracy in predicting future states compared to models that do not involve explicit state estimation and do not use any physics prior, while only using 3\% of training data. We also show that our approach achieves more efficient manipulation, both in simulation and on a real robot, when used within a model predictive controller.

I. INTRODUCTION

The paper presents a vision-based system for manipulating linear deformable objects using explicit rope states, physics-informed state-space dynamics, and model predictive control. It addresses difficult state estimation from real images through self-supervision and reports improved prediction and manipulation efficiency over pixel-space approaches.

  • Motivation: Deformable-object manipulation is challenging because objects have high-dimensional states, complex nonlinear dynamics, and expensive forward prediction.These properties also make state estimation from images difficult.
  • System overview: The system estimates an explicit rope state from images, predicts future states in rope state space, and uses model predictive control to select manipulation actions.The system repeatedly replans from newly observed rope states until reaching the goal.
  • State-space formulation: Explicit state-space representations incorporate physics priors and can support dynamics learned in one visual setting being used with objects and environments of different appearances.The paper contrasts this with pixel- or latent-space dynamics, whose generalization to visual differences is not guaranteed.
  • Self-supervised estimation: The proposed self-supervising objective enables continuous training of deformable-object state estimation on real images without expensive ground-truth annotations.The model is initialized with a small set of synthetic images before self-supervised real-image training.
  • Coarse-to-fine estimation: The perception model uses hierarchical Spatial Transformer Networks for coarse-to-fine rope-state estimation, refining eight initial segments into a higher-resolution representation.Regions extracted from feature maps are used to estimate endpoints and middle points, which are concatenated and averaged across neighboring regions.

III. METHOD

The method estimates rope states from images with a coarse-to-fine network, then adapts a differentiable rendering and GMM-based objective for self-supervised refinement on real images. Curriculum learning, temporal consistency, and robustness to varied visual appearances support training and test-time refinement.

  • System overview: At each time step, a CNN estimates the rope state from an image, after training on rendered images and finetuning on real images with the self-supervising objective.The estimated states are used by the downstream manipulation system.
  • Coarse-to-fine state estimation: The coarse-to-fine estimator predicts 8 segments, refines them through spatial transformer regions, and repeats until the rope is represented by 64 segments.Neighboring-region endpoints are averaged when converting predictions back to image coordinates.
  • Self-supervising learning objective: The RGB clustering formulation uses two Gaussian components for rope and background, while requiring only good color contrast rather than fixed rope or background colors.The GMM parameters are estimated separately for each image and are not memorized.
  • Training and refinement: The loss supports both network finetuning and test-time state refinement, while gradient clipping handles pixels whose estimated rope regions may be occluded.Temporal consistency transfers a neighboring frame’s better prediction to a worse frame during training.
  • Self-supervising learning objective: The self-supervising objective renders membership weights from the estimated rope state, updates GMM parameters with an M step, computes new weights with an E step, and minimizes their image loss.The loss can update the rope state directly or the perception-network weights.
  • Generalization: The objective is designed for textured or multicolored ropes and backgrounds, distractors, occlusions, and multiple ropes; L2 loss failed to converge on some real images.The proposed objective is illustrated as robust across these visual variations.

C. A dynamics model with a physics prior

The dynamics model predicts future rope states for action planning while encoding a chain-like mass-spring physics prior. A recurrent structure applies the same physical law across rope nodes, and simulation parameters are calibrated using a small set of real data.

  • Model design: A dynamics model is required to predict future rope states under hypothetical actions so manipulation trajectories can be planned.The model operates after perception estimates and refines the rope state.
  • Model design: The neural dynamics model uses a bidirectional LSTM to encode the structure of a chain-like mass-spring system while providing faster inference and parallel processing than a physics-based simulator.The model is trained as a computationally efficient alternative to available deformable-object simulators.
  • Training: Recurrently applying the same LSTM cell to each rope node enforces a shared physical law across endpoint and interior nodes, while simulation parameters are identified with CEM on a small real dataset.Random-action simulation sequences are then generated for training.

D. Rope manipulation with MPC

The controller uses explicit rope states to plan grasping points and planar gripper motions toward image-specified goals. MPPI performs nested sampling and optimization over grasp locations and movement trajectories using the learned dynamics model.

  • Action formulation: Model predictive control plans actions that move the rope from its current configuration to a goal configuration, with both states estimated from images.An action selects a rope grasping point and a 2D planar movement vector.
  • MPPI planning: The MPPI planner uses nested optimization to select grasping points and movement trajectories, rolling out sampled trajectories over a horizon with the dynamics model.Each trajectory is evaluated by the rope-state distance to the goal.
  • State-space planning: Explicit rope states make it straightforward to define an informative cost and sample promising action candidates compared with image-space methods.The planning objective directly uses rope-state distance.

IV. EXPERIMENTS AND RESULTS

Experiments evaluate the perception and manipulation components and show improved generalization from the coarse-to-fine estimator. The full system is evaluated for efficient rope manipulation in simulation and on real robots.

  • Evaluation scope: The experiments evaluate the system components and report efficient rope manipulation toward visually specified goals in simulation and on real robots.The evaluation includes perception, dynamics, and their interaction in manipulation.
  • Perception networks comparison: Table I reports root mean square Euclidean distance in meters between estimated and ground-truth rope-point positions.The metric evaluates perception-network estimation accuracy.
  • Perception networks comparison: 30% less error on a held-out test set was achieved by the coarse-to-fine network than by the direct-estimation baseline.Both models were trained on 10000 rendered images of b-spline curves.
  • Perception networks comparison: Hierarchical spatial transformer networks are important for generalization because dense sampling of the high-dimensional rope state space would require exponentially large datasets.The rope state space grows with the number of rope points.

B. Self-supervised finetuning

The proposed objective finetunes rope state estimation on real images, while curriculum learning and temporal consistency improve perception. The resulting state-space dynamics model is accurate, physics-informed, and substantially faster than simulation.

  • Self-supervised finetuning: 5122 real images are used to finetune the perception network without annotations, addressing robot-arm occlusions absent from rendered training images.Ablations report significant improvements from both automatic curriculum learning and temporal consistency.
  • Dynamics evaluation: The dynamics model is evaluated using average and maximum deviations between corresponding rope points over long-horizon predictions.Each state contains 65 corresponding points, and the metrics are mean(di) and max(di).
  • Dynamics evaluation: The learned neural dynamics model has prediction accuracy comparable to the simulator on real data, despite noise from partially occluded rope state estimates.The comparison uses deviations from predicted states to estimated states and image-derived ground truth states.
  • Dynamics evaluation: The neural model predicts an action in 0.03 seconds on average, versus 1.15 seconds for the simulator, and supports GPU batches up to 32000.These properties support parallel mental rollouts required by model predictive control.
  • Dynamics evaluation: With the same simulated training dataset, the explicit-state dynamics model is significantly more accurate than the image-based DVF model.The comparison uses 50-action sequences from the same starting image, while both models are trained on 0.5M simulated actions.

D. Manipulation results

The state-space method achieves manipulation goals more efficiently than the pixel-space baseline in simulation and succeeds across visually different rope settings on a real robot. Its perception system still faces generalization limits for rope states outside the training distribution.

  • Simulation results: Within 60 steps, the proposed method reaches the goal state in most simulated experiments, whereas the pixel-space baseline often fails within 100 steps.The proposed method leaves very small residual distances, while the baseline retains large residual distances at t = 100.
  • Generalization: Generalization remains limited for rope states outside the training distribution, with most overfitting occurring at the coarsest prediction layer.The refinement layers generalize well when given a reasonable coarse prediction, and temporal information improves generalization.
  • Real-robot results: All 3 real-robot tasks succeeded despite differences in rope appearance, length, thickness, and background.The experiments used visually specified “S,” “W,” and “Ω” goal shapes with a different rope and background.
  • System-level result: The system combines explicit rope-state estimation, state-space dynamics, and physics priors to support efficient manipulation of visually specified goals.The dynamics model uses a network structure informed by mass-spring systems and is evaluated within model-based visual manipulation.
  • Scope: The demonstrated manipulation is limited to a horizontal plane, although the authors state that extension to 3D tasks requires minor modifications.A future direction is using estimated 2D states with Kinect depth values to reproject points into 3D.

Appendix

The appendix details the perception and dynamics components, including coarse-to-fine rope estimation, self-supervised finetuning, and a bidirectional LSTM dynamics model for repeated action prediction.

  • Perception model: The perception network combines pretrained VGG16 features with fully connected layers for initial coarse rope-state estimation.It processes 224 × 224 images and predicts 9 points defining 8 rope segments.
  • Perception model: Hierarchical refinement doubles the number of regions of interest from 8 to 16, 32, and 64 using spatial transformer networks.Each cropped feature map is transformed to 7×7 resolution before further fully connected prediction.
  • Perception model: Finetuning minimizes image loss while automatic curriculum learning includes only samples below a negative loss threshold.The threshold is adjusted during training to expand the effective training set.
  • Perception model: Temporal consistency trains neighboring image pairs by using the better prediction as a stop-gradient target when only one sample meets the threshold.The procedure otherwise trains accepted samples with image loss and updates the network from the combined objective.
  • Dynamics model: A bidirectional LSTM with 256-unit memory processes the node sequence in both directions and predicts each node’s post-action position.Its outputs are combined with node inputs, and the model can be applied repeatedly for long-horizon prediction.
  • Dynamics model: The dynamics model feeds each of 65 rope nodes a 5-dimensional input combining position, action, and an action indicator.Positions come from simulation during training and from images during evaluation.

IV. HEURISTICS USED FOR ACTION PLANNING

The action-planning heuristics use estimated current and goal rope states to sample grasp points and generate exploratory ten-action sequences for model predictive path integral control.

  • Candidate action sequences: The planner samples every other rope node, from node 1 through node 65, as candidate grasping points.For each candidate, it generates 30 action sequences containing 10 displacement vectors.
  • Candidate action sequences: Each sequence initially moves toward the corresponding goal point along a unit direction with displacement magnitude 0.8amax.When the goal is reachable before the tenth action, final actions are clipped to reach the goal position.
  • Candidate action sequences: Exploration noise shifts the trajectory endpoint in the plane or curves the path without changing its endpoint.δx and δy control endpoint displacement, while δc changes straight trajectories into curves.

V. DATA COLLECTION

The experiments use rendered spline images, simulated manipulation sequences, and real robot-camera data to train and evaluate rope perception and manipulation.

  • Datasets: The rendered pretraining dataset contains 10000 red b-spline rope images with 65-point ground-truth states.The generated rope lengths range from 0.63m to 1.25m.
  • Datasets: The simulated dataset contains rope manipulation sequences with ground-truth states, actions, and rendered images.Each sampled start state is used to generate a 100-step sequence with randomized grasp locations and displacements.
  • Datasets: The real dataset records rope manipulation sequences with a Franka Panda arm and Kinect RGB camera.The camera is calibrated to the robot base and images are projected to top-down views without requiring depth data.
  • Evaluation data: The manipulation sequence visualization evaluates state estimation and tracking when robot actions create an intersection with the rope.The result demonstrates use of the perception method in more complex tasks such as knotting.

VI. ADDITIONAL EXPERIMENT RESULTS

Additional experiments examine state tracking and the effects of automatic curriculum-learning thresholds on finetuning losses, including comparisons with training without curriculum learning.

  • State tracking: State estimation and tracking remain visualized during a manipulation sequence designed to create a rope intersection.The authors identify this as evidence that the perception method can support complex manipulation such as knotting.
  • Curriculum-learning thresholds: Curriculum-learning thresholds are evaluated at the 5%, 10%, 20%, and 40% quantiles of pre-finetuning training losses.Final average image losses are compared across these thresholds and against no-curriculum baselines.
  • Curriculum-learning thresholds: The best threshold lies between the 10% and 20% quantiles, although differences among thresholds are small relative to improvement over baselines.This supports effectiveness across a wide range of threshold choices.
  • Training dynamics: Training curves with curriculum learning and temporal consistency report image losses averaged over samples below the current threshold.Stricter thresholds begin with lower effective-set losses, while staircase patterns mark threshold increases that add samples.

C. Data efficiency of dynamics models

The proposed dynamics model achieves strong prediction accuracy with only 3% of the available training data, whereas the baseline continues to require larger datasets. In simulated manipulation, the proposed method also reaches states closer to specified goals after 100 actions.

  • Data efficiency: 3% of the total training data is sufficient for the proposed dynamics model to train well, with overfitting appearing only at 1%.Training sizes are relative to the largest dataset, which contains 0.5M simulated actions.
  • Data efficiency: The baseline model requires larger datasets to continue improving, but generating more data was resource-intensive.
  • Manipulation outcome: After 100 simulated actions, the proposed method achieves states much closer to the specified goals than the baseline method.The comparison uses start states, specified goal states, and states achieved by each method at t = 100.
Loading 1911.06283v3…