PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visible Connectivity Dynamics for Cloth Smoothing

Xingyu Lin, Yufei Wang, Zixuan Huang, David Held

arXiv:2105.10389v4cs.ROcs.LG

TL;DR

Robotic cloth manipulation is difficult because cloth has complex dynamics and crumpled configurations are partially observed. VCD infers visible mesh connectivity from point clouds and learns particle dynamics over the resulting graph for planning. The paper reports strong simulation performance and zero-shot transfer to real cloth smoothing on a Franka arm.

  • Problem

    Cloth's complex dynamics, lack of a low-dimensional state representation, and self-occlusions make robotic manipulation challenging.

    Source: §1 · p. 1

  • Method

    VCD infers mesh connectivity among points in a partial point cloud and learns a particle-based dynamics model over that visible graph for planning.

    Source: §1 · p. 2

  • Results

    VCD greatly outperforms previous state-of-the-art cloth-smoothing methods in simulation and successfully transfers zero-shot to smoothing varied cloth on a Franka arm.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The particle-based representation has cloth-physics inductive bias, is invariant to visual features, and is interpretable.

    Source: §5 · p. 8

    The graph assumes mesh edges satisfy distance threshold R, and planning treats a grasped point as rigidly attached to the gripper.

    Source: §3.1 · p. 4, §3.4 · p. 5

Abstract

from arXiv · show

Robotic manipulation of cloth remains challenging for robotics due to the complex dynamics of the cloth, lack of a low-dimensional state representation, and self-occlusions. In contrast to previous model-based approaches that learn a pixel-based dynamics model or a compressed latent vector dynamics, we propose to learn a particle-based dynamics model from a partial point cloud observation. To overcome the challenges of partial observability, we infer which visible points are connected on the underlying cloth mesh. We then learn a dynamics model over this visible connectivity graph. Compared to previous learning-based approaches, our model poses strong inductive bias with its particle based representation for learning the underlying cloth physics; it is invariant to visual features; and the predictions can be more easily visualized. We show that our method greatly outperforms previous state-of-the-art model-based and model-free reinforcement learning methods in simulation. Furthermore, we demonstrate zero-shot sim-to-real transfer where we deploy the model trained in simulation on a Franka arm and show that the model can successfully smooth different types of cloth from crumpled configurations. Videos can be found on our project website.

1 Introduction

Cloth smoothing is difficult because crumpled fabric has complex dynamics, lacks an easy low-dimensional representation, and is partly hidden by self-occlusion. The paper proposes learning particle dynamics from visible connectivity and uses the model for planning.

1 Introduction

  • Crumpled-cloth dynamics, difficult state representation, and self-occlusion make robotic cloth manipulation challenging.These difficulties arise in tasks including laundry folding and bed making.Source: §1 · p. 1
  • Full-mesh fitting is difficult for arbitrary crumpled cloth, while latent-vector and pixel-based models lack cloth-physics inductive bias.The authors link that missing bias to suboptimal performance and generalization.Source: §1 · p. 1
  • Particle-based dynamics provide a physics-aligned representation that is invariant to visual features such as cloth color and pattern.The representation also allows the gripper's effect on a grasped particle to be applied directly.Source: §1 · p. 2
  • VCD infers connectivity among visible points, learns dynamics over that graph, and uses the model to plan cloth smoothing.The paper reports that it greatly outperforms state-of-the-art model-based and model-free methods and transfers zero-shot to a Franka arm across cloth types.Source: §1 · p. 2

2 Related Work

Prior cloth-manipulation approaches use constrained configurations, hand-designed visual features, learned policies, or pixel and latent dynamics models. The paper positions VCD as a particle-based model-based alternative intended to retain cloth detail and generalize across cloth appearance.

Vision-based Cloth Manipulation

  • Funneling actions simplify perception from unknown cloth configurations but are usually scripted and do not generalize to different shapes or configurations.This work instead targets interaction from arbitrary configurations and shapes.Source: §2 · p. 2
  • Hand-designed cloth-feature detectors require task-specific vision design and are typically not robust to cloth-configuration variations.Source: §2 · p. 2

Policy Learning for Cloth Manipulation

  • Policy-learning approaches can lack generalization to novel crumpled configurations; the paper compares VCD with a state-of-the-art policy-learning approach.Policy-learning examples include demonstrations from heuristic experts [22] and model-free reinforcement learning [4] [24].Source: §2 · p. 3

Model-based RL for Cloth Manipulation

  • Pixel-based models can produce blurry images, while fixed-size latent representations can lose fine cloth details such as folds and wrinkles.The paper places VCD in model-based RL and uses particle-based dynamics, which it says better captures cloth dynamics and is invariant to visual features.Source: §2 · p. 3

3 Method

VCD builds a graph from a partial cloth point cloud, infers visible mesh connectivity, predicts particle dynamics, and plans pick-and-place actions. The method also describes simulation training and graph imitation to help the model reason about occlusion.

3 Method

  • VCD connects partial point-cloud points with nearby and inferred mesh edges, learns dynamics over the graph, and plans robot actions.Source: §3 · p. 3

3.1 Graph Representation of Cloth Dynamics

  • The cloth state graph uses particle nodes, fixed mesh edges for cloth connections, and dynamically constructed nearby edges for spatial collisions.The method assumes every mesh edge is also a nearby edge under distance threshold R.Source: §3.1 · pp. 3–4

3.2 Inferring Visible Connectivity from a Partial Point Cloud

  • A GNN classifies nearby edges as mesh edges from the voxelized partial point cloud, using simulated mesh labels for training.The voxel grid filter first replaces points within each voxel with their centroid; inferred mesh edges can then be deployed on real point clouds.Source: §3.2 · p. 4

3.3 Modeling Visible Connectivity Dynamics with a GNN

  • A dynamics GNN predicts particle accelerations from the voxelized partial point cloud and inferred visible connectivity, then updates particle positions and velocities.The design addresses partial observation and potentially inaccurate inferred mesh edges.Source: §3.3 · p. 4

3.4 Planning with Pick-and-place Actions

  • High-level pick-and-place actions are decomposed into small gripper movements that the dynamics model predicts step by step.While grasped, the picked point is assumed rigidly attached to the gripper; after release, the model rolls out without manually setting particle states.Source: §3.4 · p. 5
  • The planner greedily optimizes projected covered area after one high-level action and holds inferred mesh edges fixed during that trajectory.It updates mesh edges when replanning after the action, when newly revealed points may be observed.Source: §3.4 · p. 5

3.5 Training in Simulation

  • Simulation training matches voxelized observed points to simulator particles to assign acceleration targets and mesh-edge labels.The simulator models cloth as grid particles with spring connections for bending and stretching constraints.Source: §3.5 · p. 5

3.6 Graph Imitation Learning for Occlusion Reasoning

  • Graph-based privileged imitation trains a partial-observation student to imitate features from a full-cloth teacher, with auxiliary covered-area prediction.The authors state that feature imitation enables implicit reasoning about occluded cloth parts.Source: §3.6 · pp. 5–6

4 Experiments

Experiments evaluate cloth smoothing in simulation and on a Franka robot, with comparisons across cloth shapes and ablations of the dynamics and edge-inference components. VCD outperforms the simulation baselines and smooths varied cloths in real-world trials.

4.1 Experimental Setup

  • Simulation training used random pick-and-place trajectories on 25–28 cm square cloths, with evaluation on square, rectangular, and two-layer T-shirt cloths.Each shape was tested with 40 initial configurations; performance was summarized by the 25th, 50th, and 75th percentiles. The study reports normalized improvement in the main paper and normalized coverage in the appendix.Source: §4.1 · p. 6
  • The real-world evaluation used the simulation-trained dynamics model on a Franka Panda with RGBD point clouds and no fine-tuning.The authors evaluated cotton and silk square towels and a cotton T-shirt; they report that real-world baselines were not evaluated because transferring their RGB-based policies was difficult.Source: §4.1 · p. 6, §4.3 · p. 7

4.2 Simulation Results

  • In simulation, VCD greatly outperformed all baselines on training-shape squares and generalized better than VSF to rectangular cloth and T-shirts; graph imitation helped most on rectangles.VSF failed on T-shirts, while VCD still generalized; graph imitation gave marginal improvement and better stability on rectangles but larger variance on T-shirts.Source: §4.2 · p. 7
  • The authors connect VCD's particle-based structural inductive bias with better performance and stronger generalization across cloth shapes than the RGB-based VSF method.The paper attributes this contrast to VCD modeling cloth structure rather than relying on RGB appearance.Source: §4.2 · p. 7

4.3 Real-world Results

  • In 12 trajectories per cloth, real-world VCD smoothed cotton and silk cloths and generalized to T-shirts despite differences in appearance, shape, and dynamics.Table 1 reports normalized improvement; Figure 4 shows example trajectories for square cotton, square silk, and a cotton T-shirt.Source: Table 1 · p. 7, Figure 4 · p. 7, §4.3 · p. 8

4.4 Ablation Studies

  • Replacing the learned dynamics GNN with Flex produced worse smoothing and slower planning: about 330 seconds per pick-and-place action versus about 40 seconds with VCD.The authors attribute the weaker Flex result to the inferred-edge partial-cloud cloth departing from Flex's common cloth mesh structure; the learned GNN trains directly on partial point clouds.Source: §4.4 · p. 8
  • Without the edge GNN, distance-based edge construction performed worse whether edges were fixed or dynamically reconstructed, and test-time edge-type inference removal also worsened performance.These ablations assess the inference of underlying mesh edges from the observed point cloud.Source: §4.4 · p. 8

5 Conclusion

VCD uses a visible-connectivity graph and particle-based dynamics for cloth smoothing, with a graph neural network architecture that processes point-cloud nodes and their edges. The conclusion highlights its performance, sim-to-real transfer, and potential relevance to other deformable-object tasks.

5 Conclusion

  • VCD greatly outperforms previous state-of-the-art methods for cloth smoothing and transfers zero-shot to a Franka arm for smoothing varied cloth types.The conclusion also describes the model as physics-informed, invariant to visual features, and interpretable.Source: §5 · p. 8
  • The conclusion presents graph representations as a possible alternative to raw images or latent vectors for deformable objects and suggests future use on cables, bags, and food.Source: §5 · p. 8

A.1 GNN Architecture

  • The dynamics GNN takes voxelized point-cloud nodes, collision edges, and inferred mesh edges as input, then predicts point accelerations for position updates.Node features include past velocities, pick status, and table distance; edge features include relative positions, edge type, and displacement from rest positions.Source: §A.1 · pp. 12–13
  • For a pick-and-place action, the gripper path is split into small movements, and the picked point is assumed to remain rigidly attached as the model propagates effects through the graph.The picked point’s position and velocity are set from the gripper movement at each low-level step.Source: §A.1 · pp. 12–13
  • Both GNNs use separate node and edge encoders followed by ten stacked graph-network blocks with residual connections.The processor updates node and edge embeddings before the decoder maps final node embeddings to acceleration predictions.Source: §A.1 · p. 13
  • The edge GNN classifies collision edges as mesh edges using point-cloud geometry, with a binary classification loss.Its input graph contains voxelized points and collision edges; node features are zero, and edge features are relative position and distance.Source: §A.1 · p. 13

A.2 VCD Training Details

VCD training uses simulator transitions to supervise dynamics and edge prediction, with point-to-particle matching supplying training states and connectivity. The edge GNN’s reported accuracy supports using its predicted mesh edges without dynamics-model fine-tuning.

A.2 VCD Training Details

  • Dynamics training uses one-step transitions with simulator ground-truth velocities, while edge-GNN labels mark collision edges whose mapped particles share a simulator spring.Using ground-truth velocities permits sampling arbitrary training timesteps rather than always simulating from the initial state.Source: §A.2 · pp. 13–14
  • 91% edge-GNN prediction accuracy supports using its predicted mesh edges at test time without fine-tuning the dynamics GNN on those predictions.Source: §A.2 · p. 14
  • Bipartite matching maps voxelized cloud points to downsampled simulator particles by distance, with the simulator mesh downsampled threefold for efficiency.The experiments use more simulated particles than voxelized points.Source: §A.2 · p. 14
  • Training data comprise 2,000 one-action trajectories, and the reported voxel size is 0.0216 with a neighbor radius roughly twice that size.The dynamics GNN takes roughly four days to converge on an RTX 2080 Ti, while the edge GNN typically takes one or two days.Source: §A.2 · p. 14

A.3 Graph-imitation Training Details

Graph-imitation training uses a full-cloth teacher to provide structural information to the partial-observation student. VCD planning rolls out candidate actions through the dynamics model and chooses the action with the highest predicted reward.

A.3 Graph-imitation Training Details

  • Dynamics models trained on the full mesh usually converge faster and reach better asymptotic performance than VCD under partial observability.The passage attributes this difference to state-estimation ambiguity from self-occlusion.Source: §A.3 · p. 15
  • Privileged graph imitation injects full-cloth structural knowledge into the partial-cloud student, encouraging implicit reasoning about occlusion.Source: §A.3 · p. 15

A.3.1 Privileged Graph-imitation Learning

  • The student imitates a frozen full-cloth teacher by matching node features after bipartite node matching, while retaining ground-truth acceleration supervision.The student inherits the teacher’s encoder and decoder initialization and also matches its global embedding for reward prediction.Source: §A.3.1 · p. 16

A.3.2 Auxiliary Reward Prediction

  • An auxiliary reward model predicts one-step cloth coverage from the global embedding, using a three-layer MLP trained with mean-square error.The training reward approximates coverage over all particles.Source: §A.3.2 · p. 16
  • At test time, VCD uses a heuristic partial-cloud coverage reward because the learned global reward performs slightly worse.The heuristic models particles as spheres.Source: §A.3.2 · p. 16

A.4 VCD Planning Details

  • Planning rolls out each sampled pick-and-place action for H low-level steps, evaluates the predicted final point cloud, and returns the action with maximal reward.The procedure samples 500 actions in simulation and 100 in the real world; the simulation reward is zeroed if points remain above a height threshold.Source: §A.4 · pp. 16–17

B Baselines Implementation

The baseline implementations adapt training and observation settings to the respective methods while using trajectories for model or policy training. The described configurations include distinct data volumes and action spaces, with MVP performance reported before later training degradation.

B Baselines Implementation

  • The baseline implementations collect 7,115 VSF, 8,000 CFM, and 5,000 MVP training trajectories, compared with 2,000 VCD trajectories.VSF and CFM use 15 and 50 actions per trajectory, respectively; MVP uses 50.Source: B Baselines Implementation · p. 17

B.1 VisuoSpatial Forsight (VSF)

  • VSF uses 56 × 56 top-down images and 7,115 trajectories of 15 pick-and-place actions, following the original approach’s fixed-height lift and horizontal movement.The lift height is 0.02 m, and its horizontal movement vector is sampled from [−0.07, 0.07] × [−0.07, 0.07] m.Source: §B.1 · p. 18

B.2 Contrastive Forward Model (CFM)

  • CFM uses 64 × 64 top-down images with same-color cloth sides and trains on 8,000 trajectories of 50 actions.Its movement vector is sampled from [−0.04, 0.04] × [0, 0.04] × [−0.04, 0.04] m.Source: §B.2 · p. 18

B.3 Maximal Value under Placing (MVP)

  • MVP’s Q function diverges after 5,000 trajectories, so the reported policy is the best-performing model from that point rather than later training.Its action is five-dimensional: image-coordinate pick location and a clipped three-dimensional displacement.Source: §B.3 · p. 19

C Experimental Setup

The experiments evaluate VCD in simulation and on a real robot, then examine its predictions, planning behavior, and runtime. Across simulation metrics, VCD consistently outperforms the baselines.

C.1 Simulation Setup

  • Simulation uses 20 randomly generated initial configurations and evaluates square cloth, rectangular cloth, and a t-shirt.Training uses nearly square cloth in Nvidia Flex wrapped in SoftGym.Source: §C.1 · p. 19

C.2 Real-world Setup

  • Real-world evaluation uses a Franka Panda and Azure Kinect on two towels and one cotton shirt.The cloth is reset by three lift-and-drop cycles; each cloth receives 12 trajectories of up to 20 pick-and-place actions.Source: §C.2 · pp. 19–20

D.1.1 Normalized Improvement and Normalized Coverage in Simulation

  • Across normalized improvement and normalized coverage, VCD consistently outperforms all simulation baselines.Supplementary Tables 4 and 5 report these metrics for varying numbers of allowed pick-and-place actions.Source: Table 4 · p. 20, Table 5 · p. 20, §D.1.1 · p. 20

D.1.2 Visualization of Edge GNN

  • The edge GNN reaches 0.91 validation accuracy and can disconnect cloth layers when the cloth is folded.The model takes point clouds as input; color in the visualization is used only for display.Source: §D.1.2 · p. 20

D.1.3 Visualizations of Planned Actions in Simulation

  • All shown planned trajectories for square cloth, rectangular cloth, and a t-shirt achieve normalized improvement above 0.98.VCD favors picking edge or corner points and pulling outward, a strategy described as effective for smoothing.Source: Figure 9 · p. 21, §D.1.3 · p. 21

D.1.4 Visualizations of Open-loop Predictions in Simulation

  • With a prediction horizon of 70 steps, VCD produces relatively accurate open-loop predictions across cloth shapes.The figures compare ground-truth observable particles and mesh edges with predicted point clouds and inferred mesh edges.Source: Figure 10 · p. 22, Figure 11 · p. 22, §D.1.4 · p. 22

D.2.1 Running Time

  • Planning averages 12.7 seconds per action using 100 samples on 4 RTX 2080Ti, while the full system takes around 40 seconds per action.Franka execution averages 10.2 seconds per action.Source: §D.2.1 · pp. 22–23

D.2.2 Normalized Coverage (NC) of Robot Experiments

  • Normalized coverage for the real-world robot experiments is reported in Supplementary Table 6.The main text reports normalized improvement for those experiments.Source: Table 6 · p. 23, §D.2.2 · p. 23

D.2.3 Visualization of Sampled Actions in The Real World

Sampled pick-and-place actions are scored during smoothing, with VCD favoring outward pulls from cloth edges and corners.

D.2.3 Visualization of Sampled Actions in The Real World

  • The sampled-action visualization colors higher predicted rewards greener and lower predicted rewards redder; VCD favors outward pulls from corners or edges.The paper describes this as a natural and effective smoothing strategy, despite no explicit optimization for it.Source: Figure 13 · p. 23, §D.2.3 · pp. 23–24
  • The figure shows 50 sampled pick-and-place actions as arrows from projected pick to place locations.The color scale encodes the model's predicted reward for each action.Source: Figure 13 · p. 23
  • VCD favors picking corner or edge points and pulling outward, a strategy described as effective for smoothing.The sampled-action scores are shown during cloth smoothing.Source: §D.2.3 · pp. 23–24

E Planning with VCD for Cloth Folding

The paper applies VCD with cross-entropy-method planning to cloth folding from an initially flattened state. It evaluates three goal specifications and reports reasonable folding performance with point-cloud goals and Chamfer-distance costs.

E Planning with VCD for Cloth Folding

  • Folding is planned from an initially flattened cloth toward one-corner-in, diagonal, or arbitrary goals using cross-entropy-method planning.The paper reports average particle distance to the goal and shows qualitative outcomes for the three goals.Source: E Planning with VCD for Cloth Folding · p. 25, Figure 14 · p. 26, Figure 15 · p. 27
  • The evaluation considers simulator-particle goals, point-cloud goals with Chamfer distance, and depth-image goals with 2D intersection over union.For simulator-particle goal specification and cost, average particle error is 0.3–1.3 cm.Source: E Planning with VCD for Cloth Folding · p. 25
  • The folding model is trained on random pick-and-place actions, and the authors suggest folding performance could improve with corner-grasping bias during data collection.The proposed bias would add more folding motions to the training data.Source: E Planning with VCD for Cloth Folding · p. 25
  • With point-cloud goals and Chamfer-distance costs, VCD achieves average particle error of 0.3–2 cm across folding evaluations.The authors describe this as a practical choice for real-world folding.Source: Table 7 · p. 25, E Planning with VCD for Cloth Folding · p. 25

F Robustness to Depth Sensor Noise

The authors test robustness to depth-sensor noise and compare VCD planning with an oracle using the full FleX cloth model. VCD is reported robust within Azure Kinect noise levels, while the oracle achieves perfect results.

F Robustness to Depth Sensor Noise

  • VCD is reported as quite robust within the Azure Kinect depth-sensor noise range in simulation.The test adds Gaussian noise to depth maps and allows at most 10 pick-and-place actions; the dashed line marks the measured Azure Kinect noise level.Source: Figure 17 · p. 29, F Robustness to Depth Sensor Noise · p. 29
  • The noise figure plots normalized improvement against depth-sensor noise, with the Azure Kinect level marked by a vertical dashed line.Error bars show the 25th and 75th percentiles.Source: Figure 17 · p. 29

G Comparison to Oracle using the FleX Cloth Model

H Ablations on architectural choices

Adding a global model improves planning performance, while the dynamics model is robust to message-passing depths from 3 to 10. The authors suggest too few steps may limit action propagation and too many may encourage overfitting.

H Ablations on architectural choices

Loading 2105.10389v4…