PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E. Gonzalez, Nawid Jamali, Katsu Yamane, Soshi Iba, Ken Goldberg

arXiv:2003.12698v2cs.ROcs.CVcs.LG

TL;DR

Fabric manipulation policies are often task-specific, while sensing and control difficulties make learning reusable policies for deformable objects challenging. This paper learns visual correspondences for fabrics in simulation and uses them to transfer actions from a single demonstration; physical experiments report an average task success rate of 80.3% across 10 tasks on two robotic systems.

  • Problem

    Learning reusable policies for deformable-object manipulation remains an open problem because of sensing and control difficulties, and fabric approaches often rely on task-specific policies.

    Source: §I · p. 1

  • Method

    The paper learns visual correspondences across fabric configurations in simulation and uses them to compute equivalent actions from a single task demonstration.

    Source: §I · p. 1

  • Results

    Physical experiments across 10 fabric-manipulation tasks on two robotic systems report an average task success rate of 80.3%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The learned correspondences support smoothing and folding policies across fabric variations and transfer to two physical robotic systems without real-world training.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Robotic fabric manipulation is challenging due to the infinite dimensional configuration space, self-occlusion, and complex dynamics of fabrics. There has been significant prior work on learning policies for specific deformable manipulation tasks, but comparatively less focus on algorithms which can efficiently learn many different tasks. In this paper, we learn visual correspondences for deformable fabrics across different configurations in simulation and show that this representation can be used to design policies for a variety of tasks. Given a single demonstration of a new task from an initial fabric configuration, the learned correspondences can be used to compute geometrically equivalent actions in a new fabric configuration. This makes it possible to robustly imitate a broad set of multi-step fabric smoothing and folding tasks on multiple physical robotic systems. The resulting policies achieve 80.3% average task success rate across 10 fabric manipulation tasks on two different robotic systems, the da Vinci surgical robot and the ABB YuMi. Results also suggest robustness to fabrics of various colors, sizes, and shapes. See https://tinyurl.com/fabric-descriptors for supplementary material and videos.

I. INTRODUCTION

The paper learns dense visual correspondences for fabrics in simulation and uses a single task demonstration to transfer geometrically equivalent actions across configurations. This approach addresses the difficulty of reusing task-specific fabric policies and is evaluated across tasks and robotic systems.

II. RELATED WORK

  • Prior fabric-manipulation approaches often use task-specific policies, while simulation-based approaches also commonly optimize policies for specific tasks.The paper positions its approach as learning a visual representation without task-specific data, then using it to design controllers for several tasks.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • The learned representation is intended to generalize across fabric color, size, and pose, supporting controllers for varied fabric instances.The paper describes domain randomization as a way to provide training variety across colors, shapes, and configurations.Source: §I · p. 1
  • A single demonstration can be transferred to a different fabric configuration by using learned correspondences to compute geometrically equivalent actions.The paper proposes learning correspondences from task-agnostic RGB data collected entirely in simulation.Source: §I · p. 1
  • Physical experiments across 10 tasks report an average task success rate of 80.3% on fabrics with varying dimensions and colors.Experiments used five T-shirts and five square fabrics on the da Vinci Research Kit and ABB YuMi.Source: §I · p. 2

II. RELATED WORK

  • The approach builds on dense object descriptors and extends prior rope correspondence work from synthetic depth data to fabric configurations using synthetic RGB images.It applies the learned descriptors to 2D fabric manipulation.Source: §I · p. 1, §II · p. 2

III. PROBLEM DEFINITION

The task assumes planar fabric observed from above and a single pick-and-place demonstration. At test time, the goal is to reproduce the demonstrated action sequence on a previously unseen configuration of a fabric with similar geometry.

A. Assumptions

  • The fabric is assumed to lie on a planar workspace and be observed by an overhead camera.The image corresponds to the fabric's initial configuration.Source: A. Assumptions · p. 2
  • A task demonstration is a sequence of pick-and-place actions from an arbitrary initial configuration, and the test fabric need only have similar geometry.The demonstration and test fabric may differ in color, size, and initial configuration; examples include matching T-shirts but not T-shirts to pants or long-sleeved shirts.Source: A. Assumptions · p. 3

B. Task Definition

  • Each demonstrated action specifies a fabric grasp point and a place point, and execution moves the grasped fabric to the place point without changing end-effector orientation.The fabric at the grasp point is placed over the world coordinate associated with the place point, retaining its surface normals.Source: B. Task Definition · p. 3
  • At test time, the robot must generate a corresponding action sequence for a similar fabric in a previously unseen configuration.The correspondence function estimates a point in the test image corresponding to a point in the demonstrated image.Source: B. Task Definition · p. 3

IV. SIMULATOR

The simulator supplies fabric meshes and vertex-level ground truth for correspondence learning. A Siamese descriptor network learns pixel matches across configurations, which are aggregated to estimate correspondences.

IV. SIMULATOR

  • Blender simulations model T-shirts and square fabrics as polygonal meshes, with mesh vertices supplying ground-truth correspondences.The simulator also models fabric dynamics for simulated experiments.Source: §IV · p. 3, Figure 2 · p. 3

A. Dense Object Descriptor Training Procedure

  • A Siamese network learns descriptors that are close for corresponding fabric pixels and separated for non-corresponding pairs using a pixel-wise contrastive loss.Training uses synthetic RGB images because real-fabric correspondences across deformations are difficult to obtain; paired inputs use the same fabric type.Source: A. Dense Object Descriptor Training… · p. 3
  • The correspondence estimate selects the top k pixel matches by descriptor distance, then computes their geometric median in pixel space.Physical experiments used k = 20 for robust predictions.Source: A. Dense Object Descriptor Training… · p. 4

B. Dataset Generation and Domain Randomization

The method generates varied simulated fabric configurations and randomized images for descriptor training. The resulting task-agnostic controller uses learned geometric structure to transfer demonstrated actions to new configurations.

B. Dataset Generation and Domain Randomization

  • Simulated pinning, release, and gravity create realistic fabric deformations for a dataset of RGB images, pixel correspondences, and segmentation masks.The masks support sampling fabric matches.Source: B. Dataset Generation and Domain… · p. 4
  • Domain randomization varies mesh size, lighting, camera pose, texture, color, and specularity, while restricting z-axis rotation to reduce ambiguity.A single dataset includes both T-shirts and square fabric; 7,500 images take about two hours to generate on the described computer.Source: B. Dataset Generation and Domain… · p. 4

VI. DESCRIPTOR-PARAMETERIZED CONTROLLER

  • At execution, the controller uses descriptor-learned fabric geometry to identify relevant pixels and generate actions for a new configuration.One task-agnostic network serves varied tasks and configurations without separate networks for each fabric type.Source: §VI · p. 4

A. Fabric Smoothing

Smoothing locates fabric corners from a labeled flat reference and pulls them toward target positions; folding uses demonstrations of pick-and-place actions. Experiments assess descriptor and policy robustness across configurations and color.

A. Fabric Smoothing

  • The square-fabric smoothing policy iterates over corners and pulls them toward target positions on an underlying plane.Rather than using ground-truth fabric state as in, it locates corners on crumpled fabric from a learned descriptor network and labeled flat reference; T-shirt smoothing uses sleeve and base corners.Source: A. Fabric Smoothing · p. 4, Figure 4 · p. 5

B. Fabric Folding

  • Each folding task uses one offline demonstration of up to four human-provided pick-and-place actions, then executes the controller open-loop.Source: B. Fabric Folding · p. 5

VII. EXPERIMENTS

  • Experiments evaluate descriptor quality and training-parameter sensitivity, as well as policy performance on the dVRK and ABB YuMi.Results suggest robustness to changes in fabric configuration and color.Source: §VII · p. 5

A. Tasks

The evaluation covers ten fabric manipulation tasks on T-shirts and square fabrics, with single demonstrations of up to four actions. Experiments use two robot platforms under distinct workspace and sensing setups.

A. Tasks

  • The evaluation includes ten real-world tasks on five T-shirts and five square fabrics, spanning folds, smoothing, and combined smoothing-and-folding tasks.The listed tasks include single, double, and multi-corner folds, sleeve folds, smoothing, and two combined tasks.Source: A. Tasks · p. 5
  • A single visual demonstration of up to four actions generates a policy that attempts the same number of actions; fabrics vary in dimension or color.A subset of tasks is also executed in simulation.Source: A. Tasks · p. 5

B. Experimental Setup

  • The dVRK uses a 5” × 5” workspace with dampened square fabrics, while the YuMi uses undampened fabrics up to 12” × 12” and overhead webcam images.For both robots, images are cropped and resized to 485×485; descriptors handle lighting, camera-pose, and workspace variability without manual setup changes.Source: B. Experimental Setup · pp. 5–6
  • Smoothing is evaluated by workspace coverage, while fold success is judged by visual consistency with the goal image.The authors say intersection-over-union offers limited diagnostic information when starting configurations differ substantially.Source: B. Experimental Setup · p. 6

C. Results

Physical trials show gains in smoothing coverage and frequent success on folding tasks across the YuMi and dVRK; the discussion notes robustness across fabric attributes and identifies broader tasks for future work.

C. Results

  • Unsuccessful grasps were the most frequent failure mode and were compounded in tasks requiring more actions.The paper notes that this failure mode is independent of learned descriptor quality and points to the need for more robust grasping methods.Source: C. Results · p. 6
  • Smoothing increased coverage by 11–22% on average, and both robots completed almost all folding tasks at least 75% of the time.The smoothing-then-folding task succeeded 6/10 times on the YuMi and 8/10 times on the dVRK.Source: Table I · p. 6, Table II · p. 6, C. Results · p. 6

VIII. DISCUSSION AND FUTURE WORK

  • The authors report robustness across fabric colors, shapes, textures, and sizes, and plan to investigate hierarchical policies and more complex multi-fabric or wrapping tasks.The reported tasks were learned without real-world training on two physical robotic systems.Source: §VIII · p. 6

IX. APPENDIX

The appendix collects simulator and experiment details, simulation results, descriptor visualizations, additional physical trials, and a hyperparameter study.

IX. APPENDIX

  • Appendix A describes the fabric simulator, while Appendices B–E cover simulation experiments, simulation results, descriptor visualizations, and additional physical trials on the dVRK and YuMi.The simulator details include using Blender to render fabric configurations and retrieve vertex coordinates for dense pixel-vertex annotations.Source: §IX · p. 9
  • Appendix F studies how different hyperparameters affect the quality of learned visual correspondences.Source: §IX · p. 9

A. Fabric Simulator Details

  • The simulator pipeline uses Blender 2.8 for cloth dynamics and rendering, with vertex coordinates used to derive ground-truth pixel correspondences for descriptor training.Source: A. Fabric Simulator Details · p. 9

1) Fabric Model:

The simulation setup builds square and t-shirt cloth models, manipulates them with hooks, and generates varied starting states for folding trials.

1) Fabric Model:

  • The square cloth is a 27×27 vertex mesh with 0.02-meter thickness, and the t-shirt mesh is cut from a similarly subdivided square mesh.The simulator uses gravity, damping, stiffness, four spring types, and repulsive forces for self-collision and object collision; parameters were visually tuned during folding actions.Source: 1) Fabric Model · p. 9, Table IV · p. 9

2) Manipulation with Hook Objects:

3) Starting Configurations and Actions:

B. Simulation Experiment Details

  • Simulation experiments use 50 trials for each of the first four folding tasks, with success judged by visual consistency between the final cloth state and target image.Structural similarity and summed vertex distance were considered but were insufficient when test-time starting configurations differed substantially from demonstrations.Source: B. Simulation Experiment Details · pp. 9–10

C. Simulation Experiment Results

In simulation, folding tasks succeed in 84–96% of trials, while descriptor visualizations show corresponding cloth pixels mapping to similar colors across configurations.

C. Simulation Experiment Results

D. Descriptor Mapping Visualizations

E. Physical Trial Trajectories

The appendix presents additional physical smoothing and folding trials, descriptor-quality ablations, and a simulation visualization of descriptor-parameterized folding.

E. Physical Trial Trajectories

F. Descriptor Quality Ablations

  • Descriptor quality was generally insensitive to training choices; RGB data and domain randomization performed best, with 3-dimensional descriptors used in simulation and 9-dimensional descriptors on real fabrics.The ablations tested image type, descriptor dimension, annotation count, and domain randomization using held-out pixel-match error.Source: F. Descriptor Quality Ablations · p. 10, Figure 13 · p. 12
  • Simulation visualizations suggest descriptor-parameterized policies can execute folding instructions from previously unseen fabric configurations.The visualization covers folding tasks 2 and 4, using pick-and-place actions specified through a web interface.Source: Figure 14 · p. 12

Related papers

Loading 2003.12698v2…